Hey everyone,
I am new to this list but I stumbled across an issue on a newly
installed server that I can't figure out by myself.
Short factsheet:
OS CentOS 5.4 fully updated
Apache 2.2.3
SVN 1.6.9 (Binary package from your site)
Neon 0.28.4 (Binary package from your site)
The following symptoms happen whenever a user opens a connection to the
SVN repositories:
As soon as the connection is established and the user starts to check
out a large repository (around 5 GB) I can see the memory usage VIRT of
the apache process to go up each time the SVN descends into a new
subdirectory. It looks like it's caching some information on the
directory but it never frees it afterwards. RES also goes up but not as
fast as VIRT.
I dug around the web to find more details on what actually uses the
memory. When doing a pmap -x <pid> of the apache process handling the
SVN connection I am able to monitor the memory allocation of a [ anon ]
Mapping constantly rise when recursing into directories. This also does
not get cleared up afterwards.
Monitoring the process in TOP showed the memory to increase for the
process handling SVN from 203 MB VIRT and 10 MB RES up to 350 MB VIRT
and 126 MB RES without cleaning up the memory when SVN closed the
connection.
I disabled most of the other mods running (including mod_authz_svn) and
it still happens.
I pinned it down to the recursing in directories by checking out the SVN
on another machine over the 100 MBit connection. The faster the
connection the easier it is to monitor since the recursion into
directories happens faster too.
I also had this issue when running SVN 1.6.6 that came with CentOS 5
hence I tried upgrading to SVN 1.6.9.
I don't know what kind of information you need in order to help me but I
will be able to give you anything you ask for. I attached some info
output at the bottom of this mail.
Thank you for this great program. I hope this is nothing serious and I
don't need to worry about. If this is something that I need to worry
about hopefully it's something fixable :)
Cheers,
Sebastian
TOP output
===
PID User VIRT RES SWAP
16415 apache 337m 116m 221m /usr/sbin/httpd -DSSL -k
start
16421 apache 309m 93m 216m /usr/sbin/httpd -DSSL -k
start
16420 apache 212m 17m 194m /usr/sbin/httpd -DSSL -k
start
16423 apache 207m 14m 193m /usr/sbin/httpd -DSSL -k
start
16422 apache 206m 13m 193m /usr/sbin/httpd -DSSL -k
start
16418 apache 205m 12m 193m /usr/sbin/httpd -DSSL -k
start
15747 root 202m 10m 191m /usr/sbin/httpd -DSSL -k
start
16417 apache 203m 10m 193m /usr/sbin/httpd -DSSL -k
start
16416 apache 202m 9948 192m /usr/sbin/httpd -DSSL -k
start
16424 apache 202m 9944 193m /usr/sbin/httpd -DSSL -k
start
16419 apache 202m 9368 193m /usr/sbin/httpd -DSSL -k start
===
PMAP -x 16421 output
===
Address Kbytes RSS Anon Locked Mode Mapping
00002b8575f54000 308 - - - r-x-- httpd
00002b8575fa1000 260 - - - rw--- [ anon ]
00002b8575fe3000 260 - - - rw--- [ anon ]
00002b8576025000 196 - - - rw--- [ anon ]
00002b8576056000 28 - - - r--s-
gconv-modules.cache
00002b857605d000 1016 - - - rw--- [ anon ]
00002b85761a1000 16 - - - rw--- httpd
00002b85761a5000 12 - - - rw--- [ anon ]
00002b85761a8000 112 - - - r-x-- ld-2.5.so
00002b85761c4000 4 - - - rw--- [ anon ]
00002b85761d1000 1964 - - - rw--- [ anon ]
00002b85763c3000 4 - - - r---- ld-2.5.so
00002b85763c4000 4 - - - rw--- ld-2.5.so
00002b85763c5000 520 - - - r-x-- libm-2.5.so
00002b8576447000 2044 - - - ----- libm-2.5.so
00002b8576646000 4 - - - r---- libm-2.5.so
00002b8576647000 4 - - - rw--- libm-2.5.so
00002b8576648000 108 - - - r-x-- libpcre.so.0.0.1
00002b8576663000 2048 - - - ----- libpcre.so.0.0.1
00002b8576863000 4 - - - rw--- libpcre.so.0.0.1
00002b8576864000 84 - - - r-x-- libselinux.so.1
00002b8576879000 2048 - - - ----- libselinux.so.1
00002b8576a79000 8 - - - rw--- libselinux.so.1
00002b8576a7b000 8 - - - rw--- [ anon ]
00002b8576a7d000 112 - - - r-x--
libaprutil-1.so.0.2.7
00002b8576a99000 2044 - - - -----
libaprutil-1.so.0.2.7
00002b8576c98000 8 - - - rw---
libaprutil-1.so.0.2.7
00002b8576c9a000 36 - - - r-x-- libcrypt-2.5.so
00002b8576ca3000 2044 - - - ----- libcrypt-2.5.so
00002b8576ea2000 4 - - - r---- libcrypt-2.5.so
00002b8576ea3000 4 - - - rw--- libcrypt-2.5.so
00002b8576ea4000 184 - - - rw--- [ anon ]
00002b8576ed2000 224 - - - r-x--
libldap-2.3.so.0.2.31
00002b8576f0a000 2048 - - - -----
libldap-2.3.so.0.2.31
00002b857710a000 8 - - - rw---
libldap-2.3.so.0.2.31
00002b857710c000 4 - - - rw--- [ anon ]
00002b857710d000 52 - - - r-x--
liblber-2.3.so.0.2.31
00002b857711a000 2048 - - - -----
liblber-2.3.so.0.2.31
00002b857731a000 4 - - - rw---
liblber-2.3.so.0.2.31
00002b857731b000 964 - - - r-x-- libdb-4.3.so
00002b857740c000 2044 - - - ----- libdb-4.3.so
00002b857760b000 20 - - - rw--- libdb-4.3.so
00002b8577610000 128 - - - r-x--
libexpat.so.0.5.0
00002b8577630000 2044 - - - -----
libexpat.so.0.5.0
00002b857782f000 12 - - - rw---
libexpat.so.0.5.0
00002b8577832000 4 - - - rw--- [ anon ]
00002b8577833000 152 - - - r-x--
libapr-1.so.0.2.7
00002b8577859000 2048 - - - -----
libapr-1.so.0.2.7
00002b8577a59000 4 - - - rw---
libapr-1.so.0.2.7
00002b8577a5a000 88 - - - r-x--
libpthread-2.5.so
00002b8577a70000 2044 - - - -----
libpthread-2.5.so
00002b8577c6f000 4 - - - r----
libpthread-2.5.so
00002b8577c70000 4 - - - rw---
libpthread-2.5.so
00002b8577c71000 16 - - - rw--- [ anon ]
00002b8577c75000 8 - - - r-x-- libdl-2.5.so
00002b8577c77000 2048 - - - ----- libdl-2.5.so
00002b8577e77000 4 - - - r---- libdl-2.5.so
00002b8577e78000 4 - - - rw--- libdl-2.5.so
00002b8577e79000 4 - - - rw--- [ anon ]
00002b8577e7a000 1332 - - - r-x-- libc-2.5.so
00002b8577fc7000 2048 - - - ----- libc-2.5.so
00002b85781c7000 16 - - - r---- libc-2.5.so
00002b85781cb000 4 - - - rw--- libc-2.5.so
00002b85781cc000 20 - - - rw--- [ anon ]
00002b85781d1000 236 - - - r-x-- libsepol.so.1
00002b857820c000 2048 - - - ----- libsepol.so.1
00002b857840c000 4 - - - rw--- libsepol.so.1
00002b857840d000 40 - - - rw--- [ anon ]
00002b8578417000 12 - - - r-x-- libuuid.so.1.2
00002b857841a000 2048 - - - ----- libuuid.so.1.2
00002b857861a000 4 - - - rw--- libuuid.so.1.2
00002b857861b000 4 - - - rw--- [ anon ]
00002b857861c000 128 - - - r-x-- libpq.so.4.1
00002b857863c000 2048 - - - ----- libpq.so.4.1
00002b857883c000 8 - - - rw--- libpq.so.4.1
00002b857883e000 384 - - - r-x--
libsqlite3.so.0.8.6
00002b857889e000 2048 - - - -----
libsqlite3.so.0.8.6
00002b8578a9e000 8 - - - rw---
libsqlite3.so.0.8.6
00002b8578aa0000 68 - - - r-x-- libresolv-2.5.so
00002b8578ab1000 2048 - - - ----- libresolv-2.5.so
00002b8578cb1000 4 - - - r---- libresolv-2.5.so
00002b8578cb2000 4 - - - rw--- libresolv-2.5.so
00002b8578cb3000 12 - - - rw--- [ anon ]
00002b8578cb6000 96 - - - r-x--
libsasl2.so.2.0.22
00002b8578cce000 2048 - - - -----
libsasl2.so.2.0.22
00002b8578ece000 4 - - - rw---
libsasl2.so.2.0.22
00002b8578ecf000 276 - - - r-x-- libssl.so.0.9.8e
00002b8578f14000 2044 - - - ----- libssl.so.0.9.8e
00002b8579113000 24 - - - rw--- libssl.so.0.9.8e
00002b8579119000 1204 - - - r-x--
libcrypto.so.0.9.8e
00002b8579246000 2044 - - - -----
libcrypto.so.0.9.8e
00002b8579445000 132 - - - rw---
libcrypto.so.0.9.8e
00002b8579466000 20 - - - rw--- [ anon ]
00002b857946b000 580 - - - r-x-- libkrb5.so.3.3
00002b85794fc000 2048 - - - ----- libkrb5.so.3.3
00002b85796fc000 16 - - - rw--- libkrb5.so.3.3
00002b8579700000 84 - - - r-x-- libnsl-2.5.so
00002b8579715000 2044 - - - ----- libnsl-2.5.so
00002b8579914000 4 - - - r---- libnsl-2.5.so
00002b8579915000 4 - - - rw--- libnsl-2.5.so
00002b8579916000 12 - - - rw--- [ anon ]
00002b8579919000 176 - - - r-x--
libgssapi_krb5.so.2.2
00002b8579945000 2048 - - - -----
libgssapi_krb5.so.2.2
00002b8579b45000 8 - - - rw---
libgssapi_krb5.so.2.2
00002b8579b47000 8 - - - r-x--
libcom_err.so.2.1
00002b8579b49000 2044 - - - -----
libcom_err.so.2.1
00002b8579d48000 4 - - - rw---
libcom_err.so.2.1
00002b8579d49000 144 - - - r-x--
libk5crypto.so.3.1
00002b8579d6d000 2044 - - - -----
libk5crypto.so.3.1
00002b8579f6c000 8 - - - rw---
libk5crypto.so.3.1
00002b8579f6e000 4 - - - rw--- [ anon ]
00002b8579f6f000 80 - - - r-x-- libz.so.1.2.3
00002b8579f83000 2044 - - - ----- libz.so.1.2.3
00002b857a182000 4 - - - rw--- libz.so.1.2.3
00002b857a183000 32 - - - r-x--
libkrb5support.so.0.1
00002b857a18b000 2044 - - - -----
libkrb5support.so.0.1
00002b857a38a000 4 - - - rw---
libkrb5support.so.0.1
00002b857a38b000 8 - - - r-x--
libkeyutils-1.2.so
00002b857a38d000 2044 - - - -----
libkeyutils-1.2.so
00002b857a58c000 4 - - - rw---
libkeyutils-1.2.so
00002b857a58d000 12 - - - rw--- [ anon ]
00002b857a590000 8 - - - r-x--
mod_auth_basic.so
00002b857a592000 2044 - - - -----
mod_auth_basic.so
00002b857a791000 8 - - - rw---
mod_auth_basic.so
00002b857a793000 8 - - - r-x--
mod_authn_file.so
00002b857a795000 2044 - - - -----
mod_authn_file.so
00002b857a994000 8 - - - rw---
mod_authn_file.so
00002b857a996000 4 - - - r-x--
mod_authn_default.so
00002b857a997000 2044 - - - -----
mod_authn_default.so
00002b857ab96000 8 - - - rw---
mod_authn_default.so
00002b857ab98000 8 - - - r-x--
mod_authz_host.so
00002b857ab9a000 2044 - - - -----
mod_authz_host.so
00002b857ad99000 8 - - - rw---
mod_authz_host.so
00002b857ad9b000 4 - - - r-x--
mod_authz_user.so
00002b857ad9c000 2044 - - - -----
mod_authz_user.so
00002b857af9b000 8 - - - rw---
mod_authz_user.so
00002b857af9d000 4 - - - r-x--
mod_authz_default.so
00002b857af9e000 2044 - - - -----
mod_authz_default.so
00002b857b19d000 8 - - - rw---
mod_authz_default.so
00002b857b19f000 36 - - - r-x-- mod_include.so
00002b857b1a8000 2048 - - - ----- mod_include.so
00002b857b3a8000 8 - - - rw--- mod_include.so
00002b857b3aa000 20 - - - r-x--
mod_log_config.so
00002b857b3af000 2044 - - - -----
mod_log_config.so
00002b857b5ae000 8 - - - rw---
mod_log_config.so
00002b857b5b0000 20 - - - r-x--
mod_mime_magic.so
00002b857b5b5000 2044 - - - -----
mod_mime_magic.so
00002b857b7b4000 8 - - - rw---
mod_mime_magic.so
00002b857b7b6000 8 - - - r-x-- mod_setenvif.so
00002b857b7b8000 2048 - - - ----- mod_setenvif.so
00002b857b9b8000 8 - - - rw--- mod_setenvif.so
00002b857b9ba000 16 - - - r-x-- mod_mime.so
00002b857b9be000 2044 - - - ----- mod_mime.so
00002b857bbbd000 8 - - - rw--- mod_mime.so
00002b857bbbf000 84 - - - r-x-- mod_dav.so
00002b857bbd4000 2044 - - - ----- mod_dav.so
00002b857bdd3000 8 - - - rw--- mod_dav.so
00002b857bdd5000 16 - - - r-x-- mod_status.so
00002b857bdd9000 2044 - - - ----- mod_status.so
00002b857bfd8000 8 - - - rw--- mod_status.so
00002b857bfda000 32 - - - r-x-- mod_autoindex.so
00002b857bfe2000 2044 - - - ----- mod_autoindex.so
00002b857c1e1000 8 - - - rw--- mod_autoindex.so
00002b857c1e3000 8 - - - r-x--
mod_vhost_alias.so
00002b857c1e5000 2044 - - - -----
mod_vhost_alias.so
00002b857c3e4000 8 - - - rw---
mod_vhost_alias.so
00002b857c3e6000 28 - - - r-x--
mod_negotiation.so
00002b857c3ed000 2044 - - - -----
mod_negotiation.so
00002b857c5ec000 8 - - - rw---
mod_negotiation.so
00002b857c5ee000 8 - - - r-x-- mod_dir.so
00002b857c5f0000 2044 - - - ----- mod_dir.so
00002b857c7ef000 8 - - - rw--- mod_dir.so
00002b857c7f1000 12 - - - r-x-- mod_alias.so
00002b857c7f4000 2044 - - - ----- mod_alias.so
00002b857c9f3000 8 - - - rw--- mod_alias.so
00002b857c9f5000 56 - - - r-x-- mod_rewrite.so
00002b857ca03000 2044 - - - ----- mod_rewrite.so
00002b857cc02000 8 - - - rw--- mod_rewrite.so
00002b857cc04000 96 - - - r-x-- mod_python.so
00002b857cc1c000 2044 - - - ----- mod_python.so
00002b857ce1b000 20 - - - rw--- mod_python.so
00002b857ce2c000 1028 - - - r-x--
libpython2.4.so.1.0
00002b857cf2d000 2044 - - - -----
libpython2.4.so.1.0
00002b857d12c000 180 - - - rw---
libpython2.4.so.1.0
00002b857d159000 20 - - - rw--- [ anon ]
00002b857d15e000 8 - - - r-x-- libutil-2.5.so
00002b857d160000 2044 - - - ----- libutil-2.5.so
00002b857d35f000 4 - - - r---- libutil-2.5.so
00002b857d360000 4 - - - rw--- libutil-2.5.so
00002b857d361000 160 - - - r-x-- mod_ssl.so
00002b857d389000 2048 - - - ----- mod_ssl.so
00002b857d589000 8 - - - rw--- mod_ssl.so
00002b857d58b000 8 - - - rw--- [ anon ]
00002b857d599000 16 - - - r-x--
libdistcache.so.1.0.1
00002b857d59d000 2044 - - - -----
libdistcache.so.1.0.1
00002b857d79c000 4 - - - rw---
libdistcache.so.1.0.1
00002b857d79d000 32 - - - r-x-- libnal.so.1.0.1
00002b857d7a5000 2048 - - - ----- libnal.so.1.0.1
00002b857d9a5000 4 - - - rw--- libnal.so.1.0.1
00002b857d9a6000 136 - - - r-x-- mod_dav_svn.so
00002b857d9c8000 2048 - - - ----- mod_dav_svn.so
00002b857dbc8000 8 - - - rw--- mod_dav_svn.so
00002b857dbd6000 160 - - - r-x--
libsvn_repos-1.so.0.0.0
00002b857dbfe000 2044 - - - -----
libsvn_repos-1.so.0.0.0
00002b857ddfd000 4 - - - rw---
libsvn_repos-1.so.0.0.0
00002b857ddfe000 24 - - - r-x--
libsvn_fs-1.so.0.0.0
00002b857de04000 2044 - - - -----
libsvn_fs-1.so.0.0.0
00002b857e003000 4 - - - rw---
libsvn_fs-1.so.0.0.0
00002b857e004000 44 - - - r-x--
libsvn_delta-1.so.0.0.0
00002b857e00f000 2044 - - - -----
libsvn_delta-1.so.0.0.0
00002b857e20e000 4 - - - rw---
libsvn_delta-1.so.0.0.0
00002b857e20f000 300 - - - r-x--
libsvn_subr-1.so.0.0.0
00002b857e25a000 2044 - - - -----
libsvn_subr-1.so.0.0.0
00002b857e459000 12 - - - rw---
libsvn_subr-1.so.0.0.0
00002b857e45c000 148 - - - r-x--
libsvn_fs_fs-1.so.0.0.0
00002b857e481000 2048 - - - -----
libsvn_fs_fs-1.so.0.0.0
00002b857e681000 4 - - - rw---
libsvn_fs_fs-1.so.0.0.0
00002b857e682000 176 - - - r-x--
libsvn_fs_base-1.so.0.0.0
00002b857e6ae000 2044 - - - -----
libsvn_fs_base-1.so.0.0.0
00002b857e8ad000 8 - - - rw---
libsvn_fs_base-1.so.0.0.0
00002b857e8af000 4 - - - r-x--
libsvn_fs_util-1.so.0.0.0
00002b857e8b0000 2044 - - - -----
libsvn_fs_util-1.so.0.0.0
00002b857eaaf000 4 - - - rw---
libsvn_fs_util-1.so.0.0.0
00002b857eab0000 12 - - - r-x-- mod_authz_svn.so
00002b857eab3000 2044 - - - ----- mod_authz_svn.so
00002b857ecb2000 4 - - - rw--- mod_authz_svn.so
00002b857ecb3000 16 - - - r-x-- mod_suphp.so
00002b857ecb7000 2048 - - - ----- mod_suphp.so
00002b857eeb7000 4 - - - rw--- mod_suphp.so
00002b857eeb8000 260 - - - rw--- [ anon ]
00002b857eefa000 260 - - - rw--- [ anon ]
00002b857ef3c000 196 - - - rw--- [ anon ]
00002b857ef6d000 504 - - - rw-s- zero (deleted)
00002b857efeb000 17248 - - - rw--- [ anon ]
00002b85800f5000 40 - - - r-x--
libnss_files-2.5.so
00002b85800ff000 2044 - - - -----
libnss_files-2.5.so
00002b85802fe000 4 - - - r----
libnss_files-2.5.so
00002b85802ff000 4 - - - rw---
libnss_files-2.5.so
00002b8580300000 55120 - - - r---- locale-archive
00002b85838d4000 416 - - - rw--- [ anon ]
00002b8583952000 140 - - - rw-s- zero (deleted)
00002b8583975000 1872 - - - rw--- [ anon ]
00002b8583b7a000 260 - - - rw--- [ anon ]
00002b8583bbc000 260 - - - rw--- [ anon ]
00002b8583bfe000 2068 - - - rw--- [ anon ]
00002b8583e34000 260 - - - rw--- [ anon ]
00002b8583e76000 260 - - - rw--- [ anon ]
00002b8583eb8000 99588 - - - rw--- [ anon ]
00002b858a02c000 11288 - - - rw--- [ anon ]
00007fffb64fc000 156 - - - rw--- [ stack ]
ffffffffff600000 8192 - - - ----- [ anon ]
---------------- ------ ------ ------ ------
total kB 346764 - - -
===
HTTPD -M output
===
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
authn_file_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_default_module (shared)
include_module (shared)
log_config_module (shared)
mime_magic_module (shared)
setenvif_module (shared)
mime_module (shared)
dav_module (shared)
status_module (shared)
autoindex_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
alias_module (shared)
rewrite_module (shared)
python_module (shared)
ssl_module (shared)
dav_svn_module (shared)
authz_svn_module (shared)
suphp_module (shared)
===
Received on 2010-02-04 02:00:35 CET