On 17/12/14 09:25, Charlie Smurthwaite wrote:
> Hi,
>
> I am running an SVN HTTP server using Apache/2.4.10 with SVN 1.8.10
> (r1615264) and I am often seeing very high kernel CPU load.
>
> The CPU time seems to be consumed in the kernel by "_raw_spin_lock",
> with the httpd processes spending much of their time waiting on calls
> to "futex".
>
> Here's a syscall analysis from svn:
> http://paste.codebasehq.com/pastes/7qzt68lx2eghz2gjns
> Here's the kernel CPU time analysis: http://i.imgur.com/37Ryt5V.png
> Here's a snapshot from htop showing why this is a problem:
> http://i.imgur.com/I3mDDbi.png
>
> I'd appreciate if anyone could tell me whether I have likely
> configured something incorrectly, whether there is an obvious
> workaround, or whether this needs to be escalated as a bug, and if so,
> to whom?
>
> Thank you!
>
> Charlie
>
>
> My httpd is configured as follows:
>
> root_at_storage02:~# /opt/subversion-server/bin/httpd -V
> Server version: Apache/2.4.10 (Unix)
> Server built: Nov 7 2014 15:16:58
> Server's Module Magic Number: 20120211:36
> Server loaded: APR 1.5.1, APR-UTIL 1.5.4
> Compiled using: APR 1.5.1, APR-UTIL 1.5.4
> Architecture: 64-bit
> Server MPM: worker
> threaded: yes (fixed thread count)
> forked: yes (variable process count)
> Server compiled with....
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=256
> -D HTTPD_ROOT="/opt/subversion-server"
> -D SUEXEC_BIN="/opt/subversion-server/bin/suexec"
> -D DEFAULT_PIDLOG="logs/httpd.pid"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
It was suggested that this might be related to memory management, so I
have looked into a couple of things:
1) I disabled swap on the server in question. The host had 25GB of RAM
free and should not have been swapping active memory, however I believe
disabling swap has solved the problem.
2) The server is running NUMA, rather than SMP memory configuration. I
suspect that this is the reason for the problem (though I have no
evidence) and that it is not specific to SVN/Apache.
Thanks to Bert on Freenode for his assistance in finding this
workaround. I suspect that changing NUMA config, or changing Apache's
threading model would also prevent the problem.
Charlie
Received on 2014-12-17 19:47:13 CET
