Possible bug in svn, svn dav, neon, apache etc.

Hello!

We've encountered a strange behaviour with our quite loaded (avg 1500 http
requests per sec) SVN server. The server (RHEL5) is serving SVN repositories
via https by Apache.

apache 2.2.9
subversion 1.6.5
neon-0.28.4-1
neon-devel-0.28.4-1

The problem is that occasionally couple of httpd processes start eating huge
amounts of memory and cause server to halt after all memory and swap are
being used.

Workarounds exist but stracing the bad processes reveal quite clear bug
somewhere in the used components.

Strace output:

mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2aac445ab000
brk(0x4004c000) = 0x40029000

This repeats forever (returned mem addr changes every time of course) until
server halts. It only takes minutes to eat up 64GB of mem and 8GB of swap.

We can see this only in our production environment which makes debugging a
bit hard.

One notable thing might be that the storage is on FC and IO scheduler is
'deadline' which might help revealing/reproducing the issue.

Br, Antti Siukola
Received on 2011-04-28 08:30:03 CEST