On 11.12.2015 15:04, Stefan Sperling wrote:
> On Thu, Dec 10, 2015 at 01:41:40PM +0100, Stefan Fuhrmann wrote:
>> 0x40000015 is basically an "abort". We do this when we need
>> more memory but can't get it.
>>
>> 9000 directory entries is around the capacity of the default
>> cache configuration (16MB). Make sure, you set the cache to
>> at least 100MB ("SVNInMemoryCacheSize 102400" in httpd.conf)
>> but NOT larger than 1GB in case you are using 32 bit Apache.
>>
>> Set "MaxMemFree" in httpd.conf to something like 4096. The
>> default of 0 is problematic.
>
> Would it be feasible to avoid crashing when the cache is misconfigured,
> or even set to the default setting, so users see a performance drop
> instead of a crash?
>
> I know our code is very lazy about checking allocations in general (which
> I don't like, but that's besides the point). In this context some additional
> effort into avoiding crashes would be a worthwhile investment.
Even if the cache configuration is inadequate, the
system should never run OOM. After many hours of
digging, I found which parts are to blame:
* mod_dav, possibly mod_dav_svn
* APR
* SVN
... in that order. The root problem is that
^/httpd/httpd/trunk/modules/dav/main/props.c: dav_close_propdb
does not destroy the pool created in dav_open_propdb.
Since that is being called for every directory entry,
we allocate at least 8K per dirent. Maybe, the walker
in our mod_dav_svn could arrange it to call that only
once?
While 8k/dirent are already bad, the situation is
exacerbated by APR's pool allocator. If that one runs
out of small nodes, it will happily recycle large ones.
So, 8k easily becomes 100s of KB.
r1594729 fixes this in many but not all situations and
has not been backported to any of of the 1.* branches,
yet. I'll submit a patch and then ask for both to be
backported.
Subversion's inability to cache those directories causes
the temporary allocation of large temporary block (e.g.
txdelta windows) in the first place. The limit in 1.9
was supposed to be ~30k entries in the default config
but due to a snafu, it is still ~8k. A series of patches
will improve it to ~60k on /trunk and a simple backport
patch gets us to the intended 30k.
Meanwhile, this is what people can do:
* Set MaxMemFree to something small-ish (<= 4MB)
This limits the maximum node size and replenishes
the pool of smaller nodes (because there are fewer
large ones to inappropriately recycle).
* Set the cache size to something large enough
See 1.8 release notes for guidance. This widely
eliminates the need to allocated larger nodes,
meaning there won't be any to inappropriately re-use.
-- Stefan^2.
Received on 2015-12-20 13:54:45 CET