[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

[PATCH] Excessive memory use with "SVNPathAuthz short_circuit"

From: Roderich Schupp <roderich.schupp_at_gmail.com>
Date: Tue, 18 Sep 2012 12:50:07 +0200

Hi,

recently our production Subversion servers blew up to allocate ~20 GB
of memory per Apache worker process (normal usage would be ~2 GB at most).
The culprit turned out to repeated requests like this:

     svn log --stop-on-copy --xml --revision 10574:74748
https://server/repository_root

(Don't ask why someone would do that over and over again :)
The thing to note here is:

- we're asking for a lot of revisions (~65 k)
- there's a total of ~580 k changed paths in these revisions

When I tried this request with an otherwise idle Apache I obeserved the
following memory usage (as reported by the VIRT column in top):

290m idle worker process after start of Apache
1463m after completion of the above request

We're using "SVNPathAuthz short_circuit". If I change that to
"SVNPathAuthz on" and repeat the experiment I get

290m idle worker process after start of Apache
374m after completion of the above request

So short_circuit path authorization is to blame.
I think the explanation is:

If we don't do short_circuit, then (cf. mod_dav_svn/authz.c)

  ap_sub_req_method_uri("GET", ...)

is called foreach changed path. This runs with a freshly created
request_rec rsub
where rsub->pool is a sub-pool of the main r->pool. When the sub
request returns,
rsub and rsub->pool are destroyed. The actual work is done in
svn_repos_authz_check_access which gets passed rsub->pool.

But for short_circuit authz, svn_repos_authz_check_access is called from
subreq_bypass using the main request r->pool (cf.
mod_authz_svn/mod_authz_svn.c).
I.e. the memory used by svn_repos_authz_check_access accumulates over
the whole log-report HTTP request.

To fix, use a sub-pool in subreq_bypass when calling
svn_repos_authz_check_access.
After applying the attached patch I get (using "SVNPathAuthz short_circuit"):

290m idle worker process after start of Apache
393m after completion of the above request

Cheers, Roderich

Received on 2012-09-18 13:53:27 CEST

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.