So I'm looking at some svn related httpd crashes we've been seeing on
svn.apache.org, and it turns out that the root cause seems to be
unbounded memory use in svn_repos_get_logs3. We construct a history
object for each path, and walk backwards through them until we've
either run out of history or sent "limit" revisions back to the user.
Unfortunately, this means we actually keep two pools and a history
object per path open, which for large numbers of paths (around 1200 in
the crashes i'm seeing) results in a LOT of memory being used (about
512 megs in this case, at which point the process crashes because it
hits its per-process memory limit).
I'm starting to think we might need some sort of system where we only
open a fixed number of history objects (and thus only need a fixed
number of scratch pools around for iterating through history) at a
time. We'd still need to track some info on a per-path basis (i.e.
what location in history we were at when we last had that node's
history opened) but it would be far less. More troubling would be the
performance hit of recreating the history objects lazily, but it's
certainly better to be slow than it is to use unbounded memory.
Additionally, it seems like we can get away with not opening a
revision root per path, since they are all rooted at the same revision
anyway. This change at least reduces our per-path overhead a bit, and
basic tests seem to indicate that it's safe. Is there any reason we
can't do that (see attached patch)?
Thoughts?
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Wed Feb 8 22:52:27 2006