Story of BerkeleyDB failing

Executive summary:
==================

"svn log" on a large repository fails with "Cannot allocate memory". I
have no workaround. Can somebody provide advice?

Full story:
===========

'svn log' emited:
-----------------

  Berkeley DB error while reading node revision for filesystem
  /usr/local/svn/perl/db: Cannot allocate memory

  --or--

  svn: Berkeley DB error while reading transaction for filesystem
  /usr/local/svn/perl/db: Cannot allocate memory

Failing commands were:
----------------------

 svn log -r HEAD http://...

     asked for password, then quickly failed; error_log of apache
     logged the same error

 svn log -r HEAD file://...

     fails quickly. See below for an strace.

Commands that still worked:
---------------------------

 svnadmin recover # didn't have to recover anything

 svnadmin dump # fortunately!

 svn st -u

 svn log -r 1

My environment:
---------------

- svn's own revision was 3953. I later upgraded to 4045 with same
  results.

- Berkeley version is 4.0.14.

- Linux Kernel is 2.5.47-ac5, LARGE_FILE support is available and
  seems to be working. Later I switched to 2.5.50 and 2.4.20, same
  results.

- In the beginning I didn't touch DB_CONFIG. Later I did try to lower
  and increase values for set_lg_bsize and set_lg_max to no avail.

- The machine has 1.5 GB memory and 8 GB swap. The partition has 12 GB
  total, at that point 3 GB were unused. The process size is moderate
  (top reported 24.6 MB for SIZE immediately before the error
  happened)

- I had managed to write 7333 Berkeley log.NNNNNNNNNN files, the last
  revision was #13069. Total size of repository incl. logfiles was
  ~8GB.

- This error occured for the first time around revision 10600 but only
  for a single command (do not know which command it was, most
  probably 'log'). After that, Subversion continued to accept checkins
  and allowed growing to >13000 revisions.

- removing logfiles did not change the situation

- the dump file was > 3GB. Fortunately I could spot a checkpoint that
  seemed appropriate to recover a valuable part of the repository (rev
  10408).

What I tried as a rescue:
-------------------------

With the dump (up to rev 10408; 2.3 GB) in my hands I removed the
whole repository, created a new one and successfully loaded the dump
into it. The repository could then be used to check out a working
copy, but it still could not send a log. It failed with the same
"Cannot allocate memory" from BerkeleyDB as before.

Limits:
-------

Maybe I have a limit in the shell environment? The limit command told
me:

    cputime unlimited
    filesize unlimited
    datasize unlimited
    stacksize 8MB
    coredumpsize 0kB
    memoryuse unlimited
    maxproc 12287
    descriptors 1024
    memorylocked unlimited
    addressspace unlimited
    maxfilelocks unlimited

Doesn't look very limited. Increasing the stacksize up to 2048MB
didn't help.

Strace (v4.2) tells me:

  pread(5, "\203\1\0\0\341\221\2\0\366\0\0\0\365\0\0\0\370\0\0\0\n"..., 4096, 1007
  616) = 4096
  brk(0x926d000) = 0x926d000
  pread(5, "\200\1\0\0w\270\5\0\365\0\0\0\356\0\0\0\366\0\0\0\n\0l"..., 4096, 1003
  520) = 4096
  brk(0x9270000) = 0x9270000
  brk(0x9279000) = 0x9279000
  write(2, "subversion/libsvn_fs/err.c:47", 29subversion/libsvn_fs/err.c:47) = 29
  write(2, ": (apr_err=160029, src_err=12)\n", 31: (apr_err=160029, src_err=12)
  ) = 31
  write(2, "svn: Berkeley DB error\n", 23svn: Berkeley DB error
  ) = 23
  write(2, "svn: Berkeley DB error while rea"..., 113svn: Berkeley DB error while
  reading node revision for filesystem /usr/local/svn/perl/db:
  Cannot allocate memory
  ) = 113

log -r 1 vs log -r HEAD
-----------------------

You might believe that if 'log -r 1' works and 'log -r HEAD' doesn't,
there would be a threshold that marks a region of revisions that works
and another region that doesn't. It turns out that this is the case,
but the threshold is not stable. At one point I determined the
threshold to be 4046/4047, then after a 'svnadmin recover' it was
9685/9686. After a while it was 9642/9643. It seems to depend on the
history of actions since the last recover. Indeed, whenever I run a
recover, the threshold becomes 9685/9686 again.

What I did not try:
-------------------

I haven't tried to upgrade to Berkeley 4.1.24.

Please advise what I should try next.--Thanks!

-- 
andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org