Issue #1899 (svndumpfilter single-file-change scalability): Investigating

I took some time to investigate issue #1899:

http://subversion.tigris.org/issues/show_bug.cgi?id=1899

The reporter reported that if a large file was committed wholly at one commit,
and it was considerably larger than the amount of available memory, then
svndumpfilter would crash. He believed it was a memory leak, but as I think
after my investigation, it's just a case, that the entire file needs to be
allocated in one contiguous buffer.

After I ran the test script I provided in the issue, I do a:

(ulimit -v 100000 ; gdb --command=cmds.gdb svndumpfilter)

Where cmds.gdb is:

<<<
set args include my-test-dir < dump.txt
b main
r
>>>

After I do a "c", the SIGABRT originates from:

<<<
#13 0x4002280d in parse_text_block (stream=0x8053e58, content_length=68812800,
    is_delta=0, parse_fns=0x8053fa8, record_baton=0x8090c78,
    buffer=0x8074c30
"Ü¡\\\037\227E)Ò8Xºr¿CwVi\003]LP¶\030\233Ä.úÇ}\226\235ÍBú¤±Í/\224³Wß\005:åCØ ¡O;ê¹±¯å©\006iÊ\034ÊIQ\231ÉÕå3\017²0üÍ8\206*°\232£ëÁ\205lk\227àM!ÌF\222X\b\213|
\005RÀþ\r\212P?", buflen=102400, pool=0x8090c40)
    at subversion/libsvn_repos/load.c:410
>>>

Which is:

SVN_ERR (svn_stream_write (text_stream, buffer, &wlen));

The function reads the entire buffer from the file and writes it chunk by
chunk into the stream without stopping.

Now text_stream according to my analysis eventually points to
revision_baton_t.body_stream, which in turn is initialized as:

<<<
rb->body = svn_stringbuf_create ("", pool);
rb->body_stream = svn_stream_from_stringbuf (rb->body, pool);
>>>

So it just stores everything in memory.

So this is not in fact a memory leak, just a design decision that causes the
program to eat a lot of memory in proportion to this.

Regards,

        Shlomi Fish

-- 
---------------------------------------------------------------------
Shlomi Fish      shlomif@iglu.org.il
Homepage:        http://shlomif.il.eu.org/
Knuth is not God! It took him two days to build the Roman Empire.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org