Re: Memory leak with 1.6.x clients

On Fri, May 29, 2009 at 4:25 PM, Stefan Sperling <stsp_at_elego.de> wrote:
> On Fri, May 29, 2009 at 02:28:17PM -0500, kmradke_at_rockwellcollins.com wrote:
>> "C. Michael Pilato" <cmpilato_at_collab.net> wrote on 05/29/2009 02:12:25 PM:
>> > Mark Phippard wrote:
>> > > On Fri, May 29, 2009 at 3:02 PM, Paul Burba <ptburba_at_gmail.com> wrote:
>> > >> This makes the commit I/O bound and slows it down quite a bit -- the
>> > >> total number of failed attempts being ((N(N+1)/2)-1) -- I'll look
>> into
>> > >> speeding this up...
>> > >
>> > > This sounds horrid.  I think we've had reports about commit being much
>> > > slower in 1.6 too.  And getting slower as the number of files goes up.
>> > >
>> > > Is this fixable?
>> >
>> > We could use tempfile names that match the name of the to-be-committed
>> file,
>> > instead of having them all be generic tempfile.N type names.  That would
>> > reduce name clashes.
>> Do we already have a hash value for the file?  Could we just stringify
>> it?  That would make it even less likely to clash, right?
>
> I've been wondering for some time why we don't use unpredictable
> names for tempfiles. That would also solve this problem.
>
> Stefan

Hi Stefan,

In IRC last Friday you said you were looking into fixing
svn_io_open_uniquely_named to avoid temp file name collisions. Any
luck with this?

You also brought up the question of whether we can change
svn_io_open_uniquely_named()'s behavior from that currently described
by its doc string without revving the API. I tend to think not, the
doc string is quite explicit in spelling out what the temp file's
names will be...though maybe the language "with a unique name
***based*** on * utf-8 encoded @a filename" gives us some wiggle room,
depending on how we interpret "based on".

I care about this only because commits of large numbers of paths with
1.6.x is quite slow* and I'd really like a fix we can apply to 1.6.3.
The attached patch is a more backport friendly (though much narrower
in scope). If your work results in something that cannot be
backported to 1.6 perhaps we can go this route.

Paul

* A crude demonstration: Committing 10 newly added directories with
1000 new 4KB files in each directory using 1.6.x_at_37906 with neon takes
18:16 to complete on my box, with the attached patch it takes only
03:26.

[[[
Avoid temp file name collisions when committing with ra_serf/ra_neon.

* subversion/libsvn_ra_neon/commit.c
  (commit_apply_txdelta): Use svn_io_open_uniquely_named directly instead of
  svn_io_open_unique_file3. Avoid temp file naming collisions by basing the
  temp file name on BASE_CHECKSUM, or for new files which have no checksum,
  on FILE->RSRC->LOCAL_PATH.

* subversion/libsvn_ra_serf/commit.c
  (apply_textdelta): Same as above, but use CTX->NAME instead of
  FILE->RSRC->LOCAL_PATH.
]]]

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2357582