[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: [TSVN] RFC: New cache scheme

From: SteveKing <stefankueng_at_gmail.com>
Date: 2005-01-21 13:02:26 CET

On Fri, 21 Jan 2005 11:29:47 +0000, Will Dean <svn@indcomp.co.uk> wrote:

> The shell extension will attempt to start the cache process if it needs it,
> anyway.

Do you mean you want to start the process each time you want to fetch
a status? Sure you would first check if the process is running before
trying to start it again. Or would you only try to start it if you
don't get an answer? How big would be the timeout then?
Or starting it only when the shell dll is loaded?
What about having the status cache implemented as a windows service?

> >The other point is that this assumes that the cache is always going to be
> >used, shouldn't it be configurable though - what happens to users with a
> >huge wc, how much memory is it to grab?
> Not very much, I don't think. But it depends how huge is huge. The files
> are stored by directory, with just the filename part retained. The full
> directory names are stored.
> Lets say that a filename is on average 10 chars (20 bytes in unicode), and
> that per file there's another 40 bytes of status info stored. Even for
> 100000 files, that's only 8Meg.

The status info (if you store the whole svn_wc_status_t struct) is
more than just 40 bytes. You have the filename in there, the URL it
points to and UUID strings...

But we can just assume that whoever has very big working copies also
must have enough RAM available - I think that's reasonable.

> Then of course we've got some extra storage for the directory names,
> allocation overhead, etc.
> Personally, I don't think data memory consumption is a big deal here.

I think I can agree with that.

> But for people with small machines and vast WC's, there will always be the
> option to turn recursive caching off and shorten the lifetime of
> objects. (We can have a garbage collector that runs through the cache and
> discards old stuff.)

Problem with a garbage collector is that while it is running, it has
to be exclusively. I.e. all other requests to the cache have to wait
until the collector is finished. So depending on how long it takes to
take out the garbage you have a shell either not responding or showing
the wrong overlays...

> It might, but like all these things, we must decide if there's a real
> problem to fix before we add lots of knobs and buttons. (There is already
> a mechanism to tell TSVN to ignore or include various things for overlays
> - we would continue to support all of this, though if the cache performance
> was much better, people might be less worried about what was included and
> what wasn't.)

True. If it's fast enough with even recursive overlays turned on, then
there's no reason to have an option to turn that off.

> I think all we're doing is turning the bug report which says 'the icons
> overlays have disappeared', which could have some quite simple steps to
> check what the cache is doing, into one which says 'explorer has ground to
> a halt with the disk thrashing'. I'm not convinced that the latter is
> easier to support than the former.

Since the cache has an IPC interface, a simple "cache status"
dialog/property page could ask the cache process for specific
information like memory use, number of hold files, ...

> 1. It would be nice to take the SVN client libraries out of the shell
> extension.

As you already mentioned, that can't be done unless we provide two
different extension dlls - one for the overlays/columns and one for
the property page.

> The current mode of failure of the SVN status system is to crash whatever
> process you're running in (Explorer or whatever app's just done a file-open
> dialog.) It's believed that at least some of these crashes come from bugs
> in the SVN libraries (to the extent that Stefan wrapped them in exception
> handling blocks.)

Yep! Had to do that because under certain circumstances where files
get locked by virus scanners the status function in SVN can crash.
Since I never could provide a reliable recipe to reproduce the crash
(even though I could point out the exact location in the source where
those crashes happen) the Subversion devs don't want to 'fix' it. It's
a philosophical difference between me and them here.

> >Sounds like it could be good, if the problems Stefan mentioned can be
> >overcome.
> Blimey, I've missed those. Better go and have another look at his postings.

One problem we might encounter with this approach which I haven't mentioned yet:
You can't fetch the status while another Subversion process is
accessing the files (e.g. for an update/commit/...). This wasn't a big
problem with the current approach, since once you start
TortoiseProc.exe, the explorer already fetched the status and shows
the overlays. It is however a problem if you have multiple instances
of explorer open showing the same working copy folder and then you an
update/commit/... - then sometimes even with the current approach the
overlays get lost (showing "unversioned") because the working copy is
With a separate process fetching the status, maybe even in a separate
thread it's much more likely to get into such a situation where you
try to fetch the status of a locked working copy.

Can you commit what you already have so far? I'd like to have a look
at it and help out where I can...


  oo  // \\      "De Chelonian Mobile"
 (_,\/ \_/ \     TortoiseSVN
   \ \_/_\_/>    The coolest Interface to (Sub)Version Control
   /_/   \_\     http://tortoisesvn.tigris.org
To unsubscribe, e-mail: dev-unsubscribe@tortoisesvn.tigris.org
For additional commands, e-mail: dev-help@tortoisesvn.tigris.org
Received on Fri Jan 21 13:03:04 2005

This is an archived mail posted to the TortoiseSVN Dev mailing list.