Re: Lock overuse in stabilize_node

On Sat, Mar 09, 2002 at 12:40:14AM -0500, Daniel Berlin wrote:
> On Fri, 2002-03-08 at 18:57, Ben Collins wrote:
> > I've tracked down the overusage of locks to the stabilize_node()
> > function in dag.c. This is the call where we traverse the commited tree
> > and mark everthing immutable.
> >
> > During this portion of my import, some 200,000 locks and 885 lock
> > objects are used up, before being released. I'm trying to find out of
> > there is a way we can explicitly release locks as we go (using
> > db->lock_vec or something).
>
> Well, hmmmm.
> 200,000 locks?
>
> For a given transaction (and all it's child tranasctions, whose locks do
> *not* go away until the parent is committed/aborted, not when *the
> child* is committed/aborted), you shouldn't be getting that many locks.

The problem is that we need to lock every single record in the BTREE (for
large imorts) while we do the stabilization. Thus for large commits, we run
out of locks. The tree Ben Collins is using is on the order of 10,000 files,
1,000 dirs). Thus we need locks for each of those entries in nodes, in reps,
one for each record of the dup-key based strings, revisions, and transactions
tables. If you say there are two locks for each node in strings, then we are
up to 66,000 locks already, not counting anything that might be read-write.

Its possible that many of those files took more than two rows in strings, and
its also possible that we might need read-write locks on some of those
objects. Then there is a global lock for the process, at least one for each
table, at least one for every non-leaf node in all the BTREES.

I agree with gstein, the solution is to re-do the txn handling in terms of
stabilization, and text-delta writing.

>
> B-tree requires one lock per tree level to access something (this is of
> course, for read-only/write-only, read-write takes two locks).
> Hash requires one lock to access something.
>
> So hash might reduce it by a factor of five maximum, based on your
> earlier db stats that showed the tree level at 5.
> This is best case. It's somewhere between "not at all, and a factor of
> 5". But we still shouldn't have that many locks.
> Hmmmm.
> First, i would actually switch the DB type to hash (which supports the
> duplicate key stuff we use, before someone asks).
> If this reduces the number of locks by more than a factor of 5, it's
> gotta be a BDB bug.
> Given that i can't see where these locks are being created and staying
> alive unless we are creating child transactions, it could quite
> possibily be a BDB bug.
> If it is, switching to hash should show it.
>
> I'm really at a loss otherwise.
> --Dan
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Kevin Pilch-Bisson                    http://www.pilch-bisson.net
     "Historically speaking, the presences of wheels in Unix
     has never precluded their reinvention." - Larry Wall
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~