On Mon, 2009-12-14, C. Michael Pilato wrote:
> Julian.
>
> I want to draw your attention to something that you may or may not realize
> about designing for the BDB backend. The 'trail' objects are (generally)
> representative of Berkeley DB transactions -- that part I'm sure you know.
> But you might not realize the value of keeping transactions as small as
> possible. Berkeley DB will accumulate locks (which I believe are
> page-level, not as tight as row-level like you might hope) over the course
> of a transaction, releasing those locks only at transaction commit/abort.
> Berkeley DB backends are configured to have a maximum number of locks and
> lockers allowed, and it's easier than you might think to hit the max-locks
> thresholds (especially under high concurrency) and see an error (typically a
> "Cannot allocate memory") result from that.
>
> For example, in the loop below you are writing a bunch of rows to the
> `changes' table. Could be 10. Could be 100,000. 100,000 writes and
> associated locks might be a problem or it might not. But I use it as a way
> to encourage you to think about reducing the amount of work you spend in any
> one trail while doing your obliterate work.
Thanks for this, and the subsequent IRC chat; sorry my net connection
dropped near the end of that.
I suspected something like this, and wrote in a comment in
svn_fs_base__begin_obliteration_txn(), "### Does all of the duplication
need to be done inside the retry_txn? It is currently inside."
I will bear this in mind and may make some changes in this direction.
However, my priorities begin with learning how to do obliteration
functionally correctly (which I am doing in BDB), and then implementing
it in FSFS, and somewhere down the list comes scalability in BDB. I
appreciate that obliteration is most needed in large repositories where
scalability is essential.
> (And yeah, this information should probably live somewhere. Maybe you could
> drop it into trail.h as a comment?)
OK, dropped into subversion/libsvn_fs_base/trail.h in r890841.
- Julian
Received on 2009-12-15 16:52:39 CET