[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

A quick-and-dirty MySQL fs backend

From: Edmund Horner <chrysophylax_at_chrysophylax.cjb.net>
Date: 2004-02-19 11:55:11 CET

Hi all.

I've been playing around with replacing the BDB stuff with equivalent
MySQL stuff, that is, I've:

   - Copied the libsvn_fs/bdb directory to libsvn_fs/mysql.
   - Modified the files of libsvn_fs/mysql to use MySQL queries to
     store and retrieve data.

Bear in mind that, although they are currently compiling, things aren't
*working*. (I had forgotten to do anything with trail.c!) However I'm
at the point when I'd like some advice about how to continue. I'll
describe my thoughts and implementation. Please feel free to argue
about it.

I know Glenn Thompson is working on something
(http://www.cdrguys.com/subversion/sql_fs_docs/svn_fs_sql.htm and
http://www.cdrguys.com/subversion/fs_refactoring/index.html). But,
unlike the following, that seems to be quite an ambitious and serious

Hopefully I'll get something *working* in the next 48 hours or so, and
I'll put the code up somewhere for other people to take away and
(optionally, but hopefully) improve.

I. Database structure

   1. Keys for records: BDB generally uses base-36 strings for record
      The keys, providing a ludicrous maximum number of possible keys.
      obvious MySQL equivalent is an INT AUTO_INCREMENT field.
      (Currently I just use a VARCHAR with a base-36 string, with the
      next available key stored in a record called 'next-id', in the
      first VARCHAR() field of that record. UGLY.)

   2. Skels: Since MySQL allows more than one non-key column in a
      table, I've done away with using skels. Data that is normally
      stored in list skels now needs to reside in an additional table,
      for instance, the 'representations' and 'transactions' tables.
      (Currently I have added tables 'windows', 'transaction_props',
      and 'transaction_copies' to contain that data. Today, I also
      talked briefly with cmpilato on #svn about storing a TXN
      reference in each copy, instead of using 'transaction_copies'.)

   3. Strings: The BDB implementation stores strings as multiple
      records under a single key. I'm not sure if this is necessary
      for an SQL implementation. (Though I currently do the same
      thing, except that each string record has additional 'from'
      and 'to' fields that can be used to find appropriate string
      records for a particular segment of a string.)

   4. I'm using the InnoDB table type, which provides "transactions".
      I personally haven't had much experience with transactional
      databases, so I don't yet know if it'll be easy to provide the
      same level of ACIDity as BDB has, but I have done some testing.
      That is, I've tested how the 'next-id' record is used by
      concurrent processses, and realised that "FOR UPDATE" must be
      appended to SELECT queries that retrieve the next-id. This
      probably wouldn't be necessary if AUTO_INCREMENTs were used;
      but it might well be necessary for other records.

II. Code Integration

   5. The MySQL-specific stuff is in a mysql/ subdirectory, in the
      same vein as the bdb/ subdir. But there is still a lot of
      BDB-specific stuff in fs.c and trail.c.

   6. I haven't written code to track the memory allocated by the
      MySQL functions yet. And it is possible that a lot more
      memory is being used, since the SQL is generated and stored in
      trail->pool (if the amount is significant I'll look into using

   7. I have read about G. Thompson's fs vtable work (though I
      haven't seen it), and since it sounds like a good thing,
      I've avoided duplicating the effort.

   8. For the record, I'm also not sure how to translate the BDB error
      macros to elegant MySQL equivalents...

III. The Repository

   9. The MySQL database is stored outside the repository, and accessed
      through sockets using the MySQL client library. Where should the
      MySQL configuation data (hostname, username, password, database)
      by stored, and what should be stored in place of the BDB data?
      (Currently the MySQL data is stored in fs.c!) I'm inclined to
      keeping the code to recognise a MySQL or BDB repository, and
      creating/opening/etc'ing the database, in libsvn_fs.

To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Feb 19 11:56:12 2004

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.