[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: possible win32 fsfs performance

From: Daniel Patterson <danpat_at_danpat.net>
Date: 2004-10-23 02:41:42 CEST

Mark Benedetto King wrote:
> Also, many collection files suffer O(N) entry lookups because their
> own file tables are not indexed. This can actually be significantly worse
> than the filesystem's performance. Of course, we don't need to use
> ZIP/JAR; we could build our own format that didn't exhibit this behaviour.

   I seem to recall that for the longest time, the Oracle RDBMS preferred
   you to point your database data files to raw block devices, rather than
   in filesystem files (i.e. /dev/hdb1 rather than /var/spool/oracle.dat)
   for a similar reason. At the time it was pretty clear that Oracle
   could do a much better (read: faster) job managing all the raw
   data itself.

   These days, I haven't seen an Oracle database stored on a raw device
   for a while, but I've not been looking hard. Oracle still uses a
   very few "great big files" to store data, and manages its own indexing
   and metadata, rather than relying on the filesystem provided by the
   OS (even on Win32 with NTFS. In fact I think Oracle has always
   used one-big-file on Win32, but that's just a guess, not a fact).

   I think the key points to consider are:

     1) You can get best performance by using a raw device and totally
        managing it yourself.
     2) Doing total self-management means lots of complicated code,
        and lots of application-unrelated algorithms.
     3) The choice of storage mechanism you make is going to affect
        performance. The more generic the storage, the more likely
        it is to be slow.
     4) Each part of the storage logic you add to your application is
        probably going to boost performance, at the cost of code
        complexity.

   What all this says to me is that maybe a more customisable storage
   backend would be a useful thing (i.e. on ReiserFS, store lots of
   small files, because Reiser is good at that. On NTFS, use
   fewer, bigger files, because that works better there). But that
   means more code :(

daniel

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 23 02:44:03 2004

This is an archived mail posted to the Subversion Dev mailing list.