Mark Benedetto King wrote:
> Also, many collection files suffer O(N) entry lookups because their
> own file tables are not indexed. This can actually be significantly worse
> than the filesystem's performance. Of course, we don't need to use
> ZIP/JAR; we could build our own format that didn't exhibit this behaviour.
I seem to recall that for the longest time, the Oracle RDBMS preferred
you to point your database data files to raw block devices, rather than
in filesystem files (i.e. /dev/hdb1 rather than /var/spool/oracle.dat)
for a similar reason. At the time it was pretty clear that Oracle
could do a much better (read: faster) job managing all the raw
data itself.
These days, I haven't seen an Oracle database stored on a raw device
for a while, but I've not been looking hard. Oracle still uses a
very few "great big files" to store data, and manages its own indexing
and metadata, rather than relying on the filesystem provided by the
OS (even on Win32 with NTFS. In fact I think Oracle has always
used one-big-file on Win32, but that's just a guess, not a fact).
I think the key points to consider are:
1) You can get best performance by using a raw device and totally
managing it yourself.
2) Doing total self-management means lots of complicated code,
and lots of application-unrelated algorithms.
3) The choice of storage mechanism you make is going to affect
performance. The more generic the storage, the more likely
it is to be slow.
4) Each part of the storage logic you add to your application is
probably going to boost performance, at the cost of code
complexity.
What all this says to me is that maybe a more customisable storage
backend would be a useful thing (i.e. on ReiserFS, store lots of
small files, because Reiser is good at that. On NTFS, use
fewer, bigger files, because that works better there). But that
means more code :(
daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 23 02:44:03 2004