On Thu, 2002-08-15 at 09:09, Karl Fogel wrote:
> Your import error might be related to
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=860
>
> Anyway, obviously yes, it's a bug :-). If we can't identify it as an
> already-known bug from your description, then we'll probably ask you
> for a minimal data set that reproduces the problem (so don't throw
> those files out just yet).
>
> But I suspect someone can recognize this as a manifestation of a known
> scalability problem; does it ring a bell, anyone? (In fact, I feel
> like I ought to recognize it myself, but I don't).
After you mentioned this, I went and cleared out the repository to start
fresh. Rebooted the server so all memory was cleared. I then tried to
import the full directory tree:
105MB of mixed data (images, docs, text, etc)
6205 total files
Started off ok with 305MB of memory being used before the "Transmitting
data..." text showed up. Then it dropped to around 200MB, so 100MB was
freed at that point.
About halfway through all the little dots, memory usage jumped back to
around 340MB and steadily grew from there. Using top and ps, it was not
readily apparent what was using so much memory as no httpd was above
10MB and svn was around 50MB (60MB at its peak usage).
When the dots stopped all of a sudden an httpd process started eating
all available memory to include 1G of swap space:
I removed a lot of the ps output so it would fit without wrapping:
VSZ RSS
949056 488624 ? D 09:39 1:24 /var/securewww/bin/httpd
59656 1104 pts/1 S 09:46 0:47 svn import
996160 489664 ? R 09:39 1:26 /var/securewww/bin/httpd
59656 1116 pts/1 S 09:46 0:47 svn import
1034048 490008 ? R 09:39 1:27 /var/securewww/bin/httpd
59656 1120 pts/1 S 09:46 0:47 svn import
1115968 485204 ? R 09:39 1:29 /var/securewww/bin/httpd
59656 1124 pts/1 S 09:46 0:47 svn import
1230656 493436 ? R 09:39 1:32 /var/securewww/bin/httpd
59656 1124 pts/1 S 09:46 0:47 svn import
1278784 494124 ? D 09:39 1:33 /var/securewww/bin/httpd
59656 1128 pts/1 S 09:46 0:47 svn import
1417024 480900 ? D 09:39 1:37 /var/securewww/bin/httpd
59656 1156 pts/1 S 09:46 0:47 svn import
1447744 493896 ? R 09:39 1:38 /var/securewww/bin/httpd
59656 1148 pts/1 S 09:46 0:47 svn import
0 0 ? Z 09:39 1:40 [httpd <defunct>] #Memory exhausted
59656 11168 pts/1 D 09:46 0:47 svn import
I don't understand what is going on behind the scenes that would cause
the import of 105MB of files to use so much memory.
> Shay Harding <sharding@ccbill.com> writes:
> > Second question relates to the 'db' directory within the repository.
> > What is the 'strings' file used for? After importing only 1 of many
> > directory trees, it sits at 62M. Will it matter how big this file gets?
> > I have roughly 15 projects to import, although not all of them are
> > nearly as large as 65M.
>
> The `strings' table is where all the file contents are stored, so yes,
> it will get about as big as the tree you import (the other tables hold
> metadata such as directory entries).
>
> So it just gets larger the more data you import. In the
> multi-100s-of-megabytes range you're talking about, it shouldn't be a
> problem I think.
>
> But hmmm. Don't some systems start having problems when a single file
> is in the multi-*gigabyte* range? Or is that only when trying to
> access it through stdio? I presume Berkeley DB already has
> optimizations for large files, but will check...
>
> -K
File size probably shouldn't be a problem. Currently running under Linux
which supports 4G (maybe more?) files. I'm pretty sure BerkeleyDB will
handle files to that size since MySQL uses it on the back-end for it's
InnoDB tables (I *think*). At any rate I know we have DB files in excess
of 1G so I'm not too concerned about that.
Shay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 15 19:31:21 2002