On Mon, 13 Dec 2004 kfogel@collab.net wrote:
> It still looks like an out-of-memory error ("abort_on_pool_failure" in
> the stack trace), hmmm. Both your client and server were on the same
> box, and indeed in the same process, when you reproduced this, right?
Yes. The commands I ran to initialize the repository were:
sh$ export REPO=/u3/cphillip/dev/cuwin/subversion-os-dev/trunk-rev-12289/repo
(cwd) n/a
sh$ svnadmin --create-type fsfs repo
(cwd) /u3/cphillip/dev/cuwin/subversion-os-dev/trunk-rev-12289
sh$ svn import file://$REPO/branches/netbsd-start/
(cwd) /u3/cphillip/dev/cuwin/subversion-os-dev/netbsd-start/src/
sh$ svn cp file://$REPO/branches/netbsd-start/ \
file://$REPO/branches/netbsd/
(cwd) n/a
The command that resulted in the core dump was:
sh$ svn co file://$REPO/ wd
(cwd) /u3/cphillip/dev/cuwin/subversion-os-dev/trunk-rev-12289/
The output of 'df .' from
/u3/cphillip/dev/cuwin/subversion-os-dev/trunk-rev-12289/ is:
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/wd1a 128220292 78734656 43074624 64% /u3
The entirety of the trunk-rev-12289/ directory and its fs children is on
/u3.
> I could try to make an educated guess from the stack trace, but it
> would be great if we could narrow this down to "server code", or
> "client code", or both. (Even when they're in the same process,
> they're distinct bodies of code.)
Understood. If my answers don't provide you enough information in a
particular area please ask me for clarification. I'm happy to provide it.
> When you say "subsequent checkout", you mean a first-time checkout of
> the new branch, right?
Yes. It is the first checkout of the whole repository (not just the new
branch) that is the next activity to occur after the copy of one branch to
another.
> What can you tell us about your hardware, memory, etc? (Not because
> they're causing the bug in any sense, just to help us figure out what
> we need to reproduce it.)
The system is a Dell PowerEdge 400SC. dmesg.boot states there are 639 MB
of total memory and 616 MB of available memory. The processor is an Intel
Pentium 4 CPU (2.4GHz) with 512 KB of cache.
The software is NetBSD running a late version that has been modified to
include bug fixes from the NetBSD source code. It is built using, quote,
a GENERIC kernel configuration file with minor patches:
+uftdi0 at uhub? port ? # David's travelling FTDI FT8U100AX
+ # serial adapter
+ucom0 at uftdi0 portno 1
End quote.
uname -a reports:
NetBSD cuw.ojctech.com 2.99.10 NetBSD 2.99.10 (GENERIC.cuw) #0: Sun Nov
14 15:35:49 CST 2004 dyoung@cuw.ojctech.com:/u3/dyoung/pristine-nbsd/O/
sys/arch/i386/compile/GENERIC.cuw i386
> Can we get our hands on your data?
Our desired initial work state is based on an import of the tarball
available at:
http://che.ojctech.com/~dyoung/public/netbsd-start.tar.gz
MD5: a1ca74ad688ac280433785ac8bdacb8b
My process is to run (from within netbsd-start/src/):
sh$ svn admin --create-type fsfs /local/repo
sh$ svn import file:///local/repo/branches/netbsd-start/
sh$ svn cp file:///local/repo/branches/netbsd-start \
file:///local/repo/branches/netbsd
sh$ svn checkout file:///local/repo/
sh$ cd branches/netbsd
sh$ find . -type d -name CVS -exec rm -rf \{\} \;
sh$ svn commit
With different versions, I've experienced trouble with the import,
checkout, and the commit. In r12289, I did not have trouble with the
import and did not have the opportunity to try the commit.
> Does it reproduce with BDB instead of FSFS?
I haven't had an opportunity to try this with BDB so I don't currently
have an answer to that question. I'm happy to just get the report in at
this point.
> > The Subversion issue tracker holds 4 issues that come close to addressing
> > this but for one reason or another don't match up well enough to allow me
> > to assume they should be used as the target for this issue.
> >
> > Issue 602 - http://subversion.tigris.org/issues/show_bug.cgi?id=602
> >
> > "import of large trees can bloat memory on client side"
> >
> > The last comment to this bug was made 2002/11 and it appears purposed to
> > handle import efficiency (I've not had this resource issue doing an
> > import of the source code).
>
> Agree that this is probably not your bug.
>
> > Issue 1702 - http://subversion.tigris.org/issues/show_bug.cgi?id=1702
> >
> > "Initial checkout should scale well for large projects"
> >
> > This issue focuses on checking out a revision from a remote repository.
> > In my scenario, I am checking out from a local repository.
>
> Well, the real point is that 1702 is about time performance, not
> memory growth. The local vs remote thing is not such a big
> difference. Many problems that are first reported in remote
> operations are also present in local operations; it just means they
> are problems in the core libraries, not the transport layer libraries.
>
> > Issue 2067 - http://subversion.tigris.org/issues/show_bug.cgi?id=2067
> >
> > "Perf issues with BDB and directories with a large number of items"
> >
> > This bug mentions similar problems with FSFS, though the bug summary
> > refers strictly to BDB. Is it meant to cover only issues with BDB-based
> > repositories?
>
> It looks like it's still mainly about BDB, and anyway is mainly about
> import and commit scalability, not specifically about checkouts.
>
> > Issue 2137 - http://subversion.tigris.org/issues/show_bug.cgi?id=2137
> >
> > "svn update" extremely slow with large repository and FSFS"
> >
> > The common problems I experience are with checkouts and commits back to
> > the local repository. Again, I'm using a late revision of the trunk
> > (r12289).
>
> Your earlier description says "a subsequent checkout leads to a core
> dump". Here you say you are also having problems with commits. I
> suspect the checkout problems are unrelated to the commit ones, so we
> should have two separate threads for them. In this reply, I've only
> been talking about the checkout problem (because until this moment,
> that's the only one I knew about anyway).
Okay. IMO it's sufficient to note that I have experienced these same
resource usage issues during not just commits but other operations, as
well. These problems occurred while attempting the process I mentioned
earlier in this email against released versions of Subversion. I also
refer to them in my previous email to users@subversion:
http://svn.haxx.se/users/archive-2004-11/0180.shtml
In response to these issues at that time, the possibility of a fix existed
(thanks to Eric!). I decided to prepare for testing a new version instead
of iterating to prepare an 'official' bug report.
In this report, which describes my attempt using r12289, I did not
experience a core dump during the import, only during the checkout.
> I think a new issue would be best. As much reproduction information
> as you can give us (numbers of files, sizes of files, names of files)
> would be great.
I will compile this information asap and place it in the issue tracker.
I was hoping that the patches already in the trunk would have been enough
to remove the resource usage issues and am eager to get our team unblocked
on doing more serious NetBSD SCM.
> Thanks very much for the report -- and the care you took to make it so
> organized & comprehensible.
I've been on all sides of the stick with bug reports (not giving enough
info, giving too much, not having enough, having too much). Finding the
sweet spot for a report can be a real trick. And in some cases generating
a useful report takes so long that it's just not easy to do as a
volunteer. Makes the fast turnaround times the Subversion team pulls off
very impressive.
Thank you to the Subversion team and its volunteers for the thousands of
man-years and more that have gone into this project!
Chase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Tue Dec 14 01:57:41 2004