I have done some digging on this problem. Find attached a log of my
thoughts and
what I have tried so far. In summary it looks like if you have a large
number of
files in a directory that you get problems when doing an "svn export".
Possibly
due to memory being overwritten, the underlying
readdir()/getdirentries() system
calls on the Mac are prematurely returning the end of the directory and
thus not
all prop-base files are being removed. A standalone test program using
the APR
routines directly has no problem listing all files in the directory,
only within
the context of "svn" are problems arising.
If you can possibly get someone else with Mac OS X to create a
directory under
svn with a few hundred files and export it directly from the repository
and
see if they can duplicate it, it would be useful to see if it is a
general problem
across all Macs.
Error generated from svn is:
A export/TraceSwitch.hh
Exported revision 2.
svn: Directory not empty
svn: svn_io_remove_dir: removing `export/.svn/prop-base'
Directory contains 215 files, but the prop-base files of only 132 are
removed.
If "ktrace" is used to monitor system calls of "svn" when export is
done, see call to getdirentries() call at start of directory traversal
to remove files:
595 svn CALL open(0x7eb170,0,0x16)
595 svn NAMI "export/.svn/prop-base"
595 svn RET open 14/0xe
595 svn CALL fstat(0xe,0xbffff270)
595 svn RET fstat 0
595 svn CALL fcntl(0xe,0x2,0x1)
595 svn RET fcntl 0
595 svn CALL fstatfs(0xe,0xbffff2d0)
595 svn RET fstatfs 0
595 svn CALL getdirentries(0xe,0x7d5610,0x1000,0x7eb0c4)
595 svn RET getdirentries 4076/0xfec
595 svn CALL stat(0x7eb318,0xbffff280)
595 svn NAMI "export/.svn/prop-base/Action.hh.svn-base"
595 svn RET stat 0
595 svn CALL chmod(0x7eb318,0x1b6)
595 svn NAMI "export/.svn/prop-base/Action.hh.svn-base"
595 svn RET chmod 0
595 svn CALL unlink(0x7eb348)
595 svn NAMI "export/.svn/prop-base/Action.hh.svn-base"
595 svn RET unlink 0
595 svn CALL stat(0x7eb3b8,0xbffff280)
When that block of file entries is dealt with and next call to the
getdirentries() function is made, it returns 0 and not the next set
of file entries. This is causing the loop to prematurely exit and
thus not all files are removed.
595 svn CALL stat(0x7fa458,0xbffff280)
595 svn NAMI "export/.svn/prop-base/Pattern.hh.svn-base"
595 svn RET stat 0
595 svn CALL chmod(0x7fa458,0x1b6)
595 svn NAMI "export/.svn/prop-base/Pattern.hh.svn-base"
595 svn RET chmod 0
595 svn CALL unlink(0x7fa488)
595 svn NAMI "export/.svn/prop-base/Pattern.hh.svn-base"
595 svn RET unlink 0
595 svn CALL getdirentries(0xe,0x7d5610,0x1000,0x7eb0c4)
595 svn RET getdirentries 0
595 svn CALL lseek(0xe,0,0,0)
595 svn RET lseek 0
595 svn CALL close(0xe)
595 svn RET close 0
595 svn CALL rmdir(0x7eb170)
595 svn NAMI "export/.svn/prop-base"
595 svn RET rmdir -1 errno 66 Directory not empty
The filesystem type is the default for Mac OS X, ie., HFS+.
The function getdirentries() isn't called directly by APR code so assume
that readdir() calls it underneath. The getdirentries() function isn't
called on each readdir() though, only when next filesystem block needs
to be read.
On first glance it would appear that if there are enough files in a
directory, or if combination of number of files and length of filenames
results in getdirentries() having to be called more than once, second
getdirentries() is returning 0 when it shouldn't.
If 256 files are created with names 000 up to 255, same error occurs but
with 169 out of 256 files removed. Use of simple name ensures it isn't
anything related to the actual names of files used. Have to remember though
that the prop-base files have .svn-base appended which makes the names
somewhat longer.
My C++ library code for scanning contents of a directory can correctly
display all the entries and obviously "ls" works, so possibly something to
do with how APR wraps it up. Note that APR is using readdir() and not
the readdir_r() system function.
If 1000 files are created with names 000 up to 999, same error occurs but
it was possible to remove 509 of the files which is more than before so
somehow related to the total number in the directory as well.
If the apr_dir_read() function is augmented with a printf(), see that
it has no problem traversing all 1000 actual files under version control.
Ie., those in directory where .svn is subdirectory, but when traversing the
.svn/prob-base directory it doesn't see all names.
The "dirstruct" pointer value doesn't itself change as files are traversed
so it isn't getting overwritten in any way, but still possible that what
it points at is and getdirentries() thus thinks it is at the end of the
directory when it isn't. The "dirstruct" is allocated and returned by
opendir().
Need to construct code which calls APR routines for directory traversal
into separate test program to see what happens then.
apr_pool_t *p;
int main()
{
apr_status_t rv;
apr_dir_t *dir;
apr_finfo_t entry;
apr_int32_t flags = APR_FINFO_TYPE | APR_FINFO_NAME;
apr_initialize();
atexit(apr_terminate);
apr_pool_create(&p, NULL);
rv = apr_dir_open(&dir, ".", p);
for (rv = apr_dir_read (&entry, flags, dir);
rv == APR_SUCCESS;
rv = apr_dir_read (&entry, flags, dir))
{
fprintf(stderr,"%s\n",entry.name);
}
apr_dir_close(dir);
return 0;
}
With this though, there is no problem with traversing prop-base directory
with 256 files. Using "ktrace" see that getdirentries() was called multiple
times successfully whereas on second time previously it returns that end
of directory has been found.
This suggests that APR routines are only having a problem with the context
of the "svn" application?
Where to go from here?
On Thursday, April 17, 2003, at 10:45 PM, Graham Dumpleton wrote:
>
> On Tuesday, April 15, 2003, at 01:35 AM, Ben Collins-Sussman wrote:
>
>> Graham Dumpleton <grahamd@dscpl.com.au> writes:
>>
>>> ~/Projects/tmpdir [511]$ svn --version
>>> svn, version 0.19.1 (r5301)
>>> compiled Apr 12 2003, 20:23:12
>>>
>>> The platform is Mac OS X:
>>
>> You're nearly 3 releases behind. Upgrade to the latest release
>> (0.20.1 or later) and see if 'svn export' still fails for you. If so,
>> we'll take a look at it.
>
> With just released 0.21.0 it still happens.
>
> Exported revision 27.
> svn: Directory not empty
> svn: svn_io_remove_dir: removing
> `/tmp/ose-7.0pl6/include/OTC/.svn/prop-base'
> ~ [512]$ /usr/local/subversion-0.21.0/bin/svn --version
> svn, version 0.21.0 (r5639)
> compiled Apr 17 2003, 22:16:55
>
> BTW, on the Mac I found that 0.20.1 and 0.21.0 would fail to build if
> 0.19.1 had been
> installed under /usr/local and the later versions were also targeted
> to install under
> /usr/local. The problem was that the already installed libraries were
> somehow being
> picked up when building shared libraries in new version being built.
> This was resulting
> in undefined and multiply defined symbols depending on which version
> you were trying
> to build.
>
> Also, 0.20.1 in fink unstable tree, although it would build and
> install, would fail
> with the same problem I think the subversion web site repository was
> having. Ie.,
>
> ~ [514]$ /sw/bin/svn list file:///Users/grahamd/Projects/Subversion
> svn: Couldn't open a repository.
> svn: Unable to open an ra_local session to URL
> svn: Unable to open repository
> 'file:///Users/grahamd/Projects/Subversion'
> svn: Berkeley DB error
> svn: Berkeley DB error while opening environment for filesystem
> /Users/grahamd/Projects/Subversion/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> ~ [515]$ /sw/bin/svn --version
> svn, version 0.20.1 (r5467)
> compiled Apr 15 2003, 17:49:01
>
> Although it suggested database recovery needed to be done, it didn't
> seem to really
> need it as running 0.19.1 or 0.21.0 was still able to access the same
> database without
> needing to do anything.
>
> Anyway, now I have a recipe for building versions which avoids linker
> symbol problems,
> I can start trying to track down what it is about that directory I am
> trying to export
> which is a problem. I will let you know what I find out and if
> possible construct a
> working minimal repository with just what triggers the problem.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Fri Apr 18 04:27:41 2003