[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: mass rearrange and berkeley db lock exhaustion

From: J Robert Ray <jrray_at_imageworks.com>
Date: 2004-10-12 22:56:04 CEST

kfogel@collab.net wrote:
> J Robert Ray <jrray@imageworks.com> writes:
>
>>Maybe unrelated, my script segfaults periodically in libneon
>>_init(). Here's a sample stack trace:
>
>
> Oooh. No idea what's going on there, sorry. I suspect the neon init
> segfaults are unrelated to the server-side lock count problem, unless
> you notice some correlation between them (?).

Hi Karl,

I'm waiting for approval from my manager to share my script with this
list. In the meantime, I'll copy here an email I sent privately to Ben.
  This is focused more on the segfault problem (and severe memory leak).

I have a couple more observations to share with you.

I have moved my test environment up to 1.10 client/server and migrated
over to fsfs, to side-step the berkeley db grief.

Still getting nailed by the periodic segfault, I hacked up my build
script to force neon to be compiled with just '-g', to better debug the
problem. After doing so, the segfault goes away and is replaced by a
"friendlier" error message:

RA layer request failed: PROPFIND request failed on '/svn/repo/moo':
PROPFIND of '/svn/repo/moo': Could not read status line: Bad file
descriptor (http://pippen) at ./svn_restructure.pl line 39

Okay, time to try to bypass neon. I switched over to using file://...
First hurdle was the is_canonical assertion.

$svn->ls('http://foo/svn/repo/bar') fails assertion.
$svn->ls('http://foo/svn/repo/bar/') passes assertion.
$svn->ls('file:///repo/bar') passes assertion.
$svn->ls('file:///repo/bar/') fails assertion.

This is frustrating! The svn command line tool doesn't care either way.
  But no matter, I adjusted my script to make it happy.

The big surprise here is when using file:// URLs it leaks memory like
crazy. It ate up all 2GB of ram in less than a minute of runtime
(client side). So much for that idea.

Here's a stack trace for the 'RA layer request...' problem I mentioned
above:

Breakpoint 1, aborted (req=0xae10620,
     doing=0x407f754b "Could not read status line", code=-1)
     at ne_request.c:197
197 ne_session *sess = req->session;
(gdb) where
#0 aborted (req=0xae10620, doing=0x407f754b "Could not read status line",
     code=-1) at ne_request.c:197
#1 0x407e7705 in read_status_line (req=0xae10620, status=0xae12744,
retry=0)
     at ne_request.c:923
#2 0x407e7a45 in send_request (req=0xae10620, request=0xadf4220)
     at ne_request.c:995
#3 0x407e8240 in ne_begin_request (req=0xae10620) at ne_request.c:1179
#4 0x407e8541 in ne_request_dispatch (req=0xae10620) at ne_request.c:1263
#5 0x406e1497 in parsed_request (sess=0xae0a420,
     method=0x406e389d "PROPFIND", url=0xade9258 "/svn/birps_test",
     body=0xad81b68 "<?xml version=\"1.0\" encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., body_file=0x0,
set_parser=0x406dddb1 <set_parser>, elements=0x406e5660,
     use_neon_shim=1, validate_compat_cb=0x406dd764 <validate_element>,
     startelm_compat_cb=0x406dd8c1 <start_element>,
     endelm_compat_cb=0x406dda41 <end_element>, startelm_cb=0, cdata_cb=0,
     endelm_cb=0, baton=0xbfffdcc0, extra_headers=0xade9268,
status_code=0x0,
     pool=0x8320d18) at subversion/libsvn_ra_dav/util.c:603
#6 0x406e1848 in svn_ra_dav__parsed_request_compat (sess=0xae0a420,
     method=0x406e389d "PROPFIND", url=0xade9258 "/svn/birps_test",
     body=0xad81b68 "<?xml version=\"1.0\" encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., body_file=0x0,
set_parser=0x406dddb1 <set_parser>, elements=0x406e5660,
     validate_cb=0x406dd764 <validate_element>,
     startelm_cb=0x406dd8c1 <start_element>,
     endelm_cb=0x406dda41 <end_element>, baton=0xbfffdcc0,
     extra_headers=0xade9268, status_code=0x0, pool=0x8320d18)
     at subversion/libsvn_ra_dav/util.c:716
#7 0x406de04d in svn_ra_dav__get_props (results=0xbfffdd40, sess=0xae0a420,
     url=0xade9258 "/svn/birps_test", depth=0, label=0x0,
     which_props=0x406e5520, pool=0x8320d18)
     at subversion/libsvn_ra_dav/props.c:527
#8 0x406de0f6 in svn_ra_dav__get_props_resource (rsrc=0xbfffde5c,
     sess=0xae0a420, url=0xade9238 "/svn/birps_test", label=0x0,
     which_props=0x406e5520, pool=0x8320d18)
     at subversion/libsvn_ra_dav/props.c:554
#9 0x406de364 in svn_ra_dav__get_starting_props (rsrc=0xbfffde5c,
     sess=0xae0a420, url=0xade9238 "/svn/birps_test", label=0x0,
     pool=0x8320d18) at subversion/libsvn_ra_dav/props.c:626
#10 0x406de3f6 in svn_ra_dav__search_for_starting_props (rsrc=0xbfffde5c,
     missing_path=0xbfffde4c, sess=0xae0a420, url=0xadae6a8
"/svn/birps_test",
     pool=0x8320d18) at subversion/libsvn_ra_dav/props.c:656
#11 0x406de653 in svn_ra_dav__get_baseline_props (bc_relative=0xbfffdea8,
     bln_rsrc=0xbfffdebc, sess=0xae0a420, url=0xadae6a8 "/svn/birps_test",
     revision=-1, which_props=0x406e5548, pool=0x8320d18)
     at subversion/libsvn_ra_dav/props.c:765
#12 0x406de904 in svn_ra_dav__get_baseline_info (is_dir=0x0, bc_url=0x0,
     bc_relative=0x0, latest_rev=0xbfffdfd4, sess=0xae0a420,
     url=0xadae6a8 "/svn/birps_test", revision=-1, pool=0x8320d18)
     at subversion/libsvn_ra_dav/props.c:893
#13 0x406d7f2b in svn_ra_dav__get_latest_revnum (session_baton=0xade91c0,
     latest_revnum=0xbfffdfd4, pool=0x8320d18)
     at subversion/libsvn_ra_dav/fetch.c:1011
#14 0x4044d24c in svn_client__get_revision_number (revnum=0xbfffdfd4,
     ra_lib=0x406e57e0, sess=0xade91c0, revision=0xbfffe120, path=0x0,
     pool=0x8320d18) at subversion/libsvn_client/revisions.c:68
#15 0x4043c76f in repos_to_repos_copy (commit_info=0xbfffe11c,
     src_url=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
     dst_url=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
ctx=0xa731280, is_move=1, pool=0x8320d18)
     at subversion/libsvn_client/copy.c:370
#16 0x4043e42e in setup_copy (commit_info=0xbfffe11c,
     src_path=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
     dst_path=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
is_move=1, force=0, ctx=0xa731280, pool=0x8320d18)
     at subversion/libsvn_client/copy.c:1115
#17 0x4043e4d9 in svn_client_move (commit_info=0xbfffe11c,
     src_path=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
     dst_path=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
force=0, ctx=0xa731280, pool=0x8320d18)
     at subversion/libsvn_client/copy.c:1151
#18 0x4097b909 in _wrap_svn_client_move (cv=0x82ed360) at svn_client.c:5436
#19 0x080a314f in Perl_pp_entersub ()
#20 0x0809d3aa in Perl_runops_standard ()
#21 0x0805f117 in S_run_body ()
#22 0x0805eeaa in perl_run ()
#23 0x0805c8e3 in main ()
#24 0x40064914 in __libc_start_main () from /lib/libc.so.6

(gdb) up
#1 0x407e7705 in read_status_line (req=0xae10620, status=0xae12744,
retry=0)
     at ne_request.c:923
923 int aret = aborted(req, _("Could not read status line"),
ret);

(gdb) p *req
$1 = {method = 0xadadc08 "PROPFIND", uri = 0xadbacc0 "/svn/birps_test",
   headers = 0xadf4ce8, body_cb = 0x407e6370 <body_string_send>,
   body_ud = 0xae10620, body = {fd = 181934952, buf = {
       buffer = 0xad81b68 "<?xml version=\"1.0\"
encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., pnt = 0xad81c94 "", left = 0}},
body_size = 300, body_progress = 0,
   respbuf = '\0' <repeats 8191 times>, resp = {length = 0, left = 0,
     chunk_left = 0, total = 0, mode = R_TILLEOF}, header_catchers = 0x0,
   private = 0xadf4d98, header_handlers = {0x0, 0x0, 0x0, 0x0, 0xad72350,
     0x0 <repeats 14 times>, 0xadb8d60, 0x0, 0x0, 0xadbacd8,
     0x0 <repeats 12 times>, 0xadd9148, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
     0x0, 0x0, 0x0, 0xae01088, 0xad72320, 0x0, 0x0, 0x0, 0x0, 0x0},
   body_readers = 0xadd9178, method_is_head = 0, use_expect100 = 0,
   can_persist = 0, session = 0xae0a420, status = {major_version = 0,
     minor_version = 0, code = 0, klass = 0, reason_phrase = 0x0}}

(gdb) p *req->session
$2 = {socket = 0xae16850, connected = 1, persisted = 0, is_http11 = 0,
   scheme = 0xae00e30 "http", server = {hostname = 0xadae690 "softboss",
     port = 81, address = 0xae011a0, current = 0xadf41e8,
     hostport = 0xae07318 "softboss:81"}, proxy = {hostname = 0x0, port = 0,
     address = 0x0, current = 0x0, hostport = 0x0}, use_proxy = 0,
   no_persist = 0, use_ssl = 0, in_connect = 0, expect100_works = -1,
   progress_cb = 0, progress_ud = 0x0, notify_cb = 0, notify_ud = 0x0,
   rdtimeout = 3600, create_req_hooks = 0xadf4cd0, pre_send_hooks =
0xae00e60,
   post_send_hooks = 0xae00e78, destroy_req_hooks = 0xadf4d20,
   destroy_sess_hooks = 0xadf4d38, private = 0xadbaaa0,
   user_agent = 0xadf4818 "User-Agent: SVN/1.1.0 (dev build, r2)
neon/0.24.7\r\n", ssl_verify_fn = 0, ssl_verify_ud = 0x0, ssl_provide_fn
= 0,
   ssl_provide_ud = 0x0, error = "Unknown error.", '\0' <repeats 8177
times>}

(gdb) p *req->session->socket
$3 = {fd = 1122, error = "Bad file descriptor", '\0' <repeats 180 times>,
   progress_ud = 0x0, rdtimeout = 3600, ops = 0x407fa094,
   buffer = '\0' <repeats 4095 times>, bufpos = 0xae16928 "", bufavail = 0}

(gdb)

I'll leave the gdb session open for a while if you want me to poke at
anything in particular.

Cheers,

- Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Received on Tue Oct 12 22:56:21 2004

This is an archived mail posted to the Subversion Users mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.