Just for the record, I still can't use ra_serf, with serf from the
0.6.x branch.
$ head CHANGES
Serf 0.6.2
Fix double free abort when destroying request buckets.
Fix test server in unit test framework to avoid random test failures.
I'd like to switch to using ra_serf to test it, but in the current
state it's completely unusable for me.
This crash below happens reliably during
svn co https://svn.apache.org/repos/asf/subversion/trunk
somewhere in the middle of the checkout (not always at the same spot):
A trunk/subversion/libsvn_fs_base/notes/structure
A trunk/subversion/libsvn_fs_base/notes/TODO
A trunk/subversion/libsvn_fs_base/notes/schema-bdb-1.6.svg
A trunk/subversion/libsvn_fs_base/notes/fs-history
A trunk/subversion/libsvn_ra
A trunk/subversion/libsvn_ra/deprecated.c
A trunk/subversion/libsvn_ra/wrapper_template.h
Segmentation fault (core dumped)
I think the cause of this is either an accidental memory leak,
or wasteful handling of memory. Since a malloc return value isn't
checked, svn eventually runs into a null pointer when it fails to
allocate more memory.
Shortly before crashing, svn is already using half a GB of memory:
20016 stsp -6 0 512M 355M sleep/1 buf_nee 0:13 37.65% svn
I've also seen this error:
Out of memory - terminating application.
Abort trap (core dumped)
That time, the trace was different (see second trace below).
With neon, memory use stabilises at around 40MB during the same operation.
If ra_serf is still not usable for me when we want to release 1.7.0,
I will vote -1 for making ra_serf the default RA layer for 1.7.0 (even
if this means not using HTTPv2 by default). I know that ra_serf works
fine for others, but that is probably because the operating system is
over-committing memory (which in this case means the OS is hiding bugs).
To reproduce this, you can try setting your resource limits to a low value.
If you cannot reproduce it, I'll happily provide shell accounts on my system.
I will also put this on my own todo-list in case no one else gets to it.
Maybe I can figure out the problem.
Stefan
Trace for "Segmentation fault (core dumped)":
#0 0x00000002068ee6b6 in serf_bucket_mem_alloc (allocator=0x20662e190,
size=32) at buckets/allocator.c:192
192 active->next = head;
(gdb) p active
$1 = (apr_memnode_t *) 0x0
(gdb) bt
#0 0x00000002068ee6b6 in serf_bucket_mem_alloc (allocator=0x20662e190,
size=32) at buckets/allocator.c:192
#1 0x00000002068eb0bc in serf_bucket_aggregate_prepend (
aggregate_bucket=0x207835338, prepend_bucket=0x2290a1f38)
at buckets/aggregate_buckets.c:154
#2 0x00000002068f0084 in ssl_encrypt (baton=0x202853038, bufsize=8000,
buf=0x20285308c "\026\003\001", len=0x7f7ffffd1220)
at buckets/ssl_buckets.c:581
#3 0x00000002068ec2ed in common_databuf_prep (databuf=0x202853068,
len=0x7f7ffffd1308) at buckets/buckets.c:316
#4 0x00000002068ec363 in serf_databuf_read (databuf=0x202853068,
requested=18446744073709551615, data=0x7f7ffffd1310, len=0x7f7ffffd1308)
at buckets/buckets.c:334
#5 0x00000002068f16dc in serf_ssl_read (bucket=0x20e7740b8,
requested=18446744073709551615, data=0x7f7ffffd1310, len=0x7f7ffffd1308)
at buckets/ssl_buckets.c:1341
#6 0x00000002068ebc3f in serf_default_read_iovec (bucket=0x20e7740b8,
requested=18446744073709551615, vecs_size=1024, vecs=0x20662a0f0,
vecs_used=0x7f7ffffd138c) at buckets/buckets.c:55
#7 0x00000002068eb349 in read_aggregate (bucket=0x2078351b8,
requested=18446744073709551615, vecs_size=1024, vecs=0x20662a0f0,
vecs_used=0x20662e0f0) at buckets/aggregate_buckets.c:261
#8 0x00000002068eb555 in serf_aggregate_read_iovec (bucket=0x2078351b8,
requested=18446744073709551615, vecs_size=1024, vecs=0x20662a0f0,
vecs_used=0x20662e0f0) at buckets/aggregate_buckets.c:361
#9 0x00000002068e9e82 in write_to_connection (conn=0x20662a028)
at ./outgoing.c:646
#10 0x00000002068ea4cf in serf__process_connection (conn=0x20662a028, events=4)
at ./outgoing.c:1002
#11 0x00000002068e8808 in serf_event_trigger (s=0x20ba681c8,
serf_baton=0x20662a038, desc=0x20ba68628) at ./context.c:199
#12 0x00000002068e896b in serf_context_run (ctx=0x20ba681c8,
duration=-694967296, pool=0x20b4d4028) at ./context.c:260
#13 0x0000000202a8032c in finish_report (report_baton=0x207ebddf0,
pool=0x20b4d4028) at subversion/libsvn_ra_serf/update.c:2278
#14 0x00000002013a97e6 in svn_wc_crawl_revisions5 (wc_ctx=0x20fbfda30,
local_abspath=0x20b4d4158 "/tmp/trunk", reporter=0x202c8b880,
report_baton=0x207ebddf0, restore_files=1, depth=svn_depth_infinity,
honor_depth_exclude=0, depth_compatibility_trick=0, use_commit_times=0,
external_func=0x2108c839d <svn_client__external_info_gatherer>,
external_baton=0x7f7ffffd1930, notify_func=0x416401 <notify>,
notify_baton=0x208c68048, scratch_pool=0x20b4d4028)
at subversion/libsvn_wc/adm_crawler.c:999
#15 0x00000002108f85c6 in update_internal (result_rev=0x0,
local_abspath=0x20b4d4158 "/tmp/trunk",
anchor_abspath=0x20b4d5920 "/tmp/trunk", revision=0x7f7ffffd1e30,
depth=svn_depth_infinity, depth_is_sticky=1, ignore_externals=0,
allow_unver_obstructions=0, timestamp_sleep=0x7f7ffffd1d3c,
send_copyfrom_args=0, innerupdate=0, ctx=0x20fbfd988, pool=0x20b4d4028)
at subversion/libsvn_client/update.c:252
#16 0x00000002108f8972 in svn_client__update_internal (result_rev=0x0,
---Type <return> to continue, or q <return> to quit---
local_abspath=0x20b4d4158 "/tmp/trunk", revision=0x7f7ffffd1e30,
depth=svn_depth_infinity, depth_is_sticky=1, ignore_externals=0,
allow_unver_obstructions=0, timestamp_sleep=0x7f7ffffd1d3c,
send_copyfrom_args=0, innerupdate=0, ctx=0x20fbfd988, pool=0x20b4d4028)
at subversion/libsvn_client/update.c:337
#17 0x00000002108a9170 in initialize_area (result_rev=0x0,
local_abspath=0x20b4d4158 "/tmp/trunk", revision=0x7f7ffffd1e30,
session_url=0x20b4d41f0 "https://svn.apache.org/repos/asf/subversion/trunk", repos_root=0x20b4d41a0 "https://svn.apache.org/repos/asf",
uuid=0x20b4d41c8 "13f79535-47bb-0310-9956-ffa450edef68", revnum=964728,
depth=svn_depth_infinity, use_sleep=0x7f7ffffd1d3c, ignore_externals=0,
allow_unver_obstructions=0, innercheckout=0, ctx=0x20fbfd988,
pool=0x20b4d4028) at subversion/libsvn_client/checkout.c:73
#18 0x00000002108a98cd in svn_client__checkout_internal (result_rev=0x0,
url=0x20b4d4168 "https://svn.apache.org/repos/asf/subversion/trunk",
local_abspath=0x20b4d4158 "/tmp/trunk", peg_revision=0x7f7ffffd1e20,
revision=0x7f7ffffd1e30, ra_cache=0x0, depth=svn_depth_unknown,
ignore_externals=0, allow_unver_obstructions=0, innercheckout=0,
timestamp_sleep=0x0, ctx=0x20fbfd988, pool=0x20b4d4028)
at subversion/libsvn_client/checkout.c:189
#19 0x00000002108a9c84 in svn_client_checkout3 (result_rev=0x0,
URL=0x20b4d40a0 "https://svn.apache.org/repos/asf/subversion/trunk",
path=0x206d03fe0 "trunk", peg_revision=0x7f7ffffd1e20,
revision=0x7f7ffffd1e30, depth=svn_depth_unknown, ignore_externals=0,
allow_unver_obstructions=0, ctx=0x20fbfd988, pool=0x20b4d4028)
at subversion/libsvn_client/checkout.c:271
#20 0x00000000004084f9 in svn_cl__checkout (os=0x20fbfd278,
baton=0x7f7ffffd2140, pool=0x20fbfd028)
at subversion/svn/checkout-cmd.c:166
#21 0x0000000000414901 in main (argc=3, argv=0x7f7ffffd2290)
at subversion/svn/main.c:2312
(gdb)
Trace for "Out of memory - terminating application.
Abort trap (core dumped)":
#0 0x0000000201d5997a in kill () from /usr/lib/libc.so.55.0
(gdb) bt
#0 0x0000000201d5997a in kill () from /usr/lib/libc.so.55.0
#1 0x0000000201dae0c1 in abort () at /usr/src/lib/libc/stdlib/abort.c:68
#2 0x0000000204fdb510 in abort_on_pool_failure (retcode=Could not find the frame base for "abort_on_pool_failure".
)
at subversion/libsvn_subr/pool.c:54
#3 0x00000002013e573e in apr_palloc (pool=0x21594b028, in_size=40)
at /home/stsp/svn/src/apr-1.4.2/memory/unix/apr_pools.c:663
#4 0x0000000202905e7d in svn_ra_serf__xml_push_state (parser=0x215967068,
state=1) at subversion/libsvn_ra_serf/xml.c:314
#5 0x00000002028f5cd9 in start_propfind (parser=0x215967068,
userData=0x215966f48, name=
{namespace = 0x2293eb0c0 "DAV:", name = 0x21f448fe2 "response"},
attrs=0x227614e00) at subversion/libsvn_ra_serf/property.c:261
#6 0x0000000202903cdf in start_xml (userData=0x215967068,
raw_name=0x21f448fe0 "D:response", attrs=0x227614e00)
at subversion/libsvn_ra_serf/util.c:1010
#7 0x00000002037f7a78 in doContent (parser=0x209084000, startTagLevel=0,
enc=0x203a0cb00,
s=0x226562057 "<D:response xmlns:S=\"http://subversion.tigris.org/xmlns/svn/\" xmlns:C=\"http://subversion.tigris.org/xmlns/custom/\" xmlns:V=\"http://subversion.tigris.org/xmlns/dav/\" xmlns:lp1=\"DAV:\" xmlns:lp3=\"http://"...,
end=0x22656252a ' <repeats 200 times>..., nextPtr=0x209084030,
haveMore=1 '\001') at /usr/src/lib/libexpat/lib/xmlparse.c:2364
#8 0x00000002037f8a74 in contentProcessor (parser=0x209084000, start=Variable "start" is not available.
)
at /usr/src/lib/libexpat/lib/xmlparse.c:2027
#9 0x00000002037f9833 in doProlog (parser=0x209084000, enc=0x203a0cb00,
s=0x226562027 "<D:multistatus xmlns:D=\"DAV:\" xmlns:ns0=\"DAV:\">\n<D:response xmlns:S=\"http://subversion.tigris.org/xmlns/svn/\" xmlns:C=\"http://subversion.tigris.org/xmlns/custom/\" xmlns:V=\"http://subversion.tigris.org"...,
end=0x22656252a ' <repeats 200 times>..., tok=29,
next=0x226562027 "<D:multistatus xmlns:D=\"DAV:\" xmlns:ns0=\"DAV:\">\n<D:response xmlns:S=\"http://subversion.tigris.org/xmlns/svn/\" xmlns:C=\"http://subversion.tigris.org/xmlns/custom/\" xmlns:V=\"http://subversion.tigris.org"...,
nextPtr=0x209084030, haveMore=1 '\001')
at /usr/src/lib/libexpat/lib/xmlparse.c:3914
#10 0x00000002037fad4b in prologProcessor (parser=0x209084000,
s=0x226562000 "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<D:multistatus xmlns:D=\"DAV:\" xmlns:ns0=\"DAV:\">\n<D:response xmlns:S=\"http://subversion.tigris.org/xmlns/svn/\" xmlns:C=\"http://subversion.tigris.org/xmlns/custom/"...,
end=0x22656252a ' <repeats 200 times>..., nextPtr=0x209084030)
at /usr/src/lib/libexpat/lib/xmlparse.c:3639
#11 0x00000002037f1831 in XML_ParseBuffer (parser=0x209084000, len=Variable "len" is not available.
)
at /usr/src/lib/libexpat/lib/xmlparse.c:1573
#12 0x0000000202904079 in svn_ra_serf__handle_xml_parser (request=0x21596c3b8,
response=0x22714bab8, baton=0x215967068, pool=0x22939c028)
at subversion/libsvn_ra_serf/util.c:1108
#13 0x0000000202904abd in handle_response (request=0x21596c3b8,
response=0x22714bab8, baton=0x215966ff0, pool=0x22939c028)
at subversion/libsvn_ra_serf/util.c:1416
#14 0x0000000209cfb061 in handle_response (request=0x21596c3b8,
pool=0x22939c028) at ./outgoing.c:752
#15 0x0000000209cfb2cf in read_from_connection (conn=0x2090a5028)
---Type <return> to continue, or q <return> to quit---
at ./outgoing.c:877
#16 0x0000000209cfb448 in serf__process_connection (conn=0x2090a5028, events=1)
at ./outgoing.c:975
#17 0x0000000209cf9808 in serf_event_trigger (s=0x20d42c1c8,
serf_baton=0x2090a5038, desc=0x20d42c628) at ./context.c:199
#18 0x0000000209cf996b in serf_context_run (ctx=0x20d42c1c8,
duration=-694967296, pool=0x202221028) at ./context.c:260
#19 0x000000020290132c in finish_report (report_baton=0x204828df0,
pool=0x202221028) at subversion/libsvn_ra_serf/update.c:2278
#20 0x00000002030e57e6 in svn_wc_crawl_revisions5 (wc_ctx=0x20476ba30,
local_abspath=0x202221158 "/tmp/trunk", reporter=0x202b0c880,
report_baton=0x204828df0, restore_files=1, depth=svn_depth_infinity,
honor_depth_exclude=0, depth_compatibility_trick=0, use_commit_times=0,
external_func=0x20ff5239d <svn_client__external_info_gatherer>,
external_baton=0x7f7ffffcb630, notify_func=0x416401 <notify>,
notify_baton=0x20ad81048, scratch_pool=0x202221028)
at subversion/libsvn_wc/adm_crawler.c:999
#21 0x000000020ff825c6 in update_internal (result_rev=0x0,
local_abspath=0x202221158 "/tmp/trunk",
anchor_abspath=0x202222920 "/tmp/trunk", revision=0x7f7ffffcbb30,
depth=svn_depth_infinity, depth_is_sticky=1, ignore_externals=0,
allow_unver_obstructions=0, timestamp_sleep=0x7f7ffffcba3c,
send_copyfrom_args=0, innerupdate=0, ctx=0x20476b988, pool=0x202221028)
at subversion/libsvn_client/update.c:252
#22 0x000000020ff82972 in svn_client__update_internal (result_rev=0x0,
local_abspath=0x202221158 "/tmp/trunk", revision=0x7f7ffffcbb30,
depth=svn_depth_infinity, depth_is_sticky=1, ignore_externals=0,
allow_unver_obstructions=0, timestamp_sleep=0x7f7ffffcba3c,
send_copyfrom_args=0, innerupdate=0, ctx=0x20476b988, pool=0x202221028)
at subversion/libsvn_client/update.c:337
#23 0x000000020ff33170 in initialize_area (result_rev=0x0,
local_abspath=0x202221158 "/tmp/trunk", revision=0x7f7ffffcbb30,
session_url=0x2022211f0 "https://svn.apache.org/repos/asf/subversion/trunk", repos_root=0x2022211a0 "https://svn.apache.org/repos/asf",
uuid=0x2022211c8 "13f79535-47bb-0310-9956-ffa450edef68", revnum=964729,
depth=svn_depth_infinity, use_sleep=0x7f7ffffcba3c, ignore_externals=0,
allow_unver_obstructions=0, innercheckout=0, ctx=0x20476b988,
pool=0x202221028) at subversion/libsvn_client/checkout.c:73
#24 0x000000020ff338cd in svn_client__checkout_internal (result_rev=0x0,
url=0x202221168 "https://svn.apache.org/repos/asf/subversion/trunk",
local_abspath=0x202221158 "/tmp/trunk", peg_revision=0x7f7ffffcbb20,
revision=0x7f7ffffcbb30, ra_cache=0x0, depth=svn_depth_unknown,
ignore_externals=0, allow_unver_obstructions=0, innercheckout=0,
timestamp_sleep=0x0, ctx=0x20476b988, pool=0x202221028)
at subversion/libsvn_client/checkout.c:189
#25 0x000000020ff33c84 in svn_client_checkout3 (result_rev=0x0,
URL=0x2022210a0 "https://svn.apache.org/repos/asf/subversion/trunk",
path=0x20c60afe0 "trunk", peg_revision=0x7f7ffffcbb20,
revision=0x7f7ffffcbb30, depth=svn_depth_unknown, ignore_externals=0,
allow_unver_obstructions=0, ctx=0x20476b988, pool=0x202221028)
at subversion/libsvn_client/checkout.c:271
---Type <return> to continue, or q <return> to quit---
#26 0x00000000004084f9 in svn_cl__checkout (os=0x20476b278,
baton=0x7f7ffffcbe40, pool=0x20476b028)
at subversion/svn/checkout-cmd.c:166
#27 0x0000000000414901 in main (argc=3, argv=0x7f7ffffcbf90)
at subversion/svn/main.c:2312
(gdb)
Received on 2010-07-16 12:17:56 CEST