On 24.04.2015 14:11, Branko Čibej wrote:
>
> Hi Marc,
>
> Just a quick note: your last msg jogged my memory and I think I know
> the root cause of the leak: improper JNI frame management within a
> loop. If I'm right, I can both fix the leak and remove the
> close-stream requirement I just added.
>
> On 24 Apr 2015 11:00 am, "Marc Strapetz" <marc.strapetz_at_syntevo.com
> <mailto:marc.strapetz_at_syntevo.com>> wrote:
>
> On 24.04.2015 06 <tel:24.04.2015%2006>:34, Branko Čibej wrote:
>
> On 22.03.2015 05 <tel:22.03.2015%2005>:06, Branko Čibej wrote:
>
> On 21.03.2015 16 <tel:21.03.2015%2016>:23, Branko Čibej wrote:
>
> On 19.03.2015 11:43, Marc Strapetz wrote:
>
> Attached example performs an endless series of
> remote status against
> the Subversion repository. When invoked with
> -Xmx24M, the VM will run
> out of memory soon. Monitoring with jvisualvm
> shows that the used heap
> size constantly grows. Monitoring with the Task
> Manager shows that the
> allocated memory grows even more (significantly).
> Looks like a memory
> leak, for which a large amount of native memory is
> involved, too.
>
> Tested on Windows 8.1 with almost latest
> Subversion 1.9 JavaHL builds.
>
> I can confirm that this happens on the Mac, too, and
> it's not a garbage
> collector artefact. I'm trying to trace where the leak
> is happening ...
> valgrind with APR pool debugging doesn't tell me much
> (no surprise there).
>
> Just to make sure we weren't doing something bad in our
> libraries, I
> wrote a small C program that does the same as your Java
> example (Ev2
> shims included), and memory usage is completely steady. So
> it is
> something in JavaHL, but I have no clue yet what the
> problem is.
>
>
> I have to say this was one of the more "interesting" bug-hunts
> in my not
> entirely boring career, and that's not really obvious from the fix
> itself. :)
>
> http://svn.apache.org/r1675771
>
> Marc: this will not be in RC1, but please give the patch a
> spin and let
> me know if it fixes your problem. I tested this with the Java
> program
> you attached to your original report, and heap size no longer
> grows
> without bounds.
>
>
> Great hunt, Brane! The native leak seems to be fixed. I've run my
> remote status loop with -Xmx24M and still get an OOME after ~170
> loop iterations. The memory leak is significantly smaller and this
> time it seems to be in the Java part. According to the profiler,
> most memory is allocated by HashMap and friends, referenced from
> JNI code. Only two org.apache.subversion classes show up, but I
> guess they indicate the source of the leak:
>
> org.apache.subversion.javahl.types.Checksum (~10K instances)
> org.apache.subversion.javahl.types.NativeInputStream (~10K instances)
>
> Let me know, if you more profiler statistics will be helpful.
>
So I've been looking at this in depth. At first I thought that one of
the problems was that we didn't release JNI local references; I added
code to make sure this happens in the status callbacks (not committed
yet) and I verified that all the native wrapped objects do get
finalized. However, the Java objects still hang around.
One of the problems is that all the callbacks happen within the scope of
the ISVNReporter.finishReport call, which means that the whole edit
drive is considered a single JNI call (despite the callbacks to Java)
and the garbage collector can't reclaim space for the objects created
within JNI during that time. But even a forced GC after the report is
done and the remote session disposed won't release all the native
references. I'm a bit stumped here ... JVM's built-in memory profiler
shows the live references and where they're allocated, but doesn't show
why they're not released even when I explicitly create and destroy JNI
frames.
Did I say that debugging JNI memory management is a pain?
-- Brane
Received on 2015-04-28 07:38:16 CEST