Re: First httpd deadlock fixed, but found another one

Brandon Ehle <azverkan@yahoo.com> writes:

> Probably, one thing I've noticed since I ran this test is that when
> you commit anything over 3000 files its starts to take a really long
> time to shutdown apache. It looks like its finalizing something with
> the db before it shuts down. For the 8333 file case, its possible
> that it had been shutting down for 4 hours and hadn't finished yet
> when I got in this morning. Next time I see that stack trace I'll
> verify whether that is whats happening.

Huh? When I shutdown Apache it first sends SIGTERM to the other
processes and if that is ignored it sends SIGKILL. It's not possible
for a process to ignore SIGKILL at all, let alone hang around for 4
hours. What on earth are you doing?

> >Did you do as I suggest and check your Apache and system logs for
> >killed proceses? It's quite possible that 8333 files is pushing
> >Apache over a memory limit.
>
> No killed processes.

I just deliberately provoked an OOM on my Linux machine. The syslog
contains

Dec 10 21:02:38 debian2 kernel: Out of Memory: Killed process 28928 (httpd).

but there is no message in the Apache log, so it's easy to miss.

> >Does your test verify that the operations it runs do in fact succeed?
>
> It just checks svn's return values.
>
> >Isn't this thread still processing a request? One that would have
> >been generated by your test? Or did you fail to mention some other
> >process that accesses the repository?
>
> See above, apache might have needed more than 4 hours to shutdown.
> Next time I will give it more time.

I still don't understand why you think shutting down Apache would
cause a dav_method_put request. As far as I can tell a dav_method_put
occurs when you commit a new or modified file, and as such I would
assume it is part of a failed commit in your test.

If the hang is repeatable then I suggest you check that the database
is not locked before shutting down Apache, run an 'svn ls URL' for
example. Of course if your test is hanging the database then it
appears it is not detecting failed commands. In that case it might be
better to have the test not shut down Apache, and then check manually
whether the database has hung.

-- 
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org