Re: zombie ssh processes

On Mar 27, 2009, at 9:51 AM, Hyrum K. Wright wrote:

> On Mar 27, 2009, at 11:21 AM, Kyle McKay wrote:
>
>> From the xcode-users mailing list:
>>
>>> From: Chris Espinosa <cde_at_apple.com>
>>> Date: March 26, 2009 15:39:14 PDT
>>> To: Xcode Users <xcode-users_at_lists.apple.com>
>>> Subject: Re: Xcode 3.1.2 and Subversion 1.6
>>>
>>> On Mar 25, 2009, at 3:19 PM, Chris Espinosa wrote:
>>>> On Mar 25, 2009, at 3:07 PM, Rob Lockstone wrote:
>>>>
>>>>> Has anyone tried using Xcode 3.1.2 with the subversion 1.6.0
>>>>> client? I think I recall (but may be wrong) that newer versions of
>>>>> Xcode don't make assumptions about the version of subversion
>>>>> that's installed and simply use whatever version it finds.
>>>>
>>>> We have not yet qualified any version of Xcode with Subversion
>>>> 1.6.0 and don't recommend replacing existing Subversion library or
>>>> client code with 1.6 until we've given it the green light.
>>>
>>> We've discovered in internal testing that this patch in Subversion
>>> 1.6:
>>>
>>> http://svn.collab.net/viewvc/svn?view=revision&revision=35533
>>>
>>> can cause Subversion 1.6 to leave behind zombie ssh processes every
>>> time you save a file in Xcode, and eventually exhaust your ability
>>> to spawn new processes. We don't recommend using Subversion 1.6
>>> with Xcode 3.1.x at this time.
>>>
>>> Chris
>>
>> From:
>>
>> http://svn.apache.org/repos/asf/apr/apr/tags/1.0.0/include/apr_thread_proc.h
>>
>> APR_KILL_NEVER // process is never sent any signals
>> APR_KILL_ALWAYS // process is sent SIGKILL on apr_pool_t cleanup
>> APR_KILL_AFTER_TIMEOUT // SIGTERM, wait 3 seconds, SIGKILL
>> APR_JUST_WAIT // wait forever for the process to complete
>> APR_KILL_ONLY_ONCE // send SIGTERM and then wait
>>
>> Restoring the apr_pool_note_subprocess and using APR_KILL_NEVER would
>> allow the children to be reaped provided they exit before pool
>> cleanup.
>>
>> However, that would likely not eliminate the zombie problem in Xcode
>> as pool cleanup probably happens faster than ssh cleanup and exit in
>> some cases. How about using APR_KILL_AFTER_TIMEOUT or
>> APR_KILL_ONLY_ONCE (or even APR_JUST_WAIT) ?
>
> How would this interact with ssh connection pooling? The case which
> drove r35533 was a user who uses ssh connection pooling for svn
> connections. Having svn kill the ssh connection is obviously
> hazardous to such a scheme, how would using the other APR_KILL_*
> conditions behave there (and would they fix the problem with XCode)?

I've built with each of the above options passed to
apr_pool_note_subprocess. For each case, I:

1) Started an 'svn co' over svn+ssh (which launched the ssh control
master).
2) ssh-ed to the same server over the shared connection
3) waited for the co to complete

I also:
Launched Xcode, and did the GUI equivalent to an svn co

- APR_KILL_{ALWAYS,AFTER_TIMEOUT,ONLY_ONCE} are inappropriate in the
connection sharing case. When the svn co completed, the shared
connection was severed, and the second ssh session was disconnected,
however with those options, the ssh subprocesses of the library-using
clients (such as Xcode) were properly reaped.

- APR_KILL_NEVER is inappropriate in the GUI client case (ssh
subprocesses of Xcode were not reaped until Xcode itself was
terminated).

- APR_JUST_WAIT is inappropriate, as the process (whether it be svn or
Xcode) that started the master ssh connection blocks until any other
ssh's using the same shared connection are terminated.

It seems that OpenSSH's connection sharing and svn's use of ssh are
fundamentally incompatible. At the very least, it doesn't seem that
there is a simple fix for both issues.

-- 
Clark S. Cox III
clark.cox_at_apple.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1447105