On Sun, Aug 7, 2011 at 2:26 AM, Greg Stein <gstein_at_gmail.com> wrote:
> On Sat, Aug 6, 2011 at 15:19, Hyrum K Wright <hyrum.wright_at_wandisco.com> wrote:
>> On Sat, Aug 6, 2011 at 6:19 AM, Greg Stein <gstein_at_gmail.com> wrote:
>>...
>>> From my quick perusal... it seems that you might not be handling
>>> thread states and the GIL. Have I missed something?
>>
>> Certainly not. I've read the relevant documentation regarding the GIL
>> and thread states, but am still trying to grok how to apply it to this
>> case. Most of the docs and examples are written for calling C from
>> Python (see [1]), rather than embedding Python in C. I'm not exactly
>> sure how this changes the semantics.
>>
>> At some point, I'll need to start worrying about thread states and the
>> GIL, and while I'm happy to experiment and learn via trial and error,
>> if you've some insights, I'd certainly welcome them.
>
> Sure. It has been a long time, but in the late 90's, I did a bunch of
> embedded Python (in particular: the COM support to enable writing COM
> objects in Python). And the thread state stuff was my design, falling
> out from my work to remove the GIL from the Python 1.4 interpreter.
>
> So... if I can sandpaper off the rusty bits of my brain in this area,
> then I can help with the embedding.
/me hands Greg some 100-grit, and thanks him for the help. :)
> First off: we can ignore the specific calls to make it work, but
> structurally you want something like our *_with_transaction() type of
> invocation. For every FS vtable entry, it will use this (say)
> svn_fs_py__invoke_python() call. We can then build the appropriate
> acquisition of any locks necessary.
I arrived at a similar conclusion in thinking about the problem over
the weekend. The current svn_fs_py__call_method() function has
something of the like to provide error wrapping, and could be either
nested or extended to handle the correct locking semantics.
> There is also an "interpreter state" to be aware of, but I believe you
> can use a single interpreter for this work. Separate interpreter
> states are important when you have disparate embedded codebases. In
> this case, there is just one. But regardless of the number of
> interpreter states, you always need to deal with the GIL. It will be
> easiest to keep one interpreter state, and then N thread states. All
> of this can be managed within the invoke_python() helper.
>
> Oh. One thing to note: you *do* want to create and store your own
> interpreter state (rather than rely on a default). You'll need some
> kind of context to hold that. You cannot use a default state because
> (say) mod_python may be running in the same process, and you want to
> keep fs_py separate from that.
>
> Note that you can't really invoke most Python functions until you hold
> the GIL. This is why the invoke_python() arrangement is best. The
> callback will always run with the GIL acquired. At that point, you can
> build your argument tuple and call into the Python code.
The docs[1] for PyEval_InitThreads() say:
When only the main thread exists, no GIL operations are needed. This
is a common situation (most Python programs do not use threads), and
the lock operations slow the interpreter down a bit. Therefore, the lock
is not created initially. This situation is equivalent to having acquired the
lock: when there is only a single thread, all object accesses are safe.
Therefore, when this function initializes the global interpreter
lock, it also
acquires it.
In our (current) case, we're only using one thread, the main thread,
so I wonder whether we need to worry about interpreter states and
thread states and the like. (This is an honest question, as in my
naïveté probably don't understand all the nuances here.) It sounds
like you're implying that we can't guarantee that "our" interpreter is
the only one running, and so must start concerning ourselves with the
GIL and such.
Another thought along these lines is that we'll be calling back from
Python into C via the callbacks, such as the delta editor. Since
those callbacks are outside our control, and may be long-running, if
we do use the GIL, would it be appropriate to release it upon entering
a C callback, and then acquire it upon exit?
-Hyrum
[1] http://docs.python.org/c-api/init.html#PyEval_InitThreads
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/
Received on 2011-08-08 21:33:48 CEST