"Mark Hammond" <mhammond@skippinet.com.au> writes:
> My goal:
>
> For a multi-threaded application (generally this will be a larger
> app embedding Python, but that is irrelevant), make it reasonably
> easy to accomplish 2 things:
>
> 1) Allow "arbitrary" threads (that is, threads never before seen by
> Python) to acquire the resources necessary to call the Python C API.
>
> 2) Allow Python extensions to be written which support (1) above.
3) Allow "arbitrary" threads to acquire the resources necessary to
call the Python C API, even if they already have those resources,
and to later release them if they did not have those resources.
> Currently (2) is covered by Py_BEGIN_ALLOW_THREADS, except that it
> is kinda like only having a hammer in your toolbox <wink>. I assert
> that 2) could actually be split into discrete goals:
I'm going to ask some questions just to make sure your terminology is
clear to me:
> 2.1) Extension functions that expect to take a lot of time, but
> generally have no thread-state considerations. This includes
> sleep(), all IO functions, and many others. This is exactly what
> Py_BEGIN_ALLOW_THREADS was designed for.
In other words, functions which will not call back into the Python API?
> 2.2) Extensions that *may* take a little time, but more to the
> point, may directly and synchronously trigger callbacks.
By "callbacks", do you mean "functions which (may) use the
Python C API?"
> That is, it is not expected that much time will be spent outside of
> Python, but rather that Python will be re-entered. I can concede
> that functions that may trigger asynch callbacks need no special
> handling here, as the normal Python thread switch mechanism will
> ensure correct their dispatch.
By "trigger asynch callbacks" do you mean, "cause a callback to occur
on a different thread?"
> Currently 2.1 and 2.2 are handled the same way, but this need not be
> the case. Currently 2.2 is only supported by *always* giving up the
> lock, and at each entry point *always* re-acquiring it. This is
> obviously wasteful if indeed the same thread immediately re-enters -
> hence we are here with a request for "how do I tell if I have the
> lock?".
Yep, that pinpoints my problem.
> Combine this with the easily stated but tricky to implement (1) and
> no one understands it at all <frown>
>
> I also propose that we restrict this to applications that intend to
> use a single "PyInterpreterState" - if you truly want multiple
> threads running in multiple interpreters (and good luck to you - I'm
> not aware anyone has ever actually done it <wink>) then you are on
> your own.
Fine with me ;-). I think eventually we'll need to come up with a
more precise definition of exactly when "you're on your own", but for
now that'll do.
> Are these goals a reasonable starting point? This describes all my
> venturing into this area.
Sounds about right to me.
--
David Abrahams
dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4