[Jeremy] > >> If a programmer uses a library implement via coroutines, can > she >> call library methods from an __xxx__ method? > > GMcM> Certain situations won't work, but you knew that. > > I expected that some won't work, but no one seems willing to tell > me exactly which ones will and which ones won't. Should the > caveat in the documentation say "avoid using certain __xxx__ > methods" <0.9 wink>. Within an __xxx__ method, you cannot *use* a coroutine not created in that method. That is true in current Stackless and will be true in Stack-lite. The presence of "library" in the question is a distraction. I guess if you think of a coroutine as just another kind of callable object, this looks like a strong limitation. But you don't find yourself thinking of threads as plain old callable objects, do you? In a threaded program, no matter how carefully designed, there is a lot of thread detritus lying around. If you don't stay concious of the transfers of control that may happen, you will screw up. Despite the limitation on using coroutines in magic methods, coroutines have an advantage in that tranfers of control only happen when you want them to. So avoiding unwanted transfers of control is vastly easier. > >> Can coroutines or microthreads co-exist with callbacks > invoked by >> C extensions? > > GMcM> Again, in certain situations it won't work. Again, you > knew GMcM> that. > > Wasn't sure. It's exactly the same situation. > >> Can a program do any microthread IO in an __call__ method? > > GMcM> Considering you know the answer to that one too, you > could've GMcM> phrased it as a parsable question. > > Do I know the answer? I assume the answer is no, but I don't > feel very certain. What is "microthreaded IO"? Probably the attempt to yield control if the IO operation would block. Would doing that inside __call__ work with microthreads? No. It's not my decision over whether this particular situation needs to be documented. Somtime between the 2nd and 5th times the programmer encounters this exception, they'll say "Oh phooey, I can't do this in __call__, I need an explicit method instead." Python has never claimed that __xxx__ methods are safe as milk. Quite the contrary. > >> If any of these are the sort "in theory" problems that the > PEP >> alludes to, then we need a full spec for what is and is > not >> allowed. It doesn't make sense to tell programmers to > follow >> unspecified "reasonable" programming practices. > > GMcM> That's easy. In a nested invocation of the Python > interpreter, GMcM> you can't use a coroutine created in an > outer interpreter. > > Can we define these situations in a way that doesn't appeal to > the interpreter implementation? No, because it's implementation dependent. > If not, can we at least come up > with a list of what will and will not work at the python level? Does Python attempt to catalogue all the ways you can screw up using magic methods? Using threads? How 'bout the metaclass hook? Even stronger, do we catalogue all the ways that an end-user-programmer can get bit by using a library written by someone else that makes use of these facilities? > GMcM> In the Python 2 documentation, there are 6 caveats listed > in GMcM> the thread module. That's a couple order of magnitudes > GMcM> different from the actual number of ways you can screw up > GMcM> using the thread module. > > The caveats for the thread module seem like pretty minor stuff to > me. If you are writing a threaded application, don't expect code > to continue running after the main thread has exited. Well, the thread caveats don't mention the consequences of starting and running a thread within an __init__ method. > The caveats for microthreads seems to cover a vast swath of > territory: The use of libraries or extension modules that involve > callbacks or instances with __xxx__ methods may lead to > application failure. While your statement is true on the face of it, it is very misleading. Things will only fall apart when you code an __xxx__ method or callback that uses a pre-existing coroutine (or does a uthread swap). You can very easily get in trouble right now with threads and callbacks. But the real point is that it is *you* the programmer trying to do something that won't work (and, BTW, getting notified right away), not some library pulling a fast one on you. (Yes, the library could make things very hard for you, but that's nothing new.) Application programmers do not need magic methods. Ever. They are very handy for people creating libraries for application programmers to use, but we already presume (naively) that these people know what they're doing. > I worry about it becomes it doesn't sound > very modular. The use of coroutines in one library means I can't > use that library in certain special cases in my own code. With a little familiarity, you'll find that coroutines are a good deal more modular than threads. In order for that library to violate your expectations, that library must be concious of multiple coroutines (otherwise, it's just a plain stackfull call / return). It must have kept a coroutine from some other call, or had you pass one in. So you (if at all cluefull <wink>) will be concious that something is going on here. The issue is the same as if you used a framework which used real threads, but never documented anything about the threads. You code callbacks that naively and independently mutate a global collection. Do you blame Python? > I'm sorry if I sound grumpy, but I feel like I can't get a > straight answer despite several attempts. At some level, it's > fine to say that there are some corner cases that won't work well > with microthreads or coroutines implemented on top of stackless > python. But I think the PEP should discuss the details. I've > never written in an application that uses stackless-based > microthreads or coroutines so I don't feel confident in my > judgement of the situation. And where on the fearful to confident scale was the Jeremy just getting introduced to threads? > Which gets back to Bernd's original question: > > GMcM> > BR> """ Don't use classes and libraries that use > classes when GMcM> > BR> IO in microthreaded programs! """ > GMcM> > GMcM> > BR> which might indeed be a problem. Am I > overlooking something GMcM> > BR> fundamental here? > > and the synopsis of your answer: > > GMcM> Synopsis of my reply: this is more a problem with > uthreads GMcM> than coroutines. In any (real) thread, you're > limited to dealing GMcM> with one non-blocking IO technique > (eg, select) without going GMcM> into a busy loop. If you're > dedicating a (real) thread to select, it GMcM> makes more sense > to use coroutines than uthreads. > > I don't understand how this addresses the question, but perhaps I > haven't seen your reply yet. Mail gets through to python-dev and > stackless at different rates. Coroutines only swap voluntarily. It's very obvious where these transfers of control take place hence simple to control when they take place. My suspicion is that most people use uthreads because they use a familiar model. Not many people are used to coroutines, but many situations would be more profitably approached with coroutines than uthreads. - Gordon
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4