[Armin Rigo] > I don't remember if it was mentioned here, but maybe we'd better check > directly whether the C stack is too large rather than (or possibly in addition > to) using a limit on the number of Python iterations. How large is too large? The Windows way checks the stack for impending overflow directly. A long "stack check on Unix" thread here in 2000 fizzled out in the expected way: no two Unices appear to have anything in common <0.9 wink>. > This is not really ANSI C, but I can't think of a setting in which the > following trick would fail to be a good indication of the amount of stack > space consumed: save a pointer to a local variable "early", e.g. in > Py_Initialize(); then in any other function call, the distance between this > pointer and a pointer to some local equals the amount of stack space consumed > by Python minus a few bytes. It fails for threads, whose stacks may be anywhere in relation to the main thread's. So any gimmick like this has to have a thread-local "starting point". If "a stack" is always a contiguous slice of memory, then it can work. > If this sounds too much of a hack, No, but I'm not sure how much additional hackery it requires to make it work in all cases. > the (usual) recursion limit could be kept to limit nested Python frames; but all > C-level recursions (including eval_frame) could additionally use the above trick. > Its advantage is that it is an extremely fast check. The ways to do it that don't work are indeed extremely fast <wink>. > If the total stack size is difficult to know in advance, That's apparently the case across platforms -- and someimes the main-thread stack is much larger than secondary-thread stacks. The Windows check doesn't even try to know the current stack's size, but relies on MS-specific C exception extensions. > we can still use PyOS_CheckStack(), but more efficiently and less randomly > than now, by maintaining a "high tide" pointer that records how much stack we > are sure we have, and calling PyOS_CheckStack() only to attempt to push > the "high tide" mark further. Threads again complicate this. AFAICT, the only platform that defines USE_STACKCHECK now (and actually implements PyOS_CheckStack()) is 32-bit Windows using an MS compiler. I did boost the stack reservation for that platform in CVS, which can't solve it, but at least hides it better <wink>. > While I'm in a hacking mood, there might be a way to prevent PyErr_Clear() to > clear away "asynchronuous" exceptions like RuntimeError, MemoryError, and > KeyboardInterrupt: for these exceptions, let's just store them away instead of > clearing them, and re-raise them at the next occasion. The "next occasion" > would have to be defined more precisely, but there is probably a way to ensure > that it will at least be before the next non-trivial opcode operation starts. > It would overwrite an exception set later, like those spurious KeyError we get > for dict lookups. It might be a better-than-nothing quick fix to all these > PyErr_Clear() all around the code base. Threads probably complicate that too. It's dreadful that serious problems can get transformed into bogus KeyErrors (in the test driver I posted, I'm not sure I spelled this out, but the key always *was* in the dict when a MemoryError got turned into a KeyError; we called a comparison function to begin with because the hash codes matched, and since these were mostly small-integer keys, they were identical to their hash codes -- so these keys were in the dicts, and the KeyErrors were lying about that). That's just nuts. I don't have spare cycles to give to it, though, so you'll have to save the world on your own again.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4