Christian Tismer just did an exhaustive search for thread unsafe use
of Python operations, and found two weaknesses. One is
posix.listdir(), which I had already found; the other is
file.writelines(). Here's a program that demonstrates the bug;
basically, while writelines is walking down the list, another thread
could truncate the list, causing PyList_GetItem() to fail or a string
object to be deallocated while writelines is using it. On my SOlaris
7 system it typically crashes in the first or second iteration.
It's easy to fix: just don't use release the interpreter lock (get rid
of Py_BEGIN_ALLOW_THREADS c.s.). This would however prevent other
threads from doing any work while this thread may be blocked for I/O.
An alternative solution is to put Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS just around the fwrite() call. This is safe, but
would require a lot of lock operations and would probably slow things
down too much.
Ideas?
--Guido van Rossum (home page: http://www.python.org/~guido/)
import os
import sys
import thread
import random
import time
import tempfile
def good_guy(fp, list):
t0 = time.time()
fp.seek(0)
fp.writelines(list)
t1 = time.time()
print fp.tell(), "bytes written"
return t1-t0
def bad_guy(dt, list):
time.sleep(random.random() * dt)
del list[:]
def main():
infn = "/usr/dict/words"
if sys.argv[1:]:
infn = sys.argv[1]
print "reading %s..." % infn
fp = open(infn)
list = fp.readlines()
fp.close()
print "read %d lines" % len(list)
tfn = tempfile.mktemp()
fp = None
try:
fp = open(tfn, "w")
print "calibrating..."
dt = 0.0
n = 3
for i in range(n):
dt = dt + good_guy(fp, list)
dt = dt / n # average time it took to write the list to disk
print "dt =", round(dt, 3)
i = 0
while 1:
i = i+1
print "test", i
copy = map(lambda x: x[1:], list)
thread.start_new_thread(bad_guy, (dt, copy))
good_guy(fp, copy)
finally:
if fp:
fp.close()
try:
os.unlink(tfn)
except os.error:
pass
main()
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4