I still suspect this might explain most of what Barry saw, if not all. — Sent from Mailbox On Thu, May 16, 2013 at 2:36 PM, Brett Cannon <brett at python.org> wrote: > On Thu, May 16, 2013 at 5:19 PM, Guido van Rossum <guido at python.org> wrote: >> This reminds me of the following bug, which can happen when two >> processes are both writing the .pyc file and a third is reading it. >> First some background. >> >> When writing a .pyc file, we use the following strategy: >> - open the file for writing >> - write a dummy header (four null bytes) >> - write the .py file's mtime >> - write the marshalled code object >> - replace the dummy heaer with the correct magic word >> > Just so people know, this is how we used to do it. In importlib we > write the entire file to a temp file and then to an atomic rename. >> Even py_compile.py (used by compileall.py) uses this strategy. > py_compile as of Python 3.4 now just uses importlib directly, so it > matches its semantics. > -Brett >> >> When reading a .pyc file, we ignore it when the magic word isn't there >> (or when the mtime doesn't match that of the .py file exactly), and >> then we will write it back like described above. >> >> Now consider the following scenario. It involves *three* processes. >> >> - Two unrelated processes both start and want to import the same module. >> - They both see the .pyc file is missing/corrupt and decide to write it. >> - The first process finishing writing the file, writing the correct header. >> - Now a third process wants to import the module, sees the valid >> header, and starts reading the file. >> - However, while this is going on, the second process gets ready to >> write the file. >> - The second process truncates the file, writes the dummy header, and >> then stalls. >> - At this point the third process (which thought it was reading a >> valid file) sees an unexpected EOF because the file has been >> truncated. >> >> Now, this would explain the EOFError, but not necessarily the >> ValueError with "unknown type code". However, it looks like marshal >> doesn't always check for EOF immediately (sometimes it calls getc() >> without checking the result, and sometimes it doesn't check the error >> state after calling r_string()), so I think all the errors are >> actually explainable from this scenario. >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130516/c3c6217e/attachment.html>
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4