RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from http://mail.python.org/pipermail/python-dev/attachments/20180806/702762a1/attachment.html below:

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
Background:<o:p></o:p>
Through the implementation of an alternate runtime I've been poking around some of the class initialization routines and I found out that there was a subtle bug with PyType_Ready and the header initializer
PyVarObject_HEAD_INIT.<o:p></o:p>
<o:p> </o:p>
Looking through the codebase, I couldn't really find any pattern of when the type should be defined within PyVarObject_HEAD_INIT. Sometimes it was initialized to NULL (or 0) and PyType_Type (let's ignore Py_True
and Py_False from now).<o:p></o:p>
<o:p> </o:p>
From PyType_Ready it turns out that setting the value PyType_Type is never actually needed outside of PyType_Type and PyBaseObject type. This is clear from the code:<o:p></o:p>
if (Py_TYPE(type) == NULL && base != NULL)<o:p></o:p>
 Py_TYPE(type) = Py_TYPE(base);â¨<o:p></o:p>
<o:p> </o:p>
Given that any PyTypeObject's base is of type PyType_Type, setting PyVarObject_HEAD_INIT(&PyType_Ready) is superfluous. Therefore, setting all static PyTypeObjects to their ob_type to NULL should be a safe
assumption to make.<o:p></o:p>
<o:p> </o:p>
Uninitialized Types:<o:p></o:p>
A quick s/PyVarObject_HEAD_INIT(&PyType_Type/PyVarObject_HEAD_INIT(NULL/ shows that some objects do need to have their ob_type set from the outset, violating the previous assumption. After writing a quick
script, I found that out of the ~300 PyVarObject_HEAD_INIT present in CPython, only these 14 types segfaulted:<o:p></o:p>
<o:p> </o:p>
PyByteArrayIter_Type<o:p></o:p>
PyBytesIter_Type<o:p></o:p>
PyDictIterKey_Type <o:p></o:p>
PyDictIterValue_Type<o:p></o:p>
PyDictIterItem_Type<o:p></o:p>
PyClassMethod_Type<o:p></o:p>
PyAsyncGen_Type<o:p></o:p>
PyListIter_Type<o:p></o:p>
PyListRevIter_Type<o:p></o:p>
PyODictIter_Type<o:p></o:p>
PyLongRangeIter_Type<o:p></o:p>
PySetIter_Type<o:p></o:p>
PyTupleIter_Type<o:p></o:p>
PyUnicodeIter_Type<o:p></o:p>
<o:p> </o:p>
<o:p> </o:p>
Bug:<o:p></o:p>
It turns out that these PyTypeObjects are never initialized through PyType_Ready. However, they are used as fully initialized types. It is by pure chance that the work without having to call the initializer
on them. This though is undefined behavior. This not only might result in a weird future bug which is hard to chase down but also, it affects other runtimes as this behavior depends on implementation details of CPython.<o:p></o:p>
<o:p> </o:p>
This is a pervasive pattern that should be removed from the codebase and ideally extensions should follow as well.<o:p></o:p>
<o:p> </o:p>
Solution:<o:p></o:p>
Here are my proposed solutions in order from less controversial to most controversial. Note that all of them I already tried in a local branch and are working:<o:p></o:p>
<o:p> </o:p>
1) Initialize all uninitialized types.<o:p></o:p>
<o:p> </o:p>
Example: <a href="https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173">
https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173</a> 
 
<o:p></o:p>
2) Move all PyVarObject_HEAD_INIT to NULL except PyType_Type, PyBaseObject_Type and the booleans.<o:p></o:p>
<o:p> </o:p>
3) Special case the initialization of PyType_Type and PyBaseObject_Type within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use NULL. To enable this a small change within PyType_Ready is needed
to initialize PyType_Type PyBaseObject:<o:p></o:p>
if (base == NULL) {<o:p></o:p>
 Py_TYPE(&PyType_Type) = &PyType_Type;<o:p></o:p>
 Py_TYPE(type) = &PyType_Type;<o:p></o:p>
}<o:p></o:p>
<o:p> </o:p>
Also, the booleans have to be fully initialized without calling PyVarObject_HEAD_INIT. I propose:<o:p></o:p>
<o:p> </o:p>
struct _longobject _Py_FalseStruct = {<o:p></o:p>
 PyObject_HEAD_INIT(&PyBool_Type), 0, { 0 }<o:p></o:p>
};<o:p></o:p>
<o:p> </o:p>
This will clean-up the entire codebase of this anti-pattern.<o:p></o:p>
<o:p> </o:p>
Example: <a href="https://github.com/eduardo-elizondo/cpython/commit/542fd79e4279c64c077c127b175a8d856d3c5f0b">
https://github.com/eduardo-elizondo/cpython/commit/542fd79e4279c64c077c127b175a8d856d3c5f0b</a> 
 
<o:p></o:p>
4) Modify PyVarObject_HEAD_INIT to ignore the input and initialize to NULL and 0.<o:p></o:p>
In order to prevent this antipattern within extension code as well, we should make PyVarObject_HEAD_INIT ignore the inputs and just set the value to NULL.<o:p></o:p>
<o:p> </o:p>
#define PyVarObject_HEAD_INIT(type, size) \<o:p></o:p>
 { PyObject_HEAD_INIT(NULL) 0 },<o:p></o:p>
<o:p> </o:p>
This will prevent external code to have a semi-initialized type that is not initialized through PyType_Ready.<o:p></o:p>
<o:p> </o:p>
5) Finally, I would go even further and suggest making PyVarObject_HEAD_INIT argumentless.
<o:p></o:p>
<o:p> </o:p>
#define PyVarObject_HEAD_INIT \<o:p></o:p>
 { PyObject_HEAD_INIT(NULL) 0 },<o:p></o:p>
<o:p> </o:p>
However, this breaks backward compatibility. That being said, this will make extension maintainers aware of this antipattern.<o:p></o:p>
<o:p> </o:p>
Example: <a href="https://github.com/eduardo-elizondo/cpython/commit/3869e53843008ff096764f4adaf26efbb5625996">
https://github.com/eduardo-elizondo/cpython/commit/3869e53843008ff096764f4adaf26efbb5625996</a> 
 
Thoughts?<o:p></o:p>
</div>
</body>
</html>

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4