Add sys.set_python_compat_version(version) to enable partial compatibility with requested Python version. Add sys.get_python_compat_version().
Modify a few functions in the standard library to implement partial compatibility with Python 3.8.
Add sys.set_python_min_compat_version(version) to deny backward compatibility with Python versions older than version.
Add -X compat_version=VERSION and -X min_compat_version=VERSION command line options. Add PYTHONCOMPATVERSION and PYTHONCOMPATMINVERSION environment variables.
To remain relevant and useful, Python has to evolve frequently; some enhancements require incompatible changes. Any incompatible change can break an unknown number of Python projects. Developers can decide to not implement a feature because of that.
Users want to get the latest Python version to obtain new features and better performance. A few incompatible changes can prevent them from using their applications on the latest Python version.
This PEP proposes to add a partial compatibility with old Python versions as a tradeoff to fit both use cases.
The main issue with the migration from Python 2 to Python 3 is not that Python 3 is backward incompatible, but how incompatible changes were introduced.
Partial compatibility to minimize the Python maintenance burdenWhile technically it would be possible to provide full compatibility with old Python versions, this PEP proposes to minimize the number of functions handling backward compatibility to reduce the maintenance burden of the Python project (CPython).
Each change introducing backport compatibility to a function should be properly discussed to estimate the maintenance cost in the long-term.
Backward compatibility code will be dropped on each Python release, on a case-by-case basis. Each compatibility function can be supported for a different number of Python releases depending on its maintenance cost and the estimated risk (number of broken projects) if it’s removed.
The maintenance cost does not only come from the code implementing the backward compatibility, but also comes from the additional tests.
Cases excluded from backward compatibilityThe performance overhead of any compatibility code must be low when sys.set_python_compat_version() is not called.
The C API is out of the scope of this PEP: Py_LIMITED_API macro and the stable ABI are solving this problem differently, see the PEP 384: Defining a Stable ABI.
Security fixes which break backward compatibility on purpose will not get a compatibility layer; security matters more than compatibility. For example, http.client.HTTPSConnection was modified in Python 3.4.3 to performs all the necessary certificate and hostname checks by default. It was a deliberate change motivated by PEP 476: Enabling certificate verification by default for stdlib http clients (bpo-22417).
The Python language does not provide backward compatibility.
Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the pickle module to Protocol 4 which was first introduced in Python 3.4. This change is backward compatible up to Python 3.4. There is no need to use the Protocol 3 by default when compatibility with Python 3.8 is requested.
The new DeprecationWarning and PendingDeprecatingWarning warnings in Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a project runs its test suite using -Werror (treat any warning as an error), these warnings must be fixed, or specific deprecation warnings must be ignored on a case-by-case basis.
Without backward compatibility, all incompatible changes must be fixed at once, which can be a blocker issue. It is even worse when a project is upgraded to a newer Python which is separated by multiple releases from the old Python.
Postponing an upgrade only makes things worse: each skipped release adds more incompatible changes. The technical debt only steadily increases over time.
With backward compatibility, it becomes possible to upgrade Python incrementally in a project, without having to fix all of the issues at once.
The “all-or-nothing” is a showstopper to port large Python 2 code bases to Python 3. The list of incompatible changes between Python 2 and Python 3 is long, and it’s getting longer with each Python 3.x release.
Cleaning up Python and DeprecationWarningOne of the Zen of Python (PEP 20) motto is:
There should be one– and preferably only one –obvious way to do it.
When Python evolves, new ways inevitably emerge. DeprecationWarnings are emitted to suggest using the new way, but many developers ignore these warnings, which are silent by default (except in the __main__ module: see the PEP 565). Some developers simply ignore all warnings when there are too many warnings, thus only bother with exceptions when the deprecated code is removed.
Sometimes, supporting both ways has a minor maintenance cost, but developers prefer to drop the old way to clean up their code. These kinds of changes are backward incompatible.
Some developers can take the end of the Python 2 support as an opportunity to push even more incompatible changes than usual.
Adding an opt-in backward compatibility prevents the breaking of applications and allows developers to continue doing these cleanups.
Redistribute the maintenance burdenThe backward compatibility involves authors of incompatible changes more in the upgrade path.
Examples of backward compatibility collections ABC aliasescollections.abc aliases to ABC classes have been removed from the collections module in Python 3.9, after being deprecated since Python 3.3. For example, collections.Mapping no longer exists.
In Python 3.6, aliases were created in collections/__init__.py by from _collections_abc import *.
In Python 3.7, a __getattr__() has been added to the collections module to emit a DeprecationWarning upon first access to an attribute:
def __getattr__(name):
# For backwards compatibility, continue to make the collections ABCs
# through Python 3.6 available through the collections module.
# Note: no new collections ABCs were added in Python 3.7
if name in _collections_abc.__all__:
obj = getattr(_collections_abc, name)
import warnings
warnings.warn("Using or importing the ABCs from 'collections' instead "
"of from 'collections.abc' is deprecated since Python 3.3, "
"and in 3.9 it will be removed.",
DeprecationWarning, stacklevel=2)
globals()[name] = obj
return obj
raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
Compatibility with Python 3.8 can be restored in Python 3.9 by adding back the __getattr__() function, but only when backward compatibility is requested:
def __getattr__(name):
if (sys.get_python_compat_version() < (3, 9)
and name in _collections_abc.__all__):
...
raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
Deprecated open() “U” mode
The "U" mode of open() is deprecated since Python 3.4 and emits a DeprecationWarning. bpo-37330 proposes to drop this mode: open(filename, "rU") would raise an exception.
This change falls into the “cleanup” category: it is not required to implement a feature.
A backward compatibility mode would be trivial to implement and would be welcomed by users.
Specification sys functionsAdd 3 functions to the sys module:
sys.set_python_compat_version(version): set the Python compatibility version. If it has been called previously, use the minimum of requested versions. Raise an exception if sys.set_python_min_compat_version(min_version) has been called and version < min_version. version must be greater than or equal to (3, 0).sys.set_python_min_compat_version(min_version): set the minimum compatibility version. Raise an exception if sys.set_python_compat_version(old_version) has been called previously and old_version < min_version. min_version must be greater than or equal to (3, 0).sys.get_python_compat_version(): get the Python compatibility version. Return a tuple of 3 integers.A version must a tuple of 2 or 3 integers. (major, minor) version is equivalent to (major, minor, 0).
By default, sys.get_python_compat_version() returns the current Python version.
For example, to request compatibility with Python 3.8.0:
import collections sys.set_python_compat_version((3, 8)) # collections.Mapping alias, removed from Python 3.9, is available # again, even if collections has been imported before calling # set_python_compat_version(). parent = collections.Mapping
Obviously, calling sys.set_python_compat_version(version) has no effect on code executed before the call. Use -X compat_version=VERSION command line option or PYTHONCOMPATVERSIONVERSION=VERSION environment variable to set the compatibility version at Python startup.
Add -X compat_version=VERSION and -X min_compat_version=VERSION command line options: call respectively sys.set_python_compat_version() and sys.set_python_min_compat_version(). VERSION is a version string with 2 or 3 numbers (major.minor.micro or major.minor). For example, -X compat_version=3.8 calls sys.set_python_compat_version((3, 8)).
Add PYTHONCOMPATVERSIONVERSION=VERSION and PYTHONCOMPATMINVERSION=VERSION=VERSION environment variables: call respectively sys.set_python_compat_version() and sys.set_python_min_compat_version(). VERSION is a version string with the same format as the command line options.
Introducing the sys.set_python_compat_version() function means that an application will behave differently depending on the compatibility version. Moreover, since the version can be decreased multiple times, the application can behave differently depending on the import order.
Python 3.9 with sys.set_python_compat_version((3, 8)) is not fully compatible with Python 3.8: the compatibility is only partial.
sys.set_python_compat_version() must not disable security fixes.
An application can work around most incompatible changes which impacts it.
For example, collections aliases can be added back using:
import collections.abc collections.Mapping = collections.abc.Mapping collections.Sequence = collections.abc.SequenceHandle backward compatibility in the parser
The parser is modified to support multiple versions of the Python language (grammar).
The current Python parser cannot be easily modified for that. AST and grammar are hardcoded to a single Python version.
In Python 3.8, compile() has an undocumented _feature_version to not consider async and await as keywords.
The latest major language backward incompatible change was Python 3.7 which made async and await real keywords. It seems like Twisted was the only affected project, and Twisted had a single affected function (it used a parameter called async).
Handling backward compatibility in the parser seems quite complex, not only to modify the parser, but also for developers who have to check which version of the Python language is used.
from __future__ import python38_syntaxAdd pythonXY_syntax to the __future__ module. It would enable backward compatibility with Python X.Y syntax, but only for the current file.
With this option, there is no need to change sys.implementation.cache_tag to use a different .pyc filename, since the parser will always produce the same output for the same input (except for the optimization level).
For example:
from __future__ import python35_syntax async = 1 await = 2Update cache_tag
Modify the parser to use sys.get_python_compat_version() to choose the version of the Python language.
sys.set_python_compat_version() updates sys.implementation.cache_tag to include the compatibility version without the micro version as a suffix. For example, Python 3.9 uses 'cpython-39' by default, but sys.set_python_compat_version((3, 7, 2)) sets cache_tag to 'cpython-39-37'. Changes to the Python language are now allowed in micro releases.
One problem is that import asyncio is likely to fail if sys.set_python_compat_version((3, 6)) has been called previously. The code of the asyncio module requires async and await to be real keywords (change done in Python 3.7).
Another problem is that regular users cannot write .pyc files into system directories, and so cannot create them on demand. It means that .pyc optimization cannot be used in the backward compatibility mode.
One solution for that is to modify the Python installer and Python package installers to precompile .pyc files not only for the current Python version, but also for multiple older Python versions (up to Python 3.0?).
Each .py file would have 3n .pyc files (3 optimization levels), where n is the number of supported Python versions. For example, it means 6 .pyc files, instead of 3, to support Python 3.8 and Python 3.9.
In 2009, PEP 3003 “Python Language Moratorium” proposed a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for Python 3.1 and Python 3.2.
In May 2018, during the PEP 572 discussions, it was also proposed to slow down Python changes: see the python-dev thread Slow down…
PEP 387I don’t believe that the way for Python to remain relevant and useful for the next 10 years is to cease all language evolution. Who knows what the computing landscape will look like in 5 years, let alone 10? Something as arbitrary as a 10-year moratorium is (again, IMHO) a death sentence for the language.
PEP 387 – Backwards Compatibility Policy proposes a process to make incompatible changes. The main point is the 4th step of the process:
PEP 497See if there’s any feedback. Users not involved in the original discussions may comment now after seeing the warning. Perhaps reconsider.
PEP 497 – A standard mechanism for backward compatibility proposes different solutions to provide backward compatibility.
Except for the __past__ mechanism idea, PEP 497 does not propose concrete solutions:
Examples of incompatible changes Python 3.8When an incompatible change to core language syntax or semantics is being made, Python-dev’s policy is to prefer and expect that, wherever possible, a mechanism for backward compatibility be considered and provided for future Python versions after the breaking change is adopted by default, in addition to any mechanisms proposed for forward compatibility such as new future_statements.
Examples of Python 3.8 incompatible changes:
PyCode_New() required a new parameter: it broke all Cython extensions (all projects distributing precompiled Cython code). This change has been reverted during the 3.8 beta phase and a new PyCode_NewWithPosOnlyArgs() function was added instead.types.CodeType requires an additional mandatory parameter. The CodeType.replace() function was added to help projects to no longer depend on the exact signature of the CodeType constructor.sys.abiflags changed from 'm' to an empty string. For example, python3.8m program is gone.PyInterpreterState was made opaque.
Backward compatibility cannot be added for all these changes. For example, changes in the C API and in the build system are out of the scope of this PEP.
See What’s New In Python 3.8: API and Feature Removals for all changes.
See also the Porting to Python 3.8 section of What’s New In Python 3.8.
Python 3.7Examples of Python 3.7 incompatible changes:
async and await are now reserved keywords.os.errno is no longer available; use import errno directly instead. Note that such undocumented internal imports may be removed any time without notice, even in micro version releases.'\' and an ASCII letter in replacement templates for re.sub() were deprecated in Python 3.5, and will now cause an error.asyncio.windows_utils.socketpair() function has been removed: it was an alias to socket.socketpair().asyncio no longer exports the selectors and _overlapped modules as asyncio.selectors and asyncio._overlapped. Replace from asyncio import selectors with import selectors.StopIteration exceptions raised directly or indirectly in coroutines and generators are transformed into RuntimeError exceptions.socketserver.ThreadingMixIn.server_close() now waits until all non-daemon threads complete. Set the new block_on_close class attribute to False to get the pre-3.7 behaviour.struct.Struct.format type is now str instead of bytes.repr for datetime.timedelta has changed to include the keyword arguments in the output.tracemalloc.Traceback frames are now sorted from oldest to most recent to be more consistent with traceback.Adding backward compatibility for most of these changes would be easy.
See also the Porting to Python 3.7 section of What’s New In Python 3.7.
Micro releasesSometimes, incompatible changes are introduced in micro releases (micro in major.minor.micro) to fix bugs or security vulnerabilities. Examples include:
compileall and py_compile module: the invalidation_mode parameter’s default value is updated to None; the SOURCE_DATE_EPOCH environment variable no longer overrides the value of the invalidation_mode argument, and determines its default value instead.xml modules: the SAX parser no longer processes general external entities by default to increase security by default.os.urandom(): on Linux, if the getrandom() syscall blocks (the urandom entropy pool is not initialized yet), fall back on reading /dev/urandom.sys.setrecursionlimit(): a RecursionError exception is now raised if the new limit is too low at the current recursion depth.ssl.create_default_context(): RC4 was dropped from the default cipher string.http.client: HTTPSConnection now performs all the necessary certificate and hostname checks by default.email.message: EmailMessage.is_attachment() is now a method instead of a property, for consistency with Message.is_multipart().os.makedirs(name, mode=0o777, exist_ok=False): Before Python 3.4.1, if exist_ok was True and the directory existed, makedirs() would still raise an error if mode did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1 (bpo-21082).Examples of changes made in micro releases which are not backward incompatible:
ssl.OP_NO_TLSv1_3 constant was added to 2.7.15, 3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2.typing.AsyncContextManager was added to Python 3.6.2.zipfile module accepts a path-like object since Python 3.6.2.loop.create_future() was added to Python 3.5.2 in the asyncio module.No backward compatibility code is needed for these kinds of changes.
ReferencesAccepted PEPs:
Draft PEPs:
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4