PEP 508 specifies a mini-language for declaring package dependencies. One feature of this language is the ability to specify extras, which are optional components of a distribution that, when used, install additional dependencies. This PEP proposes a mechanism to allow one or more extras to be installed by default if none are provided explicitly.
MotivationVarious use cases for default extras and possible solutions in this PEP were discussed extensively on this DPO thread. These fall into two broad cases that provide the motivation for the present PEP.
Recommended but not required dependenciesPackage maintainers often use extras to declare optional dependencies that extend the functionality or performance of a package. In some cases, it can be difficult to determine which dependencies should be required and which should be categorized as extras. A balance must be struck between the needs of typical users, who may prefer most features to be available by default, and users who want minimal installations without large, optional dependencies. One solution with existing Python packaging infrastructure is for package maintainers to define an extra called, for example, recommended, which includes all non-essential but suggested dependencies. Users are then instructed to install the package using package[recommended], while those who prefer more control can use package. However, in practice, many users are unaware of the [recommended] syntax, placing the burden on them to know this for a typical installation. Having a way to have recommended dependencies be installed by default while providing a way for users to request a more minimal installation would satisfy this use case, and this PEP describes a solution to this.
Examples of packages that demonstrate this pattern by encouraging users to include extra dependencies by default include:
Packages supporting multiple backends or frontendsAnother common use case for using extras is to define different backends or frontends and dependencies that need to be installed for each backend or frontend. A package might need at least one backend or frontend to be installed in order to be functional, but may be flexible on which backend or frontend this is. Concrete examples of such frontends or backends include:
With current packaging standards, maintainers have to either require one of the backends or frontends or require users to always specify extras, e.g., package[backend], and therefore risk users having an unusable installation if they only install package. Having a way to specify one or more default backend or frontend and providing a way to override these defaults would provide a much better experience for users, and the approach described in this PEP will allow this.
Note that this PEP does not aim to address the issue of disallowing conflicting or incompatible extras, for example if a package requires exactly one frontend or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that.
Examples of packages that require at least one backend or frontend to work and recommend a default extra to install a backend or frontend include:
In all three cases, installing the package without any extras results in a broken installation, and this is a commonly reported support issue for some of these packages.
RationaleA number of possible solutions have been discussed extensively by the community for several years, including in this DPO thread as well as in numerous issues and pull requests. The solution that is presented below:
It is the only solution, out of all those discussed, that meets all three criteria.
Specification New key in[project] metadata table
A new key will be added to the [project] table in project metadata as originally defined in PEP 621 and now defined in the PyPA specifications. This key will be named default-optional-dependency-keys with the following description:
Default-ExtraEach string in default-optional-dependency-keys must be the name of an extra defined in optional-dependencies, and each extra in this array will be converted to a matching Default-Extra entry in the core package metadata. Examples of valid usage which would produce the example Default-Extra entries presented in the previous section are:
[project]
default-optional-dependency-keys = [
"recommended",
]
and:
[project]
default-optional-dependency-keys = [
"backend1",
"backend2",
"backend3"
]
Examples
In this section we take a look at the use cases described in the Motivation section and how these can now be addressed by using the specification outlined above.
Recommended dependencies and minimal installationsFirst, we consider the case of packages that want recommended but not strictly required dependencies installed by default, while also providing a way to only install the required dependencies.
In order to do this, a package maintainer would define an extra called recommended containing the recommended but not required dependencies, and would choose to have this be included as a default extra:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
recommended = [
"package1",
"package2"
]
If this package was called package, users installing package would then get the equivalent of package[recommended]. Users could alternatively install package[] which would install the package without the default extras.
To take a one of the concrete examples of package from the Motivation section, the astropy package defines a recommended extra that users are currently instructed to install in the default installation instructions. With this PEP, the recommended extra could be declared as being a default extra:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
recommended = [
"scipy",
"..."
]
meaning that installing:
would then install optional but recommended dependencies such as scipy. Advanced users who want a minimal install could then use:
Packages requiring at least one backend or frontendAs described in Motivation, some packages may support multiple backends and/or frontends, and in some cases it may be desirable to ensure that there is always at least one backend or frontend package installed, as the package would be unusable otherwise. Concrete examples of this might include a GUI application that needs a GUI library to be present to be usable but is able to support different ones, or a package that can rely on different computational backends but needs at least one to be installed.
In this case, package maintainers could make the choice to define an extra for each backend or frontend, and provide a default, e.g.:
[project]
default-optional-dependency-keys = [
"backend1"
]
[project.optional-dependencies]
backend1 = [
"package1",
"package2"
]
backend2 = [
"package3"
]
If packages can support e.g. multiple backends at the same time, and some of the backends should always be installed, then the dependencies for these must be given as required dependencies rather than using the default extras mechanism.
To take one of the concrete examples mentioned in Motivation, the napari package can make use of one of PyQt5, PyQt6, PySide2, or PySide6, and users currently need to explicitly specify napari[all] in order to have one of these be installed, or e.g., napari[pyqt5] to explicitly specify one of the frontend packages. Installing napari with no extras results in a non-functional package. With this PEP, napari could define the following configuration:
[project]
default-optional-dependency-keys = [
"pyqt5"
]
[project.optional-dependencies]
pyqt5 = [
"PyQt5",
"..."
]
pyside2 = [
"PySide2",
"..."
]
pyqt6 = [
"PyQt6",
"..."
]
pyside6 = [
"PySide6",
"..."
]
meaning that:
would work out-of-the-box, but there would still be a mechanism for users to explicitly specify a frontend, e.g.:
$ pip install napari[pyside6]Packages with multiple kinds of defaults
In some cases, it may be that packages need multiple kinds of defaults. As an example, in Packages requiring at least one backend or frontend, we considered the case of packages that have either backends or frontends, but in some cases, packages may have to support backends and frontends, and want to specify one or more default frontend and one or more default backend.
Ideally, one may want the following behavior:
$ pip install package # installs default backend and frontend $ pip install package[] # installs no backends or frontends $ pip install package[backend1] # installs backend1 and default frontend $ pip install package[frontend2] # installs frontend2 and default backend $ pip install package[backend1, frontend2] # installs backend1 and frontend2
However, this PEP chooses not to provide a mechanism for making it so that e.g., if backend1 is specified, the default backend would be disabled, but the default frontend would be enabled, since this adds complexity.
Maintainers should instead for now document that if a backend or frontend is explicitly specified, both backend and frontend need to be specified. Discoverability for users who want to do this should not be an issue however since users need to read the documentation in any case to find out what backends or frontends are available, so they can be shown at the same time how to properly use the extras for backends and frontends.
One option to increase user friendliness is that maintainers can create extras called for example defaultbackend and defaultfrontend which do install the default backend and frontend. They can then recommend that users do:
$ pip install package # installs default backend and frontend $ pip install package[] # installs no backends or frontends $ pip install package[backend1, defaultfrontend] # installs backend1 and default frontend $ pip install package[defaultbackend, frontend2] # installs frontend2 and default backend $ pip install package[backend1, frontend2] # installs backend1 and frontend2
This would allow (if desired) users to then get whatever the recommended backend is, even if that default changes in time.
If there is a desire to implement a better solution in future, we believe this PEP should not preclude this. For example, one could imagine in future adding the ability for an extra to specify which default extras it disables, and if this is not specified then explicitly specified extras would disable all default extras (consistent with the present PEP).
Backward Compatibility Security ImplicationsThere are no known security implications for this PEP.
How to teach thisThis section outlines information that should be made available to people in different groups in the community in relation to the implementation of this PEP. Some aspects described below will be relevant even before the PEP is fully implemented in packaging tools as there are some preparations that can be done in advance of this implementation to facilitate any potential transition later on. The groups covered below are:
Package end usersPackage users should be provided with clear installation instructions that show what extras are available for packages and how they behave, for example explaining that by default some recommended dependencies or a given frontend or backend will be installed, and how to opt out of this or override defaults, depending what is available.
Package authorsWhile the mechanism used to define extras and the associated rule about when to use it are clear, package authors need to carefully consider several points before adopting this capability in their packages, to avoid inadvertently breaking backward-compatibility.
Supporting older versions of package installersPackage installers such as pip or uv will not necessarily implement support for default extras at the same time, and once they do it is likely that package authors will want to keep supporting users who do not have the most recent version of a package installer. In this case, the following recommendations would apply:
recommended in astropy be a default extra, but in order to support users with older versions of package installers, the documentation should still mention the extra explicitly as long as possible (until it is clear that most/all users are using package installers that implement this PEP). There is no downside to keeping the extra be explicitly mentioned, but this will ensure that users with modern tooling who do not read documentation (which may be a non-negligible fraction of the user community) will start getting the recommended dependencies by default.package[] was equivalent to package, authors will be able to document package[] as a backward-compatible universal way of getting a minimal installation. For packages that define default extras, installing package[] will always give a minimal installation even with older versions of packaging tools such as pip, and releases of this package that pre-date the introduction of default extras for a specific package will also be installable with package[] (although in these cases this will be equivalent to package). For packages that do not define default extras, package[] will continue to be equivalent to package.One temptation for authors might be to include many dependencies by default since they can provide a way to opt out from these. We recommend however that authors carefully consider what is included by default to avoid unnecessarily bloating installations and complicating dependency trees. Using default extras does not mean that all extras need to be defaults, and there is still scope for users to explicitly opt in to non-default extras.
Default extras should generally be treated with the same “weight” as required dependencies. When a package is widely used, introducing a default extra will result in that extra’s dependencies being transitively included – unless all downstream packages are updated to explicitly opt out using minimal installation specifications.
As an example, the pytest package currently has nearly 1,500 plugins that depend on it. If pytest were to add a default extra and those plugins were not updated accordingly, installing a plugin would include the default extras’ dependencies. This doesn’t preclude the use of default extras, but addition of default extras requires careful evaluation of its downstream effects.
Circular dependenciesAuthors need to take special care when circular dependencies are present. For instance, consider the following dependency tree:
package1
└── package2
└── package1
If package1 has a default extra named recommended then:
will still result in the recommended extra being installed if package2 continues to depend on package1 (with no extras specified). This could be solved by changing the dependency tree to instead be:
package1
└── package2
└── package1[]
assuming that indeed package2 does not depend on any features provided by the extra dependencies of package1. Authors therefore need to carefully consider a migration plan, coordinating with the authors of package2.
The impact on individuals who repackage Python libraries for different distributions, such as conda, Homebrew, linux package installers (such as apt and yum) and so on, needs to be considered. Not all package distributions have mechanisms that would line up with the approach described. In fact, some distributions such as conda, do not even have the concept of extras.
There are two cases to consider here:
recommended dependencies by default since there is no way for users to explicitly request them otherwise).It is impossible for a PEP such as this to exhaustively consider each of the different package distributions. However, ultimately, default extras should be understood as how package authors would like their package to be installed for the majority of users, and this should inform decisions about how default extras should be handled, whether manually or automatically.
Reference ImplementationThe following repository contains a fully functional demo package that makes use of default extras:
https://github.com/wheel-next/pep_771
This makes use of modified branches of several packages, and the following links are to these branches:
In addition, this branch contains a modified version of the Flit package.
The implementations above are proofs-of-concept at this time and the existing changes have not yet been reviewed by the relevant maintainers. Nevertheless, they are functional enough to allow for interested maintainers to try these out.
Rejected Ideas Using a meta-package for recommended installationsUsing existing packaging tools and infrastructure, package maintainers who want to provide a minimal installation for some users and a default non-minimal installation for regular users (e.g. with recommended dependencies or a default backend) can technically already achieve this if they are willing to distribute two packages instead of one – for example package-core which would be the main package with minimal dependencies, and package which would be a metapackage that would depend on package-core with optional dependencies enabled.
Taking once again a concrete example from the Motivation section, the astropy package defines a recommended extra that users are currently instructed to install in the default installation instructions. In principle, one could rename the existing astropy package to e.g. astropy-core and then create a new astropy package which would be a metapackage that would contain the following dependencies section:
dependencies = [
"astropy-core[recommended]"
]
Since users may want to pin or place version constraints on the astropy meta-package (e.g. astropy>5.0), the metapackage would need to follow the same versions as the core package, and would need to use strict pinning in the dependency section, e.g.:
version = "7.1.0"
dependencies = [
"astropy-core[recommended]==7.1.0"
]
This idea may seem appealing because it is technically already feasible. However, in practice, many projects have opted not to do this, for a number of reasons, which we now take a look at. Some of these may not be applicable to future new projects, but some of them apply to all projects, old and new.
Mismatch between package and module nameIn terms of naming, there are two main options for a package that wants to use the metapackage approach:
package would provide the minimal installation, and to then create a new metapackage with a different name, such as package-all. However, this suffers from one of the problems that motivated this PEP in the first place - users are often not aware that they can do e.g. package[recommended], so in the same way, they might not realise that package-all exists. This once again places the burden on the average user to discover this, rather then shifting some of the burden to more advanced users.package-core, and for the new meta-package to be called package. This is a better option than the first one, but is not ideal, as it then introduces a non-intuitive mismatch between the package name and module name, in that package-core provides the package module, and package does not provide any module. An example of why this would lead to confusion is that an average user might think that uninstalling the package module would be done with e.g.:
but this would not be the case (the package module would still work), and it may not be obvious to this user that the package-core package even exists.
This approach requires either maintaining two repositories instead of one, or switching to using a monorepo which would contain both packages. Neither option is ideal:
package repository will need to update their remote URLs or any git clone URLs to point to the package-core repository. The alternative is to preserve the package repository to contain the package-core package, and have a different name for the meta-package, but this could lead to confusion.subdirectory= to the repo URL), and any existing workflows that clone the repository and assume the previous layout would break.Packages that need to depend on package versions that are older than the first version where the split was done will not easily be able to depend on the minimal package. Whereas with the main proposal in this PEP, downstream users will be able to depend on e.g. package[]>version where version pre-dates the introduction of default extras, with the splitting approach it will not be possible for downstream users to depend on e.g. package-core>version, since package-core did not previously exist.
A possible solution to this is for developers to release no-op metadata packages for all old versions of a package, but this is a significant additional burden on the maintainers.
UninstallationAs alluded to when referring to naming issues in Mismatch between package and module name, uninstalling packages will no longer work the way users expect. A user doing:
will still be left with package-core, but may not realise it. This is not just confusing, but is in effect a breaking change that may impact a number of existing workflows.
Having two packages instead of one would increase the long-term maintenance cost of package distributions simply by virtue of the fact that two packages would have to be released instead of one, and in some cases this would introduce extra manual work at each release.
Synchronizing metadataThe main metadata that would be important to keep synchronized between the main package and the metapackage is the version. Anytime a new release of the core package is done, the metapackage would need to have its version updated as well as the version pinning for the core package in the dependencies.
In addition, all extras defined in the core package would need to be redefined and kept in sync in the metapackage. For example, if package defines a additional extra, users should still be able to install package[additional], but users installing the package-core package should also have the option of doing package-core[additional].
Other metadata that would need to be kept in sync includes for example author information and project URLs.
SummaryOverall, this solution would imply a significantly higher maintenance burden, not just in terms of initial set-up and transition (which could already be prohibitive for large established projects), but also in terms of long-term maintenance. This also has the potential for breaking user workflows (in particular around the issue of repositories, and e.g. uninstallation). For all these reasons, we do not consider it a compelling alternative to the present PEP.
CopyrightThis document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4