Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://realpython.com/dependency-management-python-poetry/ below:

Dependency Management With Python Poetry – Real Python

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Managing Dependencies With Python Poetry

Poetry is a tool for managing Python projects that simplifies dependency management. You can use Poetry to specify, install, and resolve dependencies, ensuring that you work with the correct package versions. Poetry takes advantage of a pyproject.toml file for configuration and maintains a poetry.lock file to lock dependencies, providing a consistent environment across different machines.

By the end of this tutorial, you’ll understand that:

• Poetry is a dependency manager that ensures consistent package versions and simplifies Python project setup.
• You can install Poetry system-wide using pipx or the official installer for better environment management.
• Installing Poetry with pip in a project’s environment is not recommended due to potential dependency conflicts.
• Basic Poetry CLI commands include poetry new, poetry add, poetry install, and poetry update.
• The poetry.lock file locks dependency versions, ensuring reproducible environments.

To complete this tutorial and get the most out of it, you should have a basic understanding of virtual environments, modules and packages, and pip.

While you’ll focus on dependency management in this tutorial, Poetry can also help you build a distribution package for your project. If you want to share your work, then you can use Poetry to publish your project on the Python Packaging Index (PyPI).

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

Before diving into the nitty-gritty of Python Poetry, you’ll take care of some prerequisites. First, you’ll read a short overview of the terminology that you’ll encounter in this tutorial. Next, you’ll install Poetry itself.

If you’ve ever used an import statement in one of your Python scripts, then you’ve worked with modules and packages. Some of them might have been Python files you wrote on your own. Others could’ve been standard library modules that ship with Python, like datetime. However, sometimes, what Python provides isn’t enough. That’s when you might turn to external modules and packages maintained by third parties.

When your Python code relies on such external modules and packages, they become the requirements or dependencies of your project.

To find packages contributed by the Python community that aren’t part of the Python standard library, you can browse PyPI. Once you’ve found a package you’re interested in, you can use Poetry to manage and install that package in your project. Before seeing how this works, you need to install Poetry on your system.

Poetry is distributed as a Python package itself, which means that you can install it into a virtual environment using pip, just like any other external package:

• Windows
• Linux + macOS

This is fine if you just want to quickly try it out. However, the official documentation strongly advises against installing Poetry into your project’s virtual environment, which the tool must manage. Because Poetry depends on several external packages itself, you’d run the risk of a dependency conflict between one of your project’s dependencies and those required by Poetry. In turn, this could cause Poetry or your code to malfunction.

In practice, you always want to keep Poetry separate from any virtual environment that you create for your Python projects. You also want to install Poetry system-wide to access it as a stand-alone application regardless of the specific virtual environment or Python version that you’re currently working in.

There are several ways to get Poetry running on your computer, including:

1. A tool called pipx
2. The official installer
3. Manual installation
4. Pre-built system packages

In most cases, the recommended way to install Poetry is with the help of pipx, which takes care of creating and maintaining isolated virtual environments for command-line Python applications. After installing pipx, you can install Poetry by issuing the following command in your terminal window:

• Windows
• Linux + macOS

While this command looks very similar to the one you saw previously, it’ll install Poetry into a dedicated virtual environment that won’t be shared with other Python packages.

Crucially, pipx will also set an alias to the poetry executable in your shell so that you can invoke Poetry from any directory without manually activating the associated virtual environment:

• Windows
• Linux + macOS

When you type poetry in your terminal or PowerShell console, you’ll always refer to the executable script installed in its isolated virtual environment. In fact, you’ll often run the poetry command from within an active virtual environment of your project, and Poetry will correctly pick it up!

If, for some reason, you can’t or don’t want to use pipx, then the next best thing you can do is take advantage of the official installer, which you can download and execute with a command like this:

• Windows
• Linux + macOS

On Windows, you can use the Invoke-WebRequest PowerShell command along with the -UseBasicParsing option to download the content of a requested URL and print it onto the standard output stream (stdout). With the pipe operator (|), you can intercept the downloaded install-poetry.py script and pipe it into your Python interpreter from the standard input stream (stdin).

Note: Some users report errors on Windows 10 when they use the PowerShell command.

The curl command downloads the content of a requested URL and prints it onto the standard output stream (stdout). When you use the pipe operator (|), you define a Unix pipeline, which intercepts the downloaded install-poetry.py script and pipes it into your Python interpreter from the standard input stream (stdin).

Note: If you’re on macOS, then you may get an ssl.SSLCertVerificationError. This error can occur if you don’t have the default root certificates for the SSL module installed. You can install them by running a command script in your Python folder:

Depending on your installed Python version, the specific path to the Python interpeter might be different. In that case, you need to adjust the path in the above command accordingly.

After you run the command, the curl command above should work without any errors.

Underneath the install.python-poetry.org URL is a cross-platform Python script that, more or less, replicates what pipx does, but in a slightly different way.

If you’re interested in the technical details, then take a peek at the underlying source code of the installation script. By following its steps manually, you’ll have a chance to tweak the installation process if you ever need something specific for your setup. This can be particularly useful when you prepare a continuous integration environment.

Finally, some operating systems may bundle Poetry as a native package. For example, if you’re using a Debian-based system like Ubuntu, then you might be able to install Poetry with apt, like so:

Bear in mind that the Poetry version packaged like that might not be the latest one. Moreover, the native package may bring hundreds of megabytes of extra dependencies, such as another Python interpreter, which would be completely unnecessary if you installed Poetry using one of the earlier methods.

Once the Poetry installation is complete, you can confirm that it’s working correctly by typing poetry --version at your command prompt. This should print out your current version of Poetry.

Note: Depending on how you installed Poetry, you can try one of these commands to upgrade it:

• python3 -m pip install -upgrade poetry
• pipx upgrade poetry
• poetry self update
• sudo apt install --only-upgrade python3-poetry

All but the last one should generally ensure that you have the most recent version of Poetry installed on your computer.

Now that you’ve verified that Poetry was installed correctly, it’s time to see how it works.

In this section, you’ll learn how to create a fresh Poetry project using the tool’s command-line interface (CLI). You’ll also explore the project structure and inspect the pyproject.toml file, which houses the project’s metadata and configuration.

To create a new Poetry project from scratch, use the poetry new command followed by the desired project name. In this tutorial, you’ll work on a project named rp-poetry. Go ahead, create that project now, and then navigate into the newly created directory by entering the following commands in your terminal:

Running poetry new rp-poetry creates a new folder named rp-poetry/. When you look inside that folder, you’ll see the following structure:

rp-poetry/
│
├── rp_poetry/
│   └── __init__.py
│
├── tests/
│   └── __init__.py
│
├── README.md
└── pyproject.toml

Did you spot how Poetry translated the dash (-) in the provided name into an underscore (_) in the corresponding rp_poetry/ subfolder? The tool automatically normalizes Python package names for you. This ensures you’ll be able to import them as Python packages. Otherwise, rp-poetry wouldn’t be a valid identifier because a dash represents the minus operator in the Python syntax.

To have more control over the resulting package name, you can optionally pass the --name argument, which lets you specify a different name for the Python package than your project folder:

Note that Poetry will also normalize the package name provided through this argument should it contain a dash.

When creating the folder structure for a new project, Poetry follows the flat layout by default. In this layout, your Python package resides at the root level of the project folder. Another popular choice in Python is the src layout, which keeps the code in an additional src/ parent folder. If you prefer this layout instead, then pass the --src flag to Poetry when creating a new project:

Adding this flag will result in a slightly different folder structure:

rp-poetry/
│
├── src/
│   │
│   └── rp_poetry/
│       └── __init__.py
│
├── tests/
│   └── __init__.py
│
├── README.md
└── pyproject.toml

Now, your rp_poetry package is nested in the src/ folder. Both the flat and src layouts have their pros and cons, which you can compare by reading the Python Packaging Guide if you’re interested. However, you’ll stick to the default flat layout for the rest of this tutorial.

Note: It’s possible to specify a custom folder structure and even have multiple Python packages in one Poetry project. This can be useful in larger projects, which need to be organized into multiple, separate components. For example, the tic-tac-toe demo project comes with an engine library and several front-end components in a single distribution package.

To achieve this with Poetry, you’ll need to edit your project’s configuration file by hand, though. Refer to the official documentation for more details.

As you can see, creating a new project with Poetry is quite straightforward and convenient. You get a basic folder structure for free without having to think too much about how to organize your Python files.

The rp_poetry/ subfolder itself isn’t very spectacular yet. Inside, you’ll only find an empty __init__.py file, which turns it into an importable Python package.

Similarly, the tests/ subfolder is yet another Python package meant to contain the unit tests and possibly other kinds of tests for your project, promoting best programming practices.

Note: Older Poetry releases used to populate these two __init__.py files with sample code at this point. However, recent versions of Poetry generate these files empty to avoid making wrong assumptions about the project’s structure or your intentions as a developer.

Poetry also generates an empty README file with an .md extension by default, suggesting the use of Markdown for formatting. If you’re not fond of Markdown, then your other options are either plain text or the reStructuredText format. These are the three formats that PyPI can render at the moment.

Lastly, Poetry creates a configuration file named pyproject.toml with the minimum required metadata for your project, as described in PEP 518. You’ll take a closer look at it now.

One of the most important files for working with Poetry is the pyproject.toml file. This file isn’t an invention of Poetry but a configuration file standard defined in PEP 518:

This PEP specifies how Python software packages should specify what build dependencies they have in order to execute their chosen build system. As part of this specification, a new configuration file is introduced for software packages to use to specify their build dependencies (with the expectation that the same configuration file will be used for future configuration details). (Source)

The authors considered a few file formats for the new configuration file mentioned in the quote above. Ultimately, they’ve decided on the TOML format, which stands for Tom’s Obvious Minimal Language. In their opinion, TOML is flexible enough while having better readability and less complexity than other options like YAML, JSON, CFG, or INI.

Note: To allow Python to parse TOML files, which have recently become popular in the Python ecosystem, the standard library absorbed the third-party tomli package and rebranded it to tomllib. It’s been available since Python 3.11 but can only read TOML documents. In contrast, Poetry relies on its own tomlkit library, which supports both reading and writing while offering additional features like preserving the original file formatting.

To see what a TOML document looks like, open the pyproject.toml file that Poetry created in your project’s folder:

Currently, you can see three sections denoted with square brackets in the pyproject.toml file above. These sections are known as tables in the TOML terminology. They contain declarative instructions, which tools like Poetry can recognize and use for managing dependencies, building the project, or performing other tasks.

While PEP 518 defines a specific table name, [build-system], which contains the obligatory build requirements for your project, the standard leaves the door open for custom extensions. Third-party tools can group their configuration options under unique namespaces prefixed with the word tool.

For example, Poetry generates the [tool.poetry] and [tool.poetry.dependencies] tables. The first one is where you define your project’s metadata, such as the name and version, while the second table lets you specify external libraries managed by Poetry for your project.

Note: These two are examples of TOML’s subtables, which represent a hierarchy of configuration options. Here’s their equivalent JSON representation:

As you can see, the dot character (.) in a TOML table’s name is a delimiter, separating the different levels of the hierarchy, much like nested objects in JSON.

In this case, the two subtables belong to only one external tool—Poetry. But you’ll often find examples like [tool.black], [tool.isort], [tool.mypy], or [tool.pytest.ini_options] for their corresponding tools. While many Python tools are moving their configuration to pyproject.toml files, there are still notable exceptions. For example, flake8 didn’t support them at the time of writing, so you might need a few separate configuration files.

The pyproject.toml file starts with the [tool.poetry] subtable, where you can store general information about your project. Poetry defines a few table keys that are valid in this subtable. While some of them are optional, there are four that you must always specify:

1. name: The name of your distribution package that will appear on PyPI
2. version: The version of your package, ideally following semantic versioning
3. description: A short description of your package
4. authors: A list of authors, in the format name <email>

The subtable [tool.poetry.dependencies] on line 8 is essential for your dependency management. You’ll learn more about it and its variants in the next section when you add external dependencies. As of now, it only states the intended Python version for your project.

Note: When you create a new project with the poetry new command, the tool assumes the minimum Python version supported by your project based on the virtual environment Poetry itself was installed in. For example, if you installed Poetry using pipx on top of Python 3.12, then that interpreter version will appear in the resulting pyproject.toml file. Naturally, you can change this by editing the file.

The last table, [build-system] on line 11 in the pyproject.toml file, defines metadata that Poetry and other build tools can work with. As mentioned earlier, this table isn’t tool-specific, as it doesn’t have a prefix. It has two keys:

1. requires: A list of dependencies required to build your package, making this key mandatory
2. build-backend: The Python object used to perform the build process

If you want to learn about this section of the pyproject.toml file, then you can find out more by reading about source trees in PEP 517.

When you create a new project with Poetry, this is the pyproject.toml file that you start with. Over time, you’ll add more configuration details about your package and the tools you use. As your project grows, your pyproject.toml file will grow with it. That’s particularly true for the [tool.poetry.dependencies] subtable.

Note: PEP 621 specifies how to store project metadata in a [project] table within pyproject.toml. There’s an ongoing discussion about how to support PEP 621 in Poetry instead of using [tool.poetry].

In the next section, you’ll find out how to expand your project by adding third-party dependencies using Poetry.

Once you’ve set up a Poetry project, the real work can begin. In this section, you’ll find out how Poetry takes care of preparing an isolated virtual environment for your project, and how it manages your dependencies within it.

When you create a new Python project by hand, it’s a good practice to also create an associated virtual environment. Otherwise, you may confuse different dependencies from different projects. Poetry comes with built-in support for virtual environments to ensure that it never interferes with your global Python installation. The tool can do the bulk of virtual environment management for you.

However, Poetry doesn’t create a virtual environment right away when you start a new project. That’s by design to let you decide whether you want to manage your virtual environments yourself or let Poetry handle them for you automatically.

Poetry will detect a manually activated virtual environment when you run one of the Poetry commands in your project’s folder:

Here, after changing your working directory to rp-poetry/, you display the environment information using the poetry env info command. Because the current shell session has an activated virtual environment—as indicated by the prompt prefix (venv)—Poetry confirms it’ll use that environment for all subsequent commands within your project scope.

In other words, if you now tried adding dependencies to your project through Poetry, you’d install them into the activated environment as if with the regular pip install command. Poetry would also update the necessary metadata in pyproject.toml, as you’ll soon discover.

On the other hand, Poetry automatically creates a virtual environment—or reuses one it had created before—when you run certain commands without an activated environment in the shell. For example, it does so when you add or remove a dependency using Poetry’s command-line interface. This prevents projects from messing with your system-wide Python installation, ensuring that project dependencies remain isolated at all times.

Note: Poetry will try to find a compatible interpreter in your system for the new virtual environment based on the Python version declared in your pyproject.toml file.

Letting Poetry create virtual environments on its own is the preferred way of isolating dependencies in your projects. You’ll see how that works now.

You can list all virtual environments that Poetry manages within your project by running the following command in your project’s directory:

At this point, you won’t see any output because you haven’t executed any commands that would trigger Poetry to create a virtual environment yet. Later, you may define more than one virtual environment to test your project against several configurations or different Python versions. Note that the above command remains unaffected by the state of your shell, so it doesn’t matter if you currently have an activated virtual environment or not.

To see Poetry’s support for virtual environments in action, make sure to deactivate any virtual environment that you might have activated before:

From now on, Poetry will take care of the creation and management of virtual environments in your projects when you execute some of its commands.

If you want to have better control over the creation of a virtual environment, then you may tell Poetry explicitly which Python version you want to use up front:

With this command, you specify the path to a desired Python interpreter on your disk. Using bare python3 will work as long as you have the corresponding Python executable in your PATH. You can also use the minor Python version like 3.12 or just 3.

Note: The Python interpreter that you indicate to Poetry must satisfy the version constraint in your pyproject.toml file. Otherwise, Poetry will reject it with an error message.

When the version constraint is loose enough, then your end-users may run a different version of Python than the one you originally tested your code with. That could introduce compatibility issues, especially when external packages rely on specific Python versions.

When Poetry creates a new virtual environment, for example, after you’ve run the poetry env use <python> command for the first time, you’ll see a message similar to this one:

As you can see, Poetry constructed a unique name for your project’s virtual environment. It contains your package name from pyproject.toml and the corresponding Python version. The seemingly random string in the middle, Dod5cRxq, is a Base64-encoded hash value of the path leading up to your project’s parent directory. It ties the name of a virtual environment to your project’s location on disk.

When you move the project to another folder, Poetry will detect that and create a brand-new virtual environment behind the scenes if necessary. Thanks to the unique string in the middle, Poetry can handle multiple projects with identical names and the same Python version while keeping all virtual environments in one folder by default.

Unless you tell it otherwise, Poetry creates virtual environments in the virtualenvs/ subfolder of its cache directory, which is specific to the operating system:

If you want to change the default cache directory, then you can edit Poetry’s configuration. It can be useful when you already use virtualenvwrapper or another third-party tool for managing your virtual environments and don’t want to clutter your disk or mix a few cache locations.

To reveal your current Poetry configuration, which includes the cache-dir and virtualenvs.path settings, run this command:

Usually, you don’t have to change these paths. If you want to learn more about interacting with Poetry’s virtual environments, then the Poetry documentation contains a chapter about managing environments.

As long as you’re inside your project folder, Poetry will use the virtual environment associated with it. If you’re ever in doubt, you can check whether the virtual environment has been created by running the poetry env list command again:

Poetry indicates which of the virtual environments connected to your project is currently selected as the default one. Although it says activated, the corresponding virtual environment isn’t actually activated in your shell in the traditional sense. Instead, Poetry will temporarily activate that virtual environment in a subprocess when you run one of the Poetry commands.

Note: To switch between existing environments, you can issue the poetry env use <python> command again. To quickly remove all environments associated with your project, run the poetry env remove --all command.

With that skill under your belt, you’re ready to add some dependencies to your project and see Poetry shine.

When you created your rp-poetry project using Poetry, it scaffolded the minimal pyproject.toml file with the [tool.poetry.dependencies] subtable. So far, it only contains a declaration of the Python interpreter version, but it’s also where you can specify the external Python packages that your project requires.

Editing this file by hand can become tedious, though, and it doesn’t actually install anything into the project’s virtual environment. That’s where Poetry’s CLI comes into play again.

You may have used pip before to install packages that aren’t part of the Python standard library. If you run pip install with the package name as an argument, then pip looks for packages on the Python Package Index (PyPI). You can use Poetry in a similar way.

Running the poetry add command will automatically update your pyproject.toml file with the new dependency and install the package at the same time. In fact, you can even specify multiple packages in one go:

This will find the latest versions of both dependencies on PyPI, install them in the corresponding virtual environment, and insert the following two declarations in your pyproject.toml file:

The order of these declarations reflects the order in which you specified those packages in the command line.

Note the caret symbol (^) before the version specifiers, which indicates that Poetry is free to install any version matching the leftmost non-zero digit of the version string. For example, if the Requests library releases a new version 2.99.99, then Poetry will consider it an acceptable candidate for your project. However, version 3.0 wouldn’t be allowed.

Note: The idea behind this follows the semantic versioning scheme (major.minor.patch), where minor and patch updates shouldn’t introduce backward-incompatible changes. Still, this is just an assumption that could break your builds, and some notable Python developers disagree with such a choice of defaults on behalf of Poetry.

If you’d like to add a particular version of an external package or define custom version constraints, then Poetry lets you do that on the command line:

When you run this command, Poetry will first remove any previously installed versions of these packages and downgrade their indirect or transitive dependencies as needed. It’ll then determine the most suitable versions of these packages, taking into account other existing constraints to resolve potential conflicts.

Note: You may sometimes need to surround your package names with quotes when their version specifiers involve symbols with a special meaning. In this case, quoting the beautifulsoup4 package prevents the shell from interpreting the less-than symbol (<) as a redirection operator.

If you want to remove one or more dependencies from your project, then Poetry provides the related poetry remove command:

As you can see, it’ll remove the given dependency along with its transitive dependencies, so you don’t need to worry about leftover packages that are no longer needed by your project. This is an advantage of Poetry over plain pip, which can only uninstall the individual packages.

Notice that Poetry informs you about writing to the lock file whenever you add or remove a dependency. You’ll learn about that file later, but in a nutshell, you can think of it as a snapshot of your project’s dependencies at a given point in time. It serves the same purpose as the constraints file leveraged by pip.

So far, you’ve been adding runtime dependencies that were necessary to make your program work correctly. However, you might only need some dependencies at specific stages of development, such as during testing. Next up, you’ll see what tools Poetry gives you to manage those kinds of dependencies.

Another neat feature in Poetry, which is missing from pip, is the ability to manage groups of dependencies, allowing you to keep logically related dependencies separate from your runtime dependencies. For example, during development, you’ll often want additional packages, such as linters, type checkers, or testing frameworks, which would only bloat the final distribution package. Your users don’t need those, after all.

With Poetry, you can group dependencies under arbitrary names so that you can selectively install those groups later on when needed. Here’s how to add a few dependencies to a group called dev and some dependencies to another group called test:

Running these commands will cause Poetry to install the listed packages into the project’s virtual environment and also add two additional subtables in your pyproject.toml file, which look as follows:

The new subtables start with the name tool.poetry.group and end with the word dependencies. The part in the middle must be a unique group name.

Normally, when you don’t provide any group name, Poetry puts the specified packages in an implicit main group, which corresponds to the [tool.poetry.dependencies] subtable. Therefore, you can’t use the name main for one of your groups because it’s been reserved.

Note: Before Poetry 1.2.0, you could use the -D or --dev option as a shortcut for adding dependencies to a development group. This option has been deprecated, however, as it became incompatible with regular dependency groups. The deprecated feature used to create a separate subtable in the pyproject.toml file.

You can define optional groups by setting the corresponding attribute in the pyproject.toml file. For example, this declaration will turn your test group into an optional one:

Poetry won’t install dependencies belonging to such a group unless you explicitly instruct it to by using the --with option. Note that you must declare another TOML subtable to mark a group as optional because the group’s configuration is kept separately from its dependency list.

In addition to this, you can add the individual packages as optional to let the user choose whether to install them:

Optional dependencies are meant to be available at runtime when explicitly requested by the user during installation. It’s common to mark packages as optional when they’re platform-specific or when they provide features, such as a particular database adapter, that only some users will need.

In pyproject.toml, optional dependencies look slightly more verbose:

The mysqlclient and psycopg2-binary dependencies have their optional flag set to true, while their version string is kept in another attribute.

However, this isn’t enough to expose such optional dependencies to the user. You must also define extras in your pyproject.toml file, which are sets of optional dependencies that your users can install together:

Depending on your particular needs, you can opt in to the desired features by selecting one or more extras during installation. As you can see, it’s also possible to combine a few extras, which might be tailored to a specific use case, such as testing.

You’ve barely scratched the surface when it comes to dependency management with Poetry. There’s so much more to explore! That said, you’re already well-equipped to implement Poetry in your projects. In the next section, you’ll learn how to work with a Poetry project from a user’s perspective.

Imagine that you’ve just cloned a Git repository with the rp-poetry project from GitHub and are starting fresh with no virtual environment. To simulate that, you can remove some of Poetry’s metadata and any virtual environments associated with the project:

If you’re a developer who wants to contribute to this project, then you can execute the poetry install command inside the rp-poetry/ folder to get the ball rolling:

If other contributors have committed the poetry.lock file to the remote repository, which they generally should, then Poetry will read that file. It’ll reproduce the exact same environment on your machine with identical versions of all the dependencies listed in the most recent snapshot of poetry.lock.

Otherwise, Poetry will fall back to reading the top-level dependencies outlined in the pyproject.toml file and will resolve the set of packages satisfying the version constraints. As a result, you’ll have a new local poetry.lock file. This may potentially lead to a different state of dependencies in your virtual environment than those of other contributors who’ve installed the project at a different time.

The dependency resolution becomes essential when you have many dependencies that require several third-party packages with different versions of their own. Before installing any packages, Poetry figures out which version of a package fulfills the version constraints that other packages set as their requirements. That’s not a trivial task. In rare cases, a solution may not even exist!

By default, Poetry installs dependencies from the implicit main group as well as all dependency groups, unless you marked them as optional. It also installs your own rp-poetry distribution package in editable mode to allow for changes in your source code to be immediately reflected in the environment without the need for reinstallation.

In contrast, Poetry won’t automatically install extra sets of dependencies and optional groups of dependencies. To get those, you must use some of the following parameters:

• --all-extras
• --extras {extra}
• --with {optional groups}

You can install a few extras at the same time by repeating the --extras parameter for each desired set of optional dependencies. However, Poetry treats the individual extras as mutually exclusive until you say otherwise. So, it’ll only install those specified on the command line while removing all other extras from your virtual environment if needed. In particular, it’ll remove all extras when you don’t select at least one during installation.

Apart from that, you have a few more options, allowing you to cherry-pick exactly which dependencies to install, including:

When you specify the --only {groups} option, Poetry will ignore the --with {optional groups} and --without {groups} options.

Resolving dependencies results in updating or producing a new poetry.lock file. It’s where Poetry keeps track of all packages and their exact versions that your project uses. You’re going to have a closer look at that file now.

Whenever you interact with Poetry through its command-line interface, it updates the pyproject.toml file and pins the resolved versions in the poetry.lock file. However, you don’t have to let Poetry do all the work. You can manually modify dependencies in the pyproject.toml file and lock them afterward.

Poetry generates and refreshes the poetry.lock file when needed as long as you stick to its command-line interface. While the lock file is not meant to be changed by hand, you can edit the related pyproject.toml file at will. Unfortunately, this may sometimes cause both files to become out of sync.

Note: If you insist on manually tampering with the poetry.lock file, then you’re likely going to invalidate the underlying hashes, rendering the file broken and unusable.

Suppose you wanted to bring back the Requests library that you removed from the rp-poetry project earlier in this tutorial. You can open the pyproject.toml file in your text editor and type the necessary declaration in the main group of dependencies:

By using the asterisk (*) as the version constraint, you indicate that you’re not specifying any particular version of the Requests library, and that any version will be acceptable. But this library isn’t installed yet.

If you now open your terminal and navigate to the project’s parent directory, then you can tell Poetry to install the manually added dependencies into the associated virtual environment and update the lock file:

In this case, Poetry refuses to install the dependencies because your poetry.lock file doesn’t currently mention the Requests library present in the companion pyproject.toml file. Conversely, if you removed a declaration of a resolved and installed dependency from pyproject.toml, then you’d face a similar complaint. Why’s this happening?

Remember that Poetry always installs the resolved dependencies from the poetry.lock file, where it pinned down the exact package versions. It’ll only consider the dependencies that you’ve listed in the pyproject.toml file when it needs to update or regenerate a missing lock file.

Therefore, to fix such a discrepancy, you could delete the lock file and run poetry install again to let Poetry resolve all dependencies from scratch. That’s not the best approach, though. It’s potentially time-consuming. But even worse, it disregards the specific versions of previously resolved dependencies, removing the guarantee of reproducible builds.

A far better approach to align the two files is by manually locking the new dependencies with the poetry lock command:

This will update your poetry.lock file to match the current pyproject.toml file without installing any dependencies.

Poetry processes all dependencies in your pyproject.toml file, finds packages that satisfy the declared constraints, and pins their exact versions in the lock file. But Poetry doesn’t stop there. When you run poetry lock, it also recursively traverses and locks all dependencies of your direct dependencies.

Note: The poetry lock command also updates your existing dependencies if newer versions that still fit your version constraints are available. If you don’t want to update any dependencies that are already in the poetry.lock file, then add the --no-update option to the poetry lock command:

In this case, Poetry only resolves the new dependencies but leaves any existing dependency versions inside the poetry.lock file untouched.

It’s important to note that dependency locking is only about two things:

1. Resolving: Finding packages that satisfy all version constraints
2. Pinning: Taking a snapshot of the resolved versions in the poetry.lock file

Poetry doesn’t actually install the resolved and pinned dependencies for you after running poetry lock. To confirm this, try importing the locked Requests library from the associated virtual environment, which Poetry manages for you:

Here, you specify a one-liner program by providing the -c option to the Python interpreter in your virtual environment. The program tries to import the Requests library but fails due to a missing module, which wasn’t installed.

Now that you’ve pinned all dependencies, it’s time to install them so that you can use them in your project.

When the poetry.lock file agrees with its pyproject.toml counterpart, then you can finally install dependencies that Poetry locked for you:

By running poetry install, you make Poetry read the poetry.lock file and install all dependencies listed there. In this case, your virtual environment already had most of the required dependencies in place, so Poetry only installed the missing ones.

When you run the same command again, Poetry won’t have much left to do anymore:

As a result, the Requests library should already be available for grabs when you import it within the interactive Python REPL session started through Poetry:

This time, you can import requests without any trouble, which is great! You may now exit the interpreter by typing exit() and hitting Enter.

Note: The -q option runs Python in quiet mode, suppressing the welcome message with version and copyright information.

What if your virtual environment contains leftover packages that you previously installed but no longer need? Poetry doesn’t mind such packages, even if they’re not formally declared in pyproject.toml or poetry.lock. You could’ve installed them as optional dependency groups some time ago or completely outside of Poetry’s control by, for instance, using pip directly:

The httpie package indirectly brings ten additional dependencies, which take up space and could potentially interfere with your project’s actual dependencies. Besides, external packages might sometimes create security holes if you don’t keep them up-to-date.

To synchronize your virtual environment with the locked packages pinned in poetry.lock, you can pass the optional --sync flag to the poetry install command:

This ensures that your virtual environment only contains the packages specified in your pyproject.toml and poetry.lock files, preventing potential conflicts caused by unnecessary or outdated dependencies.

In the next section, you’ll learn how to update your project’s dependencies using Poetry.

Suppose you added the Requests library to your project in December 2020, when the latest version of this library was the 2.25.1 release. By default, Poetry configured a permissive version constraint in your pyproject.toml file, which involves a caret (^2.25.1) to allow future non-breaking updates to pass through.

Over the years, you’ve been adding many other dependencies to your project with poetry add, and Poetry automatically picked up more recent releases of Requests that still satisfied the original version constraint. Poetry then updated the lock file accordingly and installed new versions of dependencies into your virtual environment. Now, you have the 2.29.0 release of Requests pinned in the poetry.lock file and installed in your environment.

Fast forward to the time of writing, when the library’s 2.31.0 release has become a cutting-edge version. To verify this, you can ask Poetry to compare your locked dependencies against their latest releases on PyPI by running this command:

This new release continues to satisfy your version constraint for Requests, so Poetry should accept it. However, when you try to run poetry install again, it doesn’t do anything:

Recall that the poetry install command prioritizes the poetry.lock file over your version constraints declared in pyproject.toml to ensure reproducible environments. If you want to update dependencies with compatible versions, then you have the following choices:

• Remove poetry.lock and run poetry install
• Run poetry lock followed by poetry install
• Run poetry update

As mentioned earlier, the first option has its drawbacks because it forces Poetry to recalculate all dependencies from scratch. The other two options are essentially equivalent, so you might choose the latter to handle both steps at once.

Updating dependencies always carries some risk. To better understand what’s about to happen, you can ask Poetry to perform a dry run before taking the dive:

The highlighted lines indicate which dependencies will be updated and in which direction.

The poetry update command will lock and update all packages along with their dependencies to their latest compatible versions. If you want to update one or more specific packages, then you can list them as arguments:

With this command, Poetry will search for a new version of requests and a new version of beautifulsoup4 that fulfill the version constraints listed in your pyproject.toml file. Then, it’ll resolve all dependencies of your project and pin their versions in your poetry.lock file. At the same time, your pyproject.toml file won’t change because the listed constraints remain valid.

Normally, to upgrade a dependency to a version that’s outside of the version constraints declared in your pyproject.toml file, you must adjust that file beforehand. Alternatively, you can forcefully upgrade a dependency to its latest version by running the poetry add command with the at operator (@) and a special keyword, latest:

When you run the poetry add command with the latest keyword, Poetry will ignore your current version constraint in pyproject.toml and replace it with a new constraint based on the latest version found. It’s as if you’ve never added that dependency to your project before. Use this option with caution, as an incompatible version of one of your dependencies could break the project.

If you now open your pyproject.toml file, then the version constraint for the Requests library will look as follows:

Without using the latest keyword or an explicit version constraint in the poetry add command, Poetry would conclude that the requested package is already present in your project and would do nothing.

Now you’ve gotten a handle on how Poetry uses pyproject.toml and poetry.lock. Next up, you’ll have a final look at both files.

The version constraints of the dependencies declared in your pyproject.toml file can be fairly loose. This allows for some level of flexibility when incorporating bug fixes or resolving version conflicts. By having more package versions to choose from, Poetry is more likely to find a combination of compatible dependencies.

On the other hand, Poetry tracks the exact versions of the dependencies that you’re actually using in the poetry.lock file. This improves Poetry’s performance by caching the resolved package versions so that it doesn’t have to resolve them again every time you install or update your dependencies.

To ensure reproducible environments across your team, you should consider committing the poetry.lock file to your version control system like Git. By keeping this file tracked in a version control system, you ensure that all developers will use identical versions of required packages. However, there’s one notable exception.

When you develop a library rather than an application, it’s common practice not to commit the poetry.lock file. Libraries typically need to remain compatible with multiple versions of their dependencies rather than with a single, locked-down set.

When you come across a repository that contains a poetry.lock file, it’s a good idea to use Poetry to manage its dependencies. On the other hand, if other developers on your team aren’t using Poetry yet, then you should coordinate with them about the potential adoption of Poetry across the board before making any decisions. You must always maintain compatibility with other dependency management tools that the team might be using.

Chances are, you already have some projects that didn’t start their life with the poetry new command. Or maybe you inherited a project that wasn’t created with Poetry, but now you want to use Poetry for your dependency management. In these types of situations, you can add Poetry to existing Python projects.

Say you have an rp-hello/ folder with a hello.py script inside:

It’s the classic Hello, World! program, which prints the famous string on the screen. But maybe this is just the beginning of a grand project, so you decide to add Poetry to it. Instead of using the poetry new command from before, you’ll use the poetry init command inside your project folder:

The poetry init command collects the necessary information to generate a pyproject.toml file by asking you questions interactively. It gives you recommendations with sensible defaults for most of the configurations that you need to set up, and you can press Enter to accept them. When you don’t declare any dependencies, the pyproject.toml file that Poetry creates looks something like this:

This content resembles the examples that you went through in the previous sections.

Now you can use all the commands that Poetry offers. With a pyproject.toml file present, you can now run your script from an isolated virtual environment:

Because Poetry didn’t find any virtual environments to use, it created a new one before executing your script. After doing this, it displays your Hello, World! message without any errors. That means you now have a working Poetry project.

Sometimes you have a project that already comes with its own requirements file, which lists all the external dependencies. Take a look at the requirements.txt file of this Python web scraper project:

This project requires seven third-party packages to run properly. Granted, only the Requests and Beautiful Soup libraries are directly referenced in the source code, while the rest are pulled in as their transitive dependencies.

Poetry doesn’t deal with such requirements files out of the box, as it keeps your project dependencies elsewhere. However, once you’ve created a Poetry project with poetry init, you can quickly import an existing requirements.txt file into the project by using this command:

The cat utility reads the specified file and writes its content to the standard output stream. In this case, you pass that content to the poetry add command with the help of the shell’s command substitution syntax. That, in turn, installs each dependency listed in the requirements.txt file into your Poetry project.

When a requirements file is straightforward like this, using poetry add and cat can save you some manual work. However, this isn’t ideal because all dependencies end up being listed in your pyproject.toml file:

Normally, you only want to list those packages that your project directly depends on. Apart from that, you should generally avoid using exact version numbers, as it can lead to conflicts with existing packages or prevent you from getting the latest features and security updates. Instead, you should specify a version range that allows for some flexibility while still adhering to semantic versioning principles.

In a perfect world, your pyproject.toml file should contain only these two dependencies with slightly more permissive version specifiers:

Poetry will resolve the remaining dependencies and lock them in the corresponding poetry.lock file.

Now, when you show your project’s dependencies as a tree, you’ll know exactly which of them are used by your project directly and which are their transitive dependencies:

Okay. You know how to import dependencies into a Poetry project. What about exporting them the other way around?

In some situations, you must have a requirements.txt file. For example, you may want to host your Django project on Heroku, which expects the requirements.txt file to determine what dependencies to install. Poetry gives you a few options to go about creating this file.

As long as your project’s virtual environment is up-to-date, you can pin your dependency versions using the traditional pip freeze command:

To execute the command within the context of your project’s virtual environment managed by Poetry, you must use poetry run. Alternatively, you could’ve activated the virtual environment in an interactive shell and then run the same command from there. Either way, you must first ensure that all the required dependencies are installed in their expected versions by, for example, issuing the poetry install command.

If that sounds like too much of a hassle to you, then you can install Poetry’s export plugin, which replaces the legacy poetry export command. It essentially lets you export dependencies from poetry.lock to various file formats, including requirements.txt, which is the default.

Depending on how you installed Poetry itself, these are the commands that you can choose from to install the mentioned plugin:

• poetry self add poetry-plugin-export
• pipx inject poetry poetry-plugin-export
• python -m pip install poetry-plugin-export

Once the plugin is installed, you can use the following command to export dependencies from your Poetry-managed project to a requirements.txt file:

The resulting file includes hashes and environment markers by default, meaning that you can work with very strict requirements that resemble the content of your poetry.lock file. As a matter of fact, the export plugin needs the lock file in order to generate the requirements file. If that file doesn’t exist, then Poetry will create it after resolving and locking your dependencies.

Even though the plugin looks at the lock file to get a big picture of your project’s dependencies, it’ll only export dependencies from the implicit main group in your pyproject.toml file. If you’d like to include additional dependencies from dependency groups, including the optional and non-optional groups, then use the --with parameter followed by comma-separated names of those groups:

Conversely, to generate a requirements file with dependencies belonging to only a few of your dependency groups while excluding the implicit main group, use the --only option:

In all cases, the extras with optional dependencies won’t be exported. To include them as well, you must either list them explicitly by repeating the --extras parameter or enable them all at once with the --all-extras flag.

You might have noticed that the export plugin shares a few options with the poetry install command that you saw before. That’s no coincidence. To reveal the plugin’s available options and their descriptions, run poetry export --help in your terminal.

In this tutorial, you’ve gotten an introduction to Poetry’s dependency management. Along the way, you’ve used some of Poetry’s command-line interface (CLI) commands:

You can check out the Poetry CLI documentation to learn more about the commands above and many other commands that Poetry offers. You can also run poetry --help to see information right in your terminal!

In this tutorial, you practiced creating new Poetry projects and using Poetry on top of existing projects. You learned that the pyproject.toml file plays a key role in Poetry, helping to manage project dependencies and configuration in a standardized format. Additionally, the poetry.lock file ensures that you can record and maintain the exact versions of each dependency across different installations or environments.

When you track the poetry.lock file in your Git repository, you also guarantee that all other developers on your team install identical dependency versions on their machines.

In this tutorial, you learned how to:

• Create a new project using Poetry
• Add Poetry to an existing project
• Configure your project through pyproject.toml
• Pin your project’s dependency versions
• Install dependencies from a poetry.lock file
• Run basic Poetry commands using the Poetry CLI

This tutorial focused on the basics of dependency management with Poetry, but the tool can also help you build and publish your Python packages. If you want to get a taste of this capability, then you can read about publishing an open-source Python package to PyPI using Poetry.

Now that you have some experience with dependency management using Poetry, you can use the questions and answers below to check your understanding and recap what you’ve learned.

These FAQs are related to the most important concepts you’ve covered in this tutorial. Click the Show/Hide toggle beside each question to reveal the answer.

You can install Python Poetry using pipx, the official installer, or by using system packages like apt on Debian-based systems. The recommended method is to use pipx to ensure Poetry is installed in an isolated environment.

While you can install Poetry with pip for quick testing, it’s not recommended to do so in a project’s virtual environment to prevent dependency conflicts. Instead, it’s better to install it system-wide using pipx or the official installer.

The basic Poetry CLI commands include poetry new to create a project, poetry add to add dependencies, poetry install to install them, and poetry update to update them. You can also use poetry lock to pin dependencies and poetry show to list installed packages.

The poetry.lock file records the exact versions of your project’s dependencies, ensuring consistent environments across different setups. It helps maintain reproducible builds by locking dependencies to specific versions.

To resolve dependency conflicts in Poetry, update your dependencies by modifying version constraints in pyproject.toml, then run poetry update. If issues persist, manually adjust the dependencies or consult the documentation for more advanced conflict resolution strategies.

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Managing Dependencies With Python Poetry

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4