What is pyproject.toml file for?

Background

I was about to try Python package downloaded from GitHub, and realized that it did not have a setup.py, so I could not install it with

pip install -e <folder>

Instead, the package had a pyproject.toml file which seems to have very similar entries as the setup.py usually has.

What I found

Googling lead me into PEP-518 and it gives some critique to setup.py in Rationale section. However, it does not clearly tell that usage of setup.py should be avoided, or that pyproject.toml would as such completely replace setup.py.

Questions

Is the pyproject.toml something that is used to replace setup.py? Or should a package come with both, a pyproject.toml and a setup.py?
How would one install a project with pyproject.toml in an editable state?

Yes, pyproject.toml is the specified file format of PEP 518 which contains the build system requirements of Python projects.

This solves the build-tool dependency chicken and egg problem, i.e. pip can read pyproject.toml and what version of setuptools or wheel one may need.

If you need a setup.py for an editable install, you could use a shim in setup.py:

#!/usr/bin/env python

import setuptools

if __name__ == "__main__":
    setuptools.setup()

pyproject.toml is the new unified Python project settings file that replaces setup.py. Editable installs still need a setup.py: import setuptools; setuptools.setup()

To use pyproject.toml, run python -m pip install .

Then, if the project is using poetry instead of pip, you can install dependencies (into %USERPROFILE%\AppData\Local\pypoetry\Cache\virtualenvs) like this:

poetry install

And then run dependencies like pytest:

poetry run pytest tests/

And pre-commit (uses .pre-commit-config.yaml):

poetry run pre-commit install
poetry run pre-commit run --all-files

What is it for?

Currently there are multiple packaging tools being popular in Python community and while setuptools still seems to be prevalent it's not a de facto standard anymore. This situation creates a number of hassles for both end users and developers:

1. For setuptools-based packages installation from source / build of a distribution can fail if one doesn't have setuptools installed;
2. pip doesn't support the installation of packages based on other packaging tools from source, so these tools had to generate a setup.py file to produce a compatible package. To build a distribution package one has to install the packaging tool first and then use tool-specific commands;
3. If package author decides to change the packaging tool, workflows must be changed as well to use different tool-specific commands.

pyproject.toml is a new configuration file introduced by PEP 517 and PEP 518 to solve these problems:

> ... think of the (rough) steps required to produce a built artifact for a project: > > 1. The source checkout of the project. > 2. Installation of the build system. > 3. Execute the build system. > > This PEP [518] covers step #2. PEP 517 covers step #3 ...

Any tool can also extend this file with its own section (table) to accept tool-specific options, but it's up to them and not required.

PEP 621 suggests using pyproject.toml to specify package core metadata in static, tool-agnostic way. Which backends currently support this is shown in the following table:

enscons	flit_core	hatchling	pdm-pep517	poetry-core	setuptools
0.26.0+	3.2+	0.3+	0.3.0+	Issue #3332	![Experimental support] 61.0.0+

[Experimental support]: https://i.stack.imgur.com/PPQ88.png "Experimental support"

Does it replace setup.py?

For setuptools-based packages pyproject.toml is not strictly meant to replace setup.py, but rather to ensure its correct execution if it's still needed. For other packaging tools – yes, it is:

> Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it). Projects may still wish to include a setup.py for compatibility with tools that do not use this spec.

How to install a package in editable mode?

Originally "editable install" was a setuptools-specific feature and as such it was not supported by PEP 517. Later on PEP 660 extended this concept to packages using pyproject.toml.

There are two possible conditions for installing a package in editable mode using pip:

• Modern:
  Both the frontend (pip) and a backend must support PEP 660.
  pip supports it since version 21.3;
• Legacy:
  Packaging tool must provide a setup.py file which supports the develop command.
  Since version 21.1 pip can also install packages using only setup.cfg file in editable mode.

The following table describes the support of editable installs by various backends:

enscons	flit_core	hatchling	pdm-pep517	poetry-core	setuptools
0.28.0+	3.4+	0.3+	0.8.0+	1.0.8+	![Legacy support] Issue #2816

[Legacy support]: https://i.stack.imgur.com/PPQ88.png "Legacy support"

Answering this part only, as the rest has nicely been explained by others: > How would one install a project with pyproject.toml in an editable state?

Workaround (working)

1. Clone the project and run poetry install to created a normal installation of it first.

2. Get the path of the virtualenv that poetry has created using poetry env info -p. It will probably be something like <absolute path of your cloned project/.venv. Then check what is the path to site-packages directory inside it. It depends on your Python version and will be something like <absolute path of your cloned project/.venv/lib/python3.9/site-packages.

3. Create a .pth file inside this site-packages. It should contain a single line of text with the full path of the package to install in editable mode:

echo "<absolute path of your cloned project>" > "<absolute path of your cloned project/.venv/lib/python3.9/site-packages/package_name.pth"

Proper fix (not implemented as of August 2021)

In June 2021 a PEP-660 that describes editable installs for pyproject.toml-based builds has been accepted.

Its implementation for poetry is being worked on in this PR.

So we need to wait for it be finished, get merged and a new poetry version released.

Sources

Apart from the ones linked above, my main source is "pip install -e . equivalent?" - issue #34 in github.com/python-poetry/poetry.