What's the difference between dist-packages and site-packages?

I'm a bit miffed by the python package installation process. Specifically, what's the difference between packages installed in the dist-packages directory and the site-packages directory?

dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location:

/usr/lib/python2.7/dist-packages

Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here:

/usr/local/lib/python2.7/dist-packages

From the Debian Python Wiki:

> dist-packages instead of site-packages. Third party Python software > installed from Debian packages goes into dist-packages, not > site-packages. This is to reduce conflict between the system Python, > and any from-source Python build you might install manually.

This means that if you manually compile and install Python interpreter from source, it uses the site-packages directory. This allows you to keep the two installations separate, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.

dist-packages is the debian-specific directory where apt and friends install their stuff, and site-packages is the standard pip directory.

The problem is -- what happens when different versions of the same package are present in different directories?

My solution to the problem is to make dist-packages a symlink to site-packages:

for d in $(find $WORKON_HOME -type d -name dist-packages); do
  pushd $d
  cd ..
  if test -d dist-packages/__pycache__; then
    mv -v dist-packages/__pycache__/* site-packages/__pycache__/
    rmdir -v dist-packages/__pycache__
  fi
  mv -v dist-packages/* site-packages/
  rmdir -v dist-packages
  ln -sv site-packages dist-packages
  popd
done

(if you are not using gnu tools, remove the -v option).

Debian (and Ubuntu) has introduced its own convention

# python3 -m site

on Ubuntu Focal gives

sys.path = [
    '/qpid-dispatch',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

The convention, as described in the linked mailing list, is that python deb packages installed by the distribution package manager go into /usr/lib/python3/dist-packages and packages installed using sudo pip3 go into /usr/local/lib/python3.8/dist-packages.

If you compile and install your own Python interpreter, it will default to placing itself in /usr/local, with /usr/local/bin/pip3 installs going into /usr/local/lib/pythonX.Y/site-packages.

The point of the Debian convention is to keep the three sets of packages separate:

1. python packages installed by apt
2. packages installed by root user with /usr/bin/pip3
3. packages installed by root user with their own /usr/local/bin/pip3