Python setuptools: How can I list a private repository under install_requires?
PythonGithubSetuptoolsPython Problem Overview
I am creating a setup.py
file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:
from setuptools import setup
setup(name='my_project',
...,
install_requires=[
'public_package',
'other_public_package',
'private_repo_1',
'private_repo_2',
],
dependency_links=[
'https://github.com/my_account/private_repo_1/master/tarball/',
'https://github.com/my_account/private_repo_2/master/tarball/',
],
...,
)
I am using setuptools
instead of distutils
because the latter does not support the install_requires
and dependency_links
arguments per this answer.
The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools
authenticate.
Here are some things I've tried:
-
Use
git+ssh://
instead ofhttps://
independency_links
as I would if installing the repo withpip
. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Dittogit+https
andgit+http
. -
https://<username>:<password>@github.com/...
- still get a 404. (This method doesn't work withcurl
orwget
from the command line either - thoughcurl -u <username> <repo_url> -O <output_file_name>
does work.) -
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.
Currently I just have setup.py
print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)
Duplicate-ish question with no answers here.
Python Solutions
Solution 1 - Python
I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508
standard should be used, from [2] I retrieved an example which actually does work (at least for my case).
Please note; this is with pip 20.0.2
on Python 3.7.4
setup(
name='<package>',
...
install_requires=[
'<normal_dependency>',
# Private repository
'<dependency_name> @ git+ssh://[email protected]/<user>/<repo_name>@<branch>',
# Public repository
'<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>',
],
)
After specifying my package this way installation works fine (also with -e
settings and without the need to specify --process-dependency-links
).
References [1] https://github.com/pypa/pip/issues/4187 [2] https://github.com/pypa/pip/issues/5566
Solution 2 - Python
Here's what worked for me:
install_requires=[
'private_package_name==1.1',
],
dependency_links=[
'git+ssh://[email protected]/username/private_repo.git#egg=private_package_name-1.1',
]
Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.
Solution 3 - Python
I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.
GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
- For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
- You need to include some version number (here is
0
) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word. - You need to preface with
git+
to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball version
can be a branch, a tag, or a commit hash- You need to supply
--process-dependency-links
if installing from pip
Solution 4 - Python
I found a (hacky) workaround:
#!/usr/bin/env python
from setuptools import setup
import os
os.system('pip install git+https://github-private.corp.com/user/repo.git@master')
setup( name='original-name'
, ...
, install_requires=['repo'] )
I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.
Solution 5 - Python
Via Tom Hemmes' answer I found this is the only thing that worked for me:
install_requires=[
'<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']
Solution 6 - Python
Using archive URL from github works for me, for public repositories. E.g.
dependency_links = [
'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]
Solution 7 - Python
With pip 20.1.1, this works for me
install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],
in setup.py
Solution 8 - Python
Edit: This appears to only work with public github repositories, see comments.
dependency_links=[
'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],
Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.
Solution 9 - Python
This work for our scenario:
- package is on github in a private repo
- we want to install it into site-packages (not into ./src with -e)
- being able to use pip install -r requirements.txt
- being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt
https://github.com/pypa/pip/issues/3610#issuecomment-356687173