How to add a custom CA Root certificate to the CA Store used by pip in Windows?
PythonWindowsSslPipPython Problem Overview
I just installed Python3 from python.org and am having trouble installing packages with pip. By design, there is a man-in-the-middle packet inspection appliance on the network here that inspects all packets (ssl included) by resigning all ssl connections with its own certificate. Part of the GPO pushes the custom root certificate into the Windows Keystore.
When using Java, if I need to access any external https sites, I need to manually update the cacerts in the JVM to trust the Self-Signed CA certificate.
How do I accomplish that for python? Right now, when I try to install packages using pip, understandably, I get wonderful [SSL: CERTIFICATE_VERIFY_FAILED] errors.
I realize I can ignore them using the --trusted-host parameter, but I don't want to do that for every package I'm trying to install.
Is there a way to update the CA Certificate store that python uses?
Python Solutions
Solution 1 - Python
Self-Signed Certificate Authorities pip / conda
After extensively documenting a similar problem with Git (https://stackoverflow.com/questions/11621768/how-can-i-make-git-accept-a-self-signed-certificate/41253757#41253757), here we are again behind a corporate firewall with a proxy giving us a MitM "attack" that we should trust and:
> NEVER disable all SSL verification! > > This creates a bad security culture. Don't be that person.
tl;dr
pip config set global.cert path/to/ca-bundle.crt
pip config list
conda config --set ssl_verify path/to/ca-bundle.crt
conda config --show ssl_verify
# Bonus while we are here...
git config --global http.sslVerify true
git config --global http.sslCAInfo path/to/ca-bundle.crt
But where do we get ca-bundle.crt?
Get an up to date CA Bundle
cURL publishes an extract of the Certificate Authorities bundled with Mozilla Firefox
https://curl.haxx.se/docs/caextract.html
I recommend you open up this cacert.pem file in a text editor as we will need to add our self-signed CA to this file.
Certificates are a document complying with X.509 but they can be encoded to disk a few ways. The below article is a good read but the short version is that we are dealing with the base64 encoding which is often called PEM in the file extensions. You will see it has the format:
----BEGIN CERTIFICATE----
....
base64 encoded binary data
....
----END CERTIFICATE----
Getting our Self Signed Certificate
Below are a few options on how to get our self signed certificate:
- Via OpenSSL CLI
- Via Browser
- Via Python Scripting
Get our Self-Signed Certificate by OpenSSL CLI
echo quit | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 > cacert.pem
Get our Self-Signed Certificate Authority via Browser
- Acquiring your CA: https://stackoverflow.com/a/50486128/622276
Thanks to this answer and the linked blog, it shows steps (on Windows) how to view the certificate and then copy to file using the base64 PEM encoding option.
Copy the contents of this exported file and paste it at the end of your cacerts.pem file.
For consistency rename this file cacerts.pem --> ca-bundle.crt and place it somewhere easy like:
# Windows
%USERPROFILE%\certs\ca-bundle.crt
# Linux/macOS
$HOME/certs/cabundle.crt
Get our Self-Signed Certificate Authority via Python
Thanks to all the brilliant answers in:
I have put together the following to attempt to take it a step further.
https://github.com/neozenith/get-ca-py
Finally
Set the configuration in pip and conda so that it knows where this CA store resides with our extra self-signed CA.
# Windows
pip config set global.cert %USERPROFILE%\certs\ca-bundle.crt
conda config --set ssl_verify %USERPROFILE%\certs\ca-bundle.crt
OR
# Linux / macOS
pip config set global.cert $HOME/certs/ca-bundle.crt
conda config --set ssl_verify $HOME/certs/ca-bundle.crt
THEN
pip config list
conda config --show ssl_verify
# Hot tip: use -v to show where your pip config file is...
pip config list -v
# Example output for macOS and homebrew installed python
For variant 'global', will try loading '/Library/Application Support/pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/pip.conf'
Troubleshooting
Based on a great comment below
> I've tried this and still get a SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')) error. Any suggestions?
here is a troubleshooting guide:
This is the normal error message when the certificates authorities are not yet correctly setup.
It could be a variety of factors to check:
- The paths to your ca-bundle.crt have the right path separators for your OS (it has stung me),
- you may not have the latest CAs to verify normal certificates,
- you may not have added your CA in the right encoding.
Python is effectively doing those 3 steps:
- Find my CA store,
- read all entries,
- look up this certificate against my trust store.
If any of those fail you get this error message from experience.
Check this answer linked from below to display and check your ssl_cert_dir using:
python -c "import ssl; print(ssl.get_default_verify_paths())"
References
- Pip SSL: https://pip.pypa.io/en/stable/user_guide/#configuration
- Conda SSL: https://stackoverflow.com/a/35804869/622276
- Acquiring your CA: https://stackoverflow.com/a/50486128/622276
- Using Python to automatically grab your Peer CA: https://stackoverflow.com/questions/16903528/how-to-get-response-ssl-certificate-from-requests-in-python
Solution 2 - Python
Run: python -c "import ssl; print(ssl.get_default_verify_paths())" to check the current paths which are used to verify the certificate. Add your company's root certificate to one of those.
The path openssl_capath_env points to the environment variable: SSL_CERT_DIR.
If SSL_CERT_DIR doesn't exist, you will need to create it and point it to a valid folder within your filesystem. You can then add your certificate to this folder to use it.
Solution 3 - Python
Alternative solution on Windows is to install python-certifi-win32 that will allow Python to use Windows Certificate Store.
pip install python-certifi-win32
Solution 4 - Python
Not best answer but you can reuse an already created ca bundle using --cert option of pip, for instance:
pip install SQLAlchemy==1.1.15 --cert="C:\Users\myUser\certificates\my_ca-bundle.crt"
Solution 5 - Python
On Windows, I solved it by creating a pip.ini file in %APPDATA%\pip\
e.g. C:\Users\asmith\AppData\Roaming\pip\pip.ini
In the pip.ini I put the path to my certificate:
[global]
cert=C:\Users\asmith\SSL\teco-ca.crt
https://pip.pypa.io/en/stable/user_guide/#configuration has more information about the configuration file.
Solution 6 - Python
I think nt86's solution is the most appropriate because it leverages the underlying Windows infrastructure (certificate store). But it doesn't explain how to install python-certifi-win32 to start with since pip is non functional.
The trick is to use --trustedhost to install python-certifi-win32 and then after that, pip will automatically use the windows certificate store to load the certificate used by the proxy.
So in a nutshell, you should do:
pip install python-certifi-win32 -trustedhost pypi.org
and after that you should be good to go
Solution 7 - Python
Open Anaconda Navigator.
Go to File\Preferences.
Enable SSL verification Disable (not recommended)
or Enable and indicate SSL certificate path(Optional)
Update a package to a specific version:
Select Install on Top-Right
Select package click on tick
Mark for update
Mark for specific version installation
Click Apply