SSL: CERTIFICATE_VERIFY_FAILED with Python3
PythonPython 3.xMacosUrllibPython Problem Overview
I apologize if this is a silly question, but I have been trying to teach myself how to use BeautifulSoup so that I can create a few projects.
I was following this link as a tutorial: https://www.youtube.com/watch?v=5GzVNi0oTxQ
After following the exact same code as him, this is the error that I get:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1240, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 911, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 854, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1237, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "WorldCup.py", line 3, in <module>
x = urllib.request.urlopen('https://www.google.com')
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 162, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 465, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 483, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 443, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1283, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1242, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Can someone help me figure out how to fix this?
Python Solutions
Solution 1 - Python
In my case, I used the ssl
module to "workaround" the certification like so:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Then to read your link content, you can use:
urllib.request.urlopen(urllink)
Solution 2 - Python
Go to the folder where Python is installed, e.g., in my case (Mac OS) it is installed in the Applications folder with the folder name 'Python 3.6'. Now double click on 'Install Certificates.command'. You will no longer face this error.
For those not running a mac, or having a different setup and can't find this file, the file merely runs:
pip install --upgrade certifi
Hope that helps someone :)
Solution 3 - Python
On Debian 9 I had to:
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
I'm not sure why, but this enviroment variable was never set.
Solution 4 - Python
This has changed in recent versions of the ssl library. The SSLContext was moved to it's own property. This is the equivalent of Jia's answer in Python 3.8
import ssl
ssl.SSLContext.verify_mode = ssl.VerifyMode.CERT_OPTIONAL
Solution 5 - Python
When you are using a self signed cert urllib3 version 1.25.3 refuses to ignore the SSL cert
To fix remove urllib3-1.25.3 and install urllib3-1.24.3
pip3 uninstall urllib3
pip3 install urllib3==1.24.3
Tested on Linux MacOS and Window$
Solution 6 - Python
As a workaround (not secure), you can turn certificate verification off by setting PYTHONHTTPSVERIFY environment variable to 0:
export PYTHONHTTPSVERIFY=0
Solution 7 - Python
Building on the update to Jia's 2018 answer in deltree's late 2021 one I was able to achieve equivalent functionality with:
import urllib.request
import ssl
def urllib_get_2018():
# Using a protected member like this is not any more fragile
# than extending the class and using it. I would use it.
url = 'https://localhost:6667/my-endpoint'
ssl._create_default_https_context = ssl._create_unverified_context
with urllib.request.urlopen(url = url) as f:
print(f.read().decode('utf-8'))
def urllib_get_2022():
# Finally! Able to use the publice API. Happy happy!
url = 'https://localhost:6667/my-endpoint'
scontext = ssl.SSLContext(ssl.PROTOCOL_TLS)
scontext.verify_mode = ssl.VerifyMode.CERT_NONE
with urllib.request.urlopen(url = url, context=scontext) as f:
print(f.read().decode('utf-8'))
I needed to use CERT_NONE
instead of CERT_OPTIONAL
as well as creating a ssl.SSLContext(ssl.PROTOCOL_TLS)
to pass to urlopen
.
It's important to keep using urllib
as it makes sense when working with small container images where pip
might not be installed, yet.
Solution 8 - Python
I have a lib what use https://requests.readthedocs.io/en/master/ what use https://pypi.org/project/certifi/ but I have a custom CA included in my /etc/ssl/certs
.
So I solved my problem like this:
# Your TLS certificates directory (Debian like)
export SSL_CERT_DIR=/etc/ssl/certs
# CA bundle PATH (Debian like again)
export CA_BUNDLE_PATH="${SSL_CERT_DIR}/ca-certificates.crt"
# If you have a virtualenv:
. ./.venv/bin/activate
# Get the current certifi CA bundle
CERTFI_PATH=`python -c 'import certifi; print(certifi.where())'`
test -L $CERTFI_PATH || rm $CERTFI_PATH
test -L $CERTFI_PATH || ln -s $CA_BUNDLE_PATH $CERTFI_PATH
Et voilĂ !
Solution 9 - Python
I faced the same issue with Ubuntu 20.4 and have tried many solutions but nothing worked out. Finally I just checked openssl version. Even after update and upgrade, the openssl version showed OpenSSL 1.1.1h [22 Sep 2020]. But in my windows system, where the code works without any issue, openssl version is OpenSSL 1.1.1k 25 Mar 2021.
I decided to update the openssl manually and it worked! Thank God!!!
Steps are as follows(Ubuntu 20.4):
*To check openssl version
openssl version -a
*To update openssl:
sudo apt install build-essential checkinstall zlib1g-dev
cd /usr/local/src/
sudo wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz
sudo tar -xf openssl-1.1.1k.tar.gz
cd openssl-1.1.1k
sudo ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo make
sudo make test
sudo make install
cd /etc/ld.so.conf.d/
sudo nano openssl-1.1.1k.conf
*Type /usr/local/ssl/lib and save
sudo ldconfig -v
sudo nano /etc/environment
*Add ':/usr/local/ssl/bin' to the path
source /etc/environment
echo $PATH
*Now check openssl version
openssl version -a
Solution 10 - Python
I had this problem in MacOS, and I solved it by linking the brew installed python 3 version, with
brew link python3
After that, it worked without a problem.