Cloudfront custom-origin distribution returns 502 "ERROR The request could not be satisfied." for some URLs
SslCdnAmazon CloudfrontSsl Problem Overview
We have a Cloudfront distribution with custom origin that has been working just fine for quite a long time, serving static assets for one of our sites. Just this morning, we noticed that our logo was displaying as a broken link.
Upon further investigation, Cloudfront is returning a strange error message that I've never seen before for the URL in question:
> # ERROR > > ## The request could not be satisfied. > > ------ > > ------ > > Generated by cloudfront (CloudFront)
Several other Cloudfront URLs from this distribution return the same error, but then others (again, from the same distribution) are working just fine. I don't see a pattern to what works and what doesn't.
Some other data points:
- The origin URLs work just fine. There's been no recent interruption in service, to my knowledge.
- I've invalidated the logo URL specifically, to no effect.
- I've invalidated the root URL of the distribution, to no effect.
Any idea what's going on here? I've never seen Cloudfront do this before.
UPDATE:
Here's the verbatim HTTP response from Cloudfront:
$ http GET https://d2yu7foswg1yra.cloudfront.net/static/img/crossway_logo.png
HTTP/1.1 502 Bad Gateway
Age: 213
Connection: keep-alive
Content-Length: 472
Content-Type: text/html
Date: Wed, 18 Dec 2013 17:57:46 GMT
Server: CloudFront
Via: 1.1 f319e8962c0268d31d3828d4b9d41f98.cloudfront.net (CloudFront)
X-Amz-Cf-Id: H_HGBG3sTOqEomHzHubi8ruLbGXe2MRyVhGBn4apM0y_LjQa_9W2Jg==
X-Cache: Error from cloudfront
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
</BODY></HTML>
<BR clear="all">
<HR noshade size="1px">
<ADDRESS>
Generated by cloudfront (CloudFront)
</ADDRESS>
</BODY></HTML>
Ssl Solutions
Solution 1 - Ssl
I had this error today with Amazon Cloudfront. It was because the cname I used (e.g cdn.example.com) was not added to the distribution settings under "alternate cnames", I only had cdn.example.com forwarded to the cloudfront domain in my site/hosting control panel, but you need to add it to Amazon CloudFront panel too.
Solution 2 - Ssl
I had a similar issue recently which turned out to be due to ssl_ciphers that I was using.
"CloudFront forwards HTTPS requests to the origin server using the SSLv3 or TLSv1 protocols and the AES128-SHA1 or RC4-MD5 ciphers. If your origin server does not support either the AES128-SHA1 or RC4-MD5 ciphers, CloudFront cannot establish an SSL connection to your origin. "
I had to change my nginx confg to add AES128-SHA ( deprecated RC4:HIGH ) to ssl_ciphers to fix the 302 error. I hope this helps. I have pasted the line from my ssl.conf
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:RSA+3DES:AES128-SHA:!ADH:!AECDH:!MD5;
Solution 3 - Ssl
Found my answer and adding it here in case it helps David (and others).
Turns out my origin server (say www.example.com) had a 301 redirect setup on it to change HTTP to HTTPS:
HTTP/1.1 301 Moved Permanently
Location: https://www.example.com/images/Foo_01.jpg
However, my Origin Protocol Policy was set to HTTP only. This caused CloudFront to not find my file and throw a 502 error. Additionally, I think it cached the 502 error for 5 min or so as it didn't work immediately after removing that 301 redirect.
Hope that helps!
Solution 4 - Ssl
In our case, everything LOOKED ok, but it took most of the day to figure this out:
TLDR: Check your certificate paths to make sure the root certificate is correct. In the case of COMODO certificates, it should say "USERTrust" and be issued by "AddTrust External CA Root". NOT "COMODO" issued by "COMODO RSA Certification Authority".
From the CloudFront docs: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecureConnections.html
> If the origin server returns an invalid certificate or a self-signed certificate, or if the origin server returns the certificate chain in the wrong order, CloudFront drops the TCP connection, returns HTTP error code 502, and sets the X-Cache header to Error from cloudfront.
We had the right ciphers enabled as per: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#RequestCustomEncryption
Our certificate was valid according to Google, Firefox and ssl-checker: https://www.sslshopper.com/ssl-checker.html
However the last certificate in the ssl checker chain was "COMODO RSA Domain Validation Secure Server CA", issued by "COMODO RSA Certification Authority"
It seems that CloudFront does not hold the certificate for "COMODO RSA Certification Authority" and as such thinks the certificate provided by the origin server is self signed.
This was working for a long time before apparently suddenly stopping. What happened was I had just updated our certificates for the year, but during the import, something was changed in the certificate path for all the previous certificates. They all started referencing "COMODO RSA Certification Authority" whereas before the chain was longer and the root was "AddTrust External CA Root".
Because of this, switching back to the older cert did not fix the cloudfront issue.
I had to delete the extra certificate named "COMODO RSA Certification Authority", the one that did not reference AddTrust. After doing this, all my website certificates' paths updated to point back to AddTrust/USERTrust again. Note can also open up the bad root certificate from the path, click "Details" -> "Edit Properties" and then disable it that way. This updated the path immediately. You may also need to delete multiple copies of the certificate, found under "Personal" and "Trusted Root Certificate Authorities"
Finally I had to re select the certificate in IIS to get it to serve the new certificate chain.
After all this, ssl-checker started displaying a third certificate in the chain, which pointed back to "AddTrust External CA Root"
Finally, CloudFront accepted the origin server's certificate and the provided chain as being trusted. Our CDN started working correctly again!
To prevent this happening in the future, we will need to export our newly generated certificates from a machine with the correct certificate chain, i.e. distrust or delete the certificate "COMODO RSA Certification Authroity" issued by "COMODO RSA Certification Authroity" (expiring in 2038). This only seems to affect windows machines, where this certificate is installed by default.
Solution 5 - Ssl
One more possible solution: I have a staging server that serves the site and the Cloudfront assets over HTTP. I had my origin set to "Match Viewer" instead of "HTTP Only". I also use the HTTPS Everywhere extension, which redirected all the http://*.cloudfront.net URLs to the https://* version. Since the staging server isn't available over SSL and Cloudfront was matching the viewer, it couldn't find the assets at https://example.com and cached a bunch of 502s instead.
Solution 6 - Ssl
I just went through troubleshooting this issue, and in my case it indeed was related to redirects, but not related to incorrect settings in my CloudFront Origin or Behavior. This will happen if your origin server is still redirecting to origin URLs, and not what you have set up for your cloudfront URLs. Seems this is very common if you forget to change configs. For example lets say if you have www.yoursite.com CNAME to your cloudfront distribution, with an origin of www.yoursiteorigin.com. Obviously people will come to www.yoursite.com. But if your code tries to redirect to any page on www.yoursiteorigin.com you WILL get this error.
For me, my origin was still doing the http->https redirects to my origin URLs and not my Cloudfront URLs.
Solution 7 - Ssl
In my case, it was because we had an invalid ssl cert. The problem was on our staging box and we use our prod cert on that as well. It had worked for the past couple of years with this configuration, but all of a sudden we started getting this error. Strange.
If others are getting this error, check that the ssl certificate is valid. You can enable logging to s3 via the AWS CloudFront Distribution interface to aid debugging.
Also, you can refer to amazon's docs on the matter here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecureConnections.html
Solution 8 - Ssl
I ran into this problem, which resolved itself after I stopped using a proxy. Maybe CloudFront is blacklisting some IPs.
Solution 9 - Ssl
Fixed this issue by concatenating my certificates to generate a valid certificate chain (using GoDaddy Standard SSL + Nginx).
http://nginx.org/en/docs/http/configuring_https_servers.html#chains
To generate the chain:
cat 123456789.crt gd_bundle-g2-g1.crt > my.domain.com.chained.crt
Then:
ssl_certificate /etc/nginx/ssl/my.domain.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/my.domain.com.key;
Hope it helps!
Solution 10 - Ssl
The problem, in my case, was that I was using Amazon's Cloudflare and Cloudfront's Cloudfront in tandem, and Cloudfront did not like the settings that I had provided Cloudflare.
More specifically, in the Crypto settings on Cloudflare, I had set the "Minimum TLS Settings" to 1.2, without enabling the TLS 1.2 communication setting for the distribution in Cloudfront. This was enough to make Cloudfront declare a 502 Bad Gateway error when it tried to connect to the Cloudflare-protected server.
To fix this, I had to disable SSLv3 support in the Origin Settings for that Cloudfront distribution, and enable TLS 1.2 as a supported protocol for that origin server.
To debug this problem, I used command-line versions of curl, to see what Cloudfront was actually returning when you asked for an image from its CDN, and I also used the command-line version of openssl, to determine exactly which protocols Cloudflare was offering (it wasn't offering TLS 1.0).
tl:dr; make sure everything accepts and asks for TLS 1.2, or whatever latest and greatest TLS everyone is using by the time you read this.
Solution 11 - Ssl
For my particular case it was due to the fact that the Origin ALB behind my CloudFront Behavior had a DEFAULT ACM certificate which was pointing to a different domain name.
To fix this I had to:
- Go to the ALB
- Under the Listeners tab, selected my Listener and then Edit
- Under the Default SSL Certificate, choose the correct Origin Certificate.
Solution 12 - Ssl
Beware the Origin Protocol Policy:
> For HTTPS viewer requests that CloudFront forwards to this origin, one of the domain names in the SSL certificate on your origin server must match the domain name that you specify for Origin Domain Name. Otherwise, CloudFront responds to the viewer requests with an HTTP status code 502 (Bad Gateway) instead of returning the requested object.
In most cases, you probably want CloudFront to use "HTTP Only", since it fetches objects from a server probably hosted with Amazon too. No need for additional HTTPS complexity at this step.
Note that this is different to the Viewer Protocol Policy. You can read more about the differences between the two here.
Solution 13 - Ssl
In my case, I use nginx as reverse-proxy for an API Gateway URL. I got same error.
I resolved the issue when I added the following two lines to the Nginx config:
proxy_set_header Host "XXXXXX.execute-api.REGION.amazonaws.com";
proxy_ssl_server_name on;
Source is here: https://stackoverflow.com/questions/47823443/setting-up-proxy-pass-on-nginx-to-make-api-calls-to-api-gateway
Solution 14 - Ssl
In our case, we had dropped support for SSL3, TLS1.0, and TLS1.1 for PCI-DSS compliance on our origin servers. However, you have to manually add support for TLS 1.1+ on your CloudFront origin server config. The AWS console displays the client-to-CF SSL settings, but does not easily show you CF-to-origin settings until you drill down. To fix, in the AWS console under CloudFront:
- Click DISTRIBUTIONS.
- Select your distro.
- Click ORIGINS tab.
- Select your origin server.
- Click EDIT.
- Select all protocols that your origin supports under "Origin SSL Protocols"
Solution 15 - Ssl
Make sure you have correctly configured your SSL/TLS/Cipher settings. CloudFront will drop HTTPS connections if your origin server is not handshaking with the propper TLS Ciphers.
I recommend the following settings:
# Apache
SSLCipherSuite 'ECDHE+AES:@STRENGTH:+AES256'
SSLCipherSuite 'ECDHE+AES:DHE+AES:@STRENGTH:+AES256:kRSA+3DES'
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLProtocol all -SSLv3
SSLHonorCipherOrder on
# Nginx
ssl_ciphers 'ECDHE+AES:@STRENGTH:+AES256';
ssl_ciphers 'ECDHE+AES:DHE+AES:@STRENGTH:+AES256:kRSA+3DES';
ssl_protocols all -SSLv3 -TLSv1 -TLSv1.1;
ssl_protocols all -SSLv3;
ssl_prefer_server_ciphers on;
The @STRENGTH directive will sort the ciphers in order of strength.
SSLHonorCipherOrder on (apache) and ssl_prefer_server_ciphers on (Nginx) will ensure that the order is respected.
You can view the complete list of available ciphers your version of openssl is supporting by running the command on the shell:
$ openssl ciphers -v 'ALL'
You can also list the available ciphers for the above suggested directives in strength order in a similar fashion:
$ openssl ciphers -v 'ECDHE+AES:@STRENGTH:+AES256'
If you are having such issues, I strongly suggest you increase the verbosity of your web server logs, in particular, pertaining to SSL.
In Apache, you must may do this with the following directive in your conf file
LogLevel debug
Here is a list of possible Apache LogLevel directives:
emerg (emergencies - system is unusable)
alert (action must be taken immediately)
crit (critical conditions)
error (error conditions)
warn (warning conditions)
notice (normal but significant condition)
info (informational)
debug (debug-level messages)
trace1 (trace messages)
trace2 (trace messages)
trace3 (trace messages)
trace4 (trace messages)
trace5 (trace messages)
trace6 (trace messages)
trace7 (trace messages, dumping large amounts of data)
trace8 (trace messages, dumping large amounts of data)
Generally speaking, debug will be sufficient for you to identify the negotiation issue (it might be a cipher issue, it might be a cert issue)
A typical reaction from CloudFront not being able to successfully negotiate SSL with your origin server would look something like this:
[ssl:info] [pid 25091] [client xxx.xxx.xxx.xxx:15078] AH01964: Connection to child 1 established (server example.com:443)
[ssl:debug] [pid 25091] ssl_engine_kernel.c(2372): [client xxx.xxx.xxx.xx:15078] AH02043: SSL virtual host for servername example.com found
[ssl:debug] [pid 25091] ssl_engine_io.c(1368): (104)Connection reset by peer: [client xxx.xxx.xxx.xxx:15078] AH02007: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[ssl:info] [pid 25091] [client xxx.xxx.xxx.xxx.15078] AH01998: Connection closed to child 1 with abortive shutdown (server example.com:443)
In this case, Apache interprets CloudFront's dropping of the connection by the "Hint: stop button pressed in browser?!". Yeah, kinda comical.
On a side note, CloudFront can be a tricky beast. If you force HTTPS or direct HTTP to HTTPS in your distribution, you must ensure that the communication between CloudFront and your origin server is running over a valid SSL connection with a valid SSL certificate. This may mean obtaining a certificate over Amazon ACM by linking it over an Elastic Load Balancer (ELB).
... or you may also achieve the desired results by having the ACM certificate linked to your distribution (make sure you have both the apex domain and subdomains listed, ex: example.com and *.example.com). This will propagate the ACM certificate to your origin/target in the distribution, however, your Apache/Nginx server in your origin server will have to have a valid and working SSL certificate, even if it is generated by something like Letsencrypt/Certbot (our some other valid non-self-signed certificate). Make sure you have the full chain configured in your apache *.conf setup.
You may read more about issues pertaining to 502 Bad Gateway or 502 Could Not Satisfy The Request from AWS as well as requiring HTTPS for communication between CloudFront and your custom origin (this may also help). This information about ciphers was extremely helpful, as well as the list of supported ciphers on AWS
Some of these commands may also be useful for debugging your SSL situation:
openssl s_client -connect test.example.com:443 -tls1_1
(you may experiment using -tls1_2 , -tls1_3 or other protocols)
You may also want to try out using the http command line tool (you might have to install it), which would give you an output like such:
$ http https://example.com --head
HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 251
Content-Type: text/html; charset=iso-8859-1
Date: Mon, 20 Dec 2021 19:20:15 GMT
Location: https://example.com
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Via: 1.1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.cloudfront.net (CloudFront)
X-Amz-Cf-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
X-Amz-Cf-Pop: GIG51-C2
X-Cache: Miss from cloudfront
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Good luck debugging, and remember: keep digging deeper, don't give up, and increase the verbosity of your logs!