Nginx uwsgi (104: Connection reset by peer) while reading response header from upstream
DjangoNginxUwsgiDjango Problem Overview
Environment is Nginx + uwsgi.
Getting a 502 bad gateway error from Nginx on certain GET requests. Seems to be related to the length of the URL. In our particular case, it was a long list of GET parameters. Shorten the GET parameters and no 502 error.
From the nginx/error.log
[error] 22113#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.1.100, server: server.domain.com, request: "GET <long_url_here>"
No information in the uwsgi error log.
Django Solutions
Solution 1 - Django
After spending a lot of time on this, I finally figured it out. There are many references to Nginx and connection reset by peer. Most of them seemed to be related to PHP. I couldn't find an answer that was specific to Nginx and uwsgi.
I finally found a reference to fastcgi and a 502 bad gateway error (https://support.plesk.com/hc/en-us/articles/213903705). That lead me to look for a buffer size limit in the uwsgi configuration which exists as buffer-size. The default value is 4096. From the documentation, it says:
> If you plan to receive big requests with lots of headers you can increase this value up to 64k (65535).
There are many ways to configure uwsgi, I happen to use a .ini file. So in my .ini file I tried:
buffer-size=65535
This fixed the problem. You can adjust that to taste. Maybe start with the max and work back until you have an acceptable value, or just leave it at the max.
This was frustrating to track down because there was no error on the uwsgi side of things.
Solution 2 - Django
I was getting the same nginx error and also there was no information in uwsgi log. The problem was that in some cases the application was not consuming the whole request body as advised in http://uwsgi-docs.readthedocs.org/en/latest/ThingsToKnow.html:
> If an HTTP request has a body (like a POST request generated by a form), you have to read (consume) it in your application. If you do not do this, the communication socket with your webserver may be clobbered. If you are lazy you can use the post-buffering option that will automatically read data for you. For Rack applications this is automatically enabled.
Of course, this is not a problem in your case, but it may be useful for others who are getting the same nginx error.
Solution 3 - Django
We just need to increase the attribute "output_buffering" value, in php.ini, to a greater value like 65535 or another appropriate value.
Solution 4 - Django
When we receive a message like (104: Connection reset by peer) while reading response header from upstream, most often, we could blame the upstream side of this kind of error.
As described, the connection was reset by the upstream peer, not by nginx itself. Nginx as a client can barely do anything to make it right.
I'm suspecting if modifying buffer-size will do the magic. Basically the command changes the buffer size where response headers are cached. This would take effect when the response header is too big, of which case we receive a message saying upstream sent too big header while reading response header from upstream, and that is totally different thing from connection reset by peer.
Since this kind of error is trigger randomly, I would suggest you check whether nginx uses keepalive when talking to upstreams. If this was the case, the connection might be reset by upstream server when the idle timed out whereas nginx had no idea that the connection had been dropped, hence forwarding the request using the same connection.
There's no elegant solution to fix it as far as I know. You could do retry or set a keepalive_timeout value to the upstream connection pool in nginx to avoid the problem.
referencing:
http://tengine.taobao.org/document/http_upstream_keepalive_timeout.html
Solution 5 - Django
--post-buffering 32768
worked for me as suggested (and discouraged) here https://stackoverflow.com/questions/13063454/nginx-uwsgi-connection-reset-by-peer
I don't have time to investigate it further at the moment (quick prototyping mode :), but since it took me a lot of time to find this hack, it might be worth posting here.
Solution 6 - Django
It doesn't come up occasionally.
I guess the most possible reason of that is the size of your php-fpm.log is oversize. Try to change your log_level to upper level in php-fpm.conf and clear the logs.
Anyway, it works for me.
Solution 7 - Django
This can happen if your request/response headers are quite large.
To fix it, in /etc/uwsgi/apps-available/your-app.ini add buffer-size=65535
Solution 8 - Django
You need to re-install PHP:
apt-get install --reinstall php5-fpm