Docker Nginx stopped: [emerg] 1#1: host not found in upstream

I am running docker-nginx on ECS server. My nginx service is suddenly stopped because the proxy_pass of one of the servers got unreachable. The error is as follows:

[emerg] 1#1: host not found in upstream "dev-example.io" in /etc/nginx/conf.d/default.conf:988

My config file is as below:

 server {
       listen      80;
       server_name     test.com;
       location / {
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;

          ##proxy_set_header   Host             $host;
          proxy_set_header   X-Real-IP        $remote_addr;
          proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;

          proxy_buffer_size          4k;
          proxy_buffers              4 32k;
          proxy_busy_buffers_size    64k;
          proxy_temp_file_write_size 64k;
       }
}

server {
   listen       80 default_server;
   server_name  localhost;

   #charset koi8-r;
   #access_log  /var/log/nginx/log/host.access.log  main;

   location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
   }

   #error_page  404              /404.html;

   # redirect server error pages to the static page /50x.html
   #
   error_page   500 502 503 504  /50x.html;
   location = /50x.html {
      root   /usr/share/nginx/html;
   }
}

I have many servers in the config file, even if one server was down, I need to have running nginx. Is there any way to fix it?

Any suggestion to fix this issue would be appreciated.

Just adding a resolver did not resolve the issue in my case. But I was able to work around it by using a variable for the host. Also, I guess it makes more sense to use Docker's DNS at 127.0.0.11 (this is a fixed IP).

Example:

server {
  listen 80;
  server_name test.com;
  
  location / {
    resolver 127.0.0.11;
    set $example dev-example.io:5016;
    proxy_pass http://$example;
  }
}

I found the variable workaround on this page.

Include to prevent Nginx from crashing if your site is down, include a resolver directive, as follows:

 server {
       listen                80;
       server_name           test.com;
       location / {
          resolver           8.8.8.8;
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;
 ...

WARNING! Using a public DNS create a security risk in your backend since your DNS requests can be spoofed. If this is an issue, you should point the resolver to a secure DNS server.

This usually means that the dns name you provided as upstream server cannot be resolved. To test it, log on nginx server and try pinging upstream server provided and see if the name resolution completes correctly, If its a docker container try docker exec -it to get a shell, then try pinging the upstream to test the name resolution. If the contianer is stopped try to use IP address instead of dns name in your server block.

proxy_pass         http://<IP ADDRESS>:5016/;

You can also use the resolver directive if you want to use different dns server for this location than the host system:

resolver 8.8.8.8;

When using nginx plus, you can get around this as well by adding a zone to your upstream with resolve. When use this test in your proxypass. When the server some-server starts resolving, it will starting pass traffic to it.

Make sure to as stated above, put a resolver in other parts of your config. For docker, I use resolver 127.0.0.11 valid=1s;

upstream test {
        zone test-zone 64k;
        server some-server:1234 resolve;
}