Docker Network Nginx Resolver
NetworkingNginxDockerNetworking Problem Overview
I am trying to get rid of deprecated Docker links in my configuration. What's left is getting rid of those Bad Gateway
nginx reverse proxy errors when I recreated a container.
Note: I am using Docker networks in bridge mode. (docker network create nettest
)
I am using the following configuration snippet inside nginx:
location / {
resolver 127.0.0.1 valid=30s;
set $backend "http://confluence:8090";
proxy_pass $backend;
- I started a container with hostname
confluence
on my Docker network with namenettest
. - Then I started the nginx container on network
nettest
. - I can ping
confluence
from inside the nginx container confluence
is listed inside the nginx container's/etc/hosts
file- nginx log says
send() failed (111: Connection refused) while resolving, resolver: 127.0.0.1:53
- I tried the docker network default dns resolver
127.0.0.11
from/etc/resol.conf
- nginx log says
confluence could not be resolved (3: Host not found)
Anybody knows how to configure nginx resolver with Docker Networks or an alternative on how to force Nginx to correctly resolve the Docker network hostname?
Networking Solutions
Solution 1 - Networking
First off, you should be using the Docker embedded DNS server at 127.0.0.11
.
Your problem could be caused by 1 of the following:
-
nginx is trying to use IPv6 (AAAA record) for the DNS queries.
See https://stackoverflow.com/a/35516395/1529493 for the solution.
Basically something like:
http { resolver 127.0.0.11 ipv6=off; }
This is probably no longer a problem with Docker 1.11:
> Fix to not forward docker domain IPv6 queries to external servers > (#21396)
-
Take care that you don't accidentally override the
resolver
configuration directive. In my case I had in theserver
blockresolver 8.8.8.8 8.8.4.4;
from Mozilla's SSL Configuration Generator, which was overriding theresolver 127.0.0.11;
in thehttp
block. That had me scratching my head for a long time...
Solution 2 - Networking
Maybe you should check your container's /etc/resolv.conf
It shows your container's correct DNS config and then use that DNS server IP for resolver.
127.0.0.11
does not works in Rancher
Solution 3 - Networking
I was running "node:12.18-alpine" with angular frontend and hit the same problem with proxy_pass.
Locally it was working with:
resolver 127.0.0.11;
As simple as that! Just execute:
$ cat /etc/resolv.conf | grep nameserver
In your container to get this ip address.
However, when deploying to kubernetes (AWS EKS) I got the very same error:
failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53
Solution:
First solution was to find out the IP of the kube-dns service like below:
$ kubectl get service kube-dns -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 178d
Simple replacing IP for CLUSTER-IP worked like a charm.
Later, after some more doc digging, I find out that I could reference the service by name (which is little bit more elegant and resilient):
resolver kube-dns.kube-system valid=10s;
Solution 4 - Networking
My problem was $request_uri at the end. After adding it at the end of uri and changing the 127.0.0.1 to 127.0.0.11 solved my issue. I hope it will help people to not spend hours on this.
location /products {
resolver 127.0.0.11;
proxy_pass http://products:3000$request_uri;
}
Solution 5 - Networking
In several cases where I had this error, adding resolver_timeout 1s;
to the Nginx config solved the issue. Most of the time I don't have a resolver
entry.
Edit: what also worked for containers where I could explicitly define a nameserver: resolver DNS-IP valid=1s;
Solution 6 - Networking
We hit this with docker containers on windows trying to lookup host.docker.internal
using the docker internal resolver at 172.0.0.11
. All queries would resolve correctly except host.docker.internal
. Fix was to add the ipv6=off
flag to the resolver line in nginx.conf
.
Solution 7 - Networking
I solved this problem with the following way:
docker run --rm -d --network host --name "my_domain" nginx
Solution 8 - Networking
You need a local dns server like dnsmasq
to resolve using 127.0.0.1. Try installing it using apk add --update dnsmasq
and set it up if you're using an alpine (nginx:alpine
) variant.