Network calls fail during image build on corporate network
DnsDockerDns Problem Overview
I'm having a problem building Docker images on my corporate network. I'm just getting started with Docker, so I have the following Dockerfile for a hello-world type app:
# DOCKER-VERSION 0.3.4
FROM centos:6.4
# Enable EPEL for Node.js
RUN rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# Install Node.js and npm
RUN yum install -y npm
# Bundle app source
ADD . /src
# Install app dependencies
RUN cd /src; npm install
EXPOSE 8080
CMD ["node", "/src/index.js"]
This works fine when I build it on my laptop at home, on my own wireless network. It pulls down the requisite dependencies and builds the image correctly.
However, when I'm on my corporate network at work, this same docker build fails when trying to pull down the RPM from download.fedoraproject.org, with this error message:
> Step 2 : RUN rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm ---> Running in e0c26afe9ed5 curl: (5) Couldn't resolve proxy 'some.proxy.address' error: skipping http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm - transfer failed
On my corporate network, I can access that URL just fine from my laptop. But once Docker is trying to build the container, all of a sudden it can't resolve at all. This behavior is the same for a variety of external resources (apt-get, etc.): They all can resolve just fine on my laptop on the corporate network, but Docker can't resolve them.
I don't have the network know-how to figure out what's going on here. Does anyone know why this strange behaviour would be occurring when building Docker containers?
Dns Solutions
Solution 1 - Dns
I was able to figure out the issue. On Ubuntu, Docker sets the DNS servers for container to Google's servers at 8.8.8.x. As I understand it, this is a workaround on Ubuntu due to the fact that Ubuntu sets /etc/resolv.conf to be 127.0.0.1.
Those Google servers weren't accessible from behind our firewall, which is why we couldn't resolve any URLs.
The fix is to tell Docker which DNS servers to use. This fix depends on how you installed Docker:
Ubuntu Package
If you have the Ubuntu package installed, edit /etc/default/docker and add the following line:
DOCKER_OPTS="--dns <your_dns_server_1> --dns <your_dns_server_2>"
You can add as many DNS servers as you want to this config. Once you've edited this file you'll want to restart your Docker service:
sudo service docker restart
Binaries
If you've installed Docker via the binaries method (i.e. no package), then you set the DNS servers when you start the Docker daemon:
sudo docker -d -D --dns <your_dns_server_1> --dns <your_dns_server_2> &
Solution 2 - Dns
I advise changing the DNS settings of the Docker daemon. You can set the default options for the docker daemon by creating a daemon configuration file at /etc/docker/daemon.json. Set DNS server according to your host machine, e.g. my DNS server is 10.0.0.2:
{"dns": ["10.0.0.2", "8.8.8.8"] }
Then you need just restart docker service:
sudo service docker restart
Step-by-step explanation is available here Fix Docker's networking DNS config
Solution 3 - Dns
The following steps works for me ( for both docker build and docker run command). My linux version is Ubuntu 14.04.
- Identify DNS using following command.
nm-tool | grep DNS
This result DNS:192.168.1.1 in my case
-
Create entry in /etc/default/docker.io. My current entry looks like this
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --dns 192.168.1.1"
-
Restart docker service
sudo service docker.io restart
Solution 4 - Dns
For any Linux distribution working with SystemD (Ubuntu 16, RHEL 7...), the path will be displayed with the following command:
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-06-29 08:10:33 PDT; 2min 34s ago
Docs: https://docs.docker.com
Main PID: 1169 (dockerd)
Tasks: 19
Memory: 85.0M
CPU: 1.779s
CGroup: /system.slice/docker.service
├─1169 /usr/bin/dockerd --dns 172.18.20.11 --dns 172.20.100.15 --dns 8.8.8.8 --dns 8.8.4.4 -H fd://
└─1232 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --met
The path would be /lib/systemd/system/docker.service
. Add the DOCKER_OPTS
values, which can have any of the --dns
, in the line where the daemon is started.
cat /lib/systemd/system/docker.service | grep dns
ExecStart=/usr/bin/dockerd --dns 172.18.20.11 --dns 172.20.100.15 --dns 8.8.8.8 --dns 8.8.4.4 -H fd://
Solution 5 - Dns
Docker (at least >=1.13, probably earlier) on Mac and Windows allow you configure the DNS in Preferences -> Daemon -> Advanced:
The following config sets two corporate DNS servers (use your own values here) with fallback to Google public DNS servers.
Solution 6 - Dns
Specify your DNS to the Docker daemon.
First of all get your DNS address
$ nmcli dev show | grep 'IP4.DNS'
IP4.DNS[1]: 10.0.0.2
Test if the problem is really with the DNS by launching a docker container forcing this new DNS
$ docker run --dns 10.0.0.2 <image_name> <command_name>
If this solves the problem, you can apply this fix for all the docker daemons in the following way
Edit or create a file /etc/docker/daemon.json
Add the following line to this file
{
"dns": ["10.0.0.2", "8.8.8.8"]
}
Restart docker
$ sudo service docker restart
A very nice guide for doing ALL this process can be found here.
https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/
Solution 7 - Dns
Solution without restarting Docker service
It is possible to modify the DNS settings for a single Docker image without affecting other docker build
calls (and without restarting the Docker service) by overriding the resolv.conf
at build time:
FROM ubuntu:18.04
RUN echo "nameserver 123.123.123.123" > /etc/resolv.conf && apt update
Replace the IP 123.123.123.123
with the one which is used within your corporate network (use nmcli dev show | grep 'IP4.DNS'
to get the currently used DNS server).
Downsides:
- This does not affect any other line from the Dockerfile. Hence, you have to prefix every line with the fix, if it depends on DNS resolution
Solution 8 - Dns
On my Ubuntu 16.04 machine, sometimes, Google's DNS do not work for building Docker images.
cat /etc/docker/daemon.json
{"dns": [""8.8.8.8"] }
I have to manually find out my Service Providers DNS using the following command
nmcli device show <interfacename> | grep IP4.DNS
125.22.47.102
and add it to my daemon.json
as show below
cat /etc/docker/daemon.json
{"dns": ["125.22.47.102","8.8.8.8"] }
restart docker
sudo service docker restart
(PS nm-tool is deprecated from Ubuntu 15.04)
Solution 9 - Dns
Updated info September 2021
Inspired by Jason's answer; setting DNS server in the JSON didn't work for me in the current version, but there's now another place to set it:
When you turn on the toggle, the 8.8.8.8 is already there, so I just left it and it works well enough for me in my dev environment. I didn't research it but if wanted, there may be a way to add a list, perhaps separated by commas/semicolons/spaces etc.