Docker Machine: No space left on device
DockerDocker ComposeDocker MachineDocker Problem Overview
I'm trying to set up Docker Machine with Docker Compose.
Scenario 1 (without Docker Machine)
If I run docker-compose up -d without Docker Machine, it creates my 3 linked containers as intented (nginx+mongodb+nodejs).
Scenario 2 (with Docker Machine)
Then I create a VM using Docker Machine and tell Docker to talk to that machine with eval $(docker-machine env streambacker-dev).
At this point, if I ssh to my docker machine and run df -h, I get:
If I then run docker-compose up -d, I get a "no space left on device" error while downloading the last container.
"tmpfs" seems to be indeed a bit full after that:
Checking the --virtualbox-disk-size option shows that it defaults to 20000 MB, which I think is what we can see as "/dev/sda1" on both pictures. So why are containers filling up "tmpfs" n and what exactly is "tmpfs"? Is is a temporary download directory? How can I create more space for my containers?
Thanks!
For information, I'm using Docker Machine 0.4.0-rc2 and Docker Compose 1.3.2.
Docker Solutions
Solution 1 - Docker
I had the same error ([ERROR] InnoDB: Error number 28 means 'No space left on device') and solve it this way:
1 . Delete the orphaned volumes in Docker, you can use the built-in docker volume command. The built-in command also deletes any directory in /var/lib/docker/volumes that is not a volume so make sure you didn't put anything in there you want to save.
Warning be very careful with this if you have some data you want to keep
Cleanup:
$ docker volume rm $(docker volume ls -qf dangling=true)
Additional commands:
List dangling volumes:
$ docker volume ls -qf dangling=true
List all volumes:
$ docker volume ls
2 . Also consider removing all the unused Images.
First get rid of the <none> images (those are sometimes generated while building an image and if for any reason the image building was interrupted, they stay there).
here's a nice script I use to remove them
docker rmi $(docker images | grep "^<none>" | awk '{print $3}')
Then if you are using Docker Compose to build Images locally for every project. You will end up with a lot of images usually named like your folder (example if your project folder named Hello, you will find images name Hello_blablabla). so also consider removing all these images
you can edit the above script to remove them or remove them manually with
docker rmi {image-name}
Solution 2 - Docker
Like said above, the tmpfs has nothing to do with --virtualbox-disk-size. It seems like boot2docker mounts tmpfs into memory, so you need to dedicate more memory to your virtualbox vm. You can do it by specifying the --virtualbox-memory parameter.
--virtualbox-memory "1024"
Size of memory for host in MB [$VIRTUALBOX_MEMORY_SIZE]
Defaults:
$ docker-machine create --driver virtualbox testA
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testA
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testA:~$ df -h /
Filesystem Size Used Available Use% Mounted on
tmpfs 896.6M 112.7M 783.9M 13% /
With --virtualbox-memory set to 8096
$ docker-machine create --driver virtualbox --virtualbox-memory 8096 testB
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testB
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testB:~$ df -h /
Filesystem Size Used Available Use% Mounted on
tmpfs 6.9G 112.4M 6.8G 2% /
Solution 3 - Docker
If you are using Docker Community Edition:
docker system prune --volumes
If you are using boot2docker (docker-machine) clear the volumes that are orphaned:
docker volume rm $(docker volume ls -qf dangling=true)
Clear unused images:
docker rmi $(docker images -q -f "dangling=true")
Solution 4 - Docker
A. REMOVE UNUSED IMAGES
using the docker rm or docker rmi commands you can remove the images that you don't need. Actually exist an image that helps in this task (martin/docker-cleanup-volumes). The basis is to start selectig from your images and containers list:
> docker ps -a -s
B. MODIFY THE DOCKER JSON DESCRIPTOR
it's mentioned in some forums. The idea is to increment the descriptor located in ~/.docker/machine/machines/default/config.json . The param seems to be DiskSize but i don't know if it works in other OSs (not in windows).
C. LINUX RESIZE:
in Windows OS, docker machine or boot2docker is in fact a virtualbox vm, then you can follow the procedure to resize the disk. Take care to backup the files. The general procedure is to make a resize in virtualbox and then use an utilitary called gpartd to modify the space perceived by linux in its partitions. There are some links to do this procedure referenced below:
D. RECREATE THE DOCKER-MACHINE / BOOT2DOCKER
The idea is recreate the default docker-machine. The following commands can illustrate you. Note that as you are re-creating the boot2docker, you will lost the previous downloaded docker images.
> docker-machine rm default > > docker-machine create --driver virtualbox --virtualbox-disk-size "100100" default > > docker-machine env default
then you can go to virtual box and see the boot2docker space with the command "df -h"
Solution 5 - Docker
On docker osx / I was able to press a button [Move Disk Image] and it successfully moved the Docker.qcow2 (presumably containing containers / images)
[![enter image description here][1]][1] [1]: https://i.stack.imgur.com/zUmNT.png
initially - when machines started - I was still getting a No space left on device error but it resolved shortly after.
Solution 6 - Docker
I ran into this problem and could not add additional space with the docker UI for mac, I installed docker with homebrew and ran the following command when creating my machine:
docker-machine create --driver virtualbox --virtualbox-memory "2048" --virtualbox-disk-size "40000" default
this adds double the space for memory and disk size to the virtualbox that I had before and you can add the settings size here that you need as you see fit
Solution 7 - Docker
Run docker system df to see what is taking up space on your docker
Then run:
docker system prune --all --force
to remove all hidden and unused containers.
docker system prune does not remove all unused containers.
Solution 8 - Docker
Open file /lib/systemd/system/docker.service with your favorite text editor and replace the following line where /new/path/docker is a location of your new chosen docker directory:
FROM:
ExecStart=/usr/bin/docker daemon -H fd://
TO:
ExecStart=/usr/bin/docker daemon -g /new/path/docker -H fd://
When ready stop docker service:
# systemctl stop docker
check that docker service is stopped, the following command should yield no result
# ps aux | grep -i docker | grep -v grep
reload daemon
# systemctl daemon-reload
Once this is done create a new directory you specified above and optionally rsync current docker data to a new directory:
# mkdir /new/path/docker
# rsync -aqxP /var/lib/docker/ /new/path/docker
# systemctl start docker
confirms that docker runs into new directory
# ps aux | grep -i docker | grep -v grep