how to clean up docker overlay directory?

I'm running docker via CoreOS and AWS's ECS. I had a failing image that got restarted many times, and the containers are still around- they filled my drive partition. Specifically, /var/lib/docker/overlay/ contains a large number of files/directories.

I know that docker-cleanup-volumes is a thing, but it cleans the /volumes directory, not the /overlay directory.

docker ps -a shows over 250 start attempts on my bad docker container. They aren't running, though.

Aside from rm -rf /var/lib/docker/overlay/*, how can I/should I clean this up?

From our side we used:

sudo docker system prune -a -f

Which saved me 3Go!

We also used the famous commands:

sudo docker rm -v $(sudo docker ps -a -q -f status=exited)
sudo docker rmi -f  $(sudo docker images -f "dangling=true" -q)
docker volume ls -qf dangling=true | xargs -r docker volume rm

We put that on cron to manage a little bit more efficently our disk space.

Reference: https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604/4

Here's the hacky way I'm doing this right now. I'm not going to accept it as an answer because I'm hoping there's a better way.

# delete old docker processes
docker rm `docker ps -a | grep Exited | awk '{print $1 }'`
  ignore_errors: true

# delete old images. will complain about still-in-use images.
docker rmi `docker images -aq`

I have added this to bashrc in my dev environment, and gotten used to running it every day or so.

function cleanup_docker() {
  docker ps -f status=exited -q | xargs -r docker rm
  docker images -f dangling=true -q | xargs -r docker rmi
}

In some cases, the following script can free up more space, as it will try to remove all images, and just fail silently:

function cleanup_docker_aggressive() {
  for i in $(docker images --no-trunc -q | sort -u)
  do
    docker rmi $i 2> /dev/null
  done
}

Sadly, they're not significantly cleaner than your solution.

EDIT: Starting with Docker 1.13, you can use docker system:

docker system df    # to check what is using space
docker system prune # cleans up also networks, build cache, etc

EDIT: Starting with Docker 2017.09, you can also use container and image

docker container prune
docker image prune -a

the latter you can use with fancy filters like --filter "until=24h"

We just started having this problem, and btafarelo's answer got me part of the way, or at least made me feel better about removing the sha256 entries.

System info: ec2 instances running CoreOS 1.12 behind an ELB

• Drain the docker instance from the ELB

• Shutdown docker

    systemctl stop docker
    rm -rf /var/lib/docker/overlay/*
  

• Execute the results of the commands

    for d in $(find /var/lib/docker/image/overlay -type d -name '*sha256*'); do echo rm -rf $d/* ; done
    
  

• reboot (easiest way to bring everything back up)

This recovered about 25% of the disk after the services restarted with no ill side affects.

docker ps

• --quiet

• --all

• --filter status=exited

---

docker rm

• --force

---

docker images

• --quiet
• --all
• --filter dangling=true

---

docker rmi

• -- force

---

Your hacky way is fine.

docker rm `docker ps -a | grep Exited | awk '{print $1 }'`

My hacky way is

docker rm $(docker ps --all | awk '/ago/{print $1}')

A slightly cleaner way is to run docker ps with the --quiet (-q) flag to get just the id number and --filter status=exited to --filter just the exited ones.

docker rm $(docker ps --filter status=exited --quiet) # remove stopped docker processes

or to run docker rm with the --force (-f) flag and docker ps with the --all (-a) flag to shut down even the running ones

docker rm --force $(docker ps --all --quiet) # remove all docker processes

---

What's probably taking up all that disk space after several failed builds is the images. To conserve disk space on the docker host, periodically remove unused docker images with

docker rmi $(docker images --filter dangling=true --quiet) # clean dangling docker images

or to get more aggressive, you can --force (-f) it to clean up --all (-a) images

docker rmi --force $(docker images --all --quiet) # clean all possible docker images

---

@analytik 's way of putting it into a .bashrc function seems like a practical idea

function cleanup_docker() {
  docker rm --force $(docker ps --all --quiet) # remove all docker processes
  docker rmi $(docker images --filter dangling=true --quiet) # clean dangling docker images
}

and if you're in the habit of generating lots of docker images that you don't need, add it to .bash_logout

---

Docker garbage collection can be done in an easy way using another docker container https://github.com/spotify/docker-gc

You could make it run as a cron using https://github.com/flaccid/docker-docker-gc-crond

Here is a working option:

docker rm -f $(docker ps -a |awk 'NR>1&&/Exited/{print $1}')

I followed these simple steps

Step 1: df -h [checked the memory used, to be sure, memory is used by overlay folder].

Step 2: sudo docker system prune [this cmd removes all unused containers/images/networks]

Step 3: sudo docker image prune -a [for any dangling images, if present]

Step 4: df -h [to be sure, overlay data is removed].

here is resolution to clean docker overlay directory from https://lebkowski.name/docker-volumes/

docker images --no-trunc | grep '<none>' | awk '{ print $3 }' | xargs -r docker rmi

docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v

for Docker < 1.9 :

find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d | grep -vFf <(docker ps -aq | xargs docker inspect | jq -r '.[]|.Mounts|.[]|.Name|select(.)')

Or for Docker >=1.9 :

docker volume ls -qf dangling=true | xargs -r docker volume rm

Please be aware that docker prune commands do not clean /var/lib/docker/overlay2 directory. It is also not advised to remove only the overlay directory as it may impact existing containers. I have searched for a lot of articles but couldn't find any solution to clean overlay directory other than cleaning the entire docker state:

# Please understand that this will restart the docker engine in a completely empty state 
# i.e. you will lose all images, containers, volumes, networks, swarm state, etc. 
# You can obviously first take the backup of the directories that you want to keep and copy the contents back after restarting docker service.

service stop docker
rm -rf /var/lib/docker
service start docker

The last command brings the docker service back up with all the folders inside /var/lib/docker again.

BE CAREFUL, this solutions seems remove other important stuffs.

You need inspect your images/containers and remove the folders in /opt/docker/overlay that wasn't found in inspect's execution

docker inspect $(docker ps -qa) | grep -oE '[a-f0-9]{64}' >> inspect-hashs.txt

docker inspect $(docker images -qa) | grep -oE '[a-f0-9]{64}' >> inspect-hashs.txt

sudo ls -l /var/lib/docker/overlay > overlays.txt

diff -u  inspect-hashs.txt overlays.txt | grep -E '^\+' | grep -oE '[a-f0-9]{64}' | xargs sudo rm -rf /opt/docker/overlay/