How to know the reason why a docker container exits?

DockerContainersExit

Docker Problem Overview


I have a Docker container running in a host of 1G RAM (there are also other containers running in the same host). The application in this Docker container will decode some images, which may consume memory a lot.

From time to time, this container will exit. I doubt it is due to out of memory but not very sure. I need a method to find the root cause. So is there any way to know what happened for this container's death?

Docker Solutions


Solution 1 - Docker

Others have mentioned docker logs $container_id to view the output of the application. This would always be my first thing to check.

Next, you can run a docker inspect $container_id to view details on the state, e.g.:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 2,
        "Error": "",
        "StartedAt": "2016-06-28T21:26:53.477229071Z",
        "FinishedAt": "2016-06-28T21:26:53.478066987Z"
    },

The important line there is "OOMKilled" which will be true if you exceed the container memory limits and Docker kills your app. You may also want to lookup the exit code to see if it identifies a cause for the exit by your app.

Note, this only indicates if docker itself kills your process, and requires that you have set a memory limit on your container. Outside of docker, the Linux kernel can kill your process if the host itself runs out of memory. Linux often writes to a log in /var/log when this happens. With Docker Desktop on Windows and Mac, you can adjust the memory allocated to the embedded Linux VM in the docker settings.

Solution 2 - Docker

You can find out whether the process inside the container was OOMkilled by reading the logs. OOMkills are initiated by the kernel so every time it happens there's a bunch of lines in /var/log/kern.log, for example:

python invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
oom_kill_process+0x22e/0x450
Memory cgroup out of memory: Kill process 31204 (python) score 1994 or sacrifice child
Killed process 31204 (python) total-vm:7350860kB, anon-rss:4182920kB, file-rss:2356kB, shmem-rss:0kB

Solution 3 - Docker

While the accepted answer is the best option, sometimes it can be useful to inspect from the host the content of the journal too (on linux).

You can do that by typing:

sudo journalctl -u docker

or tailing it

sudo journalctl -u docker -f

or piping the output to less if it is too long for your terminal buffer

sudo journalctl -xn -u docker | less

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionLi BinView Question on Stackoverflow
Solution 1 - DockerBMitchView Answer on Stackoverflow
Solution 2 - Dockerstyrofoam flyView Answer on Stackoverflow
Solution 3 - DockerRoberto ManfredaView Answer on Stackoverflow