How to view docker-compose healthcheck logs?
LoggingDockerDocker ComposeHealth MonitoringLogging Problem Overview
Inside my docker-compose.yml, I have the following service healthcheck section. I want to know if MariaDB is actually ready to handle queries. A service named cmd is configured to depend on condition: service_healthy.
db:
image: mariadb:10
environment:
MYSQL_RANDOM_ROOT_PASSWORD: 1
MYSQL_USER: user
MYSQL_PASSWORD: password
MYSQL_DATABASE: database
healthcheck:
test: ["CMD", "mysql", "--user=user", "--password=password", "--execute='SELECT 1'", "--host=127.0.0.1", "--port=3306"]
interval: 1s
retries: 30
This healthcheck does not work, shows that the service is unhealthy.
How do I check the output of the test CMD?
Logging Solutions
Solution 1 - Logging
You can use:
docker inspect --format "{{json .State.Health }}" <container name> | jq
Output:
{
"Status": "unhealthy",
"FailingStreak": 63,
"Log": [
{
"Start": "2017-03-11T20:49:19.668895201+03:30",
"End": "2017-03-11T20:49:19.735722044+03:30",
"ExitCode": 1,
"Output": "ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''SELECT 1'' at line 1\n"
}
]
}
And look for the output section.
To get the Output only:
docker inspect --format "{{json .State.Health }}" mariadb_db_1 | jq '.Log[].Output'
For swarm mode use the folling format (thanks for pointing it out @shotgunner):
{{json.Spec.TaskTemplate.ContainerSpec.Healthcheck}}
Feel free to swap jq for whatever tool you use for json pretty print.
Solution 2 - Logging
docker-compose ps will indicate the status of each service, including its health if healthcheck is defined. It's good for a basic overview.
% docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------------------------------------------------
remix-theme-editor_analytics_1 /bin/sh -c /analytics/run. ... Up
remix-theme-editor_base_1 /bin/bash Exit 0
remix-theme-editor_flower_1 /entrypoint --environment ... Exit 137
remix-theme-editor_frontend_1 /bin/sh -c perl -p -i -e ' ... Exit 137
remix-theme-editor_js-app_1 npm run Exit 0
remix-theme-editor_mq_1 docker-entrypoint.sh rabbi ... Up (healthy) 15671/tcp, 15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp
remix-theme-editor_mysql-migration_1 /entrypoint_mysql-migratio ... Exit 0
remix-theme-editor_mysql_1 /bin/sh -c /entrypoint_wra ... Up (health: starting) 127.0.0.2:3308->3306/tcp
remix-theme-editor_page-renderer_1 npm run start:watch Up
remix-theme-editor_python-app_1 /entrypoint Exit 2
remix-theme-editor_redis_1 docker-entrypoint.sh /bin/ ... Up (health: starting) 6379/tcp
remix-theme-editor_scheduler_1 /entrypoint --environment ... Exit 137
remix-theme-editor_socket_1 /entrypoint --environment ... Exit 1
remix-theme-editor_static-builder_1 npm run watch Up
remix-theme-editor_static-http_1 nginx -g daemon off; Up 127.0.0.2:6544->443/tcp, 80/tcp
remix-theme-editor_web_1 /entrypoint --environment ... Exit 1
remix-theme-editor_worker_1 /entrypoint --environment ... Exit 1
remix-theme-editor_worker_screenshots_1 /entrypoint --environment ... Exit 1
If you want more details, use docker inspect in conjuction with docker ps -q <service-name>.
% docker inspect --format "{{json .State.Health }}" $(docker-compose ps -q mq) | jq
{
"Status": "starting",
"FailingStreak": 48,
"Log": [
{
"Start": "2018-10-03T00:40:18.671527745-05:00",
"End": "2018-10-03T00:40:18.71729051-05:00",
"ExitCode": -1,
"Output": "OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused \"exec: \\\"nc\\\": executable file not found in $PATH\": unknown"
},
...
You can always debug the healthcheck yourself by simply executing your healthcheck code yourself. For example:
% docker exec -it $(docker-compose ps -q socket) nc -w2 127.0.0.1 5672
(UNKNOWN) [127.0.0.1] 5672 (?) : Connection refused
You can also do the same in shell:
% docker exec -it $(docker-compose ps -q socket) bash
root@b5da5207d344:~/src# nc -w2 127.0.0.1 5672
(UNKNOWN) [127.0.0.1] 5672 (?) : Connection refused
root@b5da5207d344:~/src# echo $?
1
Finally, you can simply use docker-compose up in the first terminal window, and docker-compose logs -f in another. This will display all logs from docker-compose-managed containers.
Solution 3 - Logging
In Swarm Mode:
- First use
docker service ps service_namein the manager to find the failed task id and the corresponding node.
manager$ docker service ps service_name
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
liwww3qzg9dz service_name.1 image_name:1.3 s-3 Running Running 27 seconds ago
hcgxmwk2efj0 \_ service_name.1 image_name:1.3 s-3 Shutdown Failed about a minute ago "task: non-zero exit (137): do…"
In this example, hcgxmwk2efj0 is the task id and s-3 is the node name.
- Then use
docker inspect --format "{{json .Status.ContainerStatus.ContainerID }}" task_idin the manager to get the container id.
manager$ docker inspect --format "{{json .Status.ContainerStatus.ContainerID }}" hcgxmwk2efj0
"412b09d5244047b31471248fd9a0807e5ea42406fb8f5b1701df2244933e30c8"
- Then ssh to that node, and use the command
docker inspect --format "{{json .State.Health }}" container_id | jqto get the log of the healthcheck. (| jqis not necessary in this command)
s-3$ docker inspect --format "{{json .State.Health }}" 412b09d5244047b31471248fd9a0807e5ea42406fb8f5b1701df2244933e30c8 | jq
{
"Status": "unhealthy",
"FailingStreak": 3,
"Log": [
{
"Start": "2021-09-07T06:10:05.233163051Z",
"End": "2021-09-07T06:10:07.585487343Z",
"ExitCode": 0,
"Output": "... log 1 ..."
},
{
"Start": "2021-09-07T06:10:37.644936244Z",
"End": "2021-09-07T06:10:39.881196276Z",
"ExitCode": 0,
"Output": "... log 2 ..."
},
{
"Start": "2021-09-07T06:11:10.16172012Z",
"End": "2021-09-07T06:11:25.161912411Z",
"ExitCode": -1,
"Output": "Health check exceeded timeout (15s)"
},
{
"Start": "2021-09-07T06:11:55.297395088Z",
"End": "2021-09-07T06:12:10.302928565Z",
"ExitCode": -1,
"Output": "Health check exceeded timeout (15s)"
},
{
"Start": "2021-09-07T06:12:40.371234778Z",
"End": "2021-09-07T06:12:55.371393914Z",
"ExitCode": -1,
"Output": "Health check exceeded timeout (15s)"
}
]
}