How to persist data in Prometheus running in a Docker container?

DockerPersistencePrometheus

Docker Problem Overview


I'm developing something that needs Prometheus to persist its data between restarts. Having followed the instructions

$ docker volume create a-new-volume
$ docker run \
    --publish 9090:9090 \
    --volume a-new-volume:/prometheus-data \
    --volume "$(pwd)"/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

I have a valid prometheus.yml in the right directory on the host machine and it's being read by Prometheus from within the container. I'm just scraping a couple of HTTP endpoints for testing purposes at the moment.

But when I restart the container it's empty, no data from the previous run. What am I missing from my docker run ... command to persist the data into the a-new-volume volume?

Docker Solutions


Solution 1 - Docker

Use the default data dir, which is /prometheus. To do that, use this line instead of what you have in your command:

...
--volume a-new-volume:/prometheus \
...

Found here: https://github.com/prometheus/prometheus/blob/master/Dockerfile

Surprisingly is not mentioned in the image docs

Solution 2 - Docker

I had the same issue a today, but I was using a docker composer file. So wrapping up all what was in comments of other answers and what worked for me. In case setting up the Prometheus docker via yaml compose file...

First create a folder for the volume on the host machine, e.g.:

$ mkdir /tmp/prometheus

Then change the folder owner to nobody, like (use sudo if needed):

$ chown 65534:65534 /tmp/prometheus

Then add volume to the yaml configuration file:

prometheus:
  image: prom/prometheus
  container_name: prometheus
  ports:
    - 9090:9090
  volumes:
    - /tmp/prometheus:/prometheus
    - ./prometheus.yml:/etc/prometheus/prometheus.yml

That should do it.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionMatthewView Question on Stackoverflow
Solution 1 - DockerRobertView Answer on Stackoverflow
Solution 2 - DockerAntonio PratesView Answer on Stackoverflow