Backup/Restore a dockerized PostgreSQL database
DatabasePostgresqlBackupDockerDatabase Problem Overview
I'm trying to backup/restore a PostgreSQL database as is explained on the Docker website, but the data is not restored.
The volumes used by the database image are:
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
and the CMD is:
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
I create the DB container with this command:
docker run -it --name "$DB_CONTAINER_NAME" -d "$DB_IMAGE_NAME"
Then I connect another container to insert some data manually:
docker run -it --rm --link "$DB_CONTAINER_NAME":db "$DB_IMAGE_NAME" sh -c 'exec bash'
psql -d test -h $DB_PORT_5432_TCP_ADDR
# insert some data in the db
<CTRL-D>
<CTRL-D>
The tar archive is then created:
$ sudo docker run --volumes-from "$DB_CONTAINER_NAME" --rm -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /etc/postgresql /var/log/postgresql /var/lib/postgresql
Now I remove the container used for the db and create another one, with the same name, and try to restore the data inserted before:
$ sudo docker run --volumes-from "$DB_CONTAINER_NAME" --rm -v $(pwd):/backup ubuntu tar xvf /backup/backup.tar
But the tables are empty, why is the data not properly restored ?
Database Solutions
Solution 1 - Database
Backup your databases
docker exec -t your-db-container pg_dumpall -c -U postgres > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
Restore your databases
cat your_dump.sql | docker exec -i your-db-container psql -U postgres
Solution 2 - Database
I think you can also use a postgres backup container which would backup your databases within a given time duration.
pgbackups:
container_name: Backup
image: prodrigestivill/postgres-backup-local
restart: always
volumes:
- ./backup:/backups
links:
- db:db
depends_on:
- db
environment:
- POSTGRES_HOST=db
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_EXTRA_OPTS=-Z9 --schema=public --blobs
- SCHEDULE=@every 0h30m00s
- BACKUP_KEEP_DAYS=7
- BACKUP_KEEP_WEEKS=4
- BACKUP_KEEP_MONTHS=6
- HEALTHCHECK_PORT=81
Solution 3 - Database
Backup Database
generate sql:
docker exec -t your-db-container pg_dumpall -c -U your-db-user > dump_$(date +%Y-%m-%d_%H_%M_%S).sql
to reduce the size of the sql you can generate a compress:
docker exec -t your-db-container pg_dumpall -c -U your-db-user | gzip > ./dump_$(date +"%Y-%m-%d_%H_%M_%S").gz
Restore Database
cat your_dump.sql | docker exec -i your-db-container psql -U your-db-user -d your-db-name
to restore a compressed sql:
gunzip < your_dump.sql.gz | docker exec -i your-db-container psql -U your-db-user -d your-db-name
PD: this is a compilation of what worked for me, and what I got from here and elsewhere. I am beginning to make contributions, any feedback will be appreciated.
Solution 4 - Database
cat db.dump | docker exec ...
way didn't work for my dump (~2Gb). It took few hours and ended up with out-of-memory error.
Instead, I cp'ed dump into container and pg_restore'ed it from within.
Assuming that container id is CONTAINER_ID
and db name is DB_NAME
:
# copy dump into container
docker cp local/path/to/db.dump CONTAINER_ID:/db.dump
# shell into container
docker exec -it CONTAINER_ID bash
# restore it from within
pg_restore -U postgres -d DB_NAME --no-owner -1 /db.dump
Solution 5 - Database
Okay, I've figured this out. Postgresql does not detect changes to the folder /var/lib/postgresql once it's launched, at least not the kind of changes I want it do detect.
The first solution is to start a container with bash instead of starting the postgres server directly, restore the data, and then start the server manually.
The second solution is to use a data container. I didn't get the point of it before, now I do. This data container allows to restore the data before starting the postgres container. Thus, when the postgres server starts, the data are already there.
Solution 6 - Database
The below command can be used to take dump from docker postgress container
docker exec -t <postgres-container-name> pg_dump --no-owner -U <db-username> <db-name> > file-name-to-backup-to.sql
Solution 7 - Database
Another approach (based on docker-postgresql-workflow)
Local running database (not in docker, but same approach would work) to export:
pg_dump -F c -h localhost mydb -U postgres export.dmp
Container database to import:
docker run -d -v /local/path/to/postgres:/var/lib/postgresql/data postgres #ex runs container as `CONTAINERNAME` #find via `docker ps`
docker run -it --link CONTAINERNAME:postgres --volume $PWD/:/tmp/ postgres bash -c 'exec pg_restore -h postgres -U postgres -d mydb -F c /tmp/sonar.dmp'
Solution 8 - Database
The top answer didn't work for me. I kept getting this error:
psql: error: FATAL: Peer authentication failed for user "postgres"
To get it to work I had to specify a user for the docker container:
Backup
docker exec -t --user postgres your-db-container pg_dumpall -c -U postgres > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
Restore
cat your_dump.sql | docker exec -i --user postgres your-db-container psql -U postgres
Solution 9 - Database
I had this issue while trying to use a db_dump to restore a db. I normally use dbeaver to restore- however received a psql dump, so had to figure out a method to restore using the docker container.
The methodology recommended by Forth and edited by Soviut worked for me:
cat your_dump.sql | docker exec -i your-db-container psql -U postgres -d dbname
(since this was a single db dump and not multiple db's i included the name)
However, in order to get this to work, I had to also go into the virtualenv that the docker container and project were in. This eluded me for a bit before figuring it out- as I was receiving the following docker error.
read unix @->/var/run/docker.sock: read: connection reset by peer
This can be caused by the file /var/lib/docker/network/files/local-kv.db .I don't know the accuracy of this statement: but I believe I was seeing this as I do not user docker locally, so therefore did not have this file, which it was looking for, using Forth's answer.
I then navigated to correct directory (with the project) activated the virtualenv and then ran the accepted answer. Boom, worked like a top. Hope this helps someone else out there!
Solution 10 - Database
dksnap
(https://github.com/kelda/dksnap) automates the process of running pg_dumpall
and loading the dump via /docker-entrypoint-initdb.d
.
It shows you a list of running containers, and you pick which one you want to backup. The resulting artifact is a regular Docker image, so you can then docker run
it, or share it by pushing it to a Docker registry.
(disclaimer: I'm a maintainer on the project)
Solution 11 - Database
This is the command worked for me.
cat your_dump.sql | sudo docker exec -i {docker-postgres-container} psql -U {user} -d {database_name}
for example
cat table_backup.sql | docker exec -i 03b366004090 psql -U postgres -d postgres
Reference: Solution given by GMartinez-Sisti in this discussion. https://gist.github.com/gilyes/525cc0f471aafae18c3857c27519fc4b
Solution 12 - Database
I would like to add the official docker documentation for backups and restores. This applies to all kinds of data within a volume, not just postegres.
> > > ## Backup a container > > Create a new container named dbstore: > > $ docker run -v /dbdata --name dbstore ubuntu /bin/bash > > Then in the next command, we: > > - Launch a new container and mount the volume from the dbstore container > - Mount a local host directory as /backup > - Pass a command that tars the contents of the dbdata volume to a backup.tar file inside our /backup directory. > > $ docker run --rm --volumes-from dbstore -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata > > When the command completes and the container stops, we are left with a backup of our dbdata volume. > > ## Restore container from backup > > With the backup just created, you can restore it to the same container, or another that you made elsewhere. > > For example, create a new container named dbstore2: > > $ docker run -v /dbdata --name dbstore2 ubuntu /bin/bash > > Then un-tar the backup file in the new container`s data volume: > > $ docker run --rm --volumes-from dbstore2 -v $(pwd):/backup ubuntu bash -c "cd /dbdata && tar xvf /backup/backup.tar --strip 1" > > You can use the techniques above to automate backup, migration and restore testing using your preferred tools. > >
Solution 13 - Database
Using a File System Level Backup on Docker Volumes
Example Docker Compose
version: "3.9"
services:
db:
container_name: pg_container
image: platerecognizer/parkpow-postgres
# restart: always
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
POSTGRES_USER: admin
POSTGRES_PASSWORD: admin
POSTGRES_DB: admin
volumes:
postgres_data:
Backup Postgresql Volume
docker run --rm \
--user root \
--volumes-from pg_container \
-v /tmp/db-bkp:/backup \
ubuntu tar cvf /backup/db.tar /var/lib/postgresql/data
Then copy /tmp/db-bkp
to second host
Restore Postgresql Volume
docker run --rm \
--user root \
--volumes-from pg_container \
-v /tmp/db-bkp:/backup \
ubuntu bash -c "cd /var && tar xvf /backup/db.tar --strip 1"
Solution 14 - Database
Solution for docker-compose
users:
- At First run the docker-compose file by any on of following commands:
$ docker-compose -f loca.yml up
ORdocker-compose -f loca.yml up -d
- For taking backup:
$ docker-compose -f local.yml exec postgres backup
- To see list of backups inside container:
$ docker-compose -f local.yml exec postgres backups
- Open another terminal and run following command:
$ docker ps
- Look for the CONTAINER ID of
postgres
image and copy the ID. Let's assume the CONTAINER ID is: ba78c0f9bcee - Now to bring that backup into your local file system, run the following command:
$ docker cp ba78c0f9bcee:/backups ./local_backupfolder
Hope this will help someone who was lost just like me..
N.B: The full details of this solution can be found here.