How to use environment variables in docker compose

Docker Problem Overview

I would like to be able to use env variables inside docker-compose.yml, with values passed in at the time of docker-compose up. This is the example.

I am doing this today with basic docker run command, which is wrapped around my own script. Is there a way to achieve it with compose, without any such bash wrappers?

proxy:
  hostname: $hostname
  volumes:
    - /mnt/data/logs/$hostname:/logs
    - /mnt/data/$hostname:/data

Docker Solutions

Solution 1 - Docker

#The DOCKER solution: It looks like docker-compose 1.5+ has enabled variables substitution: https://github.com/docker/compose/releases

The latest Docker Compose allows you to access environment variables from your compose file. So you can source your environment variables, then run Compose like so:

set -a
source .my-env
docker-compose up -d

Then you can reference the variables in docker-compose.yml using ${VARIABLE}, like so:

db:
  image: "postgres:${POSTGRES_VERSION}"

And here is more info from the docs, taken here: https://docs.docker.com/compose/compose-file/#variable-substitution

> When you run docker-compose up with this configuration, Compose looks > for the POSTGRES_VERSION environment variable in the shell and > substitutes its value in. For this example, Compose resolves the image > to postgres:9.3 before running the configuration. > > If an environment variable is not set, Compose substitutes with an > empty string. In the example above, if POSTGRES_VERSION is not set, > the value for the image option is postgres:. > > Both $VARIABLE and ${VARIABLE} syntax are supported. Extended > shell-style features, such as ${VARIABLE-default} and > ${VARIABLE/foo/bar}, are not supported. > > If you need to put a literal dollar sign in a configuration value, use > a double dollar sign ($$).

And I believe this feature was added in this pull request: https://github.com/docker/compose/pull/1765

#The BASH solution:

I notice folks have issues with Docker's environment variables support. Instead of dealing with environment variables in Docker, let's go back to basics, like bash! Here is a more flexible method using a bash script and a .env file.

An example .env file:

EXAMPLE_URL=http://example.com
# Note that the variable below is commented out and will not be used:
# EXAMPLE_URL=http://example2.com 
SECRET_KEY=ABDFWEDFSADFWWEFSFSDFM

# You can even define the compose file in an env variable like so:
COMPOSE_CONFIG=my-compose-file.yml
# You can define other compose files, and just comment them out
# when not needed:
# COMPOSE_CONFIG=another-compose-file.yml

then run this bash script in the same directory, which should deploy everything properly:

#!/bin/bash

docker rm -f `docker ps -aq -f name=myproject_*`
set -a
source .env
cat ${COMPOSE_CONFIG} | envsubst | docker-compose -f - -p "myproject" up -d

Just reference your env variables in your compose file with the usual bash syntax (ie ${SECRET_KEY} to insert the SECRET_KEY from the .env file).

Note the COMPOSE_CONFIG is defined in my .env file and used in my bash script, but you can easily just replace {$COMPOSE_CONFIG} with the my-compose-file.yml in the bash script.

Also note that I labeled this deployment by naming all of my containers with the "myproject" prefix. You can use any name you want, but it helps identify your containers so you can easily reference them later. Assuming that your containers are stateless, as they should be, this script will quickly remove and redeploy your containers according to your .env file params and your compose YAML file.

Update Since this answer seems pretty popular, I wrote a blog post that describes my Docker deployment workflow in more depth: http://lukeswart.net/2016/03/lets-deploy-part-1/ This might be helpful when you add more complexity to a deployment configuration, like nginx configs, LetsEncrypt certs, and linked containers.

Solution 2 - Docker

It seems that docker-compose has native support now for default environment variables in file.

all you need to do is declare your variables in a file named .env and they will be available in docker-compose.yml.

For example, for .env file with contents:

MY_SECRET_KEY=SOME_SECRET
IMAGE_NAME=docker_image

You could access your variable inside docker-compose.yml or forward them into the container:

my-service:
  image: ${IMAGE_NAME}
  environment:
    MY_SECRET_KEY: ${MY_SECRET_KEY}

Solution 3 - Docker

Create a template.yml, which is your docker-compose.yml with environment variable.
Suppose your environment variables are in a file 'env.sh'
Put the below piece of code in a sh file and run it.

> source env.sh; >rm -rf docker-compose.yml; > envsubst < "template.yml" > "docker-compose.yml";

A new file docker-compose.yml will be generated with the correct values of environment variables.

Sample template.yml file:

oracledb:
        image: ${ORACLE_DB_IMAGE}
        privileged: true
        cpuset: "0"
        ports:
                - "${ORACLE_DB_PORT}:${ORACLE_DB_PORT}"
        command: /bin/sh -c "chmod 777 /tmp/start; /tmp/start"
        container_name: ${ORACLE_DB_CONTAINER_NAME}

Sample env.sh file:

#!/bin/bash 
export ORACLE_DB_IMAGE=<image-name> 
export ORACLE_DB_PORT=<port to be exposed> 
export ORACLE_DB_CONTAINER_NAME=ORACLE_DB_SERVER

Solution 4 - Docker

The best way is to specify environment variables outside the docker-compose.yml file. You can use env_file setting, and define your environment file within the same line. Then doing a docker-compose up again should recreate the containers with the new environment variables.

Here is how my docker-compose.yml looks like:

services:
  web:
    env_file: variables.env

> Note: docker-compose expects each line in an env file to be in VAR=VAL format. Avoid using export inside the .env file. Also, the .env file should be placed in the folder where the docker-compose command is executed.

Solution 5 - Docker

The following is applicable for docker-compose 3.x Set environment variables inside the container

> method - 1 Straight method

web:
  environment:
    - DEBUG=1
      POSTGRES_PASSWORD: 'postgres'
      POSTGRES_USER: 'postgres'

> method - 2 The “.env” file

Create a .env file in the same location as the docker-compose.yml

$ cat .env
TAG=v1.5
POSTGRES_PASSWORD: 'postgres'

and your compose file will be like

$ cat docker-compose.yml
version: '3'
services:
  web:
    image: "webapp:${TAG}"
    postgres_password: "${POSTGRES_PASSWORD}"

source

Solution 6 - Docker

When using environment variables for volumes you need:

create .env file in the same folder which contains docker-compose.yaml file
declare variable in the .env file:
```
HOSTNAME=your_hostname
```

Change $hostname to ${HOSTNAME} at docker-compose.yaml file

proxy:
  hostname: ${HOSTNAME}
  volumes:
    - /mnt/data/logs/${HOSTNAME}:/logs
    - /mnt/data/${HOSTNAME}:/data

Of course you can do that dynamically on each build like:

echo "HOSTNAME=your_hostname" > .env && sudo docker-compose up

Solution 7 - Docker

Since 1.25.4, docker-compose supports the option --env-file that enables you to specify a file containing variables.

Yours should look like this:

hostname=my-host-name

And the command:

docker-compose --env-file /path/to/my-env-file config

Solution 8 - Docker

You cannot ... yet. But this is an alternative, think like a docker-composer.yml generator:

https://gist.github.com/Vad1mo/9ab63f28239515d4dafd

Basically a shell script that will replace your variables. Also you can use Grunt task to build your docker compose file at the end of your CI process.

Solution 9 - Docker

env SOME_VAR="I am some var" OTHER_VAR="I am other var" docker stack deploy -c docker-compose.yml

Use the version 3.6 :

version: "3.6"
services:
  one:
    image: "nginx:alpine"
    environment:
      foo: "bar"
      SOME_VAR:
      baz: "${OTHER_VAR}"
    labels:
      some-label: "$SOME_VAR"
  two:
    image: "nginx:alpine"
    environment:
      hello: "world"
      world: "${SOME_VAR}"
    labels:
      some-label: "$OTHER_VAR"

I got it form this link https://github.com/docker/cli/issues/939

Solution 10 - Docker

To add and env variable you may define an env_file (let's call it var.env) as:

ENV_A=A
ENV_B=B

and add it to the docker compose manifest service. Moreover, you can define env variables directly with environment.

For instance in docker-compose.yaml:

version: '3.8'
services:
  myservice:
    build:
      context: .
      dockerfile: ./docker/Dockerfile.myservice
    image: myself/myservice
    env_file:
     - ./var.env
    environment:
     - VAR_C=C
     - VAR_D=D
    volumes:
     - $HOME/myfolder:/myfolder
    ports:
     - "5000:5000"

Please check here for more/updated information : https://docs.docker.com/compose/environment-variables/

Solution 11 - Docker

I have a simple bash script I created for this it just means running it on your file before use: https://github.com/antonosmond/subber

Basically just create your compose file using double curly braces to denote environment variables e.g:

app:
    build: "{{APP_PATH}}"
ports:
    - "{{APP_PORT_MAP}}"

Anything in double curly braces will be replaced with the environment variable of the same name so if I had the following environment variables set:

APP_PATH=~/my_app/build
APP_PORT_MAP=5000:5000

on running subber docker-compose.yml the resulting file would look like:

app:
    build: "~/my_app/build"
ports:
    - "5000:5000"

Solution 12 - Docker

As far as I know, this is a work-in-progress. They want to do it, but it's not released yet. See 1377 (the "new" 495 that was mentioned by @Andy).

I ended up implementing the "generate .yml as part of CI" approach as proposed by @Thomas.

Solution 13 - Docker

add env to .env file

Such as

VERSION=1.0.0

then save it to `deploy.sh`

INPUTFILE=docker-compose.yml
RESULT_NAME=docker-compose.product.yml
NAME=test

prepare() {
        local inFile=$(pwd)/$INPUTFILE
        local outFile=$(pwd)/$RESULT_NAME
        cp $inFile $outFile
        while read -r line; do
            OLD_IFS="$IFS"
            IFS="="
            pair=($line)
            IFS="$OLD_IFS"
               sed -i -e "s/\${${pair[0]}}/${pair[1]}/g" $outFile
            done <.env
     }
       
deploy() {
        docker stack deploy -c $outFile $NAME
}

        
prepare
deploy

Solution 14 - Docker

To focus solely on the issue of default and mandatory values for environment variables, and as an update to @modulito's answer:

Using default values and enforcing mandatory values within the docker-compose.yml file is now supported (from the docs):

> Both $VARIABLE and ${VARIABLE} syntax are supported. Additionally when using the 2.1 file format, it is possible to provide inline default values using typical shell syntax: > >${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment. ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment.

>Similarly, the following syntax allows you to specify mandatory variables: > >${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.

>Other extended shell-style features, such as ${VARIABLE/foo/bar}, are not supported.

Solution 15 - Docker

Use .env file to define dynamic values in docker-compse.yml. Be it port or any other value.

Sample docker-compose:

testcore.web:
       image: xxxxxxxxxxxxxxx.dkr.ecr.ap-northeast-2.amazonaws.com/testcore:latest
       volumes: 
            - c:/logs:c:/logs
       ports:
            - ${TEST_CORE_PORT}:80
       environment:
            - CONSUL_URL=http://${CONSUL_IP}:8500 
            - HOST=${HOST_ADDRESS}:${TEST_CORE_PORT}

Inside .env file you can define the value of these variables:

CONSUL_IP=172.31.28.151
HOST_ADDRESS=172.31.16.221
TEST_CORE_PORT=10002

Solution 16 - Docker

I ended up using "sed" in my deploy.sh script to accomplish this, though my requirements were slightly different since docker-compose is being called by Terrafom: https://stackoverflow.com/questions/67428084/passing-variables-to-docker-compose-via-a-terraform-script-for-an-azure-app-serv/67436683#67436683

eval "sed -i 's/MY_VERSION/$VERSION/' ../docker-compose.yaml"

cat ../docker-compose.yaml

terraform init 
terraform apply -auto-approve \
    -var "app_version=$VERSION" \
    -var "client_id=$ARM_CLIENT_ID" \
    -var "client_secret=$ARM_CLIENT_SECRET" \
    -var "tenant_id=$ARM_TENANT_ID" \
    -var "subscription_id=$ARM_SUBSCRIPTION_ID"

eval "sed -i 's/$VERSION/MY_VERSION/' ../docker-compose.yaml"

Content Type	Original Author	Original Content on Stackoverflow
Question	Dmitry z	View Question on Stackoverflow
Solution 1 - Docker	modulitos	View Answer on Stackoverflow
Solution 2 - Docker	Doody P	View Answer on Stackoverflow
Solution 3 - Docker	Saurabh Kumar	View Answer on Stackoverflow
Solution 4 - Docker	Jibu James	View Answer on Stackoverflow
Solution 5 - Docker	Gajendra D Ambi	View Answer on Stackoverflow
Solution 6 - Docker	Nerijus Gedrimas 'Nereus Calm'	View Answer on Stackoverflow
Solution 7 - Docker	papillon	View Answer on Stackoverflow
Solution 8 - Docker	Thomas Decaux	View Answer on Stackoverflow
Solution 9 - Docker	user2851100	View Answer on Stackoverflow
Solution 10 - Docker	Galuoises	View Answer on Stackoverflow
Solution 11 - Docker	Anton	View Answer on Stackoverflow
Solution 12 - Docker	anroots	View Answer on Stackoverflow
Solution 13 - Docker	foxundermon	View Answer on Stackoverflow
Solution 14 - Docker	Joey Baruch	View Answer on Stackoverflow
Solution 15 - Docker	sorabzone	View Answer on Stackoverflow
Solution 16 - Docker	bstricks	View Answer on Stackoverflow

How to use environment variables in docker compose

Docker Problem Overview

Docker Solutions

Solution 1 - Docker

Solution 2 - Docker

Solution 3 - Docker

Solution 4 - Docker

Solution 5 - Docker

Solution 6 - Docker

Solution 7 - Docker

Solution 8 - Docker

Solution 9 - Docker

Solution 10 - Docker

Solution 11 - Docker

Solution 12 - Docker

Solution 13 - Docker

add env to .env file

then save it to `deploy.sh`

Solution 14 - Docker

Solution 15 - Docker

Solution 16 - Docker

Emulator: ERROR: x86 emulation currently requires hardware acceleration

Why does rand() + rand() produce negative numbers?

Attributions

Docker Problem Overview

Docker Solutions

Solution 1 - Docker

Solution 2 - Docker

Solution 3 - Docker

Solution 4 - Docker

Solution 5 - Docker

Solution 6 - Docker

Solution 7 - Docker

Solution 8 - Docker

Solution 9 - Docker

Solution 10 - Docker

Solution 11 - Docker

Solution 12 - Docker

Solution 13 - Docker

add env to .env file

then save it to deploy.sh

Solution 14 - Docker

Solution 15 - Docker

Solution 16 - Docker

Emulator: ERROR: x86 emulation currently requires hardware acceleration

Why does rand() + rand() produce negative numbers?

Attributions

then save it to `deploy.sh`