How to read a .properties file which contains keys that have a period character using Shell script

I am trying to read a properties file from a shell script which contains a period (.) character like below:

# app.properties
db.uat.user=saple user
db.uat.passwd=secret


#/bin/sh
function pause(){
   read -p "$*"
}

file="./app.properties"

if [ -f "$file" ]
then
	echo "$file found."
 . $file

echo "User Id " $db.uat.user
echo "user password =" $db.uat.passwd
else
	echo "$file not found."
fi

I have tried to parse the file after sourcing the file but it is not working since the keys contains the "." character and there are spaces in that value also.

My properties file always resides in the same directory of the script or somewhere in /usr/share/doc

I use simple grep inside function in bash script to receive properties from .properties file.

This properties file I use in two places - to setup dev environment and as application parameters.

I believe that grep may work slow in big loops but it solves my needs when I want to prepare dev environment.

Hope, someone will find this useful.

Example:

File: setup.sh

#!/bin/bash

ENV=${1:-dev}

function prop {
    grep "${1}" env/${ENV}.properties|cut -d'=' -f2
}

docker create \
    --name=myapp-storage \
    -p $(prop 'app.storage.address'):$(prop 'app.storage.port'):9000 \
    -h $(prop 'app.storage.host') \
    -e STORAGE_ACCESS_KEY="$(prop 'app.storage.access-key')" \
    -e STORAGE_SECRET_KEY="$(prop 'app.storage.secret-key')" \
    -e STORAGE_BUCKET="$(prop 'app.storage.bucket')" \
    -v "$(prop 'app.data-path')/storage":/app/storage \
    myapp-storage:latest

docker create \
    --name=myapp-database \
    -p "$(prop 'app.database.address')":"$(prop 'app.database.port')":5432 \
    -h "$(prop 'app.database.host')" \
    -e POSTGRES_USER="$(prop 'app.database.user')" \
    -e POSTGRES_PASSWORD="$(prop 'app.database.pass')" \
    -e POSTGRES_DB="$(prop 'app.database.main')" \
    -e PGDATA="/app/database" \
    -v "$(prop 'app.data-path')/database":/app/database \
    postgres:9.5

File: env/dev.properties

app.data-path=/apps/myapp/

#==========================================================
# Server properties
#==========================================================
app.server.address=127.0.0.70
app.server.host=dev.myapp.com
app.server.port=8080

#==========================================================
# Backend properties
#==========================================================
app.backend.address=127.0.0.70
app.backend.host=dev.myapp.com
app.backend.port=8081
app.backend.maximum.threads=5

#==========================================================
# Database properties
#==========================================================
app.database.address=127.0.0.70
app.database.host=database.myapp.com
app.database.port=5432
app.database.user=dev-user-name
app.database.pass=dev-password
app.database.main=dev-database

#==========================================================
# Storage properties
#==========================================================
app.storage.address=127.0.0.70
app.storage.host=storage.myapp.com
app.storage.port=4569
app.storage.endpoint=http://storage.myapp.com:4569
app.storage.access-key=dev-access-key
app.storage.secret-key=dev-secret-key
app.storage.region=us-east-1
app.storage.bucket=dev-bucket

Usage:

./setup.sh dev

As (Bourne) shell variables cannot contain dots you can replace them by underscores. Read every line, translate . in the key to _ and evaluate.

#/bin/sh

file="./app.properties"

if [ -f "$file" ]
then
  echo "$file found."

  while IFS='=' read -r key value
  do
    key=$(echo $key | tr '.' '_')
    eval ${key}=\${value}
  done < "$file"

  echo "User Id       = " ${db_uat_user}
  echo "user password = " ${db_uat_passwd}
else
  echo "$file not found."
fi

Note that the above only translates . to _, if you have a more complex format you may want to use additional translations. I recently had to parse a full Ant properties file with lots of nasty characters, and there I had to use:

key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')

Since variable names in the BASH shell cannot contain a dot or space it is better to use an associative array in BASH like this:

#!/bin/bash

# declare an associative array
declare -A arr

# read file line by line and populate the array. Field separator is "="
while IFS='=' read -r k v; do
   arr["$k"]="$v"
done < app.properties

Testing:

Use declare -p to show the result:

  > declare -p arr  
  
        declare -A arr='([db.uat.passwd]="secret" [db.uat.user]="saple user" )'

For a very high performance, and BASH 3.0 compatible solution:

file: loadProps.sh

function loadProperties() {
  local fileName=$1
  local prefixKey=$2

  if [ ! -f "${fileName}" ]; then
    echo "${fileName} not found!"
    return 1
  fi

  while IFS='=' read -r origKey value; do
    local key=${origKey}
    key=${key//[!a-zA-Z0-9_]/_} 
    if [[ "${origKey}" == "#"*   ]]; then
      local ignoreComments
    elif [ -z "${key}" ]; then
      local emptyLine
    else
      if [[ "${prefixKey}${key}" =~ ^[0-9].* ]]; then
      	key=_${key}
      fi
      eval ${prefixKey}${key}=\${value}
    fi
  done < <(grep "" ${fileName})
}

The other solutions provided here are great and elegant, but

• @fork2execve: slow when dealing with large properties files
• @Nicolai: slow when reading lots of properties
• @anubhava: require BASH 4.0 (for the array)

I needed something working on bash 3, dealing with properties files of ~1k entries, reading ~200 properties, and whole script called MANY times.

this function also deals with

• empty lines
• commented code
• duplicated entries (last one wins)
• normalize property names
• last line without a proper new line

Testing

file: my.properties

a=value
a=override value
b=what about `!@#$%^&*()_+[]\?
c=${a} no expansion
d=another = (equal sign)
e=     5 spaces front and back     
f=
#g=commented out
#ignore new line below

.@a%^=who named this???
a1=A-ONE
1a=ONE-A
X=lastLine with no new line!

test script

. loadProps.sh

loadProperties my.properties PROP_
echo "a='${PROP_a}'"
echo "b='${PROP_b}'"
echo "c='${PROP_c}'"
echo "d='${PROP_d}'"
echo "e='${PROP_e}'"
echo "f='${PROP_f}'"
echo "g='${PROP_g}'"
echo ".@a%^='${PROP___a__}'"
echo "a1='${PROP_a1}'"
echo "1a='${PROP_1a}'"
echo "X='${PROP_X}'"

loadProperties my.properties
echo "a='${a}'"
echo "1a='${_1a}'"

output

a='override value'
b='what about `!@#$%^&*()_+[]\?'
c='${a} no expansion'
d='another = (equal sign)'
e='     5 spaces front and back     '
f=''
g=''
.@a%^='who named this???'
a1='A-ONE'
1a='ONE-A'
X='lastLine with no new line!'
a='override value'
1a='ONE-A'

Performance Test

. loadProps.sh

function fork2execve() {
  while IFS='=' read -r key value; do
    key=$(echo $key | tr .-/ _ | tr -cd 'A-Za-z0-9_')
    eval ${key}=\${value}
  done < "$1"
}

function prop {
  grep '^\s*'"$2"'=' "$1" | cut -d'=' -f2-
}

function Nicolai() {
  for i in $(seq 1 $2); do 
    prop0000=$(prop $1 "property_0000")
  done
}

function perfCase() {
  echo "perfCase $1, $2, $3"
  time for i in $(seq 1 1); do 
    eval $1 $2 $3
  done
}

function perf() {
  perfCase $1 0001.properties $2
  perfCase $1 0010.properties $2
  perfCase $1 0100.properties $2
  perfCase $1 1000.properties $2
}

perf "loadProperties"
perf "fork2execve"
perf "Nicolai" 1
perf "Nicolai" 10
perf "Nicolai" 100

with 4 NNNN.properties files with entries such as

property_0000=value_0000
property_0001=value_0001
...
property_NNNN=value_NNNN

resulted with

function   , file,   #,     real,    user,     sys
loadPropert, 0001,    ,    0.058,   0.002,   0.005
loadPropert, 0010,    ,    0.032,   0.003,   0.005
loadPropert, 0100,    ,    0.041,   0.013,   0.006
loadPropert, 1000,    ,    0.140,   0.106,   0.013

fork2execve, 0001,    ,    0.053,   0.003,   0.007
fork2execve, 0010,    ,    0.211,   0.021,   0.051
fork2execve, 0100,    ,    2.146,   0.214,   0.531
fork2execve, 1000,    ,   21.375,   2.151,   5.312

Nicolai    , 0001,   1,    0.048,   0.003,   0.009
Nicolai    , 0010,   1,    0.047,   0.003,   0.009
Nicolai    , 0100,   1,    0.044,   0.003,   0.010
Nicolai    , 1000,   1,    0.044,   0.004,   0.009

Nicolai    , 0001,  10,    0.240,   0.020,   0.056
Nicolai    , 0010,  10,    0.263,   0.021,   0.059
Nicolai    , 0100,  10,    0.272,   0.023,   0.062
Nicolai    , 1000,  10,    0.295,   0.027,   0.059

Nicolai    , 0001, 100,    2.218,   0.189,   0.528
Nicolai    , 0010, 100,    2.213,   0.193,   0.537
Nicolai    , 0100, 100,    2.247,   0.196,   0.543
Nicolai    , 1000, 100,    2.323,   0.253,   0.534

@fork2x

I have tried like this .Please review and update me whether it is right approach or not.

#/bin/sh
function pause(){
   read -p "$*"
}

file="./apptest.properties"


if [ -f "$file" ]
then
	echo "$file found."

dbUser=`sed '/^\#/d' $file | grep 'db.uat.user'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`
dbPass=`sed '/^\#/d' $file | grep 'db.uat.passwd'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`

echo database user = $dbUser
echo database pass = $dbPass

else
	echo "$file not found."
fi

I found using while IFS='=' read -r to be a bit slow (I don't know why, maybe someone could briefly explain in a comment or point to a SO answer?). I also found @Nicolai answer very neat as a one-liner, but very inefficient as it will scan the entire properties file over and over again for every single call of prop.

I found a solution that answers the question, performs well and it is a one-liner (bit verbose line though).

The solution does sourcing but massages the contents before sourcing:

#!/usr/bin/env bash

source <(grep -v '^ *#' ./app.properties | grep '[^ ] *=' | awk '{split($0,a,"="); print gensub(/\./, "_", "g", a[1]) "=" a[2]}')

echo $db_uat_user

Explanation:

grep -v '^ *#': discard comment lines grep '[^ ] *=': discards lines without = split($0,a,"="): splits line at = and stores into array a, i.e. a[1] is the key, a[2] is the value gensub(/\./, "_", "g", a[1]): replaces . with _ print gensub... "=" a[2]} concatenates the result of gensub above with = and value.

Edit: As others pointed out, there are some incompatibilities issues (awk) and also it does not validate the contents to see if every line of the property file is actually a kv pair. But the goal here is to show the general idea for a solution that is both fast and clean. Sourcing seems to be the way to go as it loads the properties once that can be used multiple times.