How do I use sed to change my configuration files, with flexible keys and values?

I want to search a configuration file for this expression: "central.database". I then want to change the setting associated with "central.database" to "SQLTEST".

The layout of the config file would look like this initially:

central.database = SQLFIRSTTEST

This is what i want it to look like after the sed replacement:

central.database = SQLTEST

I am doing this in a bash script, any suggestions, recommendations or alternative solutions are welcome!

(Actually both central.database and SQLTEST come from bash variables here.)

---

My current code (third attempt):

sshRetValue=$(ssh -p "35903" -i $HOME/sshids/idrsa-1.old ${1} <<EOF
        sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
        echo $?
EOF
)

Error message:

Pseudo-terminal will not be allocated because stdin is not a terminal.
sed: -e expression #1, char 58: unknown option to `s'
-bash: line 3: EOF: command not found

sed -i -e '/central\.database =/ s/= .*/= new_value/' /path/to/file

Explanation:

• -i tells sed to save the results to the input file. Without it sed will print the results to stdout.
• /central\.database =/ matches lines that contain the string between slashes: central.database =. The . is escaped since it's a special character in regex.
• The s/OLD/NEW/ part performs a substitution. The OLD string is a regular expression to match and the NEW part is the string to substitute in.
• In regular expressions, .* means "match anything". So = .* matches an equal sign, space, and then anything else afterward.

Here's an example expression:

sed -i 's/^\(central\.database\s*=\s*\).*$/\1SQLTEST/' file.cfg

If you want to match stuff with / in it, you can use another delimiter:

sed -i 's#^\(cent/ral\.data/base\s*=\s*\).*$#\1SQL/TEST#' file.cfg

Or with variable expansion:

VAL="SQLTEST"
sed -i "s/^\(central\.database\s*=\s*\).*\$/\1$VAL/" file.cfg

In your example:

sshRetValue=`sed -i "s/^\(\1$CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt`;

There's a \1 before $CENTRAL_DB_NAME that's invalid. Also, sed doesn't print it's return value. This is the preferred way to check return values:

sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
sed_return_value=$?

And ultimately piping to ssh (not tested):

sed_return_value=$(ssh server <<EOF
    sed -i "s/^\($CENTRAL_DB_NAME\s*=\s*\).*\$/\1$CENTRAL_DB_VALUE/" /home/testing.txt;
    echo $?
EOF
)

The -i is for replacing data in the input file. Otherwise sed writes to stdout.

Regular expressions are a field of their own. It would be impossible to explain them in depth in a stackoverflow answer, unless there is some specific function that's eluding you.

I know it is too late to add an answer to this question however, I thought to share my knowledge to you all. There is a very general approach which I have followed to solve a similar kind of problem. I have deleted the whole line which is matching the string and added the required values to that key. To your question here is the answer

replaceValue=SQLTEST
sed -i "/central.database =/d" /home/testing.txt
echo "central.database = $replaceValue"  >> /home/testing.txt

sed deletes the matching string line from the file and the immediate next line is inserting the required key and value to the file.

I like using awk for this, since it is quite easy to understand what it is doing and takes care very well of the separator (=) and also the fact that it must be done to an uncommented line:

awk -v var="my_var" -v new_val="NEW VALUE" \  # set the vars
    'BEGIN{FS=OFS="="}                        # set separator to =
     match($1, "^\\s*" var "\\s*") {          # check if it matches
         $2=" " new_val                       # if so, replace the line
     }1' conf_file                            # print all lines

This uses match() to check if the pattern occurs in any given line. If it does, it performs the replacement with the given value.

For example:

$ cat conf
hello
my_var= SOME VALUE
#my_var = ANOTHER VALUE
bye

Let's change the value in my_var to NEW VALUE:

$ awk -v var="my_var" -v new_val="NEW VALUE" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' conf
hello
my_var= NEW VALUE
#my_var = ANOTHER VALUE
bye

It is also possible to set the values in shell variables and then use them with -v:

$ var="my_var"
$ new_value="NEW VALUE"
$ awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' conf

And you can of course put all of this within a shell function that you then call normally:

#!/bin/bash

replace () {
   file=$1
   var=$2
   new_value=$3
   awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' "$file"
}

# Call the replace() function with the necessary parameters
replace "conf" "my_var" "NEW VALUE" 

Upon execution, this returns

hello
my_var= NEW VALUE
#my_var = ANOTHER VALUE
bye

While you can also make the script receive the parameters in a way like: ./script.sh "conf_file" "var_to_replace" "NEW VALUE" to then pass them to the function.

If you want to replace between 2 property files you can use this:

awk -F= 'NR==FNR{A[$1]=$2;next}$1 in A{$2=A[$1]}1' OFS='\=' /tmp/masterfile /opt/props/finalfile.properties > /tmp/tmp.txt && mv -f /tmp/tmp.txt /opt/props/finalfile.properties

I have a file called "config.php" and I wanted to change one of its definitions lines.

For example, the line:

define('SOME_CONSTANT', 'old_value');

had to be replaced with this one:

define('SOME_CONSTANT', 'new_value');

So I did that:

sed -i -e "/.*SOME_CONSTANT*./ s/.*/define('SOME_CONSTANT', 'new_value');/" path/to/config.php

In the first part, I am looking for a line that contains "SOME_CONSTANT" (thus the wildcards)

Then I replace that line with a new definition such as: define('SOME_CONSTANT', 'new_value');

Tested and works fine in Centos 7

I used this script for keeping the priorities..

The arguments $1 will have folder in which multiple config files exist. $2 will have properties which need to be replaced in $1 path and sub paths files #3 will have properties which need to be override on top of $2

It also has hidden logic to check for existence of environment variables for the keys exist in $2 and $3 and give priority to that.

i.e If a key exist in environment that would be highest priority. Next to that would $3 and next to that would $1 file.

#!/bin/bash
#Usage is propertyReplacer <CONFIG_FOLDER_PATH> <CONFIG_FILE_2ND_PRIORITY> <CONFIG_FILE_1ST_PRIORITY>
function propertyReplacer() {

  filePathToAct="$1"
  propertiesFilePath="$2"
  propertiesSecureFilePath="$3"
  
  declare -A keyValues

  while IFS='=' read -r key value; do
    if [  "$key" == "" ]; then
      continue
    elif [[  "$key" =~ ^#.*$ ]]; then
      continue
    else
      echo $key " --> " $value
      keyValues[$key]=$value
    fi
  done < "$propertiesFilePath"

  if [ ! -f "$propertiesSecureFilePath" ]; then
    continue
  else
    while IFS='=' read -r key value; do
      if [  "$key" == "" ]; then
        continue
      elif [[  "$key" =~ ^#.*$ ]]; then
        continue
      else
        echo $key " --> " $value
        keyValues[$key]=$value
      fi
    done < "$propertiesSecureFilePath"
  fi

  for key in ${!keyValues[@]}; do
    envProp=${key//[@]/}
    if [  "$(eval echo '$'$envProp)" == "" ]; then
      echo "Environment key not exist" $envProp
    else
      value=$(eval echo '$'$envProp)
      echo "From Environment " $envProp " --> "$value 
      keyValues[$key]=$value
    fi
  done 

 
find "$filePathToAct" | while read -r resultFileName; do
  if [ ! -f "$resultFileName" ]; then
    continue
  else
    echo "Acting on the file $resultFileName"

    for key in ${!keyValues[@]}; do
      value=$(echo "${keyValues[${key}]}" | sed 's/\//\\\//g')
      echo "sed -i 's/$key/$value/g' $resultFileName "
      eval "sed -i 's/$key/$value/g' $resultFileName "
    done 
  fi
done 
 
} 

Worked example

Imagine we want to update a Postgres configuration file to increase the number of connections.

We want to update this line in /var/lib/postgresql/data/postgresql.conf:

max_connections = 100

Using sed:

sed -i 's/max_connections = 100/max_connections = 250/g' /var/lib/postgresql/data/postgresql.conf

This is rather useful when updating Docker configuration files as any updates require automating.