Create new file from templates with bash script

Bash Templates Configuration Template Engine

Bash Problem Overview

I have to create conf files and init.d which are very similar. These files permit to deploy new http service on my servers. These files are the same and only some parameters change from one file to another (listen_port, domain, path on server...).

As any error in these files leads to misfunction of service I would like to create these files using a bash script.

For example:

generate_new_http_service.sh 8282 subdomain.domain.com /home/myapp/rootOfHTTPService

I am looking for a kind of templating module that I could use with bash. This templating module would use some generic conf and init.d scripts to create new ones.

Do you have hints for that? If not I could use python templating engine.

Bash Solutions

Solution 1 - Bash

You can do this using a heredoc. e.g.

generate.sh:

#!/bin/sh

#define parameters which are passed in.
PORT=$1
DOMAIN=$2

#define the template.
cat  << EOF
This is my template.
Port is $PORT
Domain is $DOMAIN
EOF

Output:

$ generate.sh 8080 domain.com

This is my template.
Port is 8080
Domain is domain.com

or save it to a file:

$ generate.sh 8080 domain.com > result

Solution 2 - Bash

Template module for bash? Use sed, Luke! Here is an example of one of millions of possible ways of doing this:

$ cat template.txt 
#!/bin/sh

echo Hello, I am a server running from %DIR% and listening for connection at %HOST% on port %PORT% and my configuration file is %DIR%/server.conf

$ cat create.sh 
#!/bin/sh

sed -e "s;%PORT%;$1;g" -e "s;%HOST%;$2;g" -e "s;%DIR%;$3;g" template.txt > script.sh

$ bash ./create.sh 1986 example.com /tmp
$ bash ./script.sh 
Hello, I am a server running from /tmp and listening for connection at example.com on port 1986 and my configuration file is /tmp/server.conf
$

Solution 3 - Bash

you can do this directly in bash, you do not even need sed. Write a script like that:

#!/bin/bash

cat <<END
this is a template
with $foo
and $bar
END

then call it like so:

foo=FOO bar=BAR ./template

Solution 4 - Bash

For simple file generation, basically doing

 . "${config_file}"
 template_str=$(cat "${template_file}")
 eval "echo \"${template_str}\""

would suffice.

Here ${config_file} contains the configuration variables in shell parseable format, and ${template_file} is the template file that looks like shell here document. The first line sources in the file ${config_file}, the second line puts the contents of the file ${template_file} into the shell variable template_str. Finally in the third line we build the shell command echo "${template_str}" (where the double quoted expression "${template_str}" is expanded) and evaluate it.

For an example of the contents of those two files, please refer to https://serverfault.com/a/699377/120756.

There are limitations what you can have in the template file or you need to perform shell escaping. Also if the template file is externally produced, then for security reasons you need to consider implementing a proper filtering prior to execution so that you will not for example lose your files when somebody injects the famous $(rm -rf /) in the template file.

Solution 5 - Bash

Here's the approach that I ended up taking to solve this problem. I found it a little more flexible than some of the above approaches, and it avoids some of the issues with quotes.

fill.sh:

#!/usr/bin/env sh

config="$1"
template="$2"
destination="$3"

cp "$template" "$destination"

while read line; do
    setting="$( echo "$line" | cut -d '=' -f 1 )"
    value="$( echo "$line" | cut -d '=' -f 2- )"

    sed -i -e "s;%${setting}%;${value};g" "$destination"
done < "$config"

template:

Template full of important %THINGS%

"Note that quoted %FIELDS% are handled correctly"

If I need %NEWLINES% then I can add them as well.

config:

THINGS=stuff
FIELDS="values work too!"
NEWLINES="those\\nnifty\\nlinebreaks"

result: Template full of important stuff

"Note that quoted "values work too!" are handled correctly"

If I need those
nifty
linebreaks then I can add them as well.

Solution 6 - Bash

[Edit] I changed my answer from the original one, that was years ago.

I like the answer from FooF above: https://stackoverflow.com/a/30872526/3538173

Yet, I prefer not to have an intermediary variable to store the whole content of the template file in memory.

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

Example

Create a template file. Let's call it example.tpl:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

Create a configuration file to store your variables. Let's call it good.conf:

NAME=John
WEATHER=good

Now, in the script where you want to render the template, you can write this:

#!/usr/bin/env bash

template_file=example.tpl
config_file=good.conf

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

# Or store the output in a file
eval "echo \"$(cat "${template_file}")\"" > out

You should see this wonderful output :)

Hello, John!
Today, the weather is good. Enjoy!

Caution with `eval`

When you use eval, if the template file contains some instructions, they will be executed, and it can be dangerous. For example, let's change the example.tpl above with this content:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

I'm a hacker, hu hu! Look, fool!
$(ls /)

Now, if you render your template file, you will see this:

Hello, John!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Now edit your file good.conf to have this content:

NAME=$(ls -l /var)
WEATHER=good

and render the template. You should see something like this:

Hello, total 8
drwxr-xr-x.  2 root root    6 Apr 11 04:59 adm
drwxr-xr-x.  5 root root   44 Sep 11 18:04 cache
drwxr-xr-x.  3 root root   34 Sep 11 18:04 db
drwxr-xr-x.  3 root root   18 Sep 11 18:04 empty
drwxr-xr-x.  2 root root    6 Apr 11 04:59 games
drwxr-xr-x.  2 root root    6 Apr 11 04:59 gopher
drwxr-xr-x.  3 root root   18 May  9 13:48 kerberos
drwxr-xr-x. 28 root root 4096 Oct  8 00:30 lib
drwxr-xr-x.  2 root root    6 Apr 11 04:59 local
lrwxrwxrwx.  1 root root   11 Sep 11 18:03 lock -> ../run/lock
drwxr-xr-x.  8 root root 4096 Oct  8 04:55 log
lrwxrwxrwx.  1 root root   10 Sep 11 18:03 mail -> spool/mail
drwxr-xr-x.  2 root root    6 Apr 11 04:59 nis
drwxr-xr-x.  2 root root    6 Apr 11 04:59 opt
drwxr-xr-x.  2 root root    6 Apr 11 04:59 preserve
lrwxrwxrwx.  1 root root    6 Sep 11 18:03 run -> ../run
drwxr-xr-x.  8 root root   87 Sep 11 18:04 spool
drwxrwxrwt.  4 root root  111 Oct  9 09:02 tmp
drwxr-xr-x.  2 root root    6 Apr 11 04:59 yp!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
swapfile
sys
tmp
usr
var

As you can see, command injection in the configuration file and the template file is possible, and that's why you have to be extra careful:

be sure of the content of the template file: check that there is NO command injection.
be sure of the content of the configuration file: check that there is NO command injection as well. If the configuration file comes from someone else, you need to know and trust that person before rendering the template.

Imagine that you are a password-less sudoer, rendering the template file could result in ruining your system with a well-placed rm -rf.

As long as you control the content of these files, it is fine to use this eval templating.

If you have an external (untrusted) incoming configuration file, you should look for templating engine, that will isolate these kind of injection. For example, Jinja2 templating is quite famous in Python.

Solution 7 - Bash

I ended up using envsubst which was available for me. It is widely available on most Linux systems in the gettext package. I used this because it will take care of selectively replacing variables and I don't have to escape them in the template as with e.g. eval. Here is an example.

$ cat dbtemplate.xml
<DB>
        <hostname>$DBHOST</hostname>
        <port>$DBPORT</port>
        <database>$DBNAME</database>
</DB>

$ export DBHOST=mydbhost1 DBPORT=1234 DBNAME=mydb; envsubst< dbtemplate.xml
<DB>
        <hostname>mydbhost1</hostname>
        <port>1234</port>
        <database>mydb</database>
</DB>

Solution 8 - Bash

You can use python class string.Template

$ echo 'before $X after' > template.txt

$ python  -c 'import string; print(string.Template(open("template.txt").read()).substitute({"X":"A"}))'

before A after

$  python  -c 'import string, sys; print(string.Template(open("template.txt").read()).substitute({"X":sys.argv[1]}))' "A"

Here $X is a placeholder in the template and {"X":"A"} is a mapping of the placeholder to a value. In the python code we read the template text from the file, create a template from it, then substitute the placeholder with the command line argument.

Alternatively you can use Ruby's ERB, if Ruby is installed on your machine.

$ echo "before <%= ENV['X'] %> after" > template.txt

$ X=A erb template.txt

before A after

Here <%= ENV['X'] %> is a placeholder. ENV['X'] reads the value from the environment variable. X=A sets the environment variable to the desired value.

Solution 9 - Bash

Elegant and short solution in one line with `perl`

I use perl to replace variables with their values:

export world=World beautiful=wonderful
echo 'I love you, $world! You are $beautiful.' >my_template.txt
perl -pe 's|\$([A-Za-z_]+)|$ENV{$1}|g' my_template.txt

The output: I love you, World! You are wonderful.

my_template.txt can contain variables prefixed with $.

Solution 10 - Bash

use bash to generate bash script from template

this problem calls for a strict separation of variable data (script header) and constant code (script body)

#!/usr/bin/env bash

# expected result

# header
a=1
b=2

# body
echo "a=$a and b=$b"

in the template file sample.tpl.sh, i have only one template variable #%TEMPLATE_CONSTANTS

#!/usr/bin/env bash

# template file

# header
#%TEMPLATE_CONSTANTS

# body
echo "a=$a and b=$b"
echo "here in the script body,
#%TEMPLATE_CONSTANTS
is not replaced"

most environments lack a fixed-string-editor, so i made my own "fsed" with grep and dd, to replace only the first match of #%TEMPLATE_CONSTANTS

#!/usr/bin/env bash

# template processor

# https://stackoverflow.com/a/69479243/10440128
fixedReplaceFirst(){ # aka fsed (fixed string editor)
  tplFile="$1"
  pattern="$2"
  replace="$3"
  match="$(grep -b -m 1 -o -E "$pattern" "$tplFile")"
  offset1=$(echo "$match" | cut -d: -f1)
  match="$(echo "$match" | cut -d: -f2-)"
  matchLength=${#match}
  offset2=$(expr $offset1 + $matchLength)
  dd bs=1 if="$tplFile" count=$offset1 status=none
  echo -n "$replace"
  dd bs=1 if="$tplFile" skip=$offset2 status=none
}

read -d '' replace <<EOF
a=1
b=2
EOF

fixedReplaceFirst "sample.tpl.sh" "^#%TEMPLATE_CONSTANTS$" "$replace"

Solution 11 - Bash

I suggest shtpl, a shell templating system: https://github.com/dontsueme/shtpl

It has a very simple syntax, is foolproof and does only need standard *nix tools.

Content Type	Original Author	Original Content on Stackoverflow
Question	kheraud	View Question on Stackoverflow
Solution 1 - Bash	dogbane	View Answer on Stackoverflow
Solution 2 - Bash	user405725	View Answer on Stackoverflow
Solution 3 - Bash	Kim Stebel	View Answer on Stackoverflow
Solution 4 - Bash	FooF	View Answer on Stackoverflow
Solution 5 - Bash	Keegs	View Answer on Stackoverflow
Solution 6 - Bash	Samuel Phan	View Answer on Stackoverflow
Solution 7 - Bash	John	View Answer on Stackoverflow
Solution 8 - Bash	Alexey	View Answer on Stackoverflow
Solution 9 - Bash	kyb	View Answer on Stackoverflow
Solution 10 - Bash	Mila Nautikus	View Answer on Stackoverflow
Solution 11 - Bash	Christian Herenz	View Answer on Stackoverflow

Create new file from templates with bash script

Bash Problem Overview

Bash Solutions

Solution 1 - Bash

Solution 2 - Bash

Solution 3 - Bash

Solution 4 - Bash

Solution 5 - Bash

Solution 6 - Bash

Example

Caution with `eval`

Solution 7 - Bash

Solution 8 - Bash

Solution 9 - Bash

Elegant and short solution in one line with `perl`

Solution 10 - Bash

use bash to generate bash script from template

Solution 11 - Bash

How to disable entire unit test in TestNG?

Python: Cut off the last word of a sentence?

Attributions

Bash Problem Overview

Bash Solutions

Solution 1 - Bash

Solution 2 - Bash

Solution 3 - Bash

Solution 4 - Bash

Solution 5 - Bash

Solution 6 - Bash

Example

Caution with eval

Solution 7 - Bash

Solution 8 - Bash

Solution 9 - Bash

Elegant and short solution in one line with perl

Solution 10 - Bash

use bash to generate bash script from template

Solution 11 - Bash

How to disable entire unit test in TestNG?

Python: Cut off the last word of a sentence?

Attributions

Caution with `eval`

Elegant and short solution in one line with `perl`