Reading null delimited strings through a Bash loop

BashDelimiterNull Character

Bash Problem Overview


I want to iterate through a list of files without caring about what characters the filenames might contain, so I use a list delimited by null characters. The code will explain things better.

# Set IFS to the null character to hopefully change the for..in
# delimiter from the space character (sadly does not appear to work).
IFS=$'\0'

# Get null delimited list of files
filelist="`find /some/path -type f -print0`"

# Iterate through list of files
for file in $filelist ; do
    # Arbitrary operations on $file here
done

The following code works when reading from a file, but I need to read from a variable containing text.

while read -d $'\0' line ; do
    # Code here
done < /path/to/inputfile

Bash Solutions


Solution 1 - Bash

The preferred way to do this is using process substitution

while IFS= read -r -d $'\0' file; do
    # Arbitrary operations on "$file" here
done < <(find /some/path -type f -print0)

If you were hell-bent on parsing a bash variable in a similar manner, you can do so as long as the list is not NUL-terminated.

Here is an example of bash var holding a tab-delimited string

$ var=$(echo -ne "foo\tbar\tbaz\t"); 
$ while IFS= read -r -d $'\t' line ; do \
    echo "#$line#"; \
  done <<<"$var"
#foo#
#bar#
#baz#

Solution 2 - Bash

Use env -0 to output the assignments by the zero byte.

env -0 | while IFS='' read -d '' line ; do
    var=${line%%=*}
    value=${line#*=}
    echo "Variable '$var' has the value '$value'"
done

Solution 3 - Bash

Pipe them to xargs -0:

files="$( find ./ -iname 'file*' -print0 | xargs -0 )"

xargs manual:

> -0, --null > Input items are terminated by a null character instead of > by whitespace, and the quotes and backslash are not > special (every character is taken literally).

Solution 4 - Bash

In terms of readability and maintainability a bash function might be cleaner:

An example that converts MOV files to MP4 using ffmpeg (works with files containing spaces and special characters):

#!/usr/bin/env bash

do_convert () { 
  new_file="${1/.mov/.mp4}"
  ffmpeg -i "$1" "$new_file" && rm "$1" 
}

export -f do_convert  # needed to make the function visible inside xargs

find . -iname '*.mov' -print0 | xargs -0 -I {} bash -c 'do_convert "{}"' _ {}

Does not apply to the OP's question but in case your input is generated by find then there is no need to pipe via xargs -0 as find is perfectly capable of handling non-ascii characters and spaces in file names. If you don't care about readability and maintainability then the command above can be simplified to:

find . -type f -iname "*.mov" -exec bash -c 'ffmpeg -i "${1}" "${1%.*}.mp4" && rm "${1}"' _ {} \;

Solution 5 - Bash

I tried working with the bash examples above, and finally gave up, and used Python, which worked the first time. For me it turned out the problem was simpler outside the shell. I know this is possibly off topic of a bash solution, but I'm posting it here anyway in case others want an alternative.

import sh
import path
files = path.Path(".").files()
for x in files:
    sh.cp("--reflink=always", x, "UUU00::%s"%(x.basename(),))
    sh.cp("--reflink=always", x, "UUU01::%s"%(x.basename(),))

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionMatthewView Question on Stackoverflow
Solution 1 - BashSiegeXView Answer on Stackoverflow
Solution 2 - BashchorobaView Answer on Stackoverflow
Solution 3 - BashMaltigoView Answer on Stackoverflow
Solution 4 - BashccpizzaView Answer on Stackoverflow
Solution 5 - BashHenry CrutcherView Answer on Stackoverflow