How to use `while read` (Bash) to read the last line in a file if there’s no newline at the end of the file?
BashNewlineEofBash Problem Overview
Let’s say I have the following Bash script:
while read SCRIPT_SOURCE_LINE; do
echo "$SCRIPT_SOURCE_LINE"
done
I noticed that for files without a newline at the end, this will effectively skip the last line.
I’ve searched around for a solution and found this:
> When read reaches end-of-file instead > of end-of-line, it does read in the > data and assign it to the variables, > but it exits with a non-zero status. > If your loop is constructed "while > read ;do stuff ;done while command sees the non-zero exit > and does not do stuff, exactly as > requested. > > So instead of testing the read exit > status directly, test a flag, and have > the read command set that flag from > within the loop body. That way > regardless of reads exit status, the > entire loop body runs, because read > was just one of the list of commands > in the loop like any other, not a > deciding factor of if the loop will > get run at all. > > DONE=false > until $DONE ;do > read || DONE=true > # process $REPLY here > done < /path/to/file.in
How can I rewrite this solution to make it behave exactly the same as the while
loop I was having earlier, i.e. without hardcoding the location of the input file?
Bash Solutions
Solution 1 - Bash
I use the following construct:
while IFS= read -r LINE || [[ -n "$LINE" ]]; do
echo "$LINE"
done
It works with pretty much anything except null characters in the input:
- Files that start or end with blank lines
- Lines that start or end with whitespace
- Files that don't have a terminating newline
Solution 2 - Bash
In your first example, I'm assuming you are reading from stdin. To do the same with the second code block, you just have to remove the redirection and echo $REPLY:
DONE=false
until $DONE ;do
read || DONE=true
echo $REPLY
done
Solution 3 - Bash
Using grep
with while loop:
while IFS= read -r line; do
echo "$line"
done < <(grep "" file)
Using grep .
instead of grep ""
will skip the empty lines.
Note:
-
Using
IFS=
keeps any line indentation intact. -
File without a newline at the end isn't a standard unix text file.
Solution 4 - Bash
Instead of read, try to use GNU Coreutils like tee, cat, etc.
from stdin
readvalue=$(tee)
echo $readvalue
from file
readvalue=$(cat filename)
echo $readvalue
Solution 5 - Bash
This is the pattern I've been using:
while read -r; do
echo "${REPLY}"
done
[[ ${REPLY} ]] && echo "${REPLY}"
Which works because even tho' the while
loop ends as the "test" from the read
exits with a non-zero code, read
still populates the inbuilt variable $REPLY
(or whatever variables you choose to assign with read
).
Solution 6 - Bash
The basic issue here is that read
will return errorlevel 1 when it encounters EOF, even if it'll still correctly feed the variable.
So you can use errorlevel of read
right away in your loop, otherwize, the last data won't be parsed. But you could do this:
eof=
while [ -z "$eof" ]; do
read SCRIPT_SOURCE_LINE || eof=true ## detect eof, but have a last round
echo "$SCRIPT_SOURCE_LINE"
done
If you want a very solid way to parse your lines, you should use:
IFS='' read -r LINE
Remember that:
- NUL character will be ignored
- if you stick to using
echo
to mimick the behavior ofcat
you'll need to force anecho -n
upon EOF detected (you can use the condition[ "$eof" == true ]
)
Solution 7 - Bash
@Netcoder's answer is good, this optimisation eliminates spurious blank lines, also allows for the last line not to have a newline, if that's how the original was.
DONE=false
NL=
until $DONE ;do
if ! read ; then DONE=true ; NL='-n ';fi
echo $NL$REPLY
done
I used a variant of this to create 2 functions to allow piping of text that includes a '[' to keep grep happy. (you can add other translations)
function grepfix(){
local x="$@";
if [[ "$x" == '-' ]]; then
local DONE=false
local xx=
until $DONE ;do
if ! IFS= read ; then DONE=true ; xx="-n "; fi
echo ${xx}${REPLY//\[/\\\[}
done
else
echo "${x//\[/\\\[}"
fi
}
function grepunfix(){
local x="$@";
if [[ "$x" == '-' ]]; then
local DONE=false
local xx=
until $DONE ;do
if ! IFS= read ; then DONE=true ; xx="-n "; fi
echo ${xx}${REPLY//\\\[/\[}
done
else
echo "${x//\\\[/\[}"
fi
}
(passing - as $1 enables pipe otherwise just translates arguments)
Solution 8 - Bash
for reading files with or without a newline at the end:
cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done
Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/