Read n lines at a time using Bash
BashBash Problem Overview
I read the help read
page, but still don't quite make sense. Don't know which option to use.
How can I read N lines at a time using Bash?
Bash Solutions
Solution 1 - Bash
With Bash≥4 you can use mapfile
like so:
while mapfile -t -n 10 ary && ((${#ary[@]})); do
printf '%s\n' "${ary[@]}"
printf -- '--- SNIP ---\n'
done < file
That's to read 10 lines at a time.
Solution 2 - Bash
While the selected answer works, there is really no need for the separate file handle. Just using the read command on the original handle will function fine.
Here are two examples, one with a string, one with a file:
# Create a dummy file
echo -e "1\n2\n3\n4" > testfile.txt
# Loop through and read two lines at a time
while read -r ONE; do
read -r TWO
echo "ONE: $ONE TWO: $TWO"
done < testfile.txt
# Create a dummy variable
STR=$(echo -e "1\n2\n3\n4")
# Loop through and read two lines at a time
while read -r ONE; do
read -r TWO
echo "ONE: $ONE TWO: $TWO"
done <<< "$STR"
Running the above as a script would output (the same output for both loops):
ONE: 1 TWO: 2
ONE: 3 TWO: 4
ONE: 1 TWO: 2
ONE: 3 TWO: 4
Solution 3 - Bash
This is harder than it looks. The problem is how to keep the file handle.
The solution is to create another, new file handle which works like stdin
(file handle 0) but is independent and then read from that as you need.
#!/bin/bash
# Create dummy input
for i in $(seq 1 10) ; do echo $i >> input-file.txt ; done
# Create new file handle 5
exec 5< input-file.txt
# Now you can use "<&5" to read from this file
while read line1 <&5 ; do
read line2 <&5
read line3 <&5
read line4 <&5
echo "Four lines: $line1 $line2 $line3 $line4"
done
# Close file handle 5
exec 5<&-
Solution 4 - Bash
Simplest method - pretty self-explanatory. It is similar to the method provided by @Fmstrat, except the second read
statement is before the do
.
while read first_line; read second_line
do
echo "$first_line" "$second_line"
done
You can use this by piping multiline input to it:
seq 1 10 | while read first_line; read second_line
do
echo "$first_line" "$second_line"
done
output:
1 2
3 4
5 6
7 8
9 10
Solution 5 - Bash
That is much more simple! :)
cat input-file.txt | xargs -L 10 ./do_something.sh
or
cat input-file.txt | xargs -L 10 echo
Solution 6 - Bash
I don't think there is a way to do it natively in bash, but one can create a convenient function for doing so:
#
# Reads N lines from input, keeping further lines in the input.
#
# Arguments:
# $1: number N of lines to read.
#
# Return code:
# 0 if at least one line was read.
# 1 if input is empty.
#
function readlines () {
local N="$1"
local line
local rc="1"
# Read at most N lines
for i in $(seq 1 $N)
do
# Try reading a single line
read line
if [ $? -eq 0 ]
then
# Output line
echo $line
rc="0"
else
break
fi
done
# Return 1 if no lines where read
return $rc
}
With this one can easily loop over N-line chunks of the data by doing something like
while chunk=$(readlines 10)
do
echo "$chunk" | ... # Whatever processing
done
In this loop $chunk will contain 10 input lines at each iteration, except for the last one, which will contain the last lines of input, which might be less than 10 but always more than 0.
Solution 7 - Bash
Depending on what you're trying to do, you can just store the previous lines.
LINE_COUNT=0
PREVLINE1=""
PREVLINE2=""
while read LINE
do LINE_COUNT=$(($LINE_COUNT+1));
if [[ $LINE_COUNT == 3 ]]; then
LINE_COUNT=0
# do whatever you want to do with the 3 lines
done
PREVLINE2="$PREVLINE1"
PREVLINE1="$LINE"
done
done < $FILE_IN
Solution 8 - Bash
I came up with something very similar to @albarji's answer, but more concise.
read_n() { for i in $(seq $1); do read || return; echo $REPLY; done; }
while lines="$(read_n 5)"; do
echo "========= 5 lines below ============"
echo "$lines"
done < input-file.txt
The read_n
function will read $1
lines from stdin
(use redirection to make it read from a file, just like the built-in read
command). Because the exit code from read
is maintained, you can use read_n
in a loop as the above example demonstrates.
Solution 9 - Bash
Just use a for
loop:
for i in $(seq 1 $N) ; do read line ; lines+=$line$'\n' ; done
In bash version 4, you can also use the mapfile
command.
Solution 10 - Bash
I know you asked about bash, but I am amazed that this works with zsh
#!/usr/bin/env zsh
cat 3-lines.txt | read -d\4 my_var my_other_var my_third_var
Unfortunately, this doesn't work with bash
, at least the versions I tried.
The "magic" here is the -d\4
(this doesn't work in bash), that sets the line delimiter to be the EOT
character, which will be found at the end of your cat
. or any command that produces output.
If you want to read an array of N
items, bash has readarray
and mapfile
that can read files with N
lines and save every line in one position of the array.
EDIT
After some tries, I just found out that this works with bash:
$ read -d# a b
Hello
World
#
$ echo $a $b
Hello World
$
However, I could not make { cat /tmp/file ; echo '#'; } | read -d# a b
to work :(
Solution 11 - Bash
The echo
simulates a file with two lines input, use head -2
before paste
if needed:
IFS=\; read A B < <(echo -en "X1 X2\nY1 Y2\n" | paste -s -d\;)
If you want to read lines in a loop and create pairs and lines have only single word in them use:
while read NAME VALUE; do
echo "$NAME=$VALUE";
done < <(echo -en "M\n1\nN\n2\nO\n3\n" | xargs -L2 echo)
Solution 12 - Bash
Here's an alternative way of doing it:
//This will open the file and users can start adding variables.
cat > file
//After finished ctrl + D will close it
cat file|while read line;
do
//do some stuff here
done
Solution 13 - Bash
Awk is a funny way for this case:
~$ cat test.txt
tom
tom@gmail.com
jack
jack@gmail.com
marry
marry@gmail.com
gogo
gogo@gmail.com
~$ cat test.txt | awk 'BEGIN{c=1}{a=c;if(a==2){print b" "$0;c=1} if(a==1){b=$0;c=2}}'
tom tom@gmail.com
jack jack@gmail.com
marry marry@gmail.com
gogo gogo@gmail.com
Solution 14 - Bash
to read n+2 Lines from a file:-
2
4
6
8
.
.
so on
you can try this way:-
cat fileName | awk '!((NR - 0) % 2)'
Solution 15 - Bash
Also you can group lines with awk
:
$ seq -s ' ' 23 > file
$ cat file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
$ awk '(NR % 6 == 1) {print; for(i=1; i<6 && getline ; i++) { print }; printf "\n"}' RS=' ' ORS=' ' file
1 2 3 4 5 6
7 8 9 10 11 12
13 14 15 16 17 18
19 20 21 22 23
Solution 16 - Bash
Another option is to use the curly brace command grouping in bash.
{ read line1; read line2; } < test-file.txt
One thing to keep in mind tho is that if you have set -u
then this will fail if the file being read has less lines than the number of variables you're attempting to fill. One solution is simply to add || true
to the end of the above line.
Solution 17 - Bash
After having looked at all the answers, I think the following is the simplest, ie more scripters would understand it better than any other solution, but only for small number of items:
while read -r var1 && read -r var2; do
echo "$var1" "$var2"
done < yourfile.txt
The multi-command approach is also excellent, but it is lesser known syntax, although still intuitive:
while read -r var1; read -r var2; do
echo "$var1" "$var2"
done < yourfile.txt
It has the advantage that you don't need line continuations for larger number of items:
while
read -r var1
read -r var2
...
read -r varN
do
echo "$var1" "$var2"
done < yourfile.txt
The xargs answer posted is also nice in theory, but in practice processing the combined lines is not so obvious. For example one solution I came up with using this technique is:
while read -r var1 var2; do
echo "$var1" "$var2"
done <<< $(cat yourfile.txt | xargs -L 2 )
but again this uses the lesser known <<<
operator. However this approach has the advantage that if your script was initially
while read -r var1; do
echo "$var1"
done <<< yourfile.txt
then extending it for multiple lines is somewhat natural:
while read -r var1 var2; do
echo "$var1" "$var2"
done <<< $(cat endpoints.txt | xargs -L 2 )
while read -r var1; do
read -r var2
echo "$var1" "$var2"
done < yourfile.txt
is the only other one that I would consider among the many given, for its simplicity, but syntactically it is not as expressive; compared to the &&
version or multi-command version it does not feel as right.