Splitting a file in linux based on content
LinuxFileBashSedAwkLinux Problem Overview
I have an email dump of around 400mb. I want to split this into .txt files, consisting of one mail in each file. Every e-mail starts with the standard HTML header specifying the doctype.
This means I will have to split my files based on the above said header. How do I go about it in linux?
Linux Solutions
Solution 1 - Linux
If you have a mail.txt
$ cat mail.txt
<html>
mail A
</html>
<html>
mail B
</html>
<html>
mail C
</html>
run csplit to split by <html>
$ csplit mail.txt '/^<html>$/' '{*}'
- mail.txt => input file
- /^<html>$/ => pattern match every `<html>` line
- {*} => repeat the previous pattern as many times as possible
check output
$ ls
mail.txt xx00 xx01 xx02 xx03
If you want do it in awk
$ awk '/<html>/{filename=NR".txt"}; {print >filename}' mail.txt
$ ls
1.txt 5.txt 9.txt mail.txt
Solution 2 - Linux
The csplit program solves your problem elegantly:
csplit '/<!DOCTYPE.*/' $FILE
Solution 3 - Linux
csplit is the best solution to this problem. Just thought I'd post a bash-solution to show that there is no need to go perl on this task:
#!/usr/bin/bash
MAIL='mail' # path to huge mail-file
#get linenumbers for all headers
line_no=$(grep -n html $MAIL | cut -d: -f1)
read -a LINES<<< $line_no
file=0
for i in $(seq 0 2 ${#LINES[@]}); do
start=${LINES[i]}
end=$((${LINES[i+1]}-1))
echo $start, $end
sed -n "${start},${end}p" $MAIL > ${MAIL}${file}.txt
file=$((file+1))
done
Solution 4 - Linux
It is doable with some perl "magic"... Many people would call this ugly but here goes.
The trick is to replace $/ with what you want and read your input, as such:
#!/usr/bin/perl -W
use strict;
my $i = 1;
$/ = <<EOF;
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head> <xmeta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
EOF
open INPUT, "/path/to/inputfile" or die;
while (my $mail = <INPUT>) {
$mail = substr($mail, 0, index($mail, $/));
open OUTPUT, ">/path/to/emailfile." . $i . ".txt" or die;
$i++;
print OUTPUT $mail;
close OUTPUT;
}
edit: fixed, I always forget that $/ is included in the input. Also, the first file will always be empty, but then it can be easily handled.
Solution 5 - Linux
I agree with fge. With perl it would be a lot simpler. You can try something like this -
#!/usr/bin/perl
undef $/;
$_ = <>;
$n = 0;
for $match (split(/(?=HEADER_FORMAT)/)) {
open(O, '>mail' . ++$n);
print O $match;
close(O);
}
Replace HEADER_FORMAT with your header type.