How to convert DOS/Windows newline (CRLF) to Unix newline (LF) in a Bash script

How can I programmatically (i.e., not using vi) convert DOS/Windows newlines to Unix?

The dos2unix and unix2dos commands are not available on certain systems. How can I emulate these with commands like sed, awk, and tr?

You can use tr to convert from DOS to Unix; however, you can only do this safely if CR appears in your file only as the first byte of a CRLF byte pair. This is usually the case. You then use:

tr -d '\015' <DOS-file >UNIX-file

Note that the name DOS-file is different from the name UNIX-file; if you try to use the same name twice, you will end up with no data in the file.

You can't do it the other way round (with standard 'tr').

If you know how to enter carriage return into a script (control-V, control-M to enter control-M), then:

sed 's/^M$//'     # DOS to Unix
sed 's/$/^M/'     # Unix to DOS

where the '^M' is the control-M character. You can also use the bash ANSI-C Quoting mechanism to specify the carriage return:

sed $'s/\r$//'     # DOS to Unix
sed $'s/$/\r/'     # Unix to DOS

However, if you're going to have to do this very often (more than once, roughly speaking), it is far more sensible to install the conversion programs (e.g. dos2unix and unix2dos, or perhaps dtou and utod) and use them.

If you need to process entire directories and subdirectories, you can use zip:

zip -r -ll zipfile.zip somedir/
unzip zipfile.zip

This will create a zip archive with line endings changed from CRLF to CR. unzip will then put the converted files back in place (and ask you file by file - you can answer: Yes-to-all). Credits to @vmsnomad for pointing this out.

Use:

tr -d "\r" < file

Take a look here for examples using sed:

# In a Unix environment: convert DOS newlines (CR/LF) to Unix format.
sed 's/.$//'               # Assumes that all lines end with CR/LF
sed 's/^M$//'              # In Bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//'            # Works on ssed, gsed 3.02.80 or higher

# In a Unix environment: convert Unix newlines (LF) to DOS format.
sed "s/$/`echo -e \\\r`/"            # Command line under ksh
sed 's/$'"/`echo \\\r`/"             # Command line under bash
sed "s/$/`echo \\\r`/"               # Command line under zsh
sed 's/$/\r/'                        # gsed 3.02.80 or higher

Use sed -i for in-place conversion, e.g., sed -i 's/..../' file.

You can use Vim programmatically with the option -c {command}:

DOS to Unix:

vim file.txt -c "set ff=unix" -c ":wq"

Unix to DOS:

vim file.txt -c "set ff=dos" -c ":wq"

"set ff=unix/dos" means change fileformat (ff) of the file to Unix/DOS end of line format.

":wq" means write the file to disk and quit the editor (allowing to use the command in a loop).

Install dos2unix, then convert a file in-place with

dos2unix <filename>

To output converted text to a different file use

dos2unix -n <input-file> <output-file>

---

You can install it on Ubuntu or Debian with

sudo apt install dos2unix

or on macOS using Homebrew

brew install dos2unix

Using AWK you can do:

awk '{ sub("\r$", ""); print }' dos.txt > unix.txt

Using Perl you can do:

perl -pe 's/\r$//' < dos.txt > unix.txt

This problem can be solved with standard tools, but there are sufficiently many traps for the unwary that I recommend you install the flip command, which was written over 20 years ago by Rahul Dhesi, the author of zoo. It does an excellent job converting file formats while, for example, avoiding the inadvertant destruction of binary files, which is a little too easy if you just race around altering every CRLF you see...

If you don't have access to dos2unix, but can read this page, then you can copy/paste dos2unix.py from here.

#!/usr/bin/env python
"""\
convert dos linefeeds (crlf) to unix (lf)
usage: dos2unix.py <input> <output>
"""
import sys

if len(sys.argv[1:]) != 2:
  sys.exit(__doc__)

content = ''
outsize = 0
with open(sys.argv[1], 'rb') as infile:
  content = infile.read()
with open(sys.argv[2], 'wb') as output:
  for line in content.splitlines():
    outsize += len(line) + 1
    output.write(line + '\n')

print("Done. Saved %s bytes." % (len(content)-outsize))

_{(Cross-posted from Super User.)}

The solutions posted so far only deal with part of the problem, converting DOS/Windows' CRLF into Unix's LF; the part they're missing is that DOS use CRLF as a line separator, while Unix uses LF as a line terminator. The difference is that a DOS file (usually) won't have anything after the last line in the file, while Unix will. To do the conversion properly, you need to add that final LF (unless the file is zero-length, i.e. has no lines in it at all). My favorite incantation for this (with a little added logic to handle Mac-style CR-separated files, and not molest files that're already in unix format) is a bit of perl:

perl -pe 'if ( s/\r\n?/\n/g ) { $f=1 }; if ( $f || ! $m ) { s/([^\n])\z/$1\n/ }; $m=1' PCfile.txt

Note that this sends the Unixified version of the file to stdout. If you want to replace the file with a Unixified version, add perl's -i flag.

It is super duper easy with PCRE;

As a script, or replace $@ with your files.

#!/usr/bin/env bash
perl -pi -e 's/\r\n/\n/g' -- $@

> This will overwrite your files in place! > > I recommend only doing this with a backup (version control or otherwise)

An even simpler AWK solution without a program:

awk -v ORS='\r\n' '1' unix.txt > dos.txt

Technically '1' is your program, because AWK requires one when the given option.

Alternatively, an internal solution is:

while IFS= read -r line;
do printf '%s\n' "${line%$'\r'}";
done < dos.txt > unix.txt

Interestingly, in my Git Bash on Windows, sed "" did the trick already:

$ echo -e "abc\r" >tst.txt
$ file tst.txt
tst.txt: ASCII text, with CRLF line terminators
$ sed -i "" tst.txt
$ file tst.txt
tst.txt: ASCII text

My guess is that sed ignores them when reading lines from the input and always writes Unix line endings to the output.

I had just to ponder that same question (on Windows-side, but equally applicable to Linux).

Surprisingly, nobody mentioned a very much automated way of doing CRLF <-> LF conversion for text-files using the good old zip -ll option (Info-ZIP):

zip -ll textfiles-lf.zip files-with-crlf-eol.*
unzip textfiles-lf.zip

NOTE: this would create a ZIP file preserving the original file names, but converting the line endings to LF. Then unzip would extract the files as zip'ed, that is, with their original names (but with LF-endings), thus prompting to overwrite the local original files if any.

The relevant excerpt from the zip --help:

zip --help
...
-l   convert LF to CR LF (-ll CR LF to LF)

sed -i.bak --expression='s/\r\n/\n/g' <file_path>

Since the question mentions sed, this is the most straightforward way to use sed to achieve this. The expression says replace all carriage-returns and line-feeds with just line-feeds only. That is what you need when you go from Windows to Unix. I verified it works.

For Mac OS X if you have Homebrew installed (http://brew.sh/):

brew install dos2unix

for csv in *.csv; do dos2unix -c mac ${csv}; done;

Make sure you have made copies of the files, as this command will modify the files in place. The -c mac option makes the switch to be compatible with OS X.

This worked for me

tr "\r" "\n" < sampledata.csv > sampledata2.csv 

TIMTOWTDI!

perl -pe 's/\r\n/\n/; s/([^\n])\z/$1\n/ if eof' PCfile.txt

Based on Gordon Davisson's answer.

One must consider the possibility of [noeol]...

You can use AWK. Set the record separator (RS) to a regular expression that matches all possible newline character, or characters. And set the output record separator (ORS) to the Unix-style newline character.

awk 'BEGIN{RS="\r|\n|\r\n|\n\r";ORS="\n"}{print}' windows_or_macos.txt > unix.txt

On Linux, it's easy to convert ^M (Ctrl + M) to *nix newlines (^J) with sed.

It will be something like this on the CLI, and there will actually be a line break in the text. However, the \ passes that ^J along to sed:

sed 's/^M/\
/g' < ffmpeg.log > new.log

You get this by using ^V (Ctrl + V), ^M (Ctrl + M) and \ (backslash) as you type:

sed 's/^V^M/\^V^J/g' < ffmpeg.log > new.log

As an extension to Jonathan Leffler's Unix to DOS solution, to safely convert to DOS when you're unsure of the file's current line endings:

sed '/^M$/! s/$/^M/'

This checks that the line does not already end in CRLF before converting to CRLF.

I made a script based on the accepted answer, so you can convert it directly without needing an additional file in the end and removing and renaming afterwards.

convert-crlf-to-lf() {
    file="$1"
    tr -d '\015' <"$file" >"$file"2
    rm -rf "$file"
    mv "$file"2 "$file"
}

Just make sure if you have a file like "file1.txt" that "file1.txt2" doesn't already exist or it will be overwritten. I use this as a temporary place to store the file in.

With Bash 4.2 and newer you can use something like this to strip the trailing CR, which only uses Bash built-ins:

if [[ "${str: -1}" == $'\r' ]]; then
    str="${str:: -1}"
fi

I tried

sed 's/^M$//' file.txt

on OS X as well as several other methods (Fixing Dos Line Endings or http://hintsforums.macworld.com/archive/index.php/t-125.html). None worked, and the file remained unchanged (by the way, Ctrl + V, Enter was needed to reproduce ^M). In the end I used TextWrangler. It's not strictly command line, but it works and it doesn't complain.