Removing colors from output

Bash Problem Overview

I have some script that produces output with colors and I need to remove the ANSI codes.

#!/bin/bash

exec > >(tee log)   # redirect the output to a file but keep it on stdout
exec 2>&1

./somescript

The output is (in log file):

java (pid  12321) is running...@[60G[@[0;32m  OK  @[0;39m]

I didn't know how to put the ESC character here, so I put @ in its place.

I changed the script into:

#!/bin/bash

exec > >(tee log)   # redirect the output to a file but keep it on stdout
exec 2>&1

./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"

But now it gives me (in log file):

java (pid  12321) is running...@[60G[  OK  ]

How can I also remove this '@[60G?

Maybe there is a way to completely disable coloring for the entire script?

Bash Solutions

Solution 1 - Bash

According to Wikipedia, the [m|K] in the sed command you're using is specifically designed to handle m (the color command) and K (the "erase part of line" command). Your script is trying to set absolute cursor position to 60 (^[[60G) to get all the OKs in a line, which your sed line doesn't cover.

(Properly, [m|K] should probably be (m|K) or [mK], because you're not trying to match a pipe character. But that's not important right now.)

If you switch that final match in your command to [mGK] or (m|G|K), you should be able to catch that extra control sequence.

./somescript | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g"

Solution 2 - Bash

IMHO, most of these answers try too hard to restrict what is inside the escape code. As a result, they end up missing common codes like [38;5;60m (foreground ANSI color 60 from 256-color mode).

They also require the -r option which enables GNU extensions. These are not required; they just make the regex read better.

Here is a simpler answer that handles the 256-color escapes and works on systems with non-GNU sed:

./somescript | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g'

This will catch anything that starts with [, has any number of decimals and semicolons, and ends with a letter. This should catch any of the common ANSI escape sequences.

For funsies, here's a larger and more general (but minimally tested) solution for all conceivable ANSI escape sequences:

./somescript | sed 's/\x1B[@A-Z\\\]^_]\|\x1B\[[0-9:;<=>?]*[-!"#$%&'"'"'()*+,.\/]*[][\\@A-Z^_`a-z{|}~]//g'

(and if you have @edi9999's SI problem, add | sed "s/\x0f//g" to the end; this works for any control char by replacing 0f with the hex of the undesired char)

Solution 3 - Bash

I couldn't get decent results from any of the other answers, but the following worked for me:

somescript | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g"

If I only removed the control char "^[", it left the rest of the color data, e.g., "33m". Including the color code and "m" did the trick. I'm puzzled with s/\x1B//g doesn't work because \x1B[31m certainly works with echo.

Solution 4 - Bash

For Mac OSX or BSD use

./somescript | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

Solution 5 - Bash

The regular expression below will miss some ANSI Escape Codes sequences, as well as 3 digit colors. Example and Fix on regex101.com.

Use this instead:

./somescript | sed -r 's/\x1B\[(;?[0-9]{1,3})+[mGK]//g'

I also had the problem that sometimes, the SI character appeared.

It happened for example with this input : echo "$(tput setaf 1)foo$(tput sgr0) bar"

Here's a way to also strip the SI character (shift in) (0x0f)

./somescript | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | sed "s/\x0f//g"

Solution 6 - Bash

I came across ansi2txt tool from colorized-logs package in Debian. The tool drops ANSI control codes from STDIN.

Usage example:

./somescript | ansi2txt

Source code http://github.com/kilobyte/colorized-logs

Solution 7 - Bash

Much simpler function in pure Bash to filter-out common ANSI codes from a text stream:

# Strips common ANSI codes from a text stream

shopt -s extglob # Enable Bash Extended Globbing expressions
ansi_filter() {
  local line
  local IFS=
  while read -r line || [[ "$line" ]]; do
    echo "${line//$'\e'[\[(]*([0-9;])[@-n]/}"
  done
}

See:

Solution 8 - Bash

I had a similar problem. All solutions I found did work well for the color codes but did not remove the characters added by "$(tput sgr0)" (resetting attributes).

Taking, for example, the solution in the comment by davemyron the length of the resulting string in the example below is 9, not 6:

#!/usr/bin/env bash

string="$(tput setaf 9)foobar$(tput sgr0)"
string_sed="$( sed -r "s/\x1B\[[0-9;]*[JKmsu]//g" <<< "${string}" )"
echo ${#string_sed}

In order to work properly, the regex had to be extend to also match the sequence added by sgr0 ("\E(B"):

string_sed="$( sed -r "s/\x1B(\[[0-9;]*[JKmsu]|\(B)//g" <<< "${string}" )"

Solution 9 - Bash

Hmm, not sure if this will work for you, but 'tr' will 'strip' (delete) control codes - try:

./somescript | tr -d '[:cntrl:]'

Solution 10 - Bash

Here's a pure Bash solution.

Save as strip-escape-codes.sh, make executable and then run <command-producing-colorful-output> | ./strip-escape-codes.sh.

Note that this strips all ANSI escape codes/sequences. If you want to strip colors only, replace [a-zA-Z] with "m".

Bash >= 4.0:

#!/usr/bin/env bash

# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
    local _input="$1" _i _char _escape=0
    local -n _output="$2"; _output=""
    for (( _i=0; _i < ${#_input}; _i++ )); do
        _char="${_input:_i:1}"
        if (( ${_escape} == 1 )); then
            if [[ "${_char}" == [a-zA-Z] ]]; then
                _escape=0
            fi
            continue
        fi
        if [[ "${_char}" == $'\e' ]]; then
            _escape=1
            continue
        fi
        _output+="${_char}"
    done
}

while read -r line; do
    strip_escape_codes "${line}" line_stripped
    echo "${line_stripped}"
done

Bash < 4.0:

#!/usr/bin/env bash

# Strip ANSI escape codes/sequences [$1: input string, $2: target variable]
function strip_escape_codes() {
    local input="${1//\"/\\\"}" output="" i char escape=0
    for (( i=0; i < ${#input}; ++i )); do         # process all characters of input string
        char="${input:i:1}"                       # get current character from input string
        if (( ${escape} == 1 )); then             # if we're currently within an escape sequence, check if
            if [[ "${char}" == [a-zA-Z] ]]; then  # end is reached, i.e. if current character is a letter
                escape=0                          # end reached, we're no longer within an escape sequence
            fi
            continue                              # skip current character, i.e. do not add to ouput
        fi
        if [[ "${char}" == $'\e' ]]; then         # if current character is '\e', we've reached the start
            escape=1                              # of an escape sequence -> set flag
            continue                              # skip current character, i.e. do not add to ouput
        fi
        output+="${char}"                         # add current character to output
    done
    eval "$2=\"${output}\""                       # assign output to target variable
}

while read -r line; do
    strip_escape_codes "${line}" line_stripped
    echo "${line_stripped}"
done

Solution 11 - Bash

@jeff-bowman's solution helped me getting rid of SOME of the color codes. I added another small portion to the regex in order to remove some more:

sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # Original. Removed Red ([31;40m[1m[error][0m)
sed -r "s/\x1B\[([0-9];)?([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" # With an addition, removed yellow and green ([1;33;40m[1m[warning][0m and [1;32;40m[1m[ok][0m)
                ^^^^^^^^^
                remove Yellow and Green (and maybe more colors)

Solution 12 - Bash

The controversial idea would be to reconfigure terminal settings for this process environment to let the process know that terminal does not support colors.

Something like TERM=xterm-mono ./somescript comes to my mind. YMMV with your specific OS and ability of your script to understand terminal color settings.

Solution 13 - Bash

Not sure what's in ./somescript but if escape sequences are not hardcoded you can set the terminal type to avoid them

TERM=dumb ./somescript

For example, if you try

TERM=dumb tput sgr0 | xxd

you'll see it produces no output while

tput sgr0 | xxd
00000000: 1b28 421b 5b6d                           .(B.[m

does (for xterm-256color).

Solution 14 - Bash

There's also a dedicated tool to handle ANSI escape sequences: ansifilter. Use the default --text output format to strip all ANSI escape sequences (note: not just coloring).

ref: https://stackoverflow.com/a/6534712

Solution 15 - Bash

I came across this question/answers trying to do something similar as the OP. I found some other useful resources and came up with a log script based on those. Posting here in case it can help others.

Digging into the links helps understand some of the redirection which I won't try and explain because I'm just starting to understand it myself.

Usage will render the colorized output to the console, while stripping the color codes out of the text going to the log file. It will also include stderr in the logfile for any commands that don't work.

Edit: adding more usage at bottom to show how to log in different ways

#!/bin/bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

. $DIR/dev.conf
. $DIR/colors.cfg

filename=$(basename ${BASH_SOURCE[0]})
# remove extension
# filename=`echo $filename | grep -oP '.*?(?=\.)'`
filename=`echo $filename | awk -F\. '{print $1}'`
log=$DIR/logs/$filename-$target

if [ -f $log ]; then
  cp $log "$log.bak"
fi

exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>$log 2>&1


# log message
log(){
	local m="$@"
	echo -e "*** ${m} ***" >&3
	echo "=================================================================================" >&3
  local r="$@"
	echo "================================================================================="
	echo -e "*** $r ***" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g"
	echo "================================================================================="
}

echo "=================================================================================" >&3
log "${Cyan}The ${Yellow}${COMPOSE_PROJECT_NAME} ${filename} ${Cyan}script has been executed${NC}"
log $(ls) #log $(<command>)

log "${Green}Apply tag to image $source with version $version${NC}"
# log $(exec docker tag $source $target 3>&2) #prints error only to console
# log $(docker tag $source $target 2>&1) #prints error to both but doesn't exit on fail
log $(docker tag $source $target 2>&1) && exit $? #prints error to both AND exits on fail
# docker tag $source $target 2>&1 | tee $log # prints gibberish to log
echo $? # prints 0 because log function was successful
log "${Purple}Push $target to acr${NC}"

Here are the other links that helped:

Solution 16 - Bash

I used perl as I have to do this frequently on many files. This will go through all files with filename*.txt and will remove any formatting. This works for my use case and may be useful for someone else too so just thought of posting here. replace whatever your file name is in place of filename*.txt or you can put file names separated by spaces in setting the FILENAME variable below.

$ FILENAME=$(ls filename*.txt) ; for file in $(echo $FILENAME); do echo $file; cat $file | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > $file-new; mv $file-new $file; done

Solution 17 - Bash

my contribution:

./somescript | sed -r "s/\\x1B[\\x5d\[]([0-9]{1,3}(;[0-9]{1,3})?(;[0-9]{1,3})?)?[mGK]?//g"

Solution 18 - Bash

I had some issues with colorized output which the other solutions here didn't process correctly, so I built this perl one liner. It looks for escape \e followed by opening bracket \[ followed by one or color codes \d+ separated by semicolons, ending on m.

perl -ple 's/\e\[\d+(;\d+)*m//g'

It seems to work really well for colorized compiler output.

Solution 19 - Bash

This works for me:

./somescript | cat

Content Type	Original Author	Original Content on Stackoverflow
Question	Pawel P.	View Question on Stackoverflow
Solution 1 - Bash	Jeff Bowman	View Answer on Stackoverflow
Solution 2 - Bash	meustrus	View Answer on Stackoverflow
Solution 3 - Bash	JoeAndrieu	View Answer on Stackoverflow
Solution 4 - Bash	grebulon	View Answer on Stackoverflow
Solution 5 - Bash	edi9999	View Answer on Stackoverflow
Solution 6 - Bash	user4674453	View Answer on Stackoverflow
Solution 7 - Bash	Léa Gris	View Answer on Stackoverflow
Solution 8 - Bash	Jarodiv	View Answer on Stackoverflow
Solution 9 - Bash	Dale_Reagan	View Answer on Stackoverflow
Solution 10 - Bash	Maxxim	View Answer on Stackoverflow
Solution 11 - Bash	zstolar	View Answer on Stackoverflow
Solution 12 - Bash	A B	View Answer on Stackoverflow
Solution 13 - Bash	Diego Torres Milano	View Answer on Stackoverflow
Solution 14 - Bash	Juan	View Answer on Stackoverflow
Solution 15 - Bash	Carlos Soriano	View Answer on Stackoverflow
Solution 16 - Bash	jimmax777	View Answer on Stackoverflow
Solution 17 - Bash	stu	View Answer on Stackoverflow
Solution 18 - Bash	CodeMonkey	View Answer on Stackoverflow
Solution 19 - Bash	spiderlama	View Answer on Stackoverflow