<0xEF,0xBB,0xBF> character showing up in files. How to remove them?

I am doing compressing of JavaScript files and the compressor is complaining that my files have  character in them.

How can I search for these characters and remove them?

You can easily remove them using vim, here are the steps:

1. In your terminal, open the file using vim:

   vim file_name

2. Remove all BOM characters:

   :set nobomb

3. Save the file:

   :wq

Another method to remove those characters - using Vim:

> vim -b fileName

Now those "hidden" characters are visible (<feff>) and can be removed.

Thanks for the previous answers, here's a sed(1) variant just in case:

sed '1s/^\xEF\xBB\xBF//'

perl -pi~ -CSD -e 's/^\x{fffe}//' file1.js path/to/file2.js

I would assume the tool will break if you have other utf-8 in your files, but if not, perhaps this workaround can help you. (Untested ...)

Edit: added the -CSD option, as per tchrist's comment.

On Unix/Linux:

sed 's/\xEF\xBB\xBF//' < inputfile > outputfile

On MacOSX

sed $'s/\xEF\xBB\xBF//' < inputfile > outputfile

Notice the $ after sed for mac.

On Windows

There is Super Sed an enhanced version of sed. For Windows this is a standalone .exe, intended for running from the command line.

Using tail might be easier:

tail --bytes=+4 filename > new_filename

@tripleee's solution didn't work for me. But changing the file encoding to ASCII and again to UTF-8 did the trick :-)

I've used vimgrep for this

:vim "[\uFEFF]" *

also normal vim search command

/[\uFEFF]

The 'file' command shows if the BOM is present:

For example: 'file myfile.xml' displays: "XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long lines, with CRLF line terminators"

dos2unix will remove the BOM.

In windows you could use backported recode utility from UnxUtils.

In Sublime Text you can install the Highlighter package and then customize the regular expression in your user settings.

Here I added \uFEFF to the end of the highlighter_regex property.

{
	"highlighter_enabled": true,
	"highlighter_regex": "(\t+ +)|( +\t+)|[\u2026\u2018\u2019\u201c\u201d\u2013\u2014\uFEFF]|[\t ]+$",
	"highlighter_scope_name": "invalid",
	"highlighter_max_file_size": 1048576,
	"highlighter_delay": 3000
}

To overwrite the default package settings place the file here:

> ~/.config/sublime-text-3/Packages/User/highlighter.sublime-settings

I'm suggest the use of "dos2unix" tool, please test to run dos2unix ./thefile.js.

If necessary try to use something like this for multiple files:

for x in $(find . -type f -exec echo {} +); do dos2unix $x ; done

My Regards.

Save the file without code signature.