`Out-File` seems to force the BOM when using UTF-8:

    $MyFile = Get-Content $MyPath
    $MyFile | Out-File -Encoding &quot;UTF8&quot; $MyPath

How can I write a file in UTF-8 with no BOM using PowerShell?

### Update 2021

PowerShell has changed a bit since I wrote this question 10 years ago. Check multiple answers below, they have a lot of good information!

Using PowerShell to write a file in UTF-8 without the BOM

I&#39;m generating an *utf-8 XML* file using `XDocument`.

    XDocument xml_document = new XDocument(
                        new XDeclaration(&quot;1.0&quot;, &quot;utf-8&quot;, null),
                        new XElement(ROOT_NAME,                    
                        new XAttribute(&quot;note&quot;, note)
                    )
                );
    ...
    xml_document.Save(@file_path);

The file is generated correctly and validated with an xsd file with success.

When I try to upload the XML file to an online service, the service says that my file is `wrong at line 1`; I have discovered that the problem is caused by the BOM on the first bytes of the file.

Do you know why the BOM is appended to the file and how can I save the file without it?  

As stated in [Byte order mark][1] Wikipedia article:

&gt; While Unicode standard allows BOM in
&gt; UTF-8  **it does not require or
&gt; recommend it**. Byte order has no
&gt; meaning in UTF-8 so a BOM only
&gt; serves to identify a text stream or
&gt; file as UTF-8 or that it was converted
&gt; from another format that has a BOM

Is it an `XDocument` problem or should I contact the guys of the online service provider to ask for a parser upgrade?  

  [1]: http://en.wikipedia.org/wiki/Byte_order_mark

XDocument: saving XML to file without BOM

I&#39;m really confused with the `codecs.open function`. When I do:

    file = codecs.open(&quot;temp&quot;, &quot;w&quot;, &quot;utf-8&quot;)
    file.write(codecs.BOM_UTF8)
    file.close()

It gives me the error

&gt; UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xef in position
&gt; 0: ordinal not in range(128)

If I do:

    file = open(&quot;temp&quot;, &quot;w&quot;)
    file.write(codecs.BOM_UTF8)
    file.close()

It works fine.

**Question** is why does the first method fail? And how do I insert the bom?

If the second method is the correct way of doing it, what the point of using `codecs.open(filename, &quot;w&quot;, &quot;utf-8&quot;)`?

Write to UTF-8 file in Python

What&#39;s different between UTF-8 and UTF-8 without a [BOM][1]? Which is better? 

  [1]: http://en.wikipedia.org/wiki/Byte_order_mark



What&#39;s the difference between UTF-8 and UTF-8 without BOM?

I&#39;ve got this very simple thing that just outputs some stuff in CSV format, but it&#39;s got to be UTF-8. I open this file in TextEdit or TextMate or Dreamweaver and it displays UTF-8 characters properly, but if I open it in Excel it&#39;s doing this silly &#237;&#196; kind of thing instead. Here&#39;s what I&#39;ve got at the head of my document:

	header(&quot;content-type:application/csv;charset=UTF-8&quot;);
	header(&quot;Content-Disposition:attachment;filename=\&quot;CHS.csv\&quot;&quot;);

This all seems to have the desired effect except Excel (Mac, 2008) doesn&#39;t want to import it properly. There&#39;s no options in Excel for me to &quot;open as UTF-8&quot; or anything, so … I&#39;m getting a little annoyed.


I can&#39;t seem to find any clear solutions to this anywhere, despite a lot of people having the same problem. The thing I see the most is to include the BOM, but I can&#39;t exactly figure out how to do that. As you can see above I&#39;m just `echo`ing this data, I&#39;m not writing any file. I can do that if I need to, I&#39;m just not because there doesn&#39;t seem like a need for it at this point. Any help?

Update: I tried echoing the BOM as `echo pack(&quot;CCC&quot;, 0xef, 0xbb, 0xbf);` which I just pulled from a site that was trying to detect the BOM. But Excel just appends those three characters to the very first cell when it imports, and still messes up the special characters.

How can I output a UTF-8 CSV in PHP that Excel will read properly?

I have javascript files that I need them to be saved in UTF-8 (without BOM), every time I convert them to the correct format in *Notepad++*, they are reverted back to UTF-8 with BOM when I open them in Visual Studio. How can I stop VS2010 from doing that?

Another question, is UTF-8 without signature in Visual Studio the same as UTF-8 without BOM?

UTF-8 without BOM

I have a CSS file that looks fine when I open it using [gedit][1], but when it&#39;s read by PHP (to merge all the CSS files into one), this CSS has the following characters prepended to it: &#239;&#187;&#191;

PHP removes all whitespace, so a random &#239;&#187;&#191; in the middle of the code messes up the entire thing. As I mentioned, I can&#39;t actually see these characters when I open the file in gedit, so I can&#39;t remove them very easily.

I googled the problem, and there is clearly something wrong with the file encoding, which makes sense being as I&#39;ve been shifting the files around to different Linux/Windows servers via ftp and [rsync][2], with a range of text editors. I don&#39;t really know much about character encoding though, so help would be appreciated.

If it helps, the file is being saved in UTF-8 format, and gedit won&#39;t let me save it in ISO-8859-15 format (the document contains one or more characters that cannot be encoded using the specified character encoding). I tried saving it with Windows and Linux line endings, but neither helped.

  [1]: http://en.wikipedia.org/wiki/Gedit
  [2]: http://en.wikipedia.org/wiki/Rsync


How do I remove &#239;&#187;&#191; from the beginning of a file?

I&#39;m trying to read CSV files using Java. Some of the files may have a byte order mark in the beginning, but not all. When present, the byte order gets read along with the rest of the first line, thus causing problems with string compares.

Is there an easy way to skip the byte order mark when it is present?

Byte order mark screws up file reading in Java

I am trying to create a text file using VB.Net with UTF8 encoding, without BOM. Can anybody help me, how to do this?
&lt;br&gt;
I can write file with UTF8 encoding but, how to remove Byte Order Mark from it?
&lt;br&gt;

edit1:
I have tried code like this;
        

    

        Dim utf8 As New UTF8Encoding()
        Dim utf8EmitBOM As New UTF8Encoding(True)
        Dim strW As New StreamWriter(&quot;c:\temp\bom\1.html&quot;, True, utf8EmitBOM)
        strW.Write(utf8EmitBOM.GetPreamble())
        strW.WriteLine(&quot;hi there&quot;)
        strW.Close()
    
            Dim strw2 As New StreamWriter(&quot;c:\temp\bom\2.html&quot;, True, utf8)
            strw2.Write(utf8.GetPreamble())
            strw2.WriteLine(&quot;hi there&quot;)
            strw2.Close()

1.html get created with UTF8 encoding only and 2.html get created with ANSI encoding format.

Simplified approach - [http://whatilearnttuday.blogspot.com/2011/10/write-text-files-without-byte-order.html][1]


  [1]: http://whatilearnttuday.blogspot.com/2011/10/write-text-files-without-byte-order.html

Write text files without Byte Order Mark (BOM)?

How would an `awk` script (presumably a one-liner) for removing a [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) look like?

Specification:

- print every line after the first (`NR &gt; 1`)
- for the first line: If it starts with `#FE #FF` or `#FF #FE`, remove those and print the rest


Using awk to remove the Byte-order mark

When I open a file in Sublime Text 3, at the bottom I have an option to set the Character Encoding as shown in the screenshot.

![setting encoding][1]

There is the option to set it to **UTF-8** , which after doing some research means UTF-8 Without BOM, but I want to set it to **UTF-8 With BOM** like shown below:

![UTF-8 with BOM][2]

How can I do this from within ST3?


  [1]: http://i.stack.imgur.com/FNvMj.jpg
  [2]: http://i.stack.imgur.com/5F4qu.jpg

Set Encoding of File to UTF8 With BOM in Sublime Text 3

Two questions here. I have a set of files which are usually UTF-8 with BOM. I&#39;d like to convert them (ideally in place) to UTF-8 with no BOM. It seems like `codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors)` would handle this. But I don&#39;t really see any good examples on usage. Would this be the best way to handle this?  

    source files:
    Tue Jan 17$ file brh-m-157.json 
    brh-m-157.json: UTF-8 Unicode (with BOM) text

Also, it would be ideal if we could handle different input encoding wihtout explicitly knowing (seen ASCII and UTF-16). It seems like this should all be feasible. Is there a solution that can take any known Python encoding and output as UTF-8 without BOM?

**edit 1**  proposed sol&#39;n from below (thanks!)

    fp = open(&#39;brh-m-157.json&#39;,&#39;rw&#39;)
    s = fp.read()
    u = s.decode(&#39;utf-8-sig&#39;)
    s = u.encode(&#39;utf-8&#39;)
    print fp.encoding  
    fp.write(s)


This gives me the following error:  

    IOError: [Errno 9] Bad file descriptor

###Newsflash
I&#39;m being told in comments that the mistake is I open the file with mode &#39;rw&#39; instead of &#39;r+&#39;/&#39;r+b&#39;, so I should eventually re-edit my question and remove the solved part.

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

I try to detect which character encoding is used in my file.

I try with this code to get the standard encoding 

    public static Encoding GetFileEncoding(string srcFile)
        {
          // *** Use Default of Encoding.Default (Ansi CodePage)
          Encoding enc = Encoding.Default;
    
          // *** Detect byte order mark if any - otherwise assume default
          byte[] buffer = new byte[5];
          FileStream file = new FileStream(srcFile, FileMode.Open);
          file.Read(buffer, 0, 5);
          file.Close();
    
          if (buffer[0] == 0xef &amp;&amp; buffer[1] == 0xbb &amp;&amp; buffer[2] == 0xbf)
            enc = Encoding.UTF8;
          else if (buffer[0] == 0xfe &amp;&amp; buffer[1] == 0xff)
            enc = Encoding.Unicode;
          else if (buffer[0] == 0 &amp;&amp; buffer[1] == 0 &amp;&amp; buffer[2] == 0xfe &amp;&amp; buffer[3] == 0xff)
            enc = Encoding.UTF32;
          else if (buffer[0] == 0x2b &amp;&amp; buffer[1] == 0x2f &amp;&amp; buffer[2] == 0x76)
            enc = Encoding.UTF7;
          else if (buffer[0] == 0xFE &amp;&amp; buffer[1] == 0xFF)      
            // 1201 unicodeFFFE Unicode (Big-Endian)
            enc = Encoding.GetEncoding(1201);      
          else if (buffer[0] == 0xFF &amp;&amp; buffer[1] == 0xFE)      
            // 1200 utf-16 Unicode
            enc = Encoding.GetEncoding(1200);
          
    
          return enc;
        }
My five first byte are 60, 118, 56, 46 and 49.

Is there a chart that shows which encoding matches those five first bytes?



How to detect the character encoding of a text file?

I need to add a UTF-8 byte-order-mark to generated text data on client side. How do I do that?

Using `new Blob([&#39;\xEF\xBB\xBF&#39; + content])` yields `&#39;&#239;&#187;&#191;&quot;my data&quot;&#39;`,  of course.

Neither did `&#39;\uBBEF\x22BF&#39;` work (with `&#39;\x22&#39; == &#39;&quot;&#39;` being the next character in `content`).

Is it possible to prepend the UTF-8 BOM in JavaScript to a generated text?

&lt;sup&gt;Yes, I really do need the UTF-8 BOM in this case.&lt;/sup&gt;

Adding UTF-8 BOM to string/Blob

I tried [this aproach][1] without any success 


the code I&#39;m using:

    // File name
    String filename = String.Format(&quot;{0:ddMMyyHHmm}&quot;, dtFileCreated);
    String filePath = Path.Combine(Server.MapPath(&quot;App_Data&quot;), filename + &quot;.txt&quot;);

    // Process       
    myObject pbs = new myObject();         
    pbs.GenerateFile();

    // pbs.GeneratedFile is a StringBuilder object

    // Save file
    Encoding utf8WithoutBom = new UTF8Encoding(true);
    TextWriter tw = new StreamWriter(filePath, false, utf8WithoutBom);
    foreach (string s in pbs.GeneratedFile.ToArray()) 
        tw.WriteLine(s);
    tw.Close();

    // Push Generated File into Client
    Response.Clear();
    Response.ContentType = &quot;application/vnd.text&quot;;
    Response.AppendHeader(&quot;Content-Disposition&quot;, &quot;attachment; filename=&quot; + filename + &quot;.txt&quot;);
    Response.TransmitFile(filePath);
    Response.End();

the result:

![enter image description here][2]

It&#39;s [writing the BOM][3] no matter what, and special chars (like &#198; &#216; &#197;) 
are not correct :-/


**I&#39;m stuck!**

My objective is create a file using **UTF-8** as Encoding and **8859-1** as CharSet

Is this so hard to accomplish or I&#39;m just getting a bad day?




*All help is greatly appreciated, thank you!*


  [1]: https://stackoverflow.com/questions/2437666/write-text-files-without-byte-order-mark-bom
  [2]: http://i.stack.imgur.com/4Pwhm.png
  [3]: http://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding

Create Text File Without BOM

I am using xsl to control the output of my xml file, but the BOM character is being added.

How do I remove the BOM character from my xml file

Using PHP5 (cgi) to output template files from the filesystem and having issues spitting out raw HTML.

    private function fetch($name) {
        $path = $this-&gt;j-&gt;config[&#39;template_path&#39;] . $name . &#39;.html&#39;;
        if (!file_exists($path)) {
            dbgerror(&#39;Could not find the template &quot;&#39; . $name . &#39;&quot; in &#39; . $path);
        }
        $f = fopen($path, &#39;r&#39;);
        $t = fread($f, filesize($path));
        fclose($f);
        if (substr($t, 0, 3) == b&#39;\xef\xbb\xbf&#39;) {
            $t = substr($t, 3);
        }
        return $t;
    }

Even though I&#39;ve added the BOM fix I&#39;m still having problems with Firefox accepting it. You can see a live copy here: http://ircb.in/jisti/ (and the template file I threw at http://ircb.in/jisti/home.html if you want to check it out)

Any idea how to fix this? o_o

How to remove multiple UTF-8 BOM sequences

I&#39;m searching (without success) for a script, which would work as a batch file and allow me to prepend a UTF-8 text file with a BOM if it doesn&#39;t have one.

Neither the language it is written in (perl, python, c, bash) nor the OS it works on, matters to me. I have access to a wide range of computers.

I&#39;ve found a lot of scripts to do the reverse (strip the BOM), which sounds to me as kind of silly, as many Windows program will have trouble reading UTF-8 text files if they don&#39;t have a BOM.

Did I miss the obvious?

Thanks!

Adding BOM to UTF-8 files

I&#39;m having a problem with UTF8 encoding in my asp.net mvc 2 application in C#. I&#39;m trying let user download a simple text file from a string. I am trying to get bytes array with the following line:

`var x = Encoding.UTF8.GetBytes(csvString);`

but when I return it for download using: 

`return File(x, ..., ...);`

I get a file which is without BOM so I don&#39;t get Croatian characters shown up correctly. This is because my bytes array does not include BOM after encoding. I triend inserting those bytes manually and then it shows up correctly, but that&#39;s not the best way to do it.

I also tried creating UTF8Encoding class instance and passing a boolean value (true) to its constructor to include BOM, but it doesn&#39;t work either.

Anyone has a solution? Thanks!

All Byte Order-Mark Solutions on Gang of Coders

Total of 19 Byte Order-Mark Solutions