I have php document signup.php which save the content from form (in form.php document) to MySQL base. The problem arises when I want to reformat the input content. I want do decode UTF-8 charachters like  &#224;-&gt;a.

      $first_name=$_POST[&#39;first_name&#39;];
      $last_name=$_POST[&#39;last_name&#39;];
      $course=$_POST[&#39;course&#39;];
    
      $chain=&quot;pr&#234;t-&#224;-porter&quot;;
    
    $pattern = array(&quot;&#39;&#233;&#39;&quot;, &quot;&#39;&#232;&#39;&quot;, &quot;&#39;&#235;&#39;&quot;, &quot;&#39;&#234;&#39;&quot;, &quot;&#39;&#201;&#39;&quot;, &quot;&#39;&#200;&#39;&quot;, &quot;&#39;&#203;&#39;&quot;, &quot;&#39;&#202;&#39;&quot;, &quot;&#39;&#225;&#39;&quot;, &quot;&#39;&#224;&#39;&quot;, &quot;&#39;&#228;&#39;&quot;, &quot;&#39;&#226;&#39;&quot;, &quot;&#39;&#229;&#39;&quot;, &quot;&#39;&#193;&#39;&quot;, &quot;&#39;&#192;&#39;&quot;, &quot;&#39;&#196;&#39;&quot;, &quot;&#39;&#194;&#39;&quot;, &quot;&#39;&#197;&#39;&quot;, &quot;&#39;&#243;&#39;&quot;, &quot;&#39;&#242;&#39;&quot;, &quot;&#39;&#246;&#39;&quot;, &quot;&#39;&#244;&#39;&quot;, &quot;&#39;&#211;&#39;&quot;, &quot;&#39;&#210;&#39;&quot;, &quot;&#39;&#214;&#39;&quot;, &quot;&#39;&#212;&#39;&quot;, &quot;&#39;&#237;&#39;&quot;, &quot;&#39;&#236;&#39;&quot;, &quot;&#39;&#239;&#39;&quot;, &quot;&#39;&#238;&#39;&quot;, &quot;&#39;&#205;&#39;&quot;, &quot;&#39;&#204;&#39;&quot;, &quot;&#39;&#207;&#39;&quot;, &quot;&#39;&#206;&#39;&quot;, &quot;&#39;&#250;&#39;&quot;, &quot;&#39;&#249;&#39;&quot;, &quot;&#39;&#252;&#39;&quot;, &quot;&#39;&#251;&#39;&quot;, &quot;&#39;&#218;&#39;&quot;, &quot;&#39;&#217;&#39;&quot;, &quot;&#39;&#220;&#39;&quot;, &quot;&#39;&#219;&#39;&quot;, &quot;&#39;&#253;&#39;&quot;, &quot;&#39;&#255;&#39;&quot;, &quot;&#39;&#221;&#39;&quot;, &quot;&#39;&#248;&#39;&quot;, &quot;&#39;&#216;&#39;&quot;, &quot;&#39;œ&#39;&quot;, &quot;&#39;Œ&#39;&quot;, &quot;&#39;&#198;&#39;&quot;, &quot;&#39;&#231;&#39;&quot;, &quot;&#39;&#199;&#39;&quot;);
    
    $replace = array(&#39;e&#39;, &#39;e&#39;, &#39;e&#39;, &#39;e&#39;, &#39;E&#39;, &#39;E&#39;, &#39;E&#39;, &#39;E&#39;, &#39;a&#39;, &#39;a&#39;, &#39;a&#39;, &#39;a&#39;, &#39;a&#39;, &#39;A&#39;, &#39;A&#39;, &#39;A&#39;, &#39;A&#39;, &#39;A&#39;, &#39;o&#39;, &#39;o&#39;, &#39;o&#39;, &#39;o&#39;, &#39;O&#39;, &#39;O&#39;, &#39;O&#39;, &#39;O&#39;, &#39;i&#39;, &#39;i&#39;, &#39;i&#39;, &#39;I&#39;, &#39;I&#39;, &#39;I&#39;, &#39;I&#39;, &#39;I&#39;, &#39;u&#39;, &#39;u&#39;, &#39;u&#39;, &#39;u&#39;, &#39;U&#39;, &#39;U&#39;, &#39;U&#39;, &#39;U&#39;, &#39;y&#39;, &#39;y&#39;, &#39;Y&#39;, &#39;o&#39;, &#39;O&#39;, &#39;a&#39;, &#39;A&#39;, &#39;A&#39;, &#39;c&#39;, &#39;C&#39;); 
    
    $chain = preg_replace($pattern, $replace, $chain);
    
    echo $chain; // print pret-a-porter
    
    $first_name =  preg_replace($pattern, $replace, $first_name);
    
    echo $first_name; // does not change the input!?!


Why it works perfectly for $chain, but for $first_name or $last_name doesnt work?




Also i try

    echo $first_name; // print &#225;&#225;&#225;&#225;&#225;&#225;b&#233;&#233;&#233;&#233;&#233;&#233;bšššš
    $trans = array(&quot;&#225;&quot; =&gt; &quot;a&quot;, &quot;&#233;&quot; =&gt; &quot;e&quot;, &quot;š&quot; =&gt; &quot;s&quot;);
    echo strtr(&quot;&#225;&#225;&#225;&#225;&#225;&#225;b&#233;&#233;&#233;&#233;&#233;&#233;bšššš&quot;, $trans); // print aaaaaabeeeeeebssss
    echo strtr($first_name,$trans);  // print &#225;&#225;&#225;&#225;&#225;&#225;b&#233;&#233;&#233;&#233;&#233;&#233;bšššš

but the problem, as you can see, is same!

PHP replacing special characters like &#224;-&gt;a, &#232;-&gt;e

How to write a Base64-encoded image to file?

I have encoded an image to a string using Base64.
First, I read the file, then convert it to a byte array and then apply Base64 encoding to convert the image to a string.

Now my problem is how to decode it.

    byte dearr[] = Base64.decodeBase64(crntImage);
    File outF = new File(&quot;c:/decode/abc.bmp&quot;);
    BufferedImage img02 = ImageIO.write(img02, &quot;bmp&quot;, outF); 

The variable `crntImage` contains the string representation of the image.


Write Base64-encoded image to file

How to decode JSON in Flutter?

The question is simple, but the answer isn&#39;t, at least for me.

I have a project that uses a lot of JSON Strings. Basically, the entire communication between the app and the server is through JSON.

I have been using `JSON.decode(json_string)` to deal with it, but today I updated the Flutter core (0.5.8-pre.178) and JSON.decode isn&#39;t available anymore.

I went to the [Flutter Docs][1] to seek help, but it still says to use JSON.decode.

So, how to decode JSON in Flutter from now on?


  [1]: https://flutter.io/json/

How to decode JSON in Flutter?

How do you decode Base64 data in Python?

I have strings like 

    var str = &#39;One &amp;amp; two &amp;amp; three&#39;;

rendered into HTML by the web server. I need to transform those strings into 

    &#39;One &amp; two &amp; three&#39;

Currently, that&#39;s what I am doing (with help of jQuery):

    $(document.createElement(&#39;div&#39;)).html(&#39;{{ driver.person.name }}&#39;).text()

However I have an unsettling feeling that I am doing it wrong.
I have tried 

    unescape(&quot;&amp;amp;&quot;)

but it doesn&#39;t seem to work, neither do decodeURI/decodeURIComponent.

Are there any other, more native and elegant ways of doing so?

Decode &amp;amp; back to &amp; in JavaScript

I&#39;m trying to get a Python 3 program to do some manipulations with a text file filled with information. However, when trying to read the file I get the following error:
```
Traceback (most recent call last):  
   File &quot;SCRIPT LOCATION&quot;, line NUMBER, in &lt;module&gt;  
     text = file.read()` 
   File &quot;C:\Python31\lib\encodings\cp1252.py&quot;, line 23, in decode  
     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: &#39;charmap&#39; codec can&#39;t decode byte 0x90 in position 2907500: character maps to `&lt;undefined&gt;`  
```

UnicodeDecodeError: &#39;charmap&#39; codec can&#39;t decode byte X in position Y: character maps to &lt;undefined&gt;

I am writing a Python (Python 3.3) program to send some data to a webpage using POST method.  Mostly for debugging process I am getting the page result and displaying it on the screen using `print()` function.

The code is like this:

    conn.request(&quot;POST&quot;, resource, params, headers)
    response = conn.getresponse()
    print(response.status, response.reason)
    data = response.read()
    print(data.decode(&#39;utf-8&#39;));

the `HTTPResponse` `.read()` method returns a `bytes` element encoding the page (which is a well formated UTF-8 document)  It seemed okay until I stopped using IDLE GUI for Windows and used the Windows console instead.  The returned page has a U+2014 character (em-dash) which the print function translates well in the Windows GUI (I presume Code Page 1252) but does not in the Windows Console (Code Page 850).  Given the `strict` default behavior I get the following error:

    UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode character &#39;\u2014&#39; in position 10248: character maps to &lt;undefined&gt;

I could fix it using this quite ugly code:

    print(data.decode(&#39;utf-8&#39;).encode(&#39;cp850&#39;,&#39;replace&#39;).decode(&#39;cp850&#39;))

Now it replace the offending character &quot;—&quot; with a `?`.  Not the ideal case (a hyphen should be a better replacement) but good enough for my purpose.

There are several things I do not like from my solution.

 1. The code is ugly with all that decoding, encoding, and decoding.
 2. It solves the problem for just this case.  If I port the program for a system using some other encoding (latin-1, cp437, back to cp1252, etc.) it should recognize the target encoding.  It does not.  (for instance, when using again the IDLE GUI, the emdash is also lost, which didn&#39;t happen before)
 3. It would be nicer if the emdash translated to a hyphen instead of a interrogation bang.

The problem is not the emdash (I can think of several ways to solve that particularly problem) but I need to write robust code.  I am feeding the page with data from a database and that data can come back.  I can anticipate many other conflicting cases: an &#39;&#193;&#39; U+00c1 (which is possible in my database) could translate into CP-850 (DOS/Windows Console encodign for Western European Languages) but not into CP-437 (encoding for US English, which is default in many Windows instalations).

So, the question:

Is there a nicer solution that makes my code agnostic from the output interface encoding?

UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode - character maps to &lt;undefined&gt;, print function

I want to encode a string into `base64` and transfer it through a socket and decode it back. 

But after decoding it gives different answer. 

Following is my code and result is &quot;77+9x6s=&quot;


    import javax.xml.bind.DatatypeConverter;
    
        public class f{
        
           public static void main(String a[]){
           
              String str = new String(DatatypeConverter.parseBase64Binary(&quot;user:123&quot;));
              String res = DatatypeConverter.printBase64Binary(str.getBytes());
              System.out.println(res);
           }
        }

Any idea about how to implement this?

Base64 Java encode and decode a string

Basically I would like to decode a given Html document, and replace all special chars, such as `&quot;&amp;nbsp;&quot;` -&gt; `&quot; &quot;`, `&quot;&amp;gt;&quot;` -&gt; `&quot;&gt;&quot;`.

In .NET we can make use of `HttpUtility.HtmlDecode`. 

What&#39;s the equivalent function in Java?



How to unescape HTML character entities in Java?

I want to base64 encode data to put it in a URL and then decode it within my HttpHandler.

I have found that [Base64 Encoding](http://en.wikipedia.org/wiki/Base64) allows for a &#39;/&#39; character which will mess up my UriTemplate matching.  Then I found that there is a concept of a &quot;modified Base64 for URL&quot; from wikipedia:

A modified Base64 for URL variant exists, where no padding &#39;=&#39; will be used, and the &#39;+&#39; and &#39;/&#39; characters of standard Base64 are respectively replaced by &#39;-&#39; and &#39;_&#39;, so that using URL encoders/decoders is no longer necessary and has no impact on the length of the encoded value, leaving the same encoded form intact for use in relational databases, web forms, and object identifiers in general.

Using .NET I want to modify my current code from doing basic base64 encoding and decoding to using the &quot;modified base64 for URL&quot; method.  Has anyone done this?

To decode, I know it starts out with something like:

    string base64EncodedText = base64UrlEncodedText.Replace(&#39;-&#39;, &#39;+&#39;).Replace(&#39;_&#39;, &#39;/&#39;);

    // Append &#39;=&#39; char(s) if necessary - how best to do this?

    // My normal base64 decoding now uses encodedText

But, I need to potentially add one or two &#39;=&#39; chars to the end which looks a little more complex.

My encoding logic should be a little simpler:

    // Perform normal base64 encoding
    byte[] encodedBytes = Encoding.UTF8.GetBytes(unencodedText);
    string base64EncodedText = Convert.ToBase64String(encodedBytes);

    // Apply URL variant
    string base64UrlEncodedText = base64EncodedText.Replace(&quot;=&quot;, String.Empty).Replace(&#39;+&#39;, &#39;-&#39;).Replace(&#39;/&#39;, &#39;_&#39;);
    
I have seen the [Guid to Base64 for URL](https://stackoverflow.com/questions/1032376/guid-to-base64-for-url) StackOverflow entry, but that has a known length and therefore they can hardcode the number of equal signs needed at the end.

Code for decoding/encoding a modified base64 URL (in ASP.NET Framework)

I&#39;m POSTing the contents of a form field via AJAX to a PHP script and using JavaScript to `escape(field_contents)`.  The problem is that any plus signs are being stripped out and replaced by spaces.  How can I safely &#39;encode&#39; the plus sign and then appropriately &#39;decode&#39; it on the PHP side?

AJAX POST and Plus Sign ( + ) -- How to Encode?

I&#39;m not sure what this is called so I&#39;m having trouble searching for it. How can I decode a string with unicode from `http\u00253A\u00252F\u00252Fexample.com` to `http://example.com` with JavaScript? I tried `unescape`, `decodeURI`, and `decodeURIComponent` so I guess the only thing left is string replace.

EDIT: The string is not typed, but rather a substring from another piece of code. So to solve the problem you have to start with something like this:

    var s = &#39;http\\u00253A\\u00252F\\u00252Fexample.com&#39;;

I hope that shows why unescape() doesn&#39;t work.

How do I decode a string with escaped unicode?

&gt; **TL;DR**: I need a way to decode a QR-code from an image file using (preferable pure) Python.

I&#39;ve got a jpg file with a QR-code which I want to decode using Python. I&#39;ve found a couple libraries which claim to do this:

**PyQRCode** ([website here][1]) which supposedly can decode qr codes from images by simply providing a path like this:

    import sys, qrcode
    d = qrcode.Decoder()
    if d.decode(&#39;out.png&#39;):
        print &#39;result: &#39; + d.result
    else:
        print &#39;error: &#39; + d.error

So I simply installed it using `sudo pip install pyqrcode`. The thing I find strange about the example code above however, is that it only imports `qrcode` (and not `pyqrcode` though) Since I think `qrcode` refers to [this library][2] which can only *generate* qr-code images it kind of confused me. So I tried the code above with both `pyqrcode` and `qrcode`, but both fail at the second line saying `AttributeError: &#39;module&#39; object has no attribute &#39;Decoder&#39;`. Furthermore, [the website][1] refers to Ubuntu 8.10 (which came out more than 6 years ago) and I can&#39;t find a public (git or other) repository of it to check the latest commit. So I moved on to the next library:


**ZBar** ([website here][3]) claims to be `&quot;an open source software suite for reading bar codes from various sources, such as image files.&quot;` So I tried installing it on Mac OSX running `sudo pip install zbar`. This fails with `error: command &#39;cc&#39; failed with exit status 1`. I tried to suggestions in the answers to [this SO question][4], but I can&#39;t seem to solve it. So I decided to move on again:

**QRTools**, which according to [this blogpost][5] can decode images easily by using the following code:

    from qrtools import QR
    myCode = QR(filename=u&quot;/home/psutton/Documents/Python/qrcodes/qrcode.png&quot;)
    if myCode.decode():
      print myCode.data
      print myCode.data_type
      print myCode.data_to_string()

So I tried installing it using `sudo pip install qrtools`, which can&#39;t find anything. I also tried it with `python-qrtools`, `qr-tools`, `python-qrtools` and a couple more combinations, but unfortunately to no avail. I suppose it refers to [this repo][6] which says it is based on ZBar (see above). Although I want to run my code on Heroku (and thus prefer a pure Python solution) I successfully installed it on a Linux box (with `sudo apt-get install python-qrtools`) and tried running it:

    from qrtools import QR
    c = QR(filename=&#39;/home/kramer65/qrcode.jpg&#39;)
    c.data  # prints u&#39;NULL&#39;
    c.data_type  # prints u&#39;text&#39;
    c.data_to_string()  # prints &#39;\xef\xbb\xbfNULL&#39; where I expect an int (being `1234567890`)


Although this seems to decode it, It doesn&#39;t seem to do it correctly. It furthermore needs ZBar and is thus not pure Python. So I decided to find yet another library.

**PyXing** ([website here][7]) is supposedly a Python port of the popular Java [ZXing library][8], but the initial and only commit is 6 years old and the project has no readme or documentation whatsoever.

For the rest I found a couple qr-**en**coders (not **de**coders) and some API endpoints which can decode for you. Since I don&#39;t like this service to be dependent on other API endpoints I would want to keep the decoding local though.

So to conclude; would anybody know how I can decode QR-codes from images in (preferable pure) Python? All tips are welcome!


  [1]: http://pyqrcode.sourceforge.net/
  [2]: https://github.com/lincolnloop/python-qrcode
  [3]: http://zbar.sourceforge.net/
  [4]: https://stackoverflow.com/questions/22697440/cc-failed-with-exit-status-1-error-when-install-python-library
  [5]: http://zleap.net/python-qr-code-reader/
  [6]: https://launchpad.net/qr-tools
  [7]: https://github.com/holizz/pyxing
  [8]: https://github.com/zxing/zxing/

How to decode a QR-code image in (preferably pure) Python?

I am getting one JWT encoded access token from my API in response. But I am not able to decode it and get it in JSON format. I tried using the angular2-jwt library for it, but it did not worked. I am writing my code below:

     setXAuthorizationToken(client){
        let requestHeader = new Headers();
        requestHeader.append(&#39;Content-Type&#39;, &#39;application/x-www-form-urlencoded&#39;);
        this.http.post(client.clientURL + &quot;oauth/token&quot;, &#39;grant_type=password&amp;client_id=toto&amp;client_secret=sec&amp;&#39; +  &#39;username=&#39; + client.username
        + &#39;&amp;password=&#39; + client.password, {
          headers: requestHeader
        }).map(res=&gt;res.json())
        .subscribe((token) =&gt;{
          if(!token.access_token){
              return;
          }
          else{
           var decompressToken = LZString.decompressFromEncodedURIComponent(token.access_token);
           console.log(decompressToken);
    }
        });
        }

Can anybody please help me solve this issue?

How to decode the JWT encoded token payload on client-side in angular?

Why is the below item failing? Why does it succeed with &quot;latin-1&quot; codec?


    o = &quot;a test of \xe9 char&quot; #I want this to remain a string as this is what I am receiving
    v = o.decode(&quot;utf-8&quot;)

Which results in:

     Traceback (most recent call last):  
     File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;  
     File &quot;C:\Python27\lib\encodings\utf_8.py&quot;,
     line 16, in decode
         return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError:
     &#39;utf8&#39; codec can&#39;t decode byte 0xe9 in position 10: invalid continuation byte

UnicodeDecodeError, invalid continuation byte

Apparently jQuery has the ability to decode a given object or string into a JSON object. However, I have a JS object that I need to POST back to the server and I find no utility in jQuery that wraps the JSON.stringify() function. That function is found in Chrome, Safari 4, FF3.6, and IE8 but is not found in earlier browsers. I can use it natively in the browsers that support it, but otherwise am forced to fall back to using Crockford&#39;s JSON scripts.

Is there some built-in with jQuery that handles JSON encoding and decoding that takes the place of the Crockford scripts?

JSON stringify missing from jQuery 1.4.1?

I need to decode HTML entities, e.g. from &amp;amp;#246; to &amp;#246;, and &amp;amp;amp; to &amp;amp;.

````URLEncoder.decode(str)```` does not do the job (convert from % notations). TextUtils has a HTMLencode, but not a HTMLdecode.

Are there any function for decoding HTML entities?

Decode HTML entities in android

I have a query which is meant to show me any rows in table A which have not been updated recently enough. (Each row should be updated within 2 months after &quot;month_no&quot;.):

    SELECT A.identifier
         , A.name
         , TO_NUMBER(DECODE( A.month_no
                 , 1, 200803 
                 , 2, 200804 
                 , 3, 200805 
                 , 4, 200806 
                 , 5, 200807 
                 , 6, 200808 
                 , 7, 200809 
                 , 8, 200810 
                 , 9, 200811 
                 , 10, 200812 
                 , 11, 200701 
                 , 12, 200702
                 , NULL)) as MONTH_NO
         , TO_NUMBER(TO_CHAR(B.last_update_date, &#39;YYYYMM&#39;)) as UPD_DATE
      FROM table_a A
         , table_b B
     WHERE A.identifier = B.identifier
       AND MONTH_NO &gt; UPD_DATE

The last line in the WHERE clause causes an &quot;ORA-00904 Invalid Identifier&quot; error. Needless to say, I don&#39;t want to repeat the entire DECODE function in my WHERE clause. Any thoughts? (Both fixes and workarounds accepted...)


Using an Alias in a WHERE clause

I need to encode some data in the Base64 encoding in Java. How do I do that? What is the name of the class that provides a Base64 encoder?

---

I tried to use the `sun.misc.BASE64Encoder` class, without success. I have the following line of Java 7 code:

    wr.write(new sun.misc.BASE64Encoder().encode(buf));

I&#39;m using Eclipse. Eclipse marks this line as an error. I imported the required libraries:

    import sun.misc.BASE64Encoder;
    import sun.misc.BASE64Decoder;

But again, both of them are shown as errors. I found [a similar post here][1].

I used Apache Commons as the solution suggested by including:

    import org.apache.commons.*;

and importing the JAR files downloaded from: http://commons.apache.org/codec/

But the problem still exists. Eclipse still shows the errors previously mentioned. What should I do?

  [1]: https://stackoverflow.com/questions/5549464/import-sun-misc-base64encoder-got-error-in-eclipse



Encoding as Base64 in Java

I found `blob:https://www.facebook.com/c7e5a634-2343-4464-a03e-4a1987301ca1` video source on Facebook&#39;s private group and I really can&#39;t download the video by entering the url nor am I able to decode. Is there any way to decode this?

&lt;!-- begin snippet: js hide: false console: true babel: false --&gt;

&lt;!-- language: lang-html --&gt;

    &lt;video height=&quot;274&quot; width=&quot;476&quot; preload=&quot;auto&quot; style=&quot;&quot; class=&quot;_ox1 _21y0 _1_d1&quot; data-video-width=&quot;476&quot; data-video-height=&quot;274&quot; data-original-aspect-ratio=&quot;1.7387058423913&quot; id=&quot;u_0_27&quot; src=&quot;blob:https://www.facebook.com/dc89feae-5b46-4103-8ee9-da7d7630ca94&quot;&gt;&lt;/video&gt;

&lt;!-- end snippet --&gt;



Decoding facebook&#39;s blob video url

At the moment, I am trying to get a Python 3 program to do some manipulations with a text file filled with information, through the Spyder IDE/GUI. However, when trying to read the file I get the following error:


      File &quot;&lt;ipython-input-13-d81e1333b8cd&gt;&quot;, line 77, in &lt;module&gt;
        parser(f)
    
      File &quot;&lt;ipython-input-13-d81e1333b8cd&gt;&quot;, line 18, in parser
        data = infile.read()
    
      File &quot;C:\ProgramData\Anaconda3\lib\encodings\cp1252.py&quot;, line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    
    UnicodeDecodeError: &#39;charmap&#39; codec can&#39;t decode byte 0x9d in position 29815: character maps to &lt;undefined&gt;


The code of the program is as follows:

    import os
    
    os.getcwd()
    
    import glob
    import re
    import sqlite3
    import csv
     
    def parser(file):
     
        # Open a TXT file. Store all articles in a list. Each article is an item
        # of the list. Split articles based on the location of such string as
        # &#39;Document PRN0000020080617e46h00461&#39;
     
        articles = []
        with open(file, &#39;r&#39;) as infile:
            data = infile.read()
        start = re.search(r&#39;\n HD\n&#39;, data).start()
        for m in re.finditer(r&#39;Document [a-zA-Z0-9]{25}\n&#39;, data):
            end = m.end()
            a = data[start:end].strip()
            a = &#39;\n   &#39; + a
            articles.append(a)
            start = end
     
        # In each article, find all used Intelligence Indexing field codes. Extract
        # content of each used field code, and write to a CSV file.
     
        # All field codes (order matters)
        fields = [&#39;HD&#39;, &#39;CR&#39;, &#39;WC&#39;, &#39;PD&#39;, &#39;ET&#39;, &#39;SN&#39;, &#39;SC&#39;, &#39;ED&#39;, &#39;PG&#39;, &#39;LA&#39;, &#39;CY&#39;, &#39;LP&#39;,
                  &#39;TD&#39;, &#39;CT&#39;, &#39;RF&#39;, &#39;CO&#39;, &#39;IN&#39;, &#39;NS&#39;, &#39;RE&#39;, &#39;IPC&#39;, &#39;IPD&#39;, &#39;PUB&#39;, &#39;AN&#39;]
     
        for a in articles:
            used = [f for f in fields if re.search(r&#39;\n   &#39; + f + r&#39;\n&#39;, a)]
            unused = [[i, f] for i, f in enumerate(fields) if not re.search(r&#39;\n   &#39; + f + r&#39;\n&#39;, a)]
            fields_pos = []
            for f in used:
                f_m = re.search(r&#39;\n   &#39; + f + r&#39;\n&#39;, a)
                f_pos = [f, f_m.start(), f_m.end()]
                fields_pos.append(f_pos)
            obs = []
            n = len(used)
            for i in range(0, n):
                used_f = fields_pos[i][0]
                start = fields_pos[i][2]
                if i &lt; n - 1:
                    end = fields_pos[i + 1][1]
                else:
                    end = len(a)
                content = a[start:end].strip()
                obs.append(content)
            for f in unused:
                obs.insert(f[0], &#39;&#39;)
            obs.insert(0, file.split(&#39;/&#39;)[-1].split(&#39;.&#39;)[0])  # insert Company ID, e.g., GVKEY
            # print(obs)
            cur.execute(&#39;&#39;&#39;INSERT INTO articles
                           (id, hd, cr, wc, pd, et, sn, sc, ed, pg, la, cy, lp, td, ct, rf,
                           co, ina, ns, re, ipc, ipd, pub, an)
                           VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
                           ?, ?, ?, ?, ?, ?, ?, ?)&#39;&#39;&#39;, obs)
     
    # Write to SQLITE
    conn = sqlite3.connect(&#39;factiva.db&#39;)
    with conn:
        cur = conn.cursor()
        cur.execute(&#39;DROP TABLE IF EXISTS articles&#39;)
        # Mirror all field codes except changing &#39;IN&#39; to &#39;INC&#39; because it is an invalid name
        cur.execute(&#39;&#39;&#39;CREATE TABLE articles
                       (nid integer primary key, id text, hd text, cr text, wc text, pd text,
                       et text, sn text, sc text, ed text, pg text, la text, cy text, lp text,
                       td text, ct text, rf text, co text, ina text, ns text, re text, ipc text,
                       ipd text, pub text, an text)&#39;&#39;&#39;)
        for f in glob.glob(&#39;*.txt&#39;):
            print(f)
            parser(f)
     
    # Write to CSV to feed Stata
    with open(&#39;factiva.csv&#39;, &#39;w&#39;, newline=&#39;&#39;) as csvfile:
        writer = csv.writer(csvfile)
        with conn:
            cur = conn.cursor()
            cur.execute(&#39;SELECT * FROM articles WHERE hd IS NOT NULL&#39;)
            colname = [desc[0] for desc in cur.description]
            writer.writerow(colname)
            for obs in cur.fetchall():
                writer.writerow(obs)


How to fix &#39;&#39;UnicodeDecodeError: &#39;charmap&#39; codec can&#39;t decode byte 0x9d in position 29815: character maps to &lt;undefined&gt;&#39;&#39;?

All Decode Solutions on Gang of Coders

Total of 21 Decode Solutions

PHP replacing special characters like à->a, è->e