I am reading this book ([NLTK][1]) and it is confusing.  **Entropy** is [defined as][2]:

&gt; Entropy is the sum of the probability of each label
&gt; times the log probability of that same label

How can I apply *entropy* and *maximum entropy* in terms of text mining?  Can someone give me a easy, simple example (visual)?


  [1]: http://www.nltk.org/book
  [2]: http://www.nltk.org/book/ch06.html#entropy-and-information-gain

What is &quot;entropy and information gain&quot;?

When do I use each ?

Also...is the NLTK lemmatization dependent upon Parts of Speech?
Wouldn&#39;t it be more accurate if it was?

What is the difference between lemmatization vs stemming?

I want to check in a Python program if a word is in the English dictionary.

I believe nltk wordnet interface might be the way to go but I have no clue how to use it for such a simple task.

    def is_english_word(word):
        pass # how to I implement is_english_word?

    is_english_word(token.lower())

In the future, I might want to check if the singular form of a word is in the dictionary (e.g., properties -&gt; property -&gt; english word). How would I achieve that?

How to check if a word is an English word with Python?

When trying to load the `punkt` tokenizer...

    import nltk.data
    tokenizer = nltk.data.load(&#39;nltk:tokenizers/punkt/english.pickle&#39;)

...a `LookupError` was raised:

    &gt; LookupError: 
    &gt;     *********************************************************************   
    &gt; Resource &#39;tokenizers/punkt/english.pickle&#39; not found.  Please use the NLTK Downloader to obtain the resource: nltk.download().   Searched in:
    &gt;         - &#39;C:\\Users\\Martinos/nltk_data&#39;
    &gt;         - &#39;C:\\nltk_data&#39;
    &gt;         - &#39;D:\\nltk_data&#39;
    &gt;         - &#39;E:\\nltk_data&#39;
    &gt;         - &#39;E:\\Python26\\nltk_data&#39;
    &gt;         - &#39;E:\\Python26\\lib\\nltk_data&#39;
    &gt;         - &#39;C:\\Users\\Martinos\\AppData\\Roaming\\nltk_data&#39;
    &gt;     **********************************************************************





Failed loading english.pickle with nltk.data.load

I&#39;m looking for a way to split a text into n-grams.
Normally I would do something like:

    import nltk
    from nltk import bigrams
    string = &quot;I really like python, it&#39;s pretty awesome.&quot;
    string_bigrams = bigrams(string)
    print string_bigrams
 
I am aware that nltk only offers bigrams and trigrams, but is there a way to split my text in four-grams, five-grams or even hundred-grams?

Thanks!




n-grams in python, four, five, six grams?

How do I find a list with all possible pos tags used by the Natural Language Toolkit (nltk)?

What are all possible pos tags of NLTK?

I&#39;m just starting to use NLTK and I don&#39;t quite understand how to get a list of words from text. If I use `nltk.word_tokenize()`, I get a list of words and punctuation. I need only the words instead. How can I get rid of punctuation? Also `word_tokenize` doesn&#39;t work with multiple sentences: dots are added to the last word.

How to get rid of punctuation using NLTK tokenizer?

So I have a dataset that I would like to remove stop words from using 
    
    stopwords.words(&#39;english&#39;)
 
I&#39;m struggling how to use this within my code to just simply take out these words. I have a list of the words from this dataset already, the part i&#39;m struggling with is comparing to this list and removing the stop words.
Any help is appreciated.


How to remove stop words using nltk or python

In shell script I am checking whether this packages are installed or not, if not installed then install it. So withing shell script:

    import nltk
    echo nltk.__version__

but it stops shell script at `import` line

in linux terminal tried to see in this manner:

    which nltk

which gives nothing thought it is installed.

Is there any other way to verify this package installation in shell script, if not installed, also install it.

how to check which version of nltk, scikit learn installed?

My Code:

    import nltk.data
    tokenizer = nltk.data.load(&#39;nltk:tokenizers/punkt/english.pickle&#39;)

ERROR Message:

	[ec2-user@ip-172-31-31-31 sentiment]$ python mapper_local_v1.0.py
	Traceback (most recent call last):
	File &quot;mapper_local_v1.0.py&quot;, line 16, in &lt;module&gt;

	    tokenizer = nltk.data.load(&#39;nltk:tokenizers/punkt/english.pickle&#39;)

	File &quot;/usr/lib/python2.6/site-packages/nltk/data.py&quot;, line 774, in load

	    opened_resource = _open(resource_url)

	File &quot;/usr/lib/python2.6/site-packages/nltk/data.py&quot;, line 888, in _open

	    return find(path_, path + [&#39;&#39;]).open()

	File &quot;/usr/lib/python2.6/site-packages/nltk/data.py&quot;, line 618, in find

	    raise LookupError(resource_not_found)

	LookupError:

	Resource u&#39;tokenizers/punkt/english.pickle&#39; not found.  Please
	use the NLTK Downloader to obtain the resource:

	    &gt;&gt;&gt;nltk.download()

	Searched in:
	- &#39;/home/ec2-user/nltk_data&#39;
	- &#39;/usr/share/nltk_data&#39;
	- &#39;/usr/local/share/nltk_data&#39;
	- &#39;/usr/lib/nltk_data&#39;
	- &#39;/usr/local/lib/nltk_data&#39;
	- u&#39;&#39;

I&#39;m trying to run this program in Unix machine:

As per the error message, I logged into python shell from my unix machine then I used the below commands:

    import nltk
    nltk.download()

and then I downloaded all the available things using d- down loader and l- list options but still the problem persists.

I tried my best to find the solution in internet but I got the same solution what I did as I mentioned in my above steps.

Resource u&#39;tokenizers/punkt/english.pickle&#39; not found

I have a difficult time using pip to install almost anything. I&#39;m new to coding, so I thought maybe this is something I&#39;ve been doing wrong and have opted out to easy_install to get most of what I needed done, which has generally worked. However, now I&#39;m trying to download the nltk library, and neither is getting the job done.

I tried entering

    sudo pip install nltk

but got the following response:

    /Library/Frameworks/Python.framework/Versions/2.7/bin/pip run on Sat May  4 00:15:38 2013
    Downloading/unpacking nltk
    
      Getting page https://pypi.python.org/simple/nltk/
      Could not fetch URL [need more reputation to post link]: There was a problem confirming the ssl certificate: &lt;urlopen error [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm&gt;
    
      Will skip URL [need more reputation to post link]/simple/nltk/ when looking for download links for nltk
    
      Getting page [need more reputation to post link]/simple/
      Could not fetch URL https://pypi.python. org/simple/: There was a problem confirming the ssl certificate: &lt;urlopen error [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm&gt;
    
      Will skip URL [need more reputation to post link] when looking for download links for nltk
    
      Cannot fetch index base URL [need more reputation to post link]
    
      URLs to search for versions for nltk:
      * [need more reputation to post link]
      Getting page [need more reputation to post link]
      Could not fetch URL [need more reputation to post link]: There was a problem confirming the ssl certificate: &lt;urlopen error [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm&gt;
    
      Will skip URL [need more reputation to post link] when looking for download links for nltk
    
      Could not find any downloads that satisfy the requirement nltk
    
    No distributions at all found for nltk
    
    Exception information:
    Traceback (most recent call last):
      File &quot;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py&quot;, line 139, in main
        status = self.run(options, args)
      File &quot;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py&quot;, line 266, in run
        requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
      File &quot;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py&quot;, line 1026, in prepare_files
        url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
      File &quot;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/index.py&quot;, line 171, in find_requirement
        raise DistributionNotFound(&#39;No distributions at all found for %s&#39; % req)
    DistributionNotFound: No distributions at all found for nltk
    
    --easy_install installed fragments of the library and the code ran into trouble very quickly upon trying to run it.

Any thoughts on this issue? I&#39;d really appreciate some feedback on how I can either get pip working or something to get around the issue in the meantime.

pip issue installing almost any library

I have read multiple posts regarding this error, but I still can&#39;t figure it out. When I try to loop through my function:

    def fix_Plan(location):
        letters_only = re.sub(&quot;[^a-zA-Z]&quot;,  # Search for all non-letters
                              &quot; &quot;,          # Replace all non-letters with spaces
                              location)     # Column and row to search    
    
        words = letters_only.lower().split()     
        stops = set(stopwords.words(&quot;english&quot;))      
        meaningful_words = [w for w in words if not w in stops]      
        return (&quot; &quot;.join(meaningful_words))    
    
    col_Plan = fix_Plan(train[&quot;Plan&quot;][0])    
    num_responses = train[&quot;Plan&quot;].size    
    clean_Plan_responses = []
    
    for i in range(0,num_responses):
        clean_Plan_responses.append(fix_Plan(train[&quot;Plan&quot;][i]))

Here is the error:

    Traceback (most recent call last):
      File &quot;C:/Users/xxxxx/PycharmProjects/tronc/tronc2.py&quot;, line 48, in &lt;module&gt;
        clean_Plan_responses.append(fix_Plan(train[&quot;Plan&quot;][i]))
      File &quot;C:/Users/xxxxx/PycharmProjects/tronc/tronc2.py&quot;, line 22, in fix_Plan
        location)  # Column and row to search
      File &quot;C:\Users\xxxxx\AppData\Local\Programs\Python\Python36\lib\re.py&quot;, line 191, in sub
        return _compile(pattern, flags).sub(repl, string, count)
    TypeError: expected string or bytes-like object

re.sub erroring with &quot;Expected string or bytes-like object&quot;

I was following a tutorial which was available at [Part 1][1] &amp; [Part 2][2]. Unfortunately the author didn&#39;t have the time for the final section which involved using cosine similarity to actually find the distance between two documents. I followed the examples in the article with the help of the following link from [stackoverflow][3], included is the code mentioned in the above link (just so as to make life easier)

    from sklearn.feature_extraction.text import CountVectorizer
    from sklearn.feature_extraction.text import TfidfTransformer
    from nltk.corpus import stopwords
    import numpy as np
    import numpy.linalg as LA
    
    train_set = [&quot;The sky is blue.&quot;, &quot;The sun is bright.&quot;]  # Documents
    test_set = [&quot;The sun in the sky is bright.&quot;]  # Query
    stopWords = stopwords.words(&#39;english&#39;)
    
    vectorizer = CountVectorizer(stop_words = stopWords)
    #print vectorizer
    transformer = TfidfTransformer()
    #print transformer
    
    trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
    testVectorizerArray = vectorizer.transform(test_set).toarray()
    print &#39;Fit Vectorizer to train set&#39;, trainVectorizerArray
    print &#39;Transform Vectorizer to test set&#39;, testVectorizerArray
    
    transformer.fit(trainVectorizerArray)
    print
    print transformer.transform(trainVectorizerArray).toarray()
    
    transformer.fit(testVectorizerArray)
    print 
    tfidf = transformer.transform(testVectorizerArray)
    print tfidf.todense()

as a result of the above code I have the following matrix

    Fit Vectorizer to train set [[1 0 1 0]
     [0 1 0 1]]
    Transform Vectorizer to test set [[0 1 1 1]]
    
    [[ 0.70710678  0.          0.70710678  0.        ]
     [ 0.          0.70710678  0.          0.70710678]]
    
    [[ 0.          0.57735027  0.57735027  0.57735027]]

I am not sure how to use this output in order to calculate cosine similarity, I know how to implement cosine similarity with respect to two vectors of similar length but here I am not sure how to identify the two vectors.

  [1]: http://blog.christianperone.com/?p=1589
  [2]: http://blog.christianperone.com/?p=1747
  [3]: https://stackoverflow.com/questions/11911469/tfidf-for-search-queries

Python: tf-idf-cosine: to find document similarity

How to config nltk data directory from code?

Is it possible to use Stanford Parser in NLTK? (I am not talking about Stanford POS.)

How to use Stanford Parser in NLTK using Python

I am following instructions for a class homework assignment and I am supposed to look up the top 200 most frequently used words in a text file. 

Here&#39;s the last part of the code:

    fdist1 = FreqDist(NSmyText)
    vocab=fdist1.keys()
    vocab[:200]


But when I press enter after the vocab 200 line, it returns:

     Traceback (most recent call last):
     File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
    TypeError: &#39;dict_keys&#39; object is not subscriptable


Any suggestions on how to fix this so it can correctly return an answer?

NLTK python error: &quot;TypeError: &#39;dict_keys&#39; object is not subscriptable&quot;

I reckoned that often the answer to my title is to go and read the documentations, but I ran through the [NLTK book](http://www.nltk.org/book) but it doesn&#39;t give the answer. I&#39;m kind of new to Python.

I have a bunch of `.txt` files and I want to be able to use the corpus functions that NLTK provides for the corpus `nltk_data`. 

I&#39;ve tried `PlaintextCorpusReader` but I couldn&#39;t get further than:

    &gt;&gt;&gt;import nltk
    &gt;&gt;&gt;from nltk.corpus import PlaintextCorpusReader
    &gt;&gt;&gt;corpus_root = &#39;./&#39;
    &gt;&gt;&gt;newcorpus = PlaintextCorpusReader(corpus_root, &#39;.*&#39;)
    &gt;&gt;&gt;newcorpus.words()

How do I segment the `newcorpus` sentences using punkt? I tried using the punkt functions but the punkt functions couldn&#39;t read `PlaintextCorpusReader` class?

Can you also lead me to how I can write the segmented data into text files?


Creating a new corpus with NLTK

I am currently looking for the way to replace words like first, second, third,...with appropriate ordinal number representation (1st, 2nd, 3rd).
I have been googling for the last week and I didn&#39;t find any useful standard tool or any function from NLTK.

So is there any or should I write some regular expressions manually?

Thanks for any advice

Ordinal numbers replacement

I get the following error when trying to install Punkt for nltk:

    nltk.download(&#39;punkt&#39;)    
     [nltk_data] Error loading Punkt: &lt;urlopen error [SSL:
     [nltk_data]     CERTIFICATE_VERIFY_FAILED] certificate verify failed
     [nltk_data]     (_ssl.c:590)&gt;
    False



NLTK download SSL: Certificate verify failed

I&#39;m playing around with the [Natural Language Toolkit][1] (NLTK).

Its documentation ([Book][2] and [HOWTO][3]) are quite bulky and the examples are sometimes slightly advanced. 

Are there any good but basic examples of uses/applications of NLTK? I&#39;m thinking of things like the [NTLK articles][4] on the *Stream Hacker* blog. 

  [1]: http://www.nltk.org/
  [2]: http://www.nltk.org/book
  [3]: http://nltk.googlecode.com/svn/trunk/doc/howto/index.html
  [4]: http://streamhacker.wordpress.com/tag/nltk/

Practical examples of NLTK use

I am trying to process a user entered text by removing stopwords using nltk toolkit, but with stopword-removal the words like &#39;and&#39;, &#39;or&#39;, &#39;not&#39; gets removed. I want these words to be present after stopword removal process as they are operators which are required for later processing text as query. I don&#39;t know which are the words which can be operators in text query, and I also want to remove unnecessary words from my text.

Stopword removal with NLTK

I am trying to download NLTK 3.0 for use with Python 3.6 on Mac OS X 10.7.5, but am getting an SSL error:

    import nltk
    nltk.download()

[![enter image description here][1]][1]

I downloaded NLTK with a pip3 command: `sudo pip3 install -U nltk`.

Changing the index in the NLTK downloader allows the downloader to show all of NLTK&#39;s files, but when one tries to download all, one gets another SSL error (see bottom of photo):

[![enter image description here][2]][2]

I am relatively new to computer science and am not at all savvy with respect to SSL.

My question is how to simply resolve this issue?



---

Here is a similar question by a user who is having the same problem:

https://stackoverflow.com/questions/38725583/unable-to-download-nltk-data

I decided to post a new question with screenshots, since my edit to that other question was rejected.

Similar questions which I did not find helpful:

https://stackoverflow.com/questions/38916452/nltk-download-ssl-certificate-verify-failed

https://stackoverflow.com/questions/27658409/downloading-error-using-nltk-download

  [1]: https://i.stack.imgur.com/rAwBe.png
  [2]: https://i.stack.imgur.com/Pt2Oq.png

SSL error downloading NLTK data

I wanted to use wordnet lemmatizer in python and I have learnt that the default pos tag is NOUN and that it does not output the correct lemma for a verb, unless the pos tag is explicitly specified as VERB.

My question is what is the best shot inorder to perform the above lemmatization accurately?


I did the pos tagging using `nltk.pos_tag` and I am lost in integrating the tree bank pos tags to wordnet compatible pos tags. Please help

    from nltk.stem.wordnet import WordNetLemmatizer
    lmtzr = WordNetLemmatizer()
    tagged = nltk.pos_tag(tokens)

I get the output tags in NN,JJ,VB,RB. How do I change these to wordnet compatible tags?

Also do I have to train `nltk.pos_tag()` with a tagged corpus or can I use it directly on my data to evaluate?


wordnet lemmatization and pos tagging in python

I trying to import the nltk package in python 2.7

      import nltk
      stopwords = nltk.corpus.stopwords.words(&#39;english&#39;)
      print(stopwords[:10])

Running this gives me the following error:

    LookupError: 
    **********************************************************************
    Resource &#39;corpora/stopwords&#39; not found.  Please use the NLTK
    Downloader to obtain the resource:  &gt;&gt;&gt; nltk.download()

So therefore I open my python termin and did the following:

    import nltk  
    nltk.download()

Which gives me:

    showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml


However this does not seem to stop. And running it again still gives me the same error. Any thoughts where this goes wrong?



Corpora/stopwords not found when import nltk library

I am playing around with NLTK to do an assignment on sentiment analysis. I am using Python 2.7. NLTK 3.0 and NumPy1.9.1 version. 

This is the code :

    __author__ = &#39;karan&#39;
    import nltk
    import re
    import sys
    
    
    
    def main():
        print(&quot;Start&quot;);
        # getting the stop words
        stopWords = open(&quot;english.txt&quot;,&quot;r&quot;);
        stop_word = stopWords.read().split();
        AllStopWrd = []
        for wd in stop_word:
            AllStopWrd.append(wd);
        print(&quot;stop words-&gt; &quot;,AllStopWrd);
    
        # sample and also cleaning it
        tweet1= &#39;Love, my new toy&#237;&#160;&#189;&#237;&#184;&#237;&#160;&#189;&#237;&#184;#iPhone6. Its good https://twitter.com/Sandra_Ortega/status/513807261769424897/photo/1&#39;
        print(&quot;old tweet-&gt; &quot;,tweet1)
        tweet1 = tweet1.lower()
        tweet1 = &#39; &#39;.join(re.sub(&quot;(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)&quot;,&quot; &quot;,tweet1).split())
        print(tweet1);
        tw = tweet1.split()
        print(tw)
    
    
        #tokenize
        sentences = nltk.word_tokenize(tweet1)
        print(&quot;tokenized -&gt;&quot;, sentences)
    
    
        #remove stop words
        Otweet =[]
        for w in tw:
            if w not in AllStopWrd:
                Otweet.append(w);
        print(&quot;sans stop word-&gt; &quot;,Otweet)
    
    
        # get taggers for neg/pos/inc/dec/inv words
        taggers ={}
        negWords = open(&quot;neg.txt&quot;,&quot;r&quot;);
        neg_word = negWords.read().split();
        print(&quot;ned words-&gt; &quot;,neg_word)
        posWords = open(&quot;pos.txt&quot;,&quot;r&quot;);
        pos_word = posWords.read().split();
        print(&quot;pos words-&gt; &quot;,pos_word)
        incrWords = open(&quot;incr.txt&quot;,&quot;r&quot;);
        inc_word = incrWords.read().split();
        print(&quot;incr words-&gt; &quot;,inc_word)
        decrWords = open(&quot;decr.txt&quot;,&quot;r&quot;);
        dec_word = decrWords.read().split();
        print(&quot;dec wrds-&gt; &quot;,dec_word)
        invWords = open(&quot;inverse.txt&quot;,&quot;r&quot;);
        inv_word = invWords.read().split();
        print(&quot;inverse words-&gt; &quot;,inv_word)
        for nw in neg_word:
            taggers.update({nw:&#39;negative&#39;});
        for pw in pos_word:
            taggers.update({pw:&#39;positive&#39;});
        for iw in inc_word:
            taggers.update({iw:&#39;inc&#39;});
        for dw in dec_word:
            taggers.update({dw:&#39;dec&#39;});
        for ivw in inv_word:
            taggers.update({ivw:&#39;inv&#39;});
        print(&quot;tagger-&gt; &quot;,taggers)
        print(taggers.get(&#39;little&#39;))
    
        # get parts of speech
        posTagger = [nltk.pos_tag(tw)]
        print(&quot;posTagger-&gt; &quot;,posTagger)
    
    main();

This is the error that I am getting when running my code:

    SyntaxError: Non-ASCII character &#39;\xc3&#39; in file C:/Users/karan/PycharmProjects/mainProject/sentiment.py on line 19, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

How do I fix this error?

I also tried the code using Python 3.4.2 and with NLTK 3.0 and NumPy 1.9.1 but then I get the error:

    Traceback (most recent call last):
      File &quot;C:/Users/karan/PycharmProjects/mainProject/sentiment.py&quot;, line 80, in &lt;module&gt;
        main();
      File &quot;C:/Users/karan/PycharmProjects/mainProject/sentiment.py&quot;, line 72, in main
        posTagger = [nltk.pos_tag(tw)]
      File &quot;C:\Python34\lib\site-packages\nltk\tag\__init__.py&quot;, line 100, in pos_tag
        tagger = load(_POS_TAGGER)
      File &quot;C:\Python34\lib\site-packages\nltk\data.py&quot;, line 779, in load
        resource_val = pickle.load(opened_resource)
    UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xcb in position 0: ordinal not in range(128)


Python NLTK: SyntaxError: Non-ASCII character &#39;\xc3&#39; in file (Sentiment Analysis -NLP)

I have a series of text items- raw HTML from a MySQL database. I want to find the most common phrases in these entries (not the single most common phrase, and ideally, not enforcing word-for-word matching). 

My example is any review on Yelp.com, that shows 3 snippets from hundreds of reviews of a given restaurant, in the format: 

&quot;Try the hamburger&quot; (in 44 reviews)

e.g., the &quot;Review Highlights&quot; section of this page: 

&lt;a href=&quot;http://www.yelp.com/biz/sushi-gen-los-angeles&quot;&gt;http://www.yelp.com/biz/sushi-gen-los-angeles/&lt;/a&gt;

I have NLTK installed and I&#39;ve played around with it a bit, but am honestly overwhelmed by the options. This seems like a rather common problem and I haven&#39;t been able to find a straightforward solution by searching here.

How to extract common / significant phrases from a series of text entries

Is there a ready-to-use English grammar that I can just load it and use in NLTK? I&#39;ve searched around examples of parsing with NLTK, but it seems like that I have to manually specify grammar before parsing a sentence. 

Thanks a lot!

English grammar for parsing in NLTK

I am trying to start a project of sentiment analysis and I will use the stop words method. I made some research and I found that nltk have stopwords but when I execute the command there is an error.

What I do is the following, in order to know which are the words that nltk use (like what you may found here http://www.nltk.org/book/ch02.html in section4.1):

    from nltk.corpus import stopwords
    stopwords.words(&#39;english&#39;)

But when I press enter I obtain

    ---------------------------------------------------------------------------
    LookupError                               Traceback (most recent call last)
    &lt;ipython-input-6-ff9cd17f22b2&gt; in &lt;module&gt;()
    ----&gt; 1 stopwords.words(&#39;english&#39;)

    C:\Users\Usuario\Anaconda\lib\site-packages\nltk\corpus\util.pyc in __getattr__(self, attr)
     66
     67     def __getattr__(self, attr):
    ---&gt; 68         self.__load()
     69         # This looks circular, but its not, since __load() changes our
     70         # __class__ to something new:

    C:\Users\Usuario\Anaconda\lib\site-packages\nltk\corpus\util.pyc in __load(self)
     54             except LookupError, e:
     55                 try: root = nltk.data.find(&#39;corpora/%s&#39; % zip_name)
    ---&gt; 56                 except LookupError: raise e
     57
     58         # Load the corpus.

    LookupError:
    **********************************************************************
      Resource &#39;corpora/stopwords&#39; not found.  Please use the NLTK
      Downloader to obtain the resource:  &gt;&gt;&gt; nltk.download()
      Searched in:
    - &#39;C:\\Users\\Meru/nltk_data&#39;
    - &#39;C:\\nltk_data&#39;
    - &#39;D:\\nltk_data&#39;
    - &#39;E:\\nltk_data&#39;
    - &#39;C:\\Users\\Meru\\Anaconda\\nltk_data&#39;
    - &#39;C:\\Users\\Meru\\Anaconda\\lib\\nltk_data&#39;
    - &#39;C:\\Users\\Meru\\AppData\\Roaming\\nltk_data&#39;
    **********************************************************************

And, because of this problem things like this cannot run properly (obtaining the same error):

    &gt;&gt;&gt; from nltk.corpus import stopwords
    &gt;&gt;&gt; stop = stopwords.words(&#39;english&#39;)
    &gt;&gt;&gt; sentence = &quot;this is a foo bar sentence&quot;
    &gt;&gt;&gt; print [i for i in sentence.split() if i not in stop]

Do you know what may be problem? I must use words in Spanish, do you recomend another method? I also thought using Goslate package with datasets in english

Thanks for reading!

P.D.: I use Ananconda

NLTK and Stopwords Fail #lookuperror

I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I&#39;ve just got up to the method like

    my_text = [&#39;This&#39;, &#39;is&#39;, &#39;my&#39;, &#39;text&#39;]

I&#39;d like to discover any way to input my &quot;text&quot; as:

    my_text = &quot;This is my text, this is a nice way to input text.&quot;

Which method, python&#39;s or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?

How do I tokenize a string sentence in NLTK?

Updated answer:NLTK works for 2.7 well. I had 3.2. I uninstalled 3.2 and installed 2.7. Now it works!!


I have installed NLTK and tried to download NLTK Data. What I did was to follow the instrution on this site: http://www.nltk.org/data.html

I downloaded NLTK, installed it, and then tried to run the following code:

    &gt;&gt;&gt; import nltk
    &gt;&gt;&gt; nltk.download()

It gave me the error message like below:

    Traceback (most recent call last):
      File &quot;&lt;pyshell#6&gt;&quot;, line 1, in &lt;module&gt;
        nltk.download()
    AttributeError: &#39;module&#39; object has no attribute &#39;download&#39;
     Directory of C:\Python32\Lib\site-packages

Tried both `nltk.download()` and `nltk.downloader()`, both gave me error messages.

Then I used `help(nltk)` to pull out the package, it shows the following info:

    NAME
        nltk

    PACKAGE CONTENTS
        align
        app (package)
        book
        ccg (package)
        chat (package)
        chunk (package)
        classify (package)
        cluster (package)
        collocations
        corpus (package)
        data
        decorators
        downloader
        draw (package)
        examples (package)
        featstruct
        grammar
        help
        inference (package)
        internals
        lazyimport
        metrics (package)
        misc (package)
        model (package)
        parse (package)
        probability
        sem (package)
        sourcedstring
        stem (package)
        tag (package)
        test (package)
        text
        tokenize (package)
        toolbox
        tree
        treetransforms
        util
        yamltags

    FILE
        c:\python32\lib\site-packages\nltk


I do see Downloader there, not sure why it does not work. Python 3.2.2, system Windows vista.


How do I download NLTK data?

I have the following code 

    import nltk, os, json, csv, string, cPickle
    from scipy.stats import scoreatpercentile

    lmtzr = nltk.stem.wordnet.WordNetLemmatizer()

    def sanitize(wordList): 
	answer = [word.translate(None, string.punctuation) for word in wordList] 
	answer = [lmtzr.lemmatize(word.lower()) for word in answer]
	return answer

    words = []
    for filename in json_list:
		words.extend([sanitize(nltk.word_tokenize(&#39; &#39;.join([tweet[&#39;text&#39;] 
                       for tweet in json.load(open(filename,READ))])))])

I&#39;ve tested lines 2-4 in a separate testing.py file when I wrote
    
    import nltk, os, json, csv, string, cPickle
    from scipy.stats import scoreatpercentile

    wordList= [&#39;\&#39;the&#39;, &#39;the&#39;, &#39;&quot;the&#39;]
    print wordList
    wordList2 = [word.translate(None, string.punctuation) for word in wordList]
    print wordList2
    answer = [lmtzr.lemmatize(word.lower()) for word in wordList2]
    print answer

    freq = nltk.FreqDist(wordList2)
    print freq

and the command prompt returns [&#39;the&#39;,&#39;the&#39;,&#39;the&#39;], which is what I wanted (removing punctuation).

However, when I put the exact same code in a different file, python returns a TypeError stating that

    File &quot;foo.py&quot;, line 8, in &lt;module&gt;
      for tweet in json.load(open(filename, READ))])))])
    File &quot;foo.py&quot;, line 2, in sanitize
      answer = [word.translate(None, string.punctuation) for word in wordList]
    TypeError: translate() takes exactly one argument (2 given)

json_list is a list of all the file paths (I printed and check that this list is valid). I&#39;m confused on this TypeError because everything works perfectly fine when I&#39;m just testing it in a different file.

str.translate gives TypeError - Translate takes one argument (2 given), worked in Python 2

My project uses the NLTK. How can I list the project&#39;s corpus &amp; model requirements so they can be automatically installed? I don&#39;t want to click through the `nltk.download()` GUI, installing packages one by one.

Also, any way to freeze that same list of requirements (like `pip freeze`)?

Programmatically install NLTK corpora / models, i.e. without the GUI downloader?

I&#39;m fairly new to Python and NLTK. I am busy with an application that can perform spell checks (replaces an incorrectly spelled word with the correct one).
I&#39;m currently using the Enchant library on Python 2.7, PyEnchant and the NLTK library. The code below is a class that handles the correction/replacement. 

    from nltk.metrics import edit_distance

    class SpellingReplacer:
        def __init__(self, dict_name=&#39;en_GB&#39;, max_dist=2):
            self.spell_dict = enchant.Dict(dict_name)
            self.max_dist = 2

        def replace(self, word):
            if self.spell_dict.check(word):
                return word
            suggestions = self.spell_dict.suggest(word)

            if suggestions and edit_distance(word, suggestions[0]) &lt;= self.max_dist:
                return suggestions[0]
            else:
                return word


I have written a function that takes in a list of words and executes replace() on each word and then returns a list of those words, but spelled correctly.

    def spell_check(word_list):
        checked_list = []
        for item in word_list:
            replacer = SpellingReplacer()
            r = replacer.replace(item)
            checked_list.append(r)
        return checked_list

    &gt;&gt;&gt; word_list = [&#39;car&#39;, &#39;colour&#39;]
    &gt;&gt;&gt; spell_check(words)
    [&#39;car&#39;, &#39;color&#39;]

Now, I don&#39;t really like this because it isn&#39;t very accurate and I&#39;m looking for a way to achieve spelling checks and replacements on words. I also need something that can pick up spelling mistakes like &quot;caaaar&quot;? Are there better ways to perform spelling checks out there? If so, what are they? How does Google do it? Because their spelling suggester is very good.

Any suggestions?

Spell Checker for Python

I am trying to extract human names from text. 

Does anyone have a method that they would recommend?

This is what I tried (code is below):
I am using `nltk` to find everything marked as a person and then generating a list of all the NNP parts of that person. I am skipping persons where there is only one NNP which avoids grabbing a lone surname.

I am getting decent results but was wondering if there are better ways to go about solving this problem.

Code:

    import nltk
    from nameparser.parser import HumanName

    def get_human_names(text):
        tokens = nltk.tokenize.word_tokenize(text)
        pos = nltk.pos_tag(tokens)
        sentt = nltk.ne_chunk(pos, binary = False)
        person_list = []
        person = []
        name = &quot;&quot;
        for subtree in sentt.subtrees(filter=lambda t: t.node == &#39;PERSON&#39;):
            for leaf in subtree.leaves():
                person.append(leaf[0])
            if len(person) &gt; 1: #avoid grabbing lone surnames
                for part in person:
                    name += part + &#39; &#39;
                if name[:-1] not in person_list:
                    person_list.append(name[:-1])
                name = &#39;&#39;
            person = []
    
        return (person_list)

    text = &quot;&quot;&quot;
    Some economists have responded positively to Bitcoin, including 
    Francois R. Velde, senior economist of the Federal Reserve in Chicago 
    who described it as &quot;an elegant solution to the problem of creating a 
    digital currency.&quot; In November 2013 Richard Branson announced that 
    Virgin Galactic would accept Bitcoin as payment, saying that he had invested 
    in Bitcoin and found it &quot;fascinating how a whole new global currency 
    has been created&quot;, encouraging others to also invest in Bitcoin.
    Other economists commenting on Bitcoin have been critical. 
    Economist Paul Krugman has suggested that the structure of the currency 
    incentivizes hoarding and that its value derives from the expectation that 
    others will accept it as payment. Economist Larry Summers has expressed 
    a &quot;wait and see&quot; attitude when it comes to Bitcoin. Nick Colas, a market 
    strategist for ConvergEx Group, has remarked on the effect of increasing 
    use of Bitcoin and its restricted supply, noting, &quot;When incremental 
    adoption meets relatively fixed supply, it should be no surprise that 
    prices go up. And that’s exactly what is happening to BTC prices.&quot;
    &quot;&quot;&quot;
    
    names = get_human_names(text)
    print &quot;LAST, FIRST&quot;
    for name in names: 
        last_first = HumanName(name).last + &#39;, &#39; + HumanName(name).first
            print last_first

Output:

    LAST, FIRST
    Velde, Francois
    Branson, Richard
    Galactic, Virgin
    Krugman, Paul
    Summers, Larry
    Colas, Nick

Apart from Virgin Galactic, this is all valid output. Of course, knowing that Virgin Galactic isn&#39;t a human name in the context of this article is the hard (maybe impossible) part.

Improving the extraction of human names with nltk

I&#39;m trying to remove all the html/javascript using bs4, however, it doesn&#39;t get rid of javascript. I still see it there with the text. How can I get around this?

I tried using `nltk` which works fine however, `clean_html` and `clean_url` will be removed moving forward. Is there a way to use soups `get_text` and get the same result?

I tried looking at these other pages:

https://stackoverflow.com/questions/10524387/beautifulsoup-get-text-does-not-strip-all-tags-and-javascript

Currently i&#39;m using the nltk&#39;s deprecated functions.

**EDIT**

Here&#39;s an example:

    import urllib
    from bs4 import BeautifulSoup
    
    url = &quot;http://www.cnn.com&quot;
    html = urllib.urlopen(url).read()
    soup = BeautifulSoup(html)
    print soup.get_text()

I still see the following for CNN:

	$j(function() {
	&quot;use strict&quot;;
	if ( window.hasOwnProperty(&#39;safaripushLib&#39;) &amp;&amp; window.safaripushLib.checkEnv() ) {
	var pushLib = window.safaripushLib,
	current = pushLib.currentPermissions();
	if (current === &quot;default&quot;) {
	pushLib.checkPermissions(&quot;helloClient&quot;, function() {});
	}
	}
	});

	/*globals MainLocalObj*/
	$j(window).load(function () {
	&#39;use strict&#39;;
	MainLocalObj.init();
	});

How can I remove the js?

Only other options I found are:

https://github.com/aaronsw/html2text

The problem with `html2text` is that it&#39;s really _really_ slow at times, and creates noticable lag, which is one thing nltk was always very good with.

BeatifulSoup4 get_text still has javascript

I am curious if there is an algorithm/method exists to generate keywords/tags from a given text, by using some weight calculations, occurrence ratio or other tools.

Additionally, I will be grateful if you point any Python based solution / library for this.

Thanks

All Nltk Solutions on Gang of Coders

Total of 36 Nltk Solutions

What is "entropy and information gain"?