I have a strange bug when trying to `urlopen` a certain page from Wikipedia. This is the page:

http://en.wikipedia.org/wiki/OpenCola_(drink)

This is the shell session:

    &gt;&gt;&gt; f = urllib2.urlopen(&#39;http://en.wikipedia.org/wiki/OpenCola_(drink)&#39;)
    Traceback (most recent call last):
      File &quot;C:\Program Files\Wing IDE 4.0\src\debug\tserver\_sandbox.py&quot;, line 1, in &lt;module&gt;
        # Used internally for debug sandbox under external interpreter
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 126, in urlopen
        return _opener.open(url, data, timeout)
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 397, in open
        response = meth(req, response)
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 510, in http_response
        &#39;http&#39;, request, response, code, msg, hdrs)
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 435, in error
        return self._call_chain(*args)
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 369, in _call_chain
        result = func(*args)
      File &quot;c:\Python26\Lib\urllib2.py&quot;, line 518, in http_error_default
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 403: Forbidden

This happened to me on two different systems in different continents. Does anyone have an idea why this happens?

Python&#39;s `urllib2`: Why do I get error 403 when I `urlopen` a Wikipedia page?

I use the following code to stream large files from the Internet into a local file:

    fp = open(file, &#39;wb&#39;)
    req = urllib2.urlopen(url)
    for line in req:
        fp.write(line)
    fp.close()

This works but it downloads quite slowly. Is there a faster way? (The files are large so I don&#39;t want to keep them in memory.)


Stream large binary files with urllib2 to file

I need to do a API call to upload a file along with a JSON string with details about the file.

I am trying to use the python requests lib to do this:

    import requests

    info = {
        &#39;var1&#39; : &#39;this&#39;,
        &#39;var2&#39;  : &#39;that&#39;,
    }

    data = json.dumps({
        &#39;token&#39; : auth_token,
        &#39;info&#39;  : info,
    })

    headers = {&#39;Content-type&#39;: &#39;multipart/form-data&#39;}

    files = {&#39;document&#39;: open(&#39;file_name.pdf&#39;, &#39;rb&#39;)}

    r = requests.post(url, files=files, data=data, headers=headers)

This throws the following error:

        raise ValueError(&quot;Data must not be a string.&quot;)
     ValueError: Data must not be a string

If I remove the &#39;files&#39; from the request, it works.&lt;br&gt;
If I remove the &#39;data&#39; from the request, it works.&lt;br&gt;
If I do not encode data as JSON it works.

For this reason I think the error is to do with sending JSON data and files in the same request.

Any ideas on how to get this working?
       

Python Requests: Post JSON and file in single request

I&#39;m trying to test the functionality of a web app by scripting a login sequence in Python, but I&#39;m having some troubles.

Here&#39;s what I need to do:

1. Do a POST with a few parameters and headers.
2. Follow a redirect
3. Retrieve the HTML body.

Now, I&#39;m relatively new to python, but the two things I&#39;ve tested so far haven&#39;t worked. First I used httplib, with putrequest() (passing the parameters within the URL), and putheader(). This didn&#39;t seem to follow the redirects.

Then I tried urllib and urllib2, passing both headers and parameters as dicts. This seems to return the login page, instead of the page I&#39;m trying to login to, I guess it&#39;s because of lack of cookies or something.

Am I missing something simple?

Thanks.

Python: urllib/urllib2/httplib confusion

Problem: When POSTing data with Python&#39;s urllib2, all data is URL encoded and sent as Content-Type: application/x-www-form-urlencoded.  When uploading files, the Content-Type should instead be set to multipart/form-data and the contents be MIME-encoded.

To get around this limitation some sharp coders created a library called MultipartPostHandler which creates an OpenerDirector you can use with urllib2 to mostly automatically POST with multipart/form-data. A copy of this library is here: [MultipartPostHandler doesn&#39;t work for Unicode files ][1]

I am new to Python and am unable to get this library to work. I wrote out essentially the following code. When I capture it in a local HTTP proxy, I can see that the data is still URL encoded and not multi-part MIME-encoded. Please help me figure out what I am doing wrong or a better way to get this done. Thanks :-)

    FROM_ADDR = &#39;my@email.com&#39;
    
    try:
    	data = open(file, &#39;rb&#39;).read()
    except:
    	print &quot;Error: could not open file %s for reading&quot; % file
    	print &quot;Check permissions on the file or folder it resides in&quot;
    	sys.exit(1)
    
    # Build the POST request
    url = &quot;http://somedomain.com/?action=analyze&quot;		
    post_data = {}
    post_data[&#39;analysisType&#39;] = &#39;file&#39;
    post_data[&#39;executable&#39;] = data
    post_data[&#39;notification&#39;] = &#39;email&#39;
    post_data[&#39;email&#39;] = FROM_ADDR
    
    # MIME encode the POST payload
    opener = urllib2.build_opener(MultipartPostHandler.MultipartPostHandler)
    urllib2.install_opener(opener)
    request = urllib2.Request(url, post_data)
    request.set_proxy(&#39;127.0.0.1:8080&#39;, &#39;http&#39;) # For testing with Burp Proxy
    
    # Make the request and capture the response
    try:
    	response = urllib2.urlopen(request)
    	print response.geturl()
    except urllib2.URLError, e:
    	print &quot;File upload failed...&quot;

EDIT1: Thanks for your response. I&#39;m aware of the ActiveState httplib solution to this (I linked to it above). I&#39;d rather abstract away the problem and use a minimal amount of code to continue using urllib2 how I have been. Any idea why the opener isn&#39;t being installed and used?


  [1]: http://peerit.blogspot.com/2007/07/multipartposthandler-doesnt-work-for.html

Using MultipartPostHandler to POST form-data with Python

I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises:

    UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode character u&#39;\u0131&#39; in position 26: ordinal not in range(128)

I know the URL is not standards compliant but I have no chance to change it. 

What is the way to access a resource pointed by a URL containing non-ascii characters using Python?

**edit:** In other words, can / how urlopen open a URL like:

    http://example.org/&#209;&#246;&#241;-&#197;Ş&#199;İİ/



How to fetch a non-ascii url with urlopen?

I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module.

I used the inbuilt 2to3 Python tool to convert the below working urllib and urllib2 Python 2.7 code:

    import urllib2
    import urllib
    
    url = &quot;https://www.customdomain.com&quot;
    d = dict(parameter1=&quot;value1&quot;, parameter2=&quot;value2&quot;)
    
    req = urllib2.Request(url, data=urllib.urlencode(d))
    f = urllib2.urlopen(req)
    resp = f.read()


The output from the 2to3 module was the below Python 3 code:


    import urllib.request, urllib.error, urllib.parse
    
    url = &quot;https://www.customdomain.com&quot;
    d = dict(parameter1=&quot;value1&quot;, parameter2=&quot;value2&quot;)
    
    req = urllib.request.Request(url, data=urllib.parse.urlencode(d))
    f = urllib.request.urlopen(req)
    resp = f.read()





 
When the Python 3 code is run the following error is produced:

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    &lt;ipython-input-56-206954140899&gt; in &lt;module&gt;()
          5 
          6 req = urllib.request.Request(url, data=urllib.parse.urlencode(d))
    ----&gt; 7 f = urllib.request.urlopen(req)
          8 resp = f.read()
    
    C:\Users\Admin\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
        159     else:
        160         opener = _opener
    --&gt; 161     return opener.open(url, data, timeout)
        162 
        163 def install_opener(opener):
    
    C:\Users\Admin\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
        459         for processor in self.process_request.get(protocol, []):
        460             meth = getattr(processor, meth_name)
    --&gt; 461             req = meth(req)
        462 
        463         response = self._open(req, data)
    
    C:\Users\Admin\Anaconda3\lib\urllib\request.py in do_request_(self, request)
       1110                 msg = &quot;POST data should be bytes or an iterable of bytes. &quot; \
       1111                       &quot;It cannot be of type str.&quot;
    -&gt; 1112                 raise TypeError(msg)
       1113             if not request.has_header(&#39;Content-type&#39;):
       1114                 request.add_unredirected_header(
    
    TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.



I have also read two other tickets ([ticket1][1] and [ticket2][2]) which mentioned encoding the date. 

When I changed the line `f = urllib.request.urlopen(req)` to `f = urllib.request.urlopen(req.encode(&#39;utf-8&#39;))` I received the following error: `AttributeError: &#39;Request&#39; object has no attribute &#39;encode&#39;`

I am stuck as to how to make the Python 3 code work. Could you please help me?


  [1]: https://stackoverflow.com/questions/28850421/post-data-should-be-bytes-or-an-iterable-of-bytes-it-cannot-be-of-type-str
  [2]: https://stackoverflow.com/questions/5440485/typeerror-post-data-should-be-bytes-or-an-iterable-of-bytes-it-cannot-be-str

Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str

So I&#39;m trying to make a Python script that downloads webcomics and puts them in a folder on my desktop.  I&#39;ve found a few similar programs on here that do something similar, but nothing quite like what I need.  The one that I found most similar is right here (http://bytes.com/topic/python/answers/850927-problem-using-urllib-download-images).  I tried using this code:

    &gt;&gt;&gt; import urllib
    &gt;&gt;&gt; image = urllib.URLopener()
    &gt;&gt;&gt; image.retrieve(&quot;http://www.gunnerkrigg.com//comics/00000001.jpg&quot;,&quot;00000001.jpg&quot;)
    (&#39;00000001.jpg&#39;, &lt;httplib.HTTPMessage instance at 0x1457a80&gt;)

I then searched my computer for a file &quot;00000001.jpg&quot;, but all I found was the cached picture of it.  I&#39;m not even sure it saved the file to my computer.  Once I understand how to get the file downloaded, I think I know how to handle the rest.  Essentially just use a for loop and split the string at the &#39;00000000&#39;.&#39;jpg&#39; and increment the &#39;00000000&#39; up to the largest number, which I would have to somehow determine.  Any reccomendations on the best way to do this or how to download the file correctly?

Thanks!


EDIT 6/15/10

Here is the completed script, it saves the files to any directory you choose.  For some odd reason, the files weren&#39;t downloading and they just did.  Any suggestions on how to clean it up would be much appreciated.  I&#39;m currently working out how to find out many comics exist on the site so I can get just the latest one, rather than having the program quit after a certain number of exceptions are raised.


    import urllib
    import os

    comicCounter=len(os.listdir(&#39;/file&#39;))+1  # reads the number of files in the folder to start downloading at the next comic
    errorCount=0

    def download_comic(url,comicName):
        &quot;&quot;&quot;
        download a comic in the form of
    
        url = http://www.example.com
        comicName = &#39;00000000.jpg&#39;
        &quot;&quot;&quot;
        image=urllib.URLopener()
        image.retrieve(url,comicName)  # download comicName at URL

    while comicCounter &lt;= 1000:  # not the most elegant solution
        os.chdir(&#39;/file&#39;)  # set where files download to
            try:
            if comicCounter &lt; 10:  # needed to break into 10^n segments because comic names are a set of zeros followed by a number
                comicNumber=str(&#39;0000000&#39;+str(comicCounter))  # string containing the eight digit comic number
                comicName=str(comicNumber+&quot;.jpg&quot;)  # string containing the file name
                url=str(&quot;http://www.gunnerkrigg.com//comics/&quot;+comicName)  # creates the URL for the comic
                comicCounter+=1  # increments the comic counter to go to the next comic, must be before the download in case the download raises an exception
                download_comic(url,comicName)  # uses the function defined above to download the comic
                print url
            if 10 &lt;= comicCounter &lt; 100:
                comicNumber=str(&#39;000000&#39;+str(comicCounter))
                comicName=str(comicNumber+&quot;.jpg&quot;)
                url=str(&quot;http://www.gunnerkrigg.com//comics/&quot;+comicName)
                comicCounter+=1
                download_comic(url,comicName)
                print url
            if 100 &lt;= comicCounter &lt; 1000:
                comicNumber=str(&#39;00000&#39;+str(comicCounter))
                comicName=str(comicNumber+&quot;.jpg&quot;)
                url=str(&quot;http://www.gunnerkrigg.com//comics/&quot;+comicName)
                comicCounter+=1
                download_comic(url,comicName)
                print url
            else:  # quit the program if any number outside this range shows up
                quit
        except IOError:  # urllib raises an IOError for a 404 error, when the comic doesn&#39;t exist
            errorCount+=1  # add one to the error count
            if errorCount&gt;3:  # if more than three errors occur during downloading, quit the program
                break
            else:
                print str(&quot;comic&quot;+ &#39; &#39; + str(comicCounter) + &#39; &#39; + &quot;does not exist&quot;)  # otherwise say that the certain comic number doesn&#39;t exist
    print &quot;all comics are up to date&quot;  # prints if all comics are downloaded





Downloading a picture via urllib and python

In Python, what are the differences between the [`urllib`][1], [`urllib2`][2], [`urllib3`][3] and [`requests`][4] modules? Why are there three? They seem to do the same thing...


[1]: https://docs.python.org/library/urllib.html
[2]: https://docs.python.org/2.7/library/urllib2.html
[3]: https://urllib3.readthedocs.io/en/latest/
[4]: https://requests.readthedocs.io

What are the differences between the urllib, urllib2, urllib3 and requests module?

In Python I&#39;m getting an error:

    Exception:  (&lt;type &#39;exceptions.AttributeError&#39;&gt;,
    AttributeError(&quot;&#39;str&#39; object has no attribute &#39;read&#39;&quot;,), &lt;traceback object at 0x1543ab8&gt;)

Given python code:

    def getEntries (self, sub):
        url = &#39;http://www.reddit.com/&#39;
        if (sub != &#39;&#39;):
            url += &#39;r/&#39; + sub
        
        request = urllib2.Request (url + 
            &#39;.json&#39;, None, {&#39;User-Agent&#39; : &#39;Reddit desktop client by /user/RobinJ1995/&#39;})
        response = urllib2.urlopen (request)
        jsonStr = response.read()
        
        return json.load(jsonStr)[&#39;data&#39;][&#39;children&#39;]

What does this error mean and what did I do to cause it?

AttributeError(&quot;&#39;str&#39; object has no attribute &#39;read&#39;&quot;)

After using `cgi.parse_qs()`, how to convert the result (dictionary) back to query string? Looking for something similar to  `urllib.urlencode()`. 

How to convert a dictionary to query string in Python?

I&#39;m running Python 3.5.1 for Mac. I want to use `urllib2` module. I tried installing it but I was told that it&#39;s been split into `urllib.request` and `urllib.error` for Python 3. 

My command (running from the framework bin directory for now because it&#39;s not in my path):

    sudo ./pip3 install urllib.request

Returns this:

    Could not find a version that satisfies the requirement urllib.request (from versions: )
    No matching distribution found for urllib.request

I got the same error before when I tried to install `urllib2` in one fell swoop.

Need to install urllib2 for Python 3.5.1

Here&#39;s my code:

    import urllib2.request
    
    response = urllib2.urlopen(&quot;http://www.google.com&quot;)
    html = response.read()
    print(html)

Any help?

Import error: No module name urllib2

How can I download a webpage with a user agent other than the default one on urllib2.urlopen?

Changing user agent on urllib2.urlopen

When using Python 2.7 with `urllib2` to retrieve data from an API, I get the error `[Errno 104] Connection reset by peer`. Whats causing the error, and how should the error be handled so that the script does not crash?

**ticker.py**

	def urlopen(url):
	    response = None
	    request = urllib2.Request(url=url)
	    try:
	        response = urllib2.urlopen(request).read()
	    except urllib2.HTTPError as err:
	        print &quot;HTTPError: {} ({})&quot;.format(url, err.code)
	    except urllib2.URLError as err:
	        print &quot;URLError: {} ({})&quot;.format(url, err.reason)
	    except httplib.BadStatusLine as err:
	        print &quot;BadStatusLine: {}&quot;.format(url)
	    return response

	def get_rate(from_currency=&quot;EUR&quot;, to_currency=&quot;USD&quot;):
	    url = &quot;https://finance.yahoo.com/d/quotes.csv?f=sl1&amp;s=%s%s=X&quot; % (
	        from_currency, to_currency)
	    data = urlopen(url)
	    if &quot;%s%s&quot; % (from_currency, to_currency) in data:
	        return float(data.strip().split(&quot;,&quot;)[1])
	    return None


	counter = 0
	while True:
	    
	    counter = counter + 1
	    if counter==0 or counter%10:
	        rateEurUsd = float(get_rate(&#39;EUR&#39;, &#39;USD&#39;))

        # does more stuff here



**Traceback**

	Traceback (most recent call last):
	  File &quot;/var/www/testApp/python/ticker.py&quot;, line 71, in &lt;module&gt;
	    rateEurUsd = float(get_rate(&#39;EUR&#39;, &#39;USD&#39;))
	  File &quot;/var/www/testApp/python/ticker.py&quot;, line 29, in get_exchange_rate
	    data = urlopen(url)
	  File &quot;/var/www/testApp/python/ticker.py&quot;, line 16, in urlopen
	    response = urllib2.urlopen(request).read()
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 126, in urlopen
	    return _opener.open(url, data, timeout)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 406, in open
	    response = meth(req, response)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 519, in http_response
	    &#39;http&#39;, request, response, code, msg, hdrs)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 438, in error
	    result = self._call_chain(*args)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 378, in _call_chain
	    result = func(*args)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 625, in http_error_302
	    return self.parent.open(new, timeout=req.timeout)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 406, in open
	    response = meth(req, response)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 519, in http_response
	    &#39;http&#39;, request, response, code, msg, hdrs)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 438, in error
	    result = self._call_chain(*args)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 378, in _call_chain
	    result = func(*args)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 625, in http_error_302
	    return self.parent.open(new, timeout=req.timeout)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 400, in open
	    response = self._open(req, data)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 418, in _open
	    &#39;_open&#39;, req)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 378, in _call_chain
	    result = func(*args)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 1207, in http_open
	    return self.do_open(httplib.HTTPConnection, req)
	  File &quot;/usr/lib/python2.7/urllib2.py&quot;, line 1180, in do_open
	    r = h.getresponse(buffering=True)
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 1030, in getresponse
	    response.begin()
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 407, in begin
	    version, status, reason = self._read_status()
	  File &quot;/usr/lib/python2.7/httplib.py&quot;, line 365, in _read_status
	    line = self.fp.readline()
	  File &quot;/usr/lib/python2.7/socket.py&quot;, line 447, in readline
	    data = self._sock.recv(self._rbufsize)
	socket.error: [Errno 104] Connection reset by peer
	error: Forever detected script exited with code: 1


Python handling socket.error: [Errno 104] Connection reset by peer

I am trying to GET a URL using Python and the response is JSON. However, when I run

    import urllib2
    response = urllib2.urlopen(&#39;https://api.instagram.com/v1/tags/pizza/media/XXXXXX&#39;)
    html=response.read()
    print html

The html is of type str and I am expecting a JSON. Is there any way I can capture the response as JSON or a python dictionary instead of a str.

Python urllib2: Receive JSON response from url

I&#39;m trying to download and save an image from the web using python&#39;s `requests` module.

Here is the (working) code I used:

    img = urllib2.urlopen(settings.STATICMAP_URL.format(**data))
    with open(path, &#39;w&#39;) as f:
        f.write(img.read())

Here is the new (non-working) code using `requests`:

    r = requests.get(settings.STATICMAP_URL.format(**data))
    if r.status_code == 200:
        img = r.raw.read()
        with open(path, &#39;w&#39;) as f:
            f.write(img)

Can you help me on what attribute from the response to use from `requests`?

How to download image using requests

I am trying to download files from google drive and all I have is the drive&#39;s URL.

I have read about google API that talks about some `drive_service` and `MedioIO`, which also requires some credentials( mainly JSON `file/OAuth`). But I am unable to get any idea about how it is working. 

Also, tried `urllib2.urlretrieve`, but my case is to get files from the drive. Tried `wget` too but no use.

Tried `PyDrive` library. It has good upload functions to drive but no download options.

Any help will be appreciated.
Thanks.

Python: download files from google drive using url

I am trying to use  urllib2 to open url and to send specific cookie text to the server. E.g. I want to open site [Solve chess problems][1], with a specific cookie, e.g. search=1. How do I do it?

I am trying to do the following:

    import urllib2
    (need to add cookie to the request somehow)
    urllib2.urlopen(&quot;http://chess-problems.prg&quot;)

Thanks in advance




  [1]: http://chess-problems.prg 

python: urllib2 how to send cookie with urlopen request

I have

    import urllib2
    try:
       urllib2.urlopen(&quot;some url&quot;)
    except urllib2.HTTPError:
       &lt;whatever&gt;

but what I end up is catching any kind of HTTP error. I want to catch only if the specified webpage doesn&#39;t exist (404?).
  



How do I catch a specific HTTP error in Python?

Update: based on Lee&#39;s comment I decided to condense my code to a really simple script and run it from the command line:

    import urllib2
    import sys
    
    username = sys.argv[1]
    password = sys.argv[2]
    url = sys.argv[3]
    print(&quot;calling %s with %s:%s\n&quot; % (url, username, password))
    
    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, url, username, password)
    urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))
    
    req = urllib2.Request(url)
    f = urllib2.urlopen(req)
    data = f.read()
    print(data)

Unfortunately it still won&#39;t generate the `Authorization` header (per Wireshark) :(

I&#39;m having a problem sending basic AUTH over urllib2. I took a look at [this article][1], and followed the example. My code:

    passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
    passman.add_password(None, &quot;api.foursquare.com&quot;, username, password)
    urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))

    req = urllib2.Request(&quot;http://api.foursquare.com/v1/user&quot;)    
    f = urllib2.urlopen(req)
    data = f.read()

I&#39;m seeing the following on the Wire via wireshark:

    GET /v1/user HTTP/1.1
    Host: api.foursquare.com
    Connection: close
    Accept-Encoding: gzip
    User-Agent: Python-urllib/2.5 

You can see the Authorization is not sent, vs. when I send a request via curl: `curl -u user:password http://api.foursquare.com/v1/user`

    GET /v1/user HTTP/1.1
    Authorization: Basic =SNIP=
    User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
    Host: api.foursquare.com
    Accept: */*

For some reason my code seems to not send the authentication - anyone see what I&#39;m missing?

thanks

-simon


  [1]: http://www.voidspace.org.uk/python/articles/authentication.shtml

Python urllib2 Basic Auth Problem

While porting code from `python2` to `3`, I get this error when reading from a URL 

&gt;TypeError: initial_value must be str or None, not bytes.

    import urllib
    import json
    import gzip
    from urllib.parse import urlencode
    from urllib.request import Request

    
    service_url = &#39;https://babelfy.io/v1/disambiguate&#39;
    text = &#39;BabelNet is both a multilingual encyclopedic dictionary and a semantic network&#39;
    lang = &#39;EN&#39;
    Key  = &#39;KEY&#39;

        params = {
            &#39;text&#39; : text,
            &#39;key&#39;  : Key,
            &#39;lang&#39; :&#39;EN&#39;

            }
        
    url = service_url + &#39;?&#39; + urllib.urlencode(params)
    request = Request(url)
    request.add_header(&#39;Accept-encoding&#39;, &#39;gzip&#39;)
    response = urllib.request.urlopen(request)
    if response.info().get(&#39;Content-Encoding&#39;) == &#39;gzip&#39;:
                buf = StringIO(response.read())
                f = gzip.GzipFile(fileobj=buf)
                data = json.loads(f.read())

The exception is thrown at this line  

    buf = StringIO(response.read())  
If I use python2, it works fine.

Python3 error: initial_value must be str or None, with StringIO

I open urls with:

`site = urllib2.urlopen(&#39;http://google.com&#39;)`

And what I want to do is connect the same way with a proxy
I got somewhere telling me:

`site = urllib2.urlopen(&#39;http://google.com&#39;, proxies={&#39;http&#39;:&#39;127.0.0.1&#39;})`

but that didn&#39;t work either.

I know urllib2 has something like a proxy handler, but I can&#39;t recall that function.

Proxy with urllib2

I&#39;m using 

     data=urllib2.urlopen(url).read()

I want to know:

 1. How can I tell if the data at a URL is gzipped?

 1. Does urllib2 automatically uncompress the data if it is gzipped? Will the data always be a string?

Does python urllib2 automatically uncompress gzip data fetched from webpage?

I receive a &#39;HTTP Error 500: Internal Server Error&#39; response, but I still want to read the data inside the error HTML.

With Python 2.6, I normally fetch a page using:

    import urllib2
    url = &quot;http://google.com&quot;
    data = urllib2.urlopen(url)
    data = data.read()

When attempting to use this on the failing URL, I get the exception `urllib2.HTTPError`:

    urllib2.HTTPError: HTTP Error 500: Internal Server Error

How can I fetch such error pages (with or without `urllib2`), all while they are returning Internal Server Errors?

Note that with Python 3, the corresponding exception is `urllib.error.HTTPError`.

Overriding urllib2.HTTPError or urllib.error.HTTPError and reading response HTML anyway

Ok so I need to download some web pages using Python and did a quick investigation of my options.

Included with Python:

[urllib][1] - seems to me that I should use urllib2 instead. urllib has no cookie support, HTTP/FTP/local files only (no SSL)

[urllib2][2] - complete HTTP/FTP client, supports most needed things like cookies, does not support all HTTP verbs (only GET and POST, no TRACE, etc.)

Full featured:

[mechanize][3] - can use/save Firefox/IE cookies, take actions like follow second link, actively maintained (0.2.5 released in March 2011)

[PycURL][4] - supports everything curl does (FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE and LDAP), bad news: not updated since Sep 9, 2008 (7.19.0)

New possibilities:

[urllib3][5] - supports connection re-using/pooling and file posting

Deprecated (a.k.a. use urllib/urllib2 instead):

[httplib][6] - HTTP/HTTPS only (no FTP)

[httplib2][7] - HTTP/HTTPS only (no FTP)

The first thing that strikes me is that urllib/urllib2/PycURL/mechanize are all pretty mature solutions that work well. mechanize and PycURL ship with a number of Linux distributions (e.g. Fedora 13) and BSDs so installation is a non issue typically (so that&#39;s good). 

urllib2 looks good but I&#39;m wondering why PycURL and mechanize both seem very popular, is there something I am missing (i.e. if I use urllib2 will I paint myself in to a corner at some point?). I&#39;d really like some feedback on the pros/cons of these things so I can make the best choice for myself. 

Edit: added note on verb support in urllib2


  [1]: http://docs.python.org/library/urllib.html
  [2]: http://docs.python.org/library/urllib2.html
  [3]: http://wwwsearch.sourceforge.net/mechanize/
  [4]: http://pycurl.sourceforge.net/
  [5]: http://code.google.com/p/urllib3/
  [6]: http://docs.python.org/library/httplib.html
  [7]: http://code.google.com/p/httplib2/

Which is best in Python: urllib2, PycURL or mechanize?

can anyone point out a tutorial that shows me how to do a POST request using urllib2 with the data being in JSON format?

urllib2 and json

I want to send a custom &quot;Accept&quot; header in my request when using urllib2.urlopen(..). How do I do that?

How do I send a custom header with urllib2 in a HTTP Request?

I was wondering, how do you close a connection with Requests (python-requests.org)? 

With `httplib` it&#39;s `HTTPConnection.close()`, but how do I do the same with Requests? 

Code:

    r = requests.post(&quot;https://stream.twitter.com/1/statuses/filter.json&quot;, data={&#39;track&#39;:toTrack}, auth=(&#39;username&#39;, &#39;passwd&#39;))
	for line in r.iter_lines():
		if line:
			self.mongo[&#39;db&#39;].tweets.insert(json.loads(line))

Python-Requests close http connection

I&#39;m using the timeout parameter within the urllib2&#39;s urlopen.

    urllib2.urlopen(&#39;http://www.example.org&#39;, timeout=1)

How do I tell Python that if the timeout expires a custom error should be raised?

___

Any ideas?

Handling urllib2&#39;s timeout? - Python

I am writing some code to interface with redmine and I need to upload some files as part of the process, but I am not sure how to do a POST request from python containing a binary file.

I am trying to mimic the commands [here][1]:

    curl --data-binary &quot;@image.png&quot; -H &quot;Content-Type: application/octet-stream&quot; -X POST -u login:password http://redmine/uploads.xml

In python (below), but it does not seem to work. I am not sure if the problem is somehow related to encoding the file or if something is wrong with the headers.

    import urllib2, os

    FilePath = &quot;C:\somefolder\somefile.7z&quot;
    FileData = open(FilePath, &quot;rb&quot;)
    length = os.path.getsize(FilePath)

    password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
    password_manager.add_password(None, &#39;http://redmine/&#39;, &#39;admin&#39;, &#39;admin&#39;)
    auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
    opener = urllib2.build_opener(auth_handler)
    urllib2.install_opener(opener)
    request = urllib2.Request( r&#39;http://redmine/uploads.xml&#39;, FileData)
    request.add_header(&#39;Content-Length&#39;, &#39;%d&#39; % length)
    request.add_header(&#39;Content-Type&#39;, &#39;application/octet-stream&#39;)
    try:
        response = urllib2.urlopen( request)
        print response.read()
    except urllib2.HTTPError as e:
        error_message = e.read()
        print error_message


I have access to the server and it looks like a encoding error:

    ...
    invalid byte sequence in UTF-8
    Line: 1
    Position: 624
    Last 80 unconsumed characters:
    7z&#188;&#175;&#39;&#197;&#208;&#208;&#183;2^&#212;&#248;&#235;4g&#184;R&lt;s&#252;&#240;&#237;6k&#196;&#164;&#170;&#182;!&#187;=}jcdjSP&#250;&#225;-&#186;#&#187;&#196;AtD&#187;H7&#202;!&#230;&#189;]j):

    (further down)

    Started POST &quot;/uploads.xml&quot; for 192.168.0.117 at 2013-01-16 09:57:49 -0800
    Processing by AttachmentsController#upload as XML
    WARNING: Can&#39;t verify CSRF token authenticity
      Current user: anonymous
    Filter chain halted as :authorize_global rendered or redirected
    Completed 401 Unauthorized in 13ms (ActiveRecord: 3.1ms)


  [1]: http://www.redmine.org/projects/redmine/wiki/Rest_api_with_curl

Python POST binary data

I want to ignore the `certification validation` during my request to the server with an internal corporate link.

With python `requests` library I would do this:

    r = requests.get(link, allow_redirects=False,verify=False)

How do I do the same with urllib2 library?

python ignore certificate validation urllib2

There&#39;s a lot of stuff out there on urllib2 and POST calls, but I&#39;m stuck on a problem.

I&#39;m trying to do a simple POST call to a service:

    url = &#39;http://myserver/post_service&#39;
    data = urllib.urlencode({&#39;name&#39; : &#39;joe&#39;,
                             &#39;age&#39;  : &#39;10&#39;})
    content = urllib2.urlopen(url=url, data=data).read()
    print content

I can see the server logs and it says that I&#39;m doing GET calls, when I&#39;m sending the data 
argument to urlopen.

The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I&#39;m also trying with a POST within a HTML form).


All Urllib2 Solutions on Gang of Coders

Total of 33 Urllib2 Solutions

Python's `urllib2`: Why do I get error 403 when I `urlopen` a Wikipedia page?