Python requests speed up using keep-alive

PythonHttpPython RequestsPython Sockets

Python Problem Overview


In the HTTP protocol you can send many requests in one socket using keep-alive and then receive the response from server at once, so that will significantly speed up whole process. Is there any way to do this in python requests lib? Or are there any other ways to speed this up that well using requests lib?

Python Solutions


Solution 1 - Python

Yes, there is. Use requests.Session and it will do keep-alive by default.

I guess I should include a quick example:

import logging
import requests

logging.basicConfig(level=logging.DEBUG)
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
s.get('http://httpbin.org/cookies/set/anothercookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)

You will note that these log message occur

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/sessioncookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 55
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/anothercookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

If you wait a little while, and repeat the last get call

INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90

Note that it resets the dropped connection, i.e. reestablishing the connection to the server to make the new request.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionPaulOverflowView Question on Stackoverflow
Solution 1 - PythonmetatoasterView Answer on Stackoverflow