How to save requests (python) cookies to a file?
PythonCookiesPython RequestsPython Problem Overview
How to use the library requests
(in python) after a request
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
bot = requests.session()
bot.get('http://google.com')
to keep all the cookies in a file and then restore the cookies from a file.
Python Solutions
Solution 1 - Python
There is no immediate way to do so, but it's not hard to do.
You can get a CookieJar
object from the session with session.cookies
, and use pickle
to store it to a file.
A full example:
import requests, pickle
session = requests.session()
# Make some calls
with open('somefile', 'wb') as f:
pickle.dump(session.cookies, f)
Loading is then:
session = requests.session() # or an existing session
with open('somefile', 'rb') as f:
session.cookies.update(pickle.load(f))
The requests
library uses the requests.cookies.RequestsCookieJar()
subclass, which explicitly supports pickling and a dict-like API. The RequestsCookieJar.update()
method can be used to update an existing session cookie jar with the cookies loaded from the pickle file.
Solution 2 - Python
After a call such as r = requests.get()
, r.cookies
will return a RequestsCookieJar
which you can directly pickle
, i.e.
import pickle
def save_cookies(requests_cookiejar, filename):
with open(filename, 'wb') as f:
pickle.dump(requests_cookiejar, f)
def load_cookies(filename):
with open(filename, 'rb') as f:
return pickle.load(f)
#save cookies
r = requests.get(url)
save_cookies(r.cookies, filename)
#load cookies and do a request
requests.get(url, cookies=load_cookies(filename))
If you want to save your cookies in human-readable format, you have to do some work to extract the RequestsCookieJar
to a LWPCookieJar
.
import cookielib
def save_cookies_lwp(cookiejar, filename):
lwp_cookiejar = cookielib.LWPCookieJar()
for c in cookiejar:
args = dict(vars(c).items())
args['rest'] = args['_rest']
del args['_rest']
c = cookielib.Cookie(**args)
lwp_cookiejar.set_cookie(c)
lwp_cookiejar.save(filename, ignore_discard=True)
def load_cookies_from_lwp(filename):
lwp_cookiejar = cookielib.LWPCookieJar()
lwp_cookiejar.load(filename, ignore_discard=True)
return lwp_cookiejar
#save human-readable
r = requests.get(url)
save_cookies_lwp(r.cookies, filename)
#you can pass a LWPCookieJar directly to requests
requests.get(url, cookies=load_cookies_from_lwp(filename))
Solution 3 - Python
Expanding on @miracle2k's answer, requests Session
s are documented to work with any cookielib CookieJar
. The LWPCookieJar
(and MozillaCookieJar
) can save and load their cookies to and from a file. Here is a complete code snippet which will save and load cookies for a requests session. The ignore_discard
parameter is used to work with httpbin for the test, but you may not want to include it your in real code.
import os
from cookielib import LWPCookieJar
import requests
s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
# Create a new cookies file and set our Session's cookies
print('setting cookies')
s.cookies.save()
r = s.get('http://httpbin.org/cookies/set?k1=v1&k2=v2')
else:
# Load saved cookies from the file and use them in a request
print('loading saved cookies')
s.cookies.load(ignore_discard=True)
r = s.get('http://httpbin.org/cookies')
print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)
Solution 4 - Python
I offer a way by json:
to save cookie -
import json
with open('cookie.txt', 'w') as f:
json.dump(requests.utils.dict_from_cookiejar(bot.cookies), f)
and to load cookie -
import json
session = requests.session() # or an existing session
with open('cookie.txt', 'r') as f:
cookies = requests.utils.cookiejar_from_dict(json.load(f))
session.cookies.update(cookies)
Solution 5 - Python
This will do the job:
session.cookies = LWPCookieJar('cookies.txt')
The CookieJar API requires you to call load() and save() manually though. If you do not care about the cookies.txt format, I have a ShelvedCookieJar implementation that will persist on change.
Solution 6 - Python
I found that the other answers had problems:
- They didn't apply to sessions.
- They didn't save and load properly. Only the cookie name and value was saved, the expiry date, domain name, etc. was all lost.
This answer fixes these two issues:
import requests.cookies
def save_cookies(session, filename):
if not os.path.isdir(os.path.dirname(filename)):
return False
with open(filename, 'w') as f:
f.truncate()
pickle.dump(session.cookies._cookies, f)
def load_cookies(session, filename):
if not os.path.isfile(filename):
return False
with open(filename) as f:
cookies = pickle.load(f)
if cookies:
jar = requests.cookies.RequestsCookieJar()
jar._cookies = cookies
session.cookies = jar
else:
return False
Then just call save_cookies(session, filename)
to save or load_cookies(session, filename)
to load. Simple as that.
Solution 7 - Python
You can pickle the cookies object directly:
cookies = pickle.dumps(session.cookies)
The dict representation misses a lot of informations: expiration, domain, path...
It depends on the usage you intend to do with the cookies, but if you don't have informations about the expiration, for example, you should implement the logic to track expiration by hand.
Pickling the library returned object lets you easily reconstruct the state, then you can relay on the library implementation.
Obviously, this way, the consumer of the pickled object needs to use the same library
Solution 8 - Python
Simple way to convert cookies into list of dicts and save to json or db.
This is methods of class which have session
attribute.
def dump_cookies(self):
cookies = []
for c in self.session.cookies:
cookies.append({
"name": c.name,
"value": c.value,
"domain": c.domain,
"path": c.path,
"expires": c.expires
})
return cookies
def load_cookies(self, cookies):
for c in cookies:
self.session.cookies.set(**c)
All we need is five parameters such as: name
, value
, domain
, path
, expires
Solution 9 - Python
code for python 3
Note that the great majority of cookies on the Internet are Netscape cookies. so if you want to save cookies to disk in the Mozilla cookies.txt file format (which is also used by the Lynx and Netscape browsers) use MozillaCookieJar
from http.cookiejar import MozillaCookieJar
import requests
s = requests.Session()
s.cookies = MozillaCookieJar('cookies.txt')
# or s.cookies = MozillaCookieJar() and later use s.cookies.filename = 'cookies.txt' or pass the file name to save method.
response = s.get('https://www.msn.com')
s.cookies.save()
the file is overwritten if it already exists, thus wiping all the cookies it contains. Saved cookies can be restored later using the load() or revert() methods.
> Note that the save() method won’t save session cookies anyway, unless > you ask otherwise by passing a true ignore_discard argument.
s.cookies.save(ignore_discard=True)
using load method:
load cookies from a file.
Old cookies are kept unless overwritten by newly loaded ones.
s.cookies.load()
using revert method:
Clear all cookies and reload cookies from a saved file.
s.cookies.revert()
you may need also to pass a true ignore_discard argument in load or revert methods.
note about using MozillaCookieJar :
> Note This loses information about RFC 2965 cookies, and also about > newer or non-standard cookie-attributes such as port.
Solution 10 - Python
dtheodor's answer got 95% there, except this:
session = requests.session(cookies=cookies)
For me this raises an exception saying session() does not takes arguments.
I worked around it by taking the keys/values on the cookie.get_dict and adding them manually to the session using:
session.cookies.set(cookies.keys()[n],cookies.values()[n])
Solution 11 - Python
For all the solutions that use Cookielib, in Python 3.0 its been changed to http.cookiejar Please lookup Python 3.2 won't import cookielib