how to store a complex object in redis (using redis-py)

The hmset function can set the value of each field, but I found that if the value itself is a complex structured object, the value return from hget is a serialized string, not the original object

e.g

images= [{'type':'big', 'url':'....'},
     {'type':'big', 'url':'....'},
     {'type':'big', 'url':'....'}]   

redis = Redis()
redis.hset('photo:1', 'images', images)

i = redis.hget('photo:1', 'images')
print type(i)

the type of i is a string, not a python object, is there any way to solve this problem besides manually parse each fields?

Actually, you can store python objects in redis using the built-in module pickle.

Here is example.

import pickle
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)
obj = ExampleObject()
pickled_object = pickle.dumps(obj)
r.set('some_key', pickled_object)
unpacked_object = pickle.loads(r.get('some_key'))
obj == unpacked_object

If your data is JSON-serializable, then that may be the better option than saving python pickles to an external database, since it's a more common standard outside of Python, is more human-readable on its own, and avoids a rather large attack vector.

JSON Example:

import json
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)

images= [
    {'type':'big', 'url':'....'},
    {'type':'big', 'url':'....'},
    {'type':'big', 'url':'....'},
]

# Convert python dict to JSON str and save to Redis
json_images = json.dumps(images)
r.set('images', json_images)

# Read saved JSON str from Redis and unpack into python dict
unpacked_images = json.loads(r.get('images'))
images == unpacked_images

python 3:

unpacked_images = json.loads(r.get('images').decode('utf-8'))
images == unpacked_images

You can't create nested structures in Redis, meaning you can't (for example) store a native redis list inside a native redis hash-map.

If you really need nested structures, you might want to just store a JSON-blob (or something similar) instead. Another option is to store an "id"/key to a different redis object as the value of the map key, but that requires multiple calls to the server to get the full object.

I created a library, SubRedis, which lets you create much more complex structures/hierarchies in redis. If you give it a redis instance and a prefix, it gives you a nearly fully capable and independent redis instance.

redis = Redis()
photoRedis = SubRedis("photo:%s" % photoId, redis)
photoRedis.hmset('image0', images[0])
photoRedis.hmset('image1', images[1])
...

SubRedis just ends up prepending the string passed into it as a prefix onto the flat redis data structure. I find this to be a convenient wrapper for a pattern I end up doing a lot in redis -- prepending some id to nest some data.

Here is a simple wrapper around Redis which pickles/unpickles data structures:

from redis import Redis
from collections import MutableMapping
from pickle import loads, dumps


class RedisStore(MutableMapping):

    def __init__(self, engine):
        self._store = Redis.from_url(engine)

    def __getitem__(self, key):
        return loads(self._store[dumps(key)])

    def __setitem__(self, key, value):
        self._store[dumps(key)] = dumps(value)

    def __delitem__(self, key):
        del self._store[dumps(key)]

    def __iter__(self):
        return iter(self.keys())

    def __len__(self):
        return len(self._store.keys())

    def keys(self):
        return [loads(x) for x in self._store.keys()]

    def clear(self):
        self._store.flushdb()


d = RedisStore('redis://localhost:6379/0')
d['a'] = {'b': 1, 'c': 10}
print repr(d.items())
# this will not work: (it updates a temporary copy and not the real data)
d['a']['b'] = 2
print repr(d.items())
# this is how to update sub-structures:
t = d['a']
t['b'] = 2
d['a'] = t
print repr(d.items())
del d['a']

# Here is another way to implement dict-of-dict eg d['a']['b']
d[('a', 'b')] = 1
d[('a', 'b')] = 2
print repr(d.items())
# Hopefully you do not need the equivalent of d['a']
print repr([{x[0][1]: x[1]} for x in d.items() if x[0][0] == 'a'])
del d[('a', 'b')]
del d[('a', 'c')]

If you prefer plaintext-readable data in redis (pickle stores a binary version of it), you can replace pickle.dumps with repr and pickle.loads with ast.literal_eval. For json, use json.dumps and json.loads.

If you always use keys which are a simple string, you can remove the pickling from the key.

You can use RedisWorks library.

pip install redisworks

>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}}  # saves it as Hash
>>> print(root.something)  # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2

It converts python types to Redis types and vice-versa.

>>> root.sides = [10, [1, 2]]  # saves it as list in Redis.
>>> print(root.sides)  # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>

Disclaimer: I wrote the library. Here is the code: <https://github.com/seperman/redisworks>

You can use RedisJSON from RedisLabs with client for python. It's supported the nested data structure. Very useful for tasks like this.

Some code from the example:

   # Set the key `obj` to some object
   obj = {
       'answer': 42,
       'arr': [None, True, 3.14],
       'truth': {
           'coord': 'out there'
       }
   }
   rj.jsonset('obj', Path.rootPath(), obj)

   # Get something
   print 'Is there anybody... {}?'.format(
       rj.jsonget('obj', Path('.truth.coord'))
   )

I have faced a similar use case recently. Storing a complex data structure in redis hash.

I think the best way to resolve the issue is by serialiasing the json object to string and store it as value for another object.

Typescript example

Object to store in hashmap

const payload = {
 "k1":"v1",
 "k2": "v2",
 "k3": {
     "k4":"v4",
     "k5":"v5"
  }
}

Store this payload as

await redis.hmset('hashMapKey', {somePayloadKey: JSON.stringify(payload) });

This can be retrieved as

      const result = await redis.hgetall('hashMapKey');
      const payload = JSON.parse(result.somePayloadKey);

hmset and hgetall are tedis equivalents to HMSET and HGETALL in redis.

Hope this helps.

 for saving object in redis first convert object into stringify JSON
 StringifyImages = json.dumps(images)
redis.set('images', StringifyImages)

# Read stringify object from redis and parse it 
ParseImages = json.loads(redis.get('images'))

You can just store your structure as is and do an 'eval' to convert from String to Object:

images= [{'type':'big', 'url':'....'},
 {'type':'big', 'url':'....'},
 {'type':'big', 'url':'....'}]   
redis = Redis()
redis.hset('photo:1', 'images', images)

i = eval(redis.hget('photo:1', 'images'))
print type(i) #type of i should be list instead of string now