Protobuf to json in python
PythonJsonProtocol BuffersPython Problem Overview
I have an object that I de-serialize using protobuf
in Python. When I print the object it looks like a python object, however when I try to convert it to json
I have all sorts of problems.
For example, if I use json.dumps()
I get that the object (the generated code from protoc) does not contain a _ dict _ error.
If I use jsonpickle I get UnicodeDecodeError: 'utf8' codec can't decode byte 0x9d in position 97: invalid start byte
.
Test code below is using jsonpickle
with the error shown above.
if len(sys.argv) < 2:
print ("Error: missing ser file")
sys.exit()
else :
fileLocation = sys.argv[1]
org = BuildOrgObject(fileLocation)
org = org.Deserialize()
#print (org)
jsonObj = jsonpickle.encode(org)
print (jsonObj)
Python Solutions
Solution 1 - Python
I'd recommend using protobuf↔json converters from google's protobuf library:
from google.protobuf.json_format import MessageToJson
json_obj = MessageToJson(org)
You can also serialise the protobuf to a dictionary:
from google.protobuf.json_format import MessageToDict
dict_obj = MessageToDict(org)
Refer to the protobuf package API documentation: https://developers.google.com/protocol-buffers/docs/reference/python/ (see module google.protobuf.json_format
).
Solution 2 - Python
If you need to go straight to json take a look at the protobuf-to-json library, but you'll have to install that manually.
But I would recommend that you use the protobuf-to-dict library instead for a few reasons:
- It is accessible from pypi so you can simply
pip install protobuf-to-dict
or include it in arequirements.txt
dict
can be converted to json and might be more useful than a json string
Solution 3 - Python
You can also user SerializeToString.
org.SerializeToString()
Solution 4 - Python
If you are using an older version that doesn't has the preserving_proto_field_name field:
from google.protobuf.json_format import MessageToJson
def proto_to_json(proto_obj):
json_obj = MessageToJson(proto_obj):
json_obj = MessageToJso, including_default_value_fields=True)
# Change lowerCamelCase of google Json conversion to the snake_case as in original protobuf
dict_obj = dict((re.sub(r'(?<!^)(?=[A-Z])', '_', k).lower(),v) for k, v in json.loads(json_obj).items())
if hasattr(proto_obj, 'uuid'):
dict_obj["uuid"] = proto_obj.uuid.encode("hex")
return json.dumps(dict_obj, indent=4, sort_keys=True)
Solution 5 - Python
Here's my function to convert a proto3 object to a JSON object (i.e. Python dictionary):
def protobuf_to_dict(proto_obj):
key_list = proto_obj.DESCRIPTOR.fields_by_name.keys()
d = {}
for key in key_list:
d[key] = getattr(proto_obj, key)
return d
Since the converters from Google's protobuf library don't seem to work in some cases with the 3.19 version, this function leverages the Descriptor class present on each Protobuf object.
Here, getattr(obj, string_attribute)
returns the value given by obj.attribute