Protobuf to json in python

PythonJsonProtocol Buffers

Python Problem Overview


I have an object that I de-serialize using protobuf in Python. When I print the object it looks like a python object, however when I try to convert it to json I have all sorts of problems.

For example, if I use json.dumps() I get that the object (the generated code from protoc) does not contain a _ dict _ error.

If I use jsonpickle I get UnicodeDecodeError: 'utf8' codec can't decode byte 0x9d in position 97: invalid start byte.

Test code below is using jsonpickle with the error shown above.

if len(sys.argv) < 2:
    print ("Error: missing ser file")
    sys.exit()
else :
    fileLocation = sys.argv[1]
    
org = BuildOrgObject(fileLocation) 

org = org.Deserialize()

       
#print (org)
jsonObj = jsonpickle.encode(org)
print (jsonObj)

Python Solutions


Solution 1 - Python

I'd recommend using protobuf↔json converters from google's protobuf library:

from google.protobuf.json_format import MessageToJson

json_obj = MessageToJson(org)

You can also serialise the protobuf to a dictionary:

from google.protobuf.json_format import MessageToDict
dict_obj = MessageToDict(org)

Refer to the protobuf package API documentation: https://developers.google.com/protocol-buffers/docs/reference/python/ (see module google.protobuf.json_format).

Solution 2 - Python

If you need to go straight to json take a look at the protobuf-to-json library, but you'll have to install that manually.

But I would recommend that you use the protobuf-to-dict library instead for a few reasons:

  1. It is accessible from pypi so you can simply pip install protobuf-to-dict or include it in a requirements.txt
  2. dict can be converted to json and might be more useful than a json string

Solution 3 - Python

You can also user SerializeToString.

org.SerializeToString()

Solution 4 - Python

If you are using an older version that doesn't has the preserving_proto_field_name field:

from google.protobuf.json_format import MessageToJson
def proto_to_json(proto_obj):
    json_obj = MessageToJson(proto_obj):
    json_obj = MessageToJso, including_default_value_fields=True)
    # Change lowerCamelCase of google Json conversion to the snake_case as in original protobuf
    dict_obj = dict((re.sub(r'(?<!^)(?=[A-Z])', '_', k).lower(),v) for k, v in json.loads(json_obj).items())
    if hasattr(proto_obj, 'uuid'):
        dict_obj["uuid"] = proto_obj.uuid.encode("hex")
    return json.dumps(dict_obj, indent=4, sort_keys=True)

Solution 5 - Python

Here's my function to convert a proto3 object to a JSON object (i.e. Python dictionary):

def protobuf_to_dict(proto_obj):
    key_list = proto_obj.DESCRIPTOR.fields_by_name.keys()
    d = {}
    for key in key_list:
        d[key] = getattr(proto_obj, key)
    return d

Since the converters from Google's protobuf library don't seem to work in some cases with the 3.19 version, this function leverages the Descriptor class present on each Protobuf object.

Here, getattr(obj, string_attribute) returns the value given by obj.attribute

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionexHashView Question on Stackoverflow
Solution 1 - Pythondenis-suminView Answer on Stackoverflow
Solution 2 - PythonKevin HillView Answer on Stackoverflow
Solution 3 - PythonShubham KanyalView Answer on Stackoverflow
Solution 4 - Pythonvr286View Answer on Stackoverflow
Solution 5 - PythonNiravView Answer on Stackoverflow