Deserialize json with known and unknown fields

Given following json result: The default json result has a known set of fields:

{
    "id": "7908",
    "name": "product name"
}

But can be extended with additional fields (in this example _unknown_field_name_1 and _unknown_field_name_2) of which the names are not known when requesting the result.

{
    "id": "7908",
    "name": "product name",
    "_unknown_field_name_1": "some value",
    "_unknown_field_name_2": "some value"
}

I would like the json result to be serialized and deserialized to and from a class with properties for the known fields and map the unknown fields (for which there are no properties) to a property (or multiple properties) like a dictionary so they can be accessed and modified.

public class Product
{
    public string id { get; set; }
    public string name { get; set; }
    public Dictionary<string, string> fields { get; set; }
}

I think I need a way to plug into a json serializer and do the mapping for the missing members myself (both for serialize and deserialize). I have been looking at various possibilities:

• json.net and custom contract resolvers (can't figure out how to do it)
• datacontract serializer (can only override onserialized, onserializing)
• serialize to dynamic and do custom mapping (this might work, but seems a lot of work)
• let product inheriting from DynamicObject (serializers work with reflection and do not invoke the trygetmember and trysetmember methods)

I'm using restsharp, but any serializer can be plugged in.

Oh, and I cannot change the json result, and this or this didn't help me either.

Update: This looks more like it: http://geekswithblogs.net/DavidHoerster/archive/2011/07/26/json.net-custom-convertersndasha-quick-tour.aspx

An even easier option to tackling this problem would be to use the JsonExtensionDataAttribute from JSON .NET

public class MyClass
{
   // known field
   public decimal TaxRate { get; set; }

   // extra fields
   [JsonExtensionData]
   private IDictionary<string, JToken> _extraStuff;
}

There's a sample of this on the project blog here

UPDATE Please note this requires JSON .NET v5 release 5 and above

See https://gist.github.com/LodewijkSioen/5101814

What you were looking for was a custom JsonConverter

This is a way you could solve it, although I don't like it that much. I solved it using Newton/JSON.Net. I suppose you could use the JsonConverter for deserialization aswell.

private const string Json = "{\"id\":7908,\"name\":\"product name\",\"_unknown_field_name_1\":\"some value\",\"_unknown_field_name_2\":\"some value\"}";

    [TestMethod]
    public void TestDeserializeUnknownMembers()
    {
        var @object = JObject.Parse(Json);

        var serializer = new Newtonsoft.Json.JsonSerializer();
        serializer.MissingMemberHandling = Newtonsoft.Json.MissingMemberHandling.Error;
        serializer.Error += (sender, eventArgs) =>
            {
                var contract = eventArgs.CurrentObject as Contract ?? new Contract();
                contract.UnknownValues.Add(eventArgs.ErrorContext.Member.ToString(), @object[eventArgs.ErrorContext.Member.ToString()].Value<string>());
                eventArgs.ErrorContext.Handled = true;
            };

        using (MemoryStream memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(Json)))
        using (StreamReader streamReader = new StreamReader(memoryStream))
        using (JsonReader jsonReader = new JsonTextReader(streamReader))
        {
            var result = serializer.Deserialize<Contract>(jsonReader);
            Assert.IsTrue(result.UnknownValues.ContainsKey("_unknown_field_name_1"));
            Assert.IsTrue(result.UnknownValues.ContainsKey("_unknown_field_name_2"));
        }
    }

    [TestMethod]
    public void TestSerializeUnknownMembers()
    {
        var deserializedObject = new Contract
        {
            id = 7908,
            name = "product name",
            UnknownValues = new Dictionary<string, string>
        {
            {"_unknown_field_name_1", "some value"},
            {"_unknown_field_name_2", "some value"}
        }
        };

        var json = JsonConvert.SerializeObject(deserializedObject, new DictionaryConverter());
        Console.WriteLine(Json);
        Console.WriteLine(json);
        Assert.AreEqual(Json, json);
    }
}

class DictionaryConverter : JsonConverter
{
    public DictionaryConverter()
    {

    }

    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(Contract);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var contract = value as Contract;
        var json = JsonConvert.SerializeObject(value);
        var dictArray = String.Join(",", contract.UnknownValues.Select(pair => "\"" + pair.Key + "\":\"" + pair.Value + "\""));

        json = json.Substring(0, json.Length - 1) + "," + dictArray + "}";
        writer.WriteRaw(json);
    }
}

class Contract
{
    public Contract()
    {
        UnknownValues = new Dictionary<string, string>();
    }

    public int id { get; set; }
    public string name { get; set; }

    [JsonIgnore]
    public Dictionary<string, string> UnknownValues { get; set; }
}
}

I thought I'd throw my hat in the ring since I had a similar problem recently. Here's an example of the JSON I wanted to deserialize:

{
    "agencyId": "agency1",
    "overrides": {
        "assumption.discount.rates": "value: 0.07",
        ".plan": {
            "plan1": {
                "assumption.payroll.growth": "value: 0.03",
                "provision.eeContrib.rate": "value: 0.35"
            },
            "plan2": {
                ".classAndTier": {
                    "misc:tier1": {
                        "provision.eeContrib.rate": "value: 0.4"
                    },
                    "misc:tier2": {
                        "provision.eeContrib.rate": "value: 0.375"
                    }
                }
            }
        }
    }
}

This is for a system where overrides apply at different levels and are inherited down the tree. In any case, the data model I wanted was something that would allow me to have a property bag with these special inheritance rules also supplied.

What I ended up with was the following:

public class TestDataModel
{
    public string AgencyId;
    public int Years;
    public PropertyBagModel Overrides;
}

public class ParticipantFilterModel
{
    public string[] ClassAndTier;
    public string[] BargainingUnit;
    public string[] Department;
}

public class PropertyBagModel
{
    [JsonExtensionData]
    private readonly Dictionary<string, JToken> _extensionData = new Dictionary<string, JToken>();

    [JsonIgnore]
    public readonly Dictionary<string, string> Values = new Dictionary<string, string>();

    [JsonProperty(".plan", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByPlan;

    [JsonProperty(".classAndTier", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByClassAndTier;

    [JsonProperty(".bargainingUnit", NullValueHandling = NullValueHandling.Ignore)]
    public Dictionary<string, PropertyBagModel> ByBarginingUnit;

    [OnSerializing]
    private void OnSerializing(StreamingContext context)
    {
        foreach (var kvp in Values)
            _extensionData.Add(kvp.Key, kvp.Value);
    }

    [OnSerialized]
    private void OnSerialized(StreamingContext context)
    {
        _extensionData.Clear();
    }

    [OnDeserialized]
    private void OnDeserialized(StreamingContext context)
    {
        Values.Clear();
        foreach (var kvp in _extensionData.Where(x => x.Value.Type == JTokenType.String))
            Values.Add(kvp.Key, kvp.Value.Value<string>());
        _extensionData.Clear();
    }
}

The basic idea is this:

1. The PropertyBagModel on deserialization by JSON.NET has the ByPlan, ByClassAndTier, etc. fields populated and also has the private _extensionData field populated.
2. Then JSON.NET calls the private OnDeserialized() method and that will move the data from _extensionData to Values as appropriate (or drop it on the floor otherwise - presumably you could log this if it was something you wanted to know). We then remove the extra gunk from _extensionData so it doesn't consume memory.
3. On serialization, the OnSerializing method gets calls where we move stuff into _extensionData so it gets saved.
4. When serialization has finished, OnSerialized gets called and we remove the extra stuff from _extensionData.

We could further delete and recreate the _extensionData Dictionary when needed but I didn't see a real value in this as I'm not using tons of these objects. To do this we'd just create on OnSerializing and delete on OnSerialized. On OnDeserializing, instead of clearing, we could free it.

I was looking into a similar issue and found this post.

Here is a way to do it using reflection.

To make it more generic, one should check the type of the property instead of simply using ToString() in propertyInfo.SetValue, unless OFC all the actual properties are strings.

Also, lowercase property names is not standard in C# but given that GetProperty is case sensitive there are few other options.

public class Product
{
	private Type _type;

	public Product()
	{
		fields = new Dictionary<string, object>();
		_type = GetType();
	}

	public string id { get; set; }
	public string name { get; set; }
	public Dictionary<string, object> fields { get; set; }

	public void SetProperty(string key, object value)
	{
		var propertyInfo = _type.GetProperty(key);
		if (null == propertyInfo)
		{
			fields.Add(key,value);
			return;
		}
		propertyInfo.SetValue(this, value.ToString());
	}
}
...
private const string JsonTest = "{\"id\":7908,\"name\":\"product name\",\"_unknown_field_name_1\":\"some value\",\"_unknown_field_name_2\":\"some value\"}";

var product = new Product();
var data = JObject.Parse(JsonTest);
foreach (var item in data)
{
	product.SetProperty(item.Key, item.Value);
}