Gson optional and required fields

How should one deal with Gson and required versus optional fields?

Since all fields are optional, I can't really fail my network request based on if the response json contains some key, Gsonwill simply parse it to null.

Method I am using gson.fromJson(json, mClassOfT);

For example if I have following json:

{"user_id":128591, "user_name":"TestUser"}

And my class:

public class User {

	@SerializedName("user_id")
	private String mId;

	@SerializedName("user_name")
	private String mName;

	public String getId() {
		return mId;
	}

	public void setId(String id) {
		mId = id;
	}

	public String getName() {
		return mName;
	}

	public void setName(String name) {
		mName = name;
	}
}

Is the any option to get Gson to fail if json would not contain user_id or user_name key?

There can be many cases where you might need at least some values to be parsed and other one could be optional?

Is there any pattern or library to be used to handle this case globally?

Thanks.

As you note, Gson has no facility to define a "required field" and you'll just get null in your deserialized object if something is missing in the JSON.

Here's a re-usable deserializer and annotation that will do this. The limitation is that if the POJO required a custom deserializer as-is, you'd have to go a little further and either pass in a Gson object in the constructor to deserialize to object itself or move the annotation checking out into a separate method and use it in your deserializer. You could also improve on the exception handling by creating your own exception and pass it to the JsonParseException so it can be detected via getCause() in the caller.

That all said, in the vast majority of cases, this will work:

public class App
{

    public static void main(String[] args)
    {
        Gson gson =
            new GsonBuilder()
            .registerTypeAdapter(TestAnnotationBean.class, new AnnotatedDeserializer<TestAnnotationBean>())
            .create();

        String json = "{\"foo\":\"This is foo\",\"bar\":\"this is bar\"}";
        TestAnnotationBean tab = gson.fromJson(json, TestAnnotationBean.class);
        System.out.println(tab.foo);
        System.out.println(tab.bar);

        json = "{\"foo\":\"This is foo\"}";
        tab = gson.fromJson(json, TestAnnotationBean.class);
        System.out.println(tab.foo);
        System.out.println(tab.bar);

        json = "{\"bar\":\"This is bar\"}";
        tab = gson.fromJson(json, TestAnnotationBean.class);
        System.out.println(tab.foo);
        System.out.println(tab.bar);
    }
}

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
@interface JsonRequired
{
}

class TestAnnotationBean
{
    @JsonRequired public String foo;
    public String bar;
}

class AnnotatedDeserializer<T> implements JsonDeserializer<T>
{

    public T deserialize(JsonElement je, Type type, JsonDeserializationContext jdc) throws JsonParseException
    {
        T pojo = new Gson().fromJson(je, type);

        Field[] fields = pojo.getClass().getDeclaredFields();
        for (Field f : fields)
        {
            if (f.getAnnotation(JsonRequired.class) != null)
            {
                try
                {
                    f.setAccessible(true);
                    if (f.get(pojo) == null)
                    {
                        throw new JsonParseException("Missing field in JSON: " + f.getName());
                    }
                }
                catch (IllegalArgumentException ex)
                {
                    Logger.getLogger(AnnotatedDeserializer.class.getName()).log(Level.SEVERE, null, ex);
                }
                catch (IllegalAccessException ex)
                {
                    Logger.getLogger(AnnotatedDeserializer.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        }
        return pojo;

    }
}

Output:

This is foo
this is bar
This is foo
null
Exception in thread "main" com.google.gson.JsonParseException: Missing field in JSON: foo

Answer of Brian Roach is very good, but sometimes it's also necessary to handle:

• properties of model's super class
• properties inside of arrays

For these purposes the following class can be used:

/**
 * Adds the feature to use required fields in models.
 *
 * @param <T> Model to parse to.
 */
public class JsonDeserializerWithOptions<T> implements JsonDeserializer<T> {

    /**
     * To mark required fields of the model:
     * json parsing will be failed if these fields won't be provided.
     * */
    @Retention(RetentionPolicy.RUNTIME) // to make reading of this field possible at the runtime
    @Target(ElementType.FIELD)          // to make annotation accessible through reflection
    public @interface FieldRequired {}

    /**
     * Called when the model is being parsed.
     *
     * @param je   Source json string.
     * @param type Object's model.
     * @param jdc  Unused in this case.
     *
     * @return Parsed object.
     *
     * @throws JsonParseException When parsing is impossible.
     * */
    @Override
    public T deserialize(JsonElement je, Type type, JsonDeserializationContext jdc)
            throws JsonParseException {
        // Parsing object as usual.
        T pojo = new Gson().fromJson(je, type);

        // Getting all fields of the class and checking if all required ones were provided.
        checkRequiredFields(pojo.getClass().getDeclaredFields(), pojo);

        // Checking if all required fields of parent classes were provided.
        checkSuperClasses(pojo);

        // All checks are ok.
        return pojo;
    }

    /**
     * Checks whether all required fields were provided in the class.
     *
     * @param fields Fields to be checked.
     * @param pojo   Instance to check fields in.
     *
     * @throws JsonParseException When some required field was not met.
     * */
    private void checkRequiredFields(@NonNull Field[] fields, @NonNull Object pojo)
            throws JsonParseException {
        // Checking nested list items too.
        if (pojo instanceof List) {
            final List pojoList = (List) pojo;
            for (final Object pojoListPojo : pojoList) {
                checkRequiredFields(pojoListPojo.getClass().getDeclaredFields(), pojoListPojo);
                checkSuperClasses(pojoListPojo);
            }
        }

        for (Field f : fields) {
            // If some field has required annotation.
            if (f.getAnnotation(FieldRequired.class) != null) {
                try {
                    // Trying to read this field's value and check that it truly has value.
                    f.setAccessible(true);
                    Object fieldObject = f.get(pojo);
                    if (fieldObject == null) {
                        // Required value is null - throwing error.
                        throw new JsonParseException(String.format("%1$s -> %2$s",
                                pojo.getClass().getSimpleName(),
                                f.getName()));
                    } else {
                        checkRequiredFields(fieldObject.getClass().getDeclaredFields(), fieldObject);
                        checkSuperClasses(fieldObject);
                    }
                }

                // Exceptions while reflection.
                catch (IllegalArgumentException | IllegalAccessException e) {
                    throw new JsonParseException(e);
                }
            }
        }
    }

    /**
     * Checks whether all super classes have all required fields.
     *
     * @param pojo Object to check required fields in its superclasses.
     *
     * @throws JsonParseException When some required field was not met.
     * */
    private void checkSuperClasses(@NonNull Object pojo) throws JsonParseException {
        Class<?> superclass = pojo.getClass();
        while ((superclass = superclass.getSuperclass()) != null) {
            checkRequiredFields(superclass.getDeclaredFields(), pojo);
        }
    }

}

First of all the interface (annotation) to mark required fields with is described, we'll see an example of its usage later:

    /**
     * To mark required fields of the model:
     * json parsing will be failed if these fields won't be provided.
     * */
    @Retention(RetentionPolicy.RUNTIME) // to make reading of this field possible at the runtime
    @Target(ElementType.FIELD)          // to make annotation accessible throw the reflection
    public @interface FieldRequired {}

Then deserialize method is implemented. It parses json strings as usual: missing properties in result pojo will have null values:

T pojo = new Gson().fromJson(je, type);

Then the recursive check of all fields of the parsed pojo is being launched:

checkRequiredFields(pojo.getClass().getDeclaredFields(), pojo);

Then we also check all fields of pojo's super classes:

checkSuperClasses(pojo);

It's required when some SimpleModel extends its SimpleParentModel and we want to make sure that all properties of SimpleModel marked as required are provided as SimpleParentModel's ones.

Let's take a look on checkRequiredFields method. First of all it checks if some property is instance of List (json array) - in this case all objects of the list should also be checked to make sure that they have all required fields provided too:

if (pojo instanceof List) {
    final List pojoList = (List) pojo;
    for (final Object pojoListPojo : pojoList) {
        checkRequiredFields(pojoListPojo.getClass().getDeclaredFields(), pojoListPojo);
        checkSuperClasses(pojoListPojo);
    }
}

Then we are iterating through all fields of pojo, checking if all fields with FieldRequired annotation are provided (what means these fields are not null). If we have encountered some null property which is required - an exception will be fired. Otherwise another recursive step of the validation will be launched for current field, and properties of parent classes of the field will be checked too:

        for (Field f : fields) {
            // If some field has required annotation.
            if (f.getAnnotation(FieldRequired.class) != null) {
                try {
                    // Trying to read this field's value and check that it truly has value.
                    f.setAccessible(true);
                    Object fieldObject = f.get(pojo);
                    if (fieldObject == null) {
                        // Required value is null - throwing error.
                        throw new JsonParseException(String.format("%1$s -> %2$s",
                                pojo.getClass().getSimpleName(),
                                f.getName()));
                    } else {
                        checkRequiredFields(fieldObject.getClass().getDeclaredFields(), fieldObject);
                        checkSuperClasses(fieldObject);
                    }
                }

                // Exceptions while reflection.
                catch (IllegalArgumentException | IllegalAccessException e) {
                    throw new JsonParseException(e);
                }
            }
        }

And the last method should be reviewed is checkSuperClasses: it just runs the similar required fields validation checking properties of pojo's super classes:

    Class<?> superclass = pojo.getClass();
    while ((superclass = superclass.getSuperclass()) != null) {
        checkRequiredFields(superclass.getDeclaredFields(), pojo);
    }

And finally lets review some example of this JsonDeserializerWithOptions's usage. Assume we have the following models:

private class SimpleModel extends SimpleParentModel {

    @JsonDeserializerWithOptions.FieldRequired Long id;
    @JsonDeserializerWithOptions.FieldRequired NestedModel nested;
    @JsonDeserializerWithOptions.FieldRequired ArrayList<ListModel> list;

}

private class SimpleParentModel {

    @JsonDeserializerWithOptions.FieldRequired Integer rev;

}

private class NestedModel extends NestedParentModel {

    @JsonDeserializerWithOptions.FieldRequired Long id;

}

private class NestedParentModel {

    @JsonDeserializerWithOptions.FieldRequired Integer rev;

}

private class ListModel {

    @JsonDeserializerWithOptions.FieldRequired Long id;

}

We can be sure that SimpleModel will be parsed correctly without exceptions in this way:

final Gson gson = new GsonBuilder()
    .registerTypeAdapter(SimpleModel.class, new JsonDeserializerWithOptions<SimpleModel>())
    .create();

gson.fromJson("{\"list\":[ { \"id\":1 } ], \"id\":1, \"rev\":22, \"nested\": { \"id\":2, \"rev\":2 }}", SimpleModel.class);

Of course, provided solution can be improved and accept more features: for example - validations for nested objects which are not marked with FieldRequired annotation. Currently it's out of answer's scope, but can be added later.

This is my simple solution that creates a generic solution with minimum coding.

1. Create @Optional annotation
2. Mark First Optional. Rest are assumed optional. Earlier are assumed required.
3. Create a generic 'loader' method that checks that source Json object has a value. The loop stops once an @Optional field is encountered.

I am using subclassing so the grunt work is done in the superclass.

Here is the superclass code.

import com.google.gson.Gson;
import java.lang.reflect.Field;
import java.lang.annotation.Annotation;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
... 
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface Optional {
  public boolean enabled() default true;
}

and the grunt work method

@SuppressWarnings ("unchecked")
  public <T> T payload(JsonObject oJR,Class<T> T) throws Exception {
  StringBuilder oSB = new StringBuilder();
  String sSep = "";
  Object o = gson.fromJson(oJR,T);
  // Ensure all fields are populated until we reach @Optional
  Field[] oFlds =  T.getDeclaredFields();
  for(Field oFld:oFlds) {
    Annotation oAnno = oFld.getAnnotation(Optional.class);
    if (oAnno != null) break;
    if (!oJR.has(oFld.getName())) {
      oSB.append(sSep+oFld.getName());
      sSep = ",";
    }
  }
  if (oSB.length() > 0) throw CVT.e("Required fields "+oSB+" mising");
  return (T)o;
}

and an example of usage

public static class Payload {
  String sUserType ;
  String sUserID   ;
  String sSecpw    ;
  @Optional
  String sUserDev  ;
  String sUserMark ;
}

and the populating code

Payload oPL = payload(oJR,Payload.class);

In this case sUserDev and sUserMark are optional and the rest required. The solution relies on the fact that the class stores the Field definitions in the declared order.

I searched a lot and found no good answer. The solution I chose is as follows:

Every field that I need to set from JSON is an object, i.e. boxed Integer, Boolean, etc. Then, using reflection, I can check that the field is not null:

public class CJSONSerializable {
    public void checkDeserialization() throws IllegalAccessException, JsonParseException {
        for (Field f : getClass().getDeclaredFields()) {
            if (f.get(this) == null) {
                throw new JsonParseException("Field " + f.getName() + " was not initialized.");
            }
        }
    }
}

From this class, I can derive my JSON object:

public class CJSONResp extends CJSONSerializable {
  @SerializedName("Status")
  public String status;

  @SerializedName("Content-Type")
  public String contentType;
}

and then after parsing with GSON, I can call checkDeserialization and it will report me if some of the fields is null.

(Inspired by Brian Roache's answer.)

It seems that Brian's answer doesn't work for primitives because the values can be initialized as something other than null (e.g. 0).

Moreover, it seems like the deserializer would have to be registered for every type. A more scalable solution uses TypeAdapterFactory (as below).

In certain circumstances, it is safer to whitelist exceptions from required fields (i.e. as JsonOptional fields) rather than annotating all fields as required.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface JsonOptional {
}

Though this approach can easily be adapted for required fields instead.

import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonParseException;
import com.google.gson.TypeAdapter;
import com.google.gson.TypeAdapterFactory;
import com.google.gson.internal.Streams;
import com.google.gson.reflect.TypeToken;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonWriter;

import java.io.IOException;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class AnnotatedTypeAdapterFactory implements TypeAdapterFactory {
  @Override
  public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> typeToken) {
    Class<? super T> rawType = typeToken.getRawType();

    Set<Field> requiredFields = Stream.of(rawType.getDeclaredFields())
            .filter(f -> f.getAnnotation(JsonOptional.class) == null)
            .collect(Collectors.toSet());

    if (requiredFields.isEmpty()) {
      return null;
    }

    final TypeAdapter<T> baseAdapter = (TypeAdapter<T>) gson.getAdapter(rawType);

    return new TypeAdapter<T>() {

      @Override
      public void write(JsonWriter jsonWriter, T o) throws IOException {
        baseAdapter.write(jsonWriter, o);
      }

      @Override
      public T read(JsonReader in) throws IOException {
        JsonElement jsonElement = Streams.parse(in);

        if (jsonElement.isJsonObject()) {
          ArrayList<String> missingFields = new ArrayList<>();
          for (Field field : requiredFields) {
            if (!jsonElement.getAsJsonObject().has(field.getName())) {
              missingFields.add(field.getName());
            }
          }
          if (!missingFields.isEmpty()) {
            throw new JsonParseException(
                    String.format("Missing required fields %s for %s",
                            missingFields, rawType.getName()));
          }
        }

        TypeAdapter<T> delegate = gson.getDelegateAdapter(AnnotatedTypeAdapterFactory.this, typeToken);
        return delegate.fromJsonTree(jsonElement);

      }
    };

  }
}