Deserializing dynamic JSON documents: Jackson ObjectMapper on steroids

Using the Jackson ObjectMapper for handling JSON documents in Java applications is a very common practice and most Java developers are familiar with the library. However, most applications I have worked with only customized the default behavior through custom serializers and deserializers for value objects, mixins, or, in some cases, JSON views. These kinds of customizations are rather trivial and provide enough power to handle most usecases.

For one of our projects at willhaben, we implemented a small markup language that allows us to render dynamic forms on our mobile clients (native apps as well as mobile web frontend). This enables us to specify at runtime, what information we want our users to provide because the UI is no longer embedded statically in the clients. Rather it is created on the fly through interpretation of the markup document. In the end, the solution is greatly inspired by HTML, but tailored for our use case.

Dynamically rendering forms on the client implies that neither the client code nor the server code knows at compile time, which fields the JSON document will contain. There were also some additional requirements that I will explain as I talk about the overall solution we implemented.

Deserializing a JSON document with unknown properties is possible in a variety of ways. The first and probably easiest way is to use a data structure as the deserialization target that supports arbitrary key-value pairs; also known as a “Map” in Java. We knew that our JSON would be flat, so deserializing the request into a Map<String, String> would have worked. At this point I want to quote a former co-worker who once said:

As soon as you pass around Map<String, ?> in a Java application you can also switch to Python or something similar, because you obviously don’t care about type safety in your application.

So, the question was, how can we use Jackson to deserialize dynamic JSON documents?

Considering the millions of features that Jackson supports, there is an interesting pair of annotations: JsonAnyGetter and JsonAnySetter. The JsonAnySetter annotation can be applied to a two-argument method of a DTO that will be called for any key-value pairs that cannot be resolved otherwise. Here is an example:

import com.fasterxml.jackson.annotation.JsonAnySetter;

class DTO {
    
    private String title;
    
    @JsonAnySetter
    public void handleUnknownProperty(String key, String value) {
        System.out.printf("JSON property: %s: %s", key, value);
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }
}

All properties that can be resolved statically (in this case, only title), will be populated. For all the others, our fallback method will be called.

Neat! But there’s more.

The markup that is sent to the client is stored in the database along with all the available properties in a hierarchical structure. Each property has associated meta-data like the type and validation rules. These properties are modeled as typed value objects in our application (e.g. URL, text, DecimalScaleTwo, etc). We wanted our deserializer to automatically instantiate the correct types for us, so we don’t have to do that ourselves in the controller. For example, if we expect the user to enter an amount of money, we include that in the markup and want our deserializer to lookup the JSON key in the database, determine the type (DecimalScaleTwo), instantiate the associated ValueObject, and store it in our DTO. Why? Having the type information at hand allows us to leverage the JSR303 validation infrastructure by annotating the ValueObjects with validation rules and our controller with @Valid . This way, the DTO is automatically validated before it is passed to our controller method. This would not be possible if our DTO only contained untyped string values.

In order to lookup the JSON key in the database, the deserializer needs access to a spring-managed bean. This is actually quite easy to achieve. Make your serializer and the Jackson module a spring-managed bean and you are good to go:

import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.module.SimpleModule;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.io.IOException;

interface LookupService {
    String lookup(String jsonKey);
}

interface ValueObject<T> {
    T getValue();
}

@Component
class MyModule extends SimpleModule {

    @Autowired
    public MyModule(ValueObjectDeserializer deserializer) {
        addDeserializer(ValueObject.class, deserializer);
    }
}

@Component
class ValueObjectDeserializer extends JsonDeserializer<ValueObject> {
    
    private final LookupService lookupService;

    @Autowired
    public ValueObjectDeserializer(LookupService lookupService) {
        this.lookupService = lookupService;
    }

    @Override
    public ValueObject deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
        return null;
    }
}

So far so good, but where do we use this deserializer? Normally Jackson automatically finds the deserializer to use based on the types of the fields in the DTO. This also works for the JsonAnySetter method. Thus, we can write the following method signature and Jackson will automatically invoke our deserializer:

@JsonAnySetter
public void handleUnknownProperty(String key, CustomValue value) {
    ...
}

There’s only one problem: Jackson passes us an instance of JsonParser into our deserializer which at first sight only provides us with accessors such as longValue() or stringValue() . These accessors only hand you the value of the current JSON node in various representations. The key is passed separately into the JsonAnySettermethod. However, we need it in the deserializer in order to access our LookupService .

Luckily, we can access an abstract representation of the JSON node that is currently being deserialized through the JsonParser . For this we need to execute the following statement:

TreeNode treeNode = jsonParser.getCodec().readTree(jsonParser);

Using the returned TreeNode , we can access the key of the current JSON node and, pass it to the LookupService , thus retrieving any meta-data we need to know about the value we are about to deserialize. Based on the meta-data, we can instantiate the correct ValueObject subtype and return it from the deserializer. Jackson then passes on the instance to our JsonAnySetter method, where we can store it in our DTO.

So what did we achieve in the end?

We deserialized a dynamic JSON document with the help of a custom deserializer that performs a database-lookup through a container-managed bean with the key of the JSON node that is currently being deserialized. Storing these fully-typed ValueObjects instead of strings in our DTO in turn allows us to apply JSR303 validation rules, thus automatically validating the DTO before our controller even gets called.

Although the individual features discussed here are rather simple in isolation, their combination allowed us to setup a very powerful deserialization process that saved us from writing a lot of glue and boilerplate code. It also once again demonstrated, how versatile, mature and thought-through libraries like Jackson are. By customizing them to our needs, we can offload the heavy lifting of tasks like deserialization to them and don’t have to clutter our application code with these concerns.