Decoding JSON with Typescript

Typescript is very well suited for adding type safety to a JavaScript program, but on its own, it's not enough to guarantee that it won't crash at run-time.

This article shows how JSON decoders can help to extend TypeScript compile-time guarantees to the run-time environment.

Runtime vs Compile time

The application you are working on deals with users, so you create a User type:

interface User {
   firstName: string;
   lastName: string;
   picture: string;
   email: string;
}

You'll use this type to annotate the /me API endpoint result, and then you'll do all sorts of things with this User but let's concentrate on the profile area of the app:

• It'll display the concatenation of firstName + lastName.
• Below the firstName + lastName you also want to display the email.
• Finally, you want to show the picture of the user or, if not present, a default image.

What could go wrong? Well, for starters the User type is not telling the truth, It does not express all the permutations of User shapes that the API can return.
Let's see a few examples:

// The result has null properties
{ firstName: "John", lastName: null, picture: null, email: "[e-mail]" }

// The API returned null
null

// The result has undefined properties
{ firstName: "John", lastName: "Doe", email: "[e-mail]" }

// The API contract changed and the UI team wasn't notified
{ fName: "John", lName: "Doe", picture: 'pic.jpg', email: "[e-mail]" }

You can cope with these issues by using defensive programming techniques, a.k.a. null / undefined checks behind if statements but, what happens when someone else wants to use the /me result elsewhere? Maybe your colleague trusts the User type, why not? What happens then? We introduced a new vector for runtime errors.

Enter Json decoders

You use Json decoders to make sure that a given run-time value complies with a specific compile-time type, and not only that but also gives you tools to apply transformations, failovers and more.

Json decoders have gained popularity lately thanks to Elm.
Elm's Json decoders are a core part of the language, and they are used all over the place to ensure a smooth JS to Elm communication.

The idea behind Json decoders is that you have a collection of basic decoders ( string, number, boolean, object, array... ) that you can compose into more complex decoders.

State-of-the-art JSON decoder libraries

There are a few JSON decoding libraries out there, but there's one that stood out from the rest when I made the research a while ago. Daniel Van Den Eijkel created something that kept the principles of the Elm decoding library while being idiomatic in TypeScript terms.

Unfortunately, the library was unmaintained and unpublished, so I decided to fork it, polish it, and release it as an npm package under the name ts.data.json.
My contribution to the library has been documentation, better error reporting, unit testing, API improvements, a few new decoders and publishing the npm package.

Using JSON decoders

Install the library:

npm install ts.data.json --save

Decoding basics

Before implementing our custom User decoder let's try decoding a string from start to finish.

import { JsonDecoder } from 'ts.data.json';

console.log( JsonDecoder.string.decode('Hi!') ); // Ok({value: 'Hi!'})

Finished! 🎉

Unwrapping decoder results

As we saw in our previous example the decoding process has two steps.

• First, we declare the decoder with JsonDecoder.string.
• Second, we execute the decoder passing a JavaScript value with *.decode('Hi!'), which returns the result wrapped in an instance of Ok.

Why are we wrapping the result in an Ok instance? because in case of failure we would wrap the result in an Err instance.
Let's see how the decode() signature looks like:

decode(json: any): Result<a>

Result<a> is a union type of Ok and Err.

type Result<a> = Ok<a> | Err;

So most of the time we won't be using decode(), instead we'll probably want to use decodePromise().
Let's see how the decodePromise() signature looks like:

decodePromise<b>(json: any): Promise<a>

Let's try decoding a string from start to finish using decodePromise():

import { JsonDecoder } from 'ts.data.json';

const json = Math.random() > 0.5 ? 'Hi!' : null;
JsonDecoder.string.decodePromise(json)
  .then(value => {
    console.log(value);
  })
  .catch(error => {
    console.log(error);
  });

Half of the time we'll go through the then() route and get Hi!, and half of the time we'll go through the catch() route get null is not a valid string.

Now that we know the basics let's get serious and build our custom User decoder.

The User decoder

Aside from the primitive decoders:

• JsonDecoder.string: Decoder<string>
• JsonDecoder.number: Decoder<number>
• JsonDecoder.boolean: Decoder<boolean>

there are also other more complex decoders, and for our User we'll be using the JsonDecoder.object decoder:

• JsonDecoder.object<a>(decoders: DecoderObject<a>, decoderName: string): Decoder<a>

What's that Decoder<a> thing all decoders are returning?

Decoders have the logic to decode a particular value, but they don't know how to execute it, this is what the Decoder class is for.
Decoder<a> has methods to execute, unwrap, chain and transform decoders / decoder values.

Let's try decoding a User from start to finish using all the tricks we've learned so far:

import { JsonDecoder } from 'ts.data.json';

interface User {
  firstName: string;
  lastName: string;
}

const userDecoder = JsonDecoder.object<User>(
  {
    firstName: JsonDecoder.string,
    lastName: JsonDecoder.string
  },
  'User'
);

const validUser = {
  firstName: 'Nils',
  lastName: 'Frahm'
};

const invalidUser = {
  firstName: null,
  lastName: 'Wagner'
};

const json = Math.random() > 0.5 ? validUser : invalidUser;

userDecoder
  .decodePromise(json)
  .then(value => {
    console.log(value);
  })
  .catch(error => {
    console.log(error);
  });

Half of the time we'll get {firstName: "Nils", lastName: "Frahm"} and half of the time we'll get <User> decoder failed at key "firstName" with error: null is not a valid string. JsonDecoder has us covered.

Going down the rabbit hole

We just started to scratch the surface of what this library is capable of, there are decoders for every type you could imagine. You can also decode:

• arrays
• dictionaries
• recursive data structures
• null
• undefined

and other fancy stuff.

Go to the GitHub repo and find out!

Comments

[Thijs Koerselman]

This seems very similar to using a runtime schema validation library like Joi.

I've been using this helper function to typecheck API payloads. It returns the payload cast to the type it is supposed to be when the schema validates. If it doesn't validate the function throws an error with messages explaining what part of the schema failed to comply.

import * as Joi from "joi";

export function validatePayload<T>(payload: T, schema: Joi.AnySchema): T {
  const { error, value } = Joi.validate(payload, schema);

  if (error) {
    /**
     * Collect all of the Joi error messages and combine them in one
     * comma-separated string.
     */
    const errorMessages = error.details.map(errorItem => errorItem.message);
    throw new Error(errorMessages.join(","));
  }

  return value;
}

[Joan Llenas Masó]

Never used Joi before but it looks much more powerful (and heavy).
This library is just a few bytes, although it has all the basic pieces you need to build more Joi-ish stuff (I guess).
Something you can do with ts.data.json that I don't see in the Joi docs is mapping over a decoded value. For instance:

type User = {
    id: number,
    name: string,
    dateOfBirth: Date
}

JsonDecoder.object<User>(
    {
        id: JsonDecoder.number,
        name: JsonDecoder.string,
        dateOfBirth: JsonDecoder.string.map(stringDate => new Date(stringDate)),
    },
    'User'
);

Very useful.

Thanks for pointing that out Thijs.
Cheers!

[Thijs Koerselman]

Mapping seems useful indeed :)

I use Joi on the server so size is not an issue there. For the browser I use yup. Similar API but much smaller footprint.

[Joan Llenas Masó]

Interesting! smaller footprint but powerful too.
I might add an alternatives section in the ts.data.json docs.
Thanks!

[H. Mendez]

HI everybody.

I am using the JsonDecoder and I have two issues that I do not know how to solv.

1. how I represent any type
2. I have class functions that are public and JsonDecoder asks me to add them. How should I handle this situation?

(alias) namespace JsonDecoder
import JsonDecoder
Argument of type '{ optionID: Decoder; name: Decoder; study: any; workflowId: Decoder; workflow: Decoder; scenarios: Decoder; disabled: Decoder; defaultTask: Decoder<...>; }' is not assignable to parameter of type 'DecoderObject'.
Type '{ optionID: Decoder; name: Decoder; study: any; workflowId: Decoder; workflow: Decoder; scenarios: Decoder; disabled: Decoder; defaultTask: Decoder<...>; }' is missing the following properties from type 'DecoderObject': getTaskByID, getScenarioByID, addScenario, updateScenario, getPreviousTaskts
Best,
Hmendezm

[Joan Llenas Masó]

You can decode the methods with JsonDecoder.succeed:

...
getTaskByID: JsonDecoder.succeed
...

but the class will lose all the instance information during the decoding process. If you are using the instanceof operator later on it won't work. Aside from that, you are good to go.

[H. Mendez]

Hey Joan, sorry for bothering you again.

I have a case where the columns from the JSON can be with different for instance the JSON can have _color: yellow or color: yellow.

I am using the keyMap for the _color and I thought that if the column does not exist will take color but it is not the case. How can I have more the one keymap option?

Best
Hmendezm

[H. Mendez]

Hi Joan, sorry for bothering you. U have a case when the property in the class can be a number or null. How I can set a default value when it is null?

Best
Hmendezm

[H. Mendez]

Thanks, Joan for the help.

[Avi Block]

Have you seen io-ts? io-ts also removes the duplication by creating a "decoder" and a type at the same time.

interface User {
  firstName: string;
  lastName: string;
  age: string;
}

const userDecoder = JsonDecoder.object<User>(
  {
    firstName: JsonDecoder.string,
    lastName: JsonDecoder.string,
    age: JsonDecoder.number
  },
  'User'
)

what would happen in this case?

[Joan Llenas Masó]

I'm aware of io-ts but I haven't jumped on the fp-ts bandwagon yet. The learning curve seems quite steep.

The decoder you just mentioned would fail at compile-time, because number and string are different types.
If the data you want to decode is a number but you want a string you could do:

const userDecoder = JsonDecoder.object<User>(
  {
    firstName: JsonDecoder.string,
    lastName: JsonDecoder.string,
    age: JsonDecoder.number.map(num => num.toString())
  },
  'User'
)

Cheers!

[Ryan L. Bell]

I had a similar idea.

github.com/kofno/jsonous

[Joan Llenas Masó]

Looks very nice 👌
Very Elmish, which I like!
It's in the related libraries section now

[Ryan L. Bell]

Awesome! Thank you!

[Rutvik Deshmukh]

actually one thing i observed while testing this package, userDecoder.decodeToPromise is the method name for getting javascript object from specified JSON object, but in your article you have written as decodePromise, this method not exist on useDecoder

[H. Mendez]

Hi Joan, sorry for bothering you. U have a case when the property in the class can be a number or null. How I can set a default value when it is null?

Best
Hmendezm

[Joan Llenas Masó]

Hi! You can use JsonDecoder.failover: github.com/joanllenas/ts.data.json...
There are other more exotic options but I think this is enough in most cases.

[H. Mendez]

Thanks Joan.
I did the failover and it is working as intending. I got problems with the JsonDecoder.Success in functions as you recommended posts back (Sep 27).

Uncaught (in promise): TypeError: Cannot read property 'toApiDTO' of undefined.

public static cvt= JsonDecoder.object<CoordinateValueType>({
    latitude: JsonDecoder.failover(0, JsonDecoder.number),
    longitude: JsonDecoder.failover(0, JsonDecoder.number),
    altitude: JsonDecoder.failover(0, JsonDecoder.number),
    equals: JsonDecoder.succeed,
    toApiDTO: JsonDecoder.succeed,
    toString: JsonDecoder.succeed
}, 'CoordinateValueType', {
    latitude: '_latitude',
    longitude: '_longitude',
    altitude: '_altitude',
    equals: 'equals',
    toApiDTO: 'toApiDTO',
    toString: 'toString'
})

Best
Hmendezm

[DGCP3]

Zod enters the room