Python to JSON and Back

Photo by David Clode on Unsplash

As JavaScript Object Notation (JSON) has become the ubiquitous language of communication between APIs on the web, being able to take Python objects and serialize them to JSON strings is an important part of communicating with other services, clients, APIs, what have you.

The default Python json module supports both dump and dumps methods - the more common one used is dumps - dumping the object to a JSON String - all based on this Conversion Table - which pointedly omits the datetime objects - most importantly because the JSON Specification has no reference to any specific date/time format.
This is left up to the implementer, based on what format they need - leaving coders everywhere figuring out how to Do The Right Thing - since any date/time mismatches have potential to cause all manner of havoc if incorrectly parsed. If you want to read more on time, check out this collection of articles.

A simple demonstration of the failure condition looks like this:

>>> import datetime
>>> import json
>>> json.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File ".../lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File ".../lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File ".../3.6.8/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable

And if you've hit this, and search the internet, you'll find all manner of approaches to resolve - a common approach seen is implementing a function that iterates through an object, and if it finds a datetime value, convert it to the ISO 8601 format, with datetime.isoformat(). Example of the conversion step:

>>> json.dumps(datetime.datetime.utcnow().isoformat())
'"2020-04-23T23:36:36.620619"'

However this then becomes custom code to iterate through objects, and sometimes we may need to iterate through deeply nested objects - for example a blog page, its posts and comments in a single JSON payload may all have timestamps that would need to be iterated through, detected, converted, and then json.dumps() the whole thing.

Some popular frameworks may handle this for you automatically - notable callouts are the Django REST Framework DateTimeField will default to ISO8601, as will the excellent pydantic library (here's the implementation under the hood).

However, oftentimes we may not want to take on importing a framework or a full-fledged library, but still want to avoid having to create custom solutions like this - and that's where we can lean on well-trodden paths from library maintainers, like the rapidjson library - and more importantly, the Python wrapper to this fast C library.

rapidjson

rapidjson aims to be a swap-in replacement for the native Python json module, and allows the coder to extend the behavior to support more behaviors with less effort. Note: This package must be installed via the python-rapidjson package name.

Keep in mind: there are a few incompatibilities you should be aware of when using rapidjson - but most implementations may not be using these, in fact the coercion of "true" strings to Python True values is actually an improvement!

So if we were to run our example from above, replacing json with rapidjson, we get the same result, albeit the traceback is a little clearer now:

>>> import datetime
>>> import rapidjson
>>> rapidjson.dumps(datetime.datetime.utcnow())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: datetime.datetime(2020, 4, 24, 0, 11, 16, 105719) is not JSON serializable

However now we can pass an argument to rapidjson.dumps() to inform it how we want to handle datetime instances, like so:

>>> rapidjson.dumps(datetime.datetime.utcnow(), datetime_mode=rapidjson.DM_ISO8601)
'"2020-04-24T00:14:35.531078"'

Voilá, we've got a valid ISO8601-formatted string of a datetime with little added!

I've put together a more complete example of full object deserialization and loading below - feel free to run, fork, extend, experiment with the code, and read more about the other datetime_mode options in rapidjson.

Enjoy!