Headless Wagtail, what are the pain points?

Last year at RIPE NCC, we rewrote RIPE Labs using Wagtail, a CMS based on Django. It ended up being a monolithic application that spits out server side rendered HTML with Django templates.

This year we're revisiting Wagtail to rewrite our main site and I proposed decoupling backend and frontend, setting up a headless CMS paired with a JavaScript frontend framework.

The reason I proposed such a structure is to take advantage of React and a metaframework like Next.js or Gatsby. I'll mostly speak in terms of React because that's what I'm more familiar with, but we also evaluated Vue and what follows applies equally well.

To better understand how to set up such a project I went through The Definitive Guide to Next.js and Wagtail by Michael Yin and this boilerplate repo by Frojd. I recommend you checkout the repo first and get the book only if you need further explanations.

In this series of posts I'll show you how to set up a project with Wagtail as a headless CMS and Next.js on the frontend. I'll try my best to highlight the pain points, so you can come out with a honest evaluation of this setup.
Here's a link to the companion repo that contains all the code from these posts.

If you're curious about our choice at RIPE NCC: we'll use Django templates this time too.

Why use a JavaScript framework?

First things first, why would I want to use a JavaScript framework instead of Django templates? There are good reasons for both approaches.

I think breaking down a UI into reusable components makes it easier to develop a consistent UI. Additionally encapsulated components are simpler to test individually and with the help of TypeScript you will catch errors, big and small, at build time that the Django templating engine will never catch.

The flip side is that it increases complexity a little bit: Django templates work out of the box with all the other features of Django, such as sessions and authentication.

Second, metaframeworks such as Next.js or Gatsby provide a great DX when developing frontends locally, much nicer than Django templates. More importantly their ability to pre-render pages and producing a static build is a perfect fit for a headless CMS.

Serving JSON from Wagtail

Ok, enough talk! What do you need to change in your Wagtail setup to make it fully headless?

Wagtail includes an optional module that exposes a public, read only, JSON-formatted API. However, it's not the most intuitive and it still assumes you have HTML templates for all your pages.

By default, on the wagtail.core.Page model the method responsible for rendering a page is serve, which returns a TemplateResponse as of Wagtail 2.16.1.

class Page(...):
    ...
    def serve(self, request, *args, **kwargs):
        request.is_preview = getattr(request, "is_preview", False)

        return TemplateResponse(
            request,
            self.get_template(request, *args, **kwargs),
            self.get_context(request, *args, **kwargs),
        )

I suggest prepending all Wagtail URLs with /api/ and overriding this method to return JSON instead. In this way, your frontend paths can map 1:1 with your headless CMS. In alternative, you could run Wagtail on a different subdomain.

For example, a page with path /careers will have a matching path /api/careers that returns the page's content in JSON format from Wagtail.

In the implementation below a request with header Content-Type: application/json to /api/careers will receive a JSON response, while a request that doesn't accept JSON will be redirected to /careers.

# models.py
class BasePage(Page):
    class Meta:
        abstract = True

    ...

    def serve(self, request, *args, **kwargs):
        """
        If the request accepts JSON, return an object with all
        the page's data. Otherwise redirect to the rendered frontend.
        """
        if request.content_type == "application/json":
            # this is very important, we'll see why later
            response = self.serialize_page()
            return JsonResponse(response)
        else:
            full_path = request.get_full_path()
            return HttpResponseRedirect(
                urllib.parse.urljoin(
                    settings.BASE_URL,
                    full_path.replace("/api", ""),
                )
            )

Before this can work, you'll need to define custom serializers with Django REST framework. For the most part this is no big deal, as we will see in the last section, but for rich text some adjustments are needed.

Rendering Wagtail's rich text

Because Wagtail has a built-in module for a JSON API, you would think you could re-use their serializer (you can for StreamField!), but doing so will return the internal database representation of rich text.

This can be a problem, because the representation of internal objects looks like this:

<!-- Link to another page -->
<a linktype="page" id="3">Contact us</a>

<!-- Embedded image -->
<embed embedtype="image" id="10" alt="A pied wagtail" format="left" />

The actual URL and images are only rendered when passing the HTML through the |richtext template filter or through the expand_db_html function from wagtail.core.rich_text.

So you'll want to override the get_api_representation method of RichTextBlock in this way:

# blocks.py
from wagtail.core.blocks import RichTextBlock
from wagtail.core.rich_text import expand_db_html

class CustomRichTextBlock(RichTextBlock):
    def get_api_representation(self, value, context=None):
        return expand_db_html(value.source).replace("/api", "")

Similarly for RichTextField, you'll want to use a custom field serializer like so:

# fields.py
from rest_framework.fields import Field
from wagtail.core.rich_text import expand_db_html


class CustomRichTextField(Field):
    def to_representation(self, value):
        return expand_db_html(value).replace("/api", "")

Note that if you have prepended /api to Wagtail URLs you'll want to remove the prefix when rendering on your frontend.

Writing page serializers with Django REST framework

The last thing we need before we can spit out JSON is a serializer. What are serializers and what do they do?

Serializers allow complex data such as querysets and model instances to be converted to native Python datatypes that can then be easily rendered into JSON

Let's see it in action by creating a serializer for our abstract BasePage. Because all our models will inherit from this class, we can add common fields here so all pages will return the same metadata.

# serializers.py
from rest_framework import serializers

from .models import BasePage


class BasePageSerializer(serializers.ModelSerializer):
    class Meta:
        model = BasePage
        fields = ("id", "slug", "title", "url", "first_published_at")

So far so good, serializing these fields is easy because they're strings, integers, and DateTime objects, which Django REST can handle automatically.

However, one of the main building blocks of Wagtail pages is the StreamField and Django REST can't handle it out of the box, but luckily Wagtail can. Let's update the serializer to handle StreamField fields.

# serializers.py
from rest_framework import serializers
from wagtail.api.v2 import serializers as wagtail_serializers
from wagtail.core import fields

from .models import BasePage


class BasePageSerializer(serializers.ModelSerializer):
    serializer_field_mapping = serializers.ModelSerializer.serializer_field_mapping.copy()
    serializer_field_mapping.update({
       fields.StreamField: wagtail_serializers.StreamField,
    })

    class Meta:
        model = BasePage
        fields = ("id", "slug", "title", "url", "first_published_at")

We now have all we need to create our first non-abstract model, which we'll be able to edit in the Wagtail interface as usual and fetch as JSON.

Putting it all together

With all that said and done, let's create a Page model to add articles to our site. Our articles will be simple for now: they'll have a summary and a body with rich text and images.

# models.py
from wagtail.core.fields import StreamField
from wagtail.images.blocks import ImageChooserBlock

from .blocks import CustomRichTextBlock

...
class ArticlePage(BasePage):
    summary = models.CharField(max_length=300)
    body = StreamField(
        [
            ("paragraph", CustomRichTextBlock()),
            ("image", ImageChooserBlock(icon="image")),
        ],
    )

    content_panels = [
        FieldPanel("title"),
        FieldPanel("summary", widget=forms.Textarea(attrs={"rows": "4"})),
        StreamFieldPanel("body"),
    ]

Remember that each model we add needs its own serializer. Let's add an ArticlePageSerializer then. Because the model inherits from BasePage, we only need to extend the Meta.fields with the two new fields on ArticlePage.

Since summary is a string, Django REST will handle it automatically; and since body is a StreamField, BasePageSerializer already knows how to handle it.

# serializers.py
from .models import ArticlePage
...
class ArticlePageSerializer(BasePageSerializer):
    class Meta:
        model = ArticlePage
        fields = BasePageSerializer.Meta.fields + (
            "summary",
            "body",
        )

If you try running this now, you'll get an error. We haven't defined the serialize_page method on BasePage. We're going to write it now, and it's the most important piece of the puzzle.

# models.py
class BasePage(Page):
    ...
     serializer_class = None

     def serialize_page(self):
         if not self.serializer_class:
             raise Exception(
                  f"serializer_class is not set {self.__class__.__name__}",
              )
         serializer_class = import_string(self.serializer_class)
         return {
             "type": self.__class__.__name__,
             "data": serializer_class(self).data,
         }
    ...


class ArticlePage(BasePage):
      serializer_class = "your_app.serializers.ArticlePageSerializer"
    ...

There's a lot going on here, so let me break it down.

1. We have added a serializer_class property which is None by default as serializers don't work with abstract models.
2. We set the serializer_class property on ArticlePage to be the path to the ArticlePageSerializer we added earlier.
3. In serialize_page we attempt to dynamically import the serializer and serialize the current object with it.
4. We return the name of the class as "type", more on this in the next part of this guide.

The end result is that serialize_page will return a dictionary along these lines:

{
  "type": "ArticlePage",
  "data": {
    "id": 3,
    "slug": "my-article",
    "title": "My article",
    "url": "/api/my-article/",
    "first_published_at": "2022-02-25T15:14:13.030300Z",
    "summary": "My article's summary",
    "body": [
      {
        "type": "paragraph",
        "value": "<p data-block-key=\"itzwt\">This is my article</p>",
        "id": "4519e337-b467-44dd-a075-d2db4df0f0c8"
      }
    ]
  }
}

And BasePage.serve will return it in JSON form if we send a request to /api/my-article.

Conclusion

This is the end of the first part of my guide about running Wagtail as a headless CMS. In this first part, we've seen how to transform the default behavior of Wagtail to return JSON instead of rendering templates.

In the next installment of this guide, we'll see in detail how to set up our frontend to consume this JSON API.

Originally posted on my blog at: https://tommasoamici.com/blog/headless-wagtail-what-are-the-pain-points

Comments

[asgervelling]

These are great ideas! Very helpful.
I did change the CustomRichTextField to extend RichTextField. That cured me of some bugs, and works as expected in places where i use a regular Field, not a StreamField.

[Japanesen]

Would love to see another part. Would also be awesome if someone uploaded a nextjs + wagtail guide at some point, it seems like an insanily good combination.