Modern multi-architecture builds with Docker

In this post I'm going to explain several ways to build docker images for multiple architectures. With the ongoing rise of ARM-architectures, for example the Raspberry Pi or Amazon's efficient EC2 A1-Instances, multi-architecture builds will probably gain more focus.

If you're on a single computer, building and running docker images is very easy. The build command analyzes a given Dockerfile and runs the specific instructions. To do so, docker uses the kernel of your OS (or your VM, depending on your setup). This can bound the architecture of the image to the host architecture, especially when you compile a binary inside it.

When using a typical desktop PC, this architecture is probably x86/amd64. So if you run this image on a different computer with the same architecture, everything is fine. But what to do when a different architecture is the target, for example ARM?

Compile programs for different architectures

Generally speaking, 3 typical techniques are used to compile software for a different architecture, which are briefly explained in this section.

1. Build on the target system

Building your software directly on the target system is obviously the easiest approach, since you probably do not have to change anything in your code.

Let's consider building software for the ARM architecture. For example, you could transfer your code to a Raspberry Pi, install the toolchain and build your code there. Since Docker is supported on this device, you can use the same commands as on your desktop computer.
Typical problems with this approach can be the limited access to ARM-powered hardware and since they are often not that powerful, a slow build performance. E.g., compiling dependencies in C is something which can take a lot of time on a Raspberry Pi (though its get faster with every new board).

2. Emulate the hardware

If you do not have access to the target system, emulation is another viable solution. While in virtualization only certain parts of a computer's hardware are simulated in order to run a guest OS, emulation simulates the complete hardware. This makes emulation slower than virtualization, but it also not limited to the underlying hardware, making it possible to simulate hardware like an ARM-processor on a x86 system.

As you might have guessed, this approach is truly powerful by enabling you to build software for various architectures, but simulating the entire hardware is a huge overhead. Wouldn't it be great to simulate only the hardware components required for building the software?

User-mode emulation

Usually when an executable file is passed to a exec-system call, the kernel expects it to be a native binary for the current system.
If you ever encountered the error exec user process caused "exec format error" in a docker image, you tried to run a binary which can't be executed on the processor.
Luckily, with binfmt_misc it is possible to register custom interpreters in the userland to handle foreign binaries.
To do so, you basically register the respective interpreter and a "magic number", which identifies these binaries in the specific format.
The idea is, that in combination with a powerful emulator, you can still run and build binaries for foreign architectures on your system, by emulating only the required parts.

QEMU

One famous emulator which is capable of such a feature, is QEMU. Besides a user-mode emulation, it supports various architectures, full-system emulation and virtualization as well.

To use user-mode emulation with QEMU, we need to register this emulator for some foreign architectures, for example ARM. The Hypriot project has explained this well in this article. They even build a docker image, which runs in privileged mode to do this registering of magic numbers with QEMU for you on your host system.
An excerpt of the register script is put below to give you a rough idea:

# Register new interpreters
# - important: using flags 'C' and 'F'
echo ':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-arm:CF' > /proc/sys/fs/binfmt_misc/register
echo ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-aarch64:CF' > /proc/sys/fs/binfmt_misc/register
echo ':qemu-ppc64le:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\x00:/qemu-ppc64le:CF' > /proc/sys/fs/binfmt_misc/register

# Show results
echo "---"
echo "Installed interpreter binaries:"
ls -al /qemu-*
echo "---"
cd /proc/sys/fs/binfmt_misc
for file in *; do
    case "${file}" in
    status|register)
        ;;
    *)
        echo "Registered interpreter=${file}"
        cat ${file}
        echo "---"
        ;;
    esac
done

3. Use a cross-compiler

The last option is using a cross-compiler to build software for different architectures.
While a standard compiler builds for the system it's running on, a cross-compiler can generate binaries for other architectures as well.
Since it does not rely on any emulation but runs natively on your system, the build performance is as good as option 1.
Modern languages put in a lot of effort to support the feature well, especially with Golang this is a breeze. For example, you specify the OS (GOOS) and the target architecture (GOARCH) to cross compile:

# build for mac
GOOS=darwin GOARCH=386 go build main.go
# build for Raspberry Pi
GOOS=linux GOARCH=arm go build main.go

If multi-arch builds are an ongoing task for you, definitely check out such programming languages.

Docker images and multi-arch

Let's take a look at how docker support multi-arch images. Afaik, two approaches are mainly used:

Separate image

One option is to make a different image for each architecture. This can be done by creating a different repository or setting a designated tag for each supported platform.
This usually requires a separate Dockerfile too, as seen in the coreos/flannel repository for example:

Dockerfile

# AMD 64
FROM alpine
ENV FLANNEL_ARCH=amd64
ADD dist/qemu-$FLANNEL_ARCH-static /usr/bin/qemu-$FLANNEL_ARCH-static
...

Dockerfile.arm

FROM arm32v6/alpine
ENV FLANNEL_ARCH=arm
ADD dist/qemu-$FLANNEL_ARCH-static /usr/bin/qemu-$FLANNEL_ARCH-static
...

By passing the architecture as build arguments or environment variables to the Dockerfile, you can run instructions specific to these platforms. In this example, an env variable FLANNEL_ARCH is used for this purpose.

Manifests

A Docker manifest is a very simple concept. Basically, it's an object containing a list of image references for each supported architecture. A docker client can then pull an image by inspecting this manifest file returned by the registry, search the list for a matching platform and then load the image by the identifying digest.

An example docker manifest file, containing images for linux/arm, linux/amd64 and linux/ppc64le, is shown below:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 424,
         "digest": "sha256:f67dcc5fc786f04f0743abfe0ee5dae9bd8caf8efa6c8144f7f2a43889dc513b",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 424,
         "digest": "sha256:b64ca0b60356a30971f098c92200b1271257f100a55b351e6bbe985638352f3a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 425,
         "digest": "sha256:df436846483aff62bad830b730a0d3b77731bcf98ba5e470a8bbb8e9e346e4e8",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

While this feature is currently experimental on the docker client, it's already integrated in the docker registry and containerd. The OCI Image specification has included manifests too.

Generating manifests

As stated above, you currently need to enable experimental features on the docker client to work with manifests. Once you've done that, docker manifest should be a valid command. Generating a manifest is then very easy via the CLI.

Let's say we developed a superapp and built an image for amd64 and arm manually.
Now we want to publish these two images, app/superapp-amd64 and app/superapp-arm, as "one" image app/superapp by using a manifest.
This can be achieved with the following two commands:

docker manifest create app/superapp app/superapp-amd64 app/superapp-arm
# Created manifest list docker.io/app/superapp
# Push the manifest
docker manifest push app/superapp 

Using Docker's buildx for ARM builds

In June 2019, Docker announced tooling support for building docker images and the ARM architecture as an experimental feature.
On Docker Desktop see https://engineering.docker.com/2019/04/multi-arch-images/ and on Linux see https://engineering.docker.com/2019/06/getting-started-with-docker-for-arm-on-linux/ for a setup guide.

This feature combines the approaches above, namely QEMU, binfmt_misc, and manifests and bundles them as a single tool buildx. It allows you to write a single Dockerfile, which can be used to build images for various platforms without changing it. And with the QEMU user-mode emulation, you can build and run images which have a different architecture than your current system.

A simple example

Let's package a simple Go-application as a docker image for arm64 and amd64. We write a simple program that reports the OS and the architecture of the system.

main.go

package main

import "fmt"
import "runtime"

func main() {
    fmt.Printf("OS: %s\nArchitecture: %s\n", runtime.GOOS, runtime.GOARCH)
}

If we take a look at the Dockerfile, we see that it is not aware of the architectures which should be supported.

FROM golang:alpine AS builder
RUN mkdir /app
ADD . /app/
WORKDIR /app
RUN go build -o report .

FROM busybox
RUN mkdir /app
WORKDIR /app
COPY --from=builder /app/report .
CMD ["./report"]

Now comes the interesting part, the cross-architecture build with buildx. Let's build an image for linux/amd64 and linux/arm64:

docker buildx build --platform linux/amd64,linux/arm64 -t foo/bar  --push .

• with the --platform flag you specify the platforms the image should be built for
• just like docker build, the -t flag lets you define the image tag
• --push pushes the image directly to the registry after an successful build

A simple architecture-aware example

Sometimes, you do not want to build everything natively in the Dockerfile, for example by downloading prebuilt third-party binaries for the target architecture. Docker's buildx has you covered here as well, as it exposes for each value in --platform arguments like TARGETARCH or TARGETOS.

For instance, this is a Dockerfile for a multi-architecture image for ipfs-cluster. In order to skip the compilation of the ipfs-cluster binary from source, we download the right version for our architecture via wget.

FROM golang:1.12-stretch AS builder

# This dockerfile builds and runs ipfs-cluster-service.
ENV SUEXEC_VERSION v0.2
ENV TINI_VERSION v0.16.1
ENV IPFS_CLUSTER_VERSION v0.11.0
ARG TARGETARCH

RUN set -x \
  && cd /tmp \
  && git clone https://github.com/ncopa/su-exec.git \
  && cd su-exec \
  && make \      ### native build ###
  && git checkout -q $SUEXEC_VERSION \
...

RUN wget https://dist.ipfs.io/ipfs-cluster-ctl/${IPFS_CLUSTER_VERSION}/ipfs-cluster-ctl_${IPFS_CLUSTER_VERSION}_linux-${TARGETARCH}.tar.gz 
...

When built with the --platform linux/amd64,linux/arm64,linux/arm flag, the TARGETARCH is set to amd64, arm, arm64 once in order to download the correct binary. Still, native builds are possible, since we're emulating the hardware. For instance, su-exec is simply compiled with make. Nice!

Caveats

As an experimental tool, buildx still has some rough edges for the developer. The main issues I encountered where:

• You can only push the images directly to a registry. A file export of the docker image is only possible in the OCI format. Issue 166, Issue 186
• When an error is thrown while building your image, it's often very hard to decipher the error message in a multi-arch error log. If you can, first try to build with the usual docker build command for your system and if it works, switch to buildx.
• Due to the emulation, the build is still magnitudes slower, especially if you need to compile dependencies of your software. Check for prebuilt dependencies/binaries online and load the correct version, using TARGETARCH for example.

Wrapping it up

I hope I could give you a brief overview of the various techniques for multi-architecture docker images or raise your interest in it.
It's really exciting to see that Docker invests in simplifying this process and while buildx is still young, it's already usable.
Besides building on your local machine, installing the user-mode emulation is also possible on CI-servers, so a multi-architecture build pipeline can be set up.
Thinking of containers as a way to package software, it's a logical next step to make it available for various architectures as easy as possible.

Further reading

• containers with binfmt_misc
• docker buildx setup
• Demo with buildx
• Multiarch images with Go and without buildx