Misusing Ninja

Originally posted on: https://dev.l1x.be/posts/2023/02/21/misusing-ninja/

Abstract

In the world of software development, build systems are essential tools for compiling and linking source code into executables. While there are many build systems available (make, cmake), one of the most popular and powerful is the Ninja build system.

Developed by Evan Martin while working on the Chromium project at Google, Ninja is a fast, scalable, and cross-platform build system that has gained widespread adoption in the software development community. In this blog post, we'll take a closer look at the Ninja build system and its key features.

How Ninja Works

At its core, Ninja is a simple, low-level build system that aims to be fast and efficient. Ninja works by generating a graph of dependencies between input files and output files and then executing a series of build commands to transform the input files into the output files.

The input files are typically source code files written in a programming language such as C++, Rust, or in our case Python. The output files are the compiled object files, libraries, and executables that result from the build process. And this is where it gets interesting. You can safely ignore this part of Ninja and just use it as a dag orchestrator that executes commands in a certain order.

Ninja uses a file format called "build.ninja" to define the build graph and specify the build commands. The build.ninja file is a text file that describes the dependencies between the input and output files or in our case the build steps.

Key Features of Ninja

One of the main advantages of Ninja is its speed. Because Ninja generates a build graph that only includes the necessary build steps, it can execute builds quickly and efficiently. Additionally, Ninja can scale to handle large projects with many dependencies, making it a popular choice for building complex software applications. For our use case, it is not that much use but in case you are using it for a C++ project it is pretty good.

Another key feature of Ninja is its cross-platform support. Ninja is designed to work on multiple operating systems, including Windows, macOS, and Linux, which makes it easy to use in a variety of development environments. This is true if you have a way of making sure that the tools that are used in the build are present and have the same CLI on all systems you run your builds on.

Getting Started with Ninja

I usually learn by example and there aren't many examples around. I have read the original documentation and tried a few things out but finally, I understand how all things hang together.

There are 3 things that you need to know about a ninja build file:

• rule
• build
• dependency

A rule is a description and a command that gets executed when the rule is invoked.

A build step is just calling a rule (or potentially multiple rules using dependencies).

A dependency is a concept of describing a relationship between rules.

Let's have a look at a basic example.

rule ok
  command = echo "ok"

build ok: ok

Invoking it:

❯ ninja ok
[1/1] echo "ok"
ok

Let's add a build that has two steps (sometimes called stages or targets).

rule one
  command = echo "one"

rule two
  command = echo "two"

build one: one
build two: two || one

With two pipe characters, we can express dependency between the build stages.

Invoking the build:

❯ ninja one
[1/1] echo "one"
one

❯ ninja two
[1/2] echo "one"
one
[2/2] echo "two"
two

This means we cannot forget to execute the first step when executing the second.

Using Ninja for building and deploying to the cloud

There is no point in writing an article without Docker in it so let's put Docker into Ninja.

version   = 0.6.0
aws_account_id = 11111111111


rule login-to-ecr
  command = aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com
  description = Logging in to ECR

rule init
  command = env | egrep AWS_PROFILE
  description = Displaying AWS_PROFILE


rule build-image
  command = docker build . -t depoxy:backend-api-${version} --file Dockerfile
  description = Building depoxy:backend-api-${version}

rule tag-image
  command = docker tag depoxy:backend-api-${version} ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/depoxy:backend-api-${version}
  description = Tagging depoxy:backend-api-${version}

rule upload-image
  command = docker push ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/depoxy:backend-api-${version}
  description = Push depoxy:backend-api-${version}



build login-to-ecr: login-to-ecr
build init: init || login-to-ecr
build build-image: build-image || init
build tag-image: tag-image || build-image
build upload-image: upload-image || tag-image

default login-to-ecr init build-image tag-image upload-image

There goes, a simple docker workflow implemented in Ninja. We use similar workflows to upload files to AWS and deploy with Terraform too. This unified our different build efforts mostly using YAML for CI/CD and made sure we do not forget anything while not needing to write a tonne of YAML.

The reduction from the YAML-based workflows is pretty significant:

+229 −1,633

To get started with Ninja

To get started with Ninja, you'll need to install the Ninja build system on your development machine. Ninja can be installed from package managers on the most popular operating systems, or you can download the source code and build it manually.

Once you have Ninja installed, you'll need to create a build.ninja file that describes your project's build process. You can create this file manually, or you can use a build system generator such as gn or Meson to generate it automatically.

Finally, you can run the Ninja build command to execute the build process and create the output files. Ninja will automatically track changes to input files and re-execute the build commands as needed, ensuring that your output files are always up-to-date.

Conclusion

The Ninja build system is a powerful and efficient tool for building software applications. With its speed, scalability, and cross-platform support, Ninja is a great choice for developers working on a wide range of projects. Whether you're building a small utility or a large, complex application, Ninja can help you streamline your build process and get your software up and running quickly. It helped us to move from YAML to a much better alternative while reducing the complexity of the CI/CD workflows. The next is to find a CI/CD platform that supports Ninja files or maybe to create a project that does exactly that.