DEV Community

Dorian D.
Dorian D.

Posted on • Edited on

Declarative and reproducible environments with colima, nix and k8s

#colima #kubernetes #nix #containers

Table of contents

Motivation

Having reproducible environments heavily impacts productivity and developer experience. Being able to test on local machines allows developers to "fail fast" and prevent possible bugs or incidents in production environments.
In an attempt to make my life easier I've studied nix combined with container tools like Colima and k8s.

Why colima and not docker?

Colima is a container runtime which can be use to run containers. There is no need to use docker to run container images and since it now requires you to buy a license I thought it would be more accessible to people who can't afford it.

Download and install nix

Download and install the nix package manager.

Usage

Before running the commands below make sure you wiped out all your global dependencies.

Here is an example in case you used homebrew:

my-computer:~$ which kubectl
/opt/homebrew/bin/kubernetes-cli
my-computer:~$ brew uninstall kubectl
Uninstalling /opt/homebrew/Cellar/kubernetes-cli/<version>...
my-computer:~$ which colima
/opt/homebrew/bin/colima
my-computer:~$ brew uninstall colima
Uninstalling /opt/homebrew/Cellar/colima/<version>...
Enter fullscreen mode Exit fullscreen mode

If you also have docker installed globally on your machine, you should also uninstall it if you want a clean slate, because in the shell.nix file there is an alias for docker that could mess up your configuration.

Make sure that colima config files are deleted as well (they are usually in the ~ directory under ~/.config/colima).
You can proceed without deleting these dependencies, but global configuration could clash with the local one and some of the commands may not work properly.

There is a way to make this work in any case without uninstalling your global packages, but it's less simple.

How to read this guide

There are commands that you will need to run inside your shell, these are prefixed by my-computer:~/colima-k8s-nix$ assuming you clone this repo in your home directory ~ or /Users/<your-username> on Mac and /users/<your-username> on Linux.

The commands that you need to run in the nix-shell are prefixed with [nix-shell:~/colima-k8s-nix]$ instead.

Commands

Start a nix-shell with colima and k8s:

my-computer:~/colima-k8s-nix$ nix-shell
Enter fullscreen mode Exit fullscreen mode

Inside the shell type:

[nix-shell:~/colima-k8s-nix]$ colima start --cpu 4 --memory 8 --network-address --kubernetes -r containerd
Enter fullscreen mode Exit fullscreen mode

Pulling docker images:

[nix-shell:~/colima-k8s-nix]$ docker pull nginx
[nix-shell:~/colima-k8s-nix]$ docker pull openjdk:alpine
[nix-shell:~/colima-k8s-nix]$ docker pull mongo
Enter fullscreen mode Exit fullscreen mode

Verifying that colima is in kubernetes mode:

[nix-shell:~/colima-k8s-nix]$ kubectl run -i --tty busybox --image=busybox -- sh
Enter fullscreen mode Exit fullscreen mode

Exit from k8s mode: ctrl + d.

Stop colima runtime to free CPU and memory:

[nix-shell:~/colima-k8s-nix]$ colima stop
Enter fullscreen mode Exit fullscreen mode

Delete and tear down colima instances:

[nix-shell:~/colima-k8s-nix]$ colima delete default #colima delete <name-of-the-instance>
Enter fullscreen mode Exit fullscreen mode

Exit from nix-shell by typing:

[nix-shell:~/colima-k8s-nix]$ exit
Enter fullscreen mode Exit fullscreen mode

Garbage collection commands

Nix is really powerful, but in its raw state it generates a lot of garbage. There are some ways to handle this gracefully, but for now just run commands that handle garbage collection.

my-computer:~/colima-k8s-nix$ nix-env --delete-generations old
my-computer:~/colima-k8s-nix$ nix-store --gc
my-computer:~/colima-k8s-nix$ nix-collect-garbage -d
Enter fullscreen mode Exit fullscreen mode

You can also run nix-collect-garbage if you're not sure which packages you should get rid of.

More on nix garbage collection can be found here.

Important

Before leaving the nix-shell make sure that you stopped and deleted all the instances of colima.

In fact although the local dependencies of the shell are separated from the one on your local machine (you can find more about this topic here here) you still share your file system, user groups etc. with the nix-shell. In this way the shell is the closest environment to your local without messing up your installed dependencies. This approach has advantages and disadvantages.

❗❗Please make sure to run the commands to delete colima instances and the garbage collection commands after you're done developing.

About nix

What's nix?

Nix is a cross-platform package manager. It uses the nix programming language. Nix and NixOs are often used in the same context, but while the first is a package manager, the latter is a linux distribution based on nix.

The nix ecosystem comes with a lot of tools. The nix-shell is one of these. Under the hood it uses the nix packages, aka nixpkgs, which are packages that can be found in this repository.

One of the advantages of using the nix ecosystem is the capacity to make reproducible, declarative and reliable systems. One of the disadvantages is that getting to know this ecosystem can be very overwhelming and therefore it has a steep learning curve when compared to other build tools.

What were you saying about isolation?

So far we know that we can tap in this "magic" shell that has everything that we need without the hassle.

But what is this exactly? Is this a container? Is this a virtual machine? Let's ask our shell to give us some information about our operating system and platform:

my-computer:~/colima-k8s-nix$ uname -a
Enter fullscreen mode Exit fullscreen mode

And now let's ask the nix-shell:

[nix-shell:~/colima-k8s-nix]$ uname -a
Enter fullscreen mode Exit fullscreen mode

You will notice that the answer is exactly the same. This is certainly not what we would obtain if we would run the same command in a docker container or in a virtual machine. So our nix-shell shares ressources with our machine.

What does this mean for us? One of the consequences of this is that when we run commands on this nix-shell (which is configured as in the shell.nix file), it's as if we would install those dependencies on our machine with homebrew for example. This also means that if we create a file from the nix-shell, we will be able to find it if we jump back to our normal shell:

[nix-shell:~/colima-k8s-nix]$ touch test.txt
Enter fullscreen mode Exit fullscreen mode 
my-computer:~/colima-k8s-nix$ ls
LICENSE   README.md     shell.nix      test.txt
Enter fullscreen mode Exit fullscreen mode

And the same goes if we first create a file through our regular shell and we want to read it from the nix-shell. So the environment1 that we have in the nix-shell is closer to our real shell. This is why we have to handle the memory responsibly and use garbage collection commands. The memory that nix uses to download and install packages is the physical memory of our machine, not the virtualized one (like in the case of containers and virtual machines).

This also means that if we exit the shell without stopping and deleting the colima instance that we started, it will continue to run even when we exit the nix-shell and even if we don't have colima installed globally.

Container tools: docker, containerd and nerdctl

You might have noticed that inside the nix-shell we've created the colima instance with the flag -r containerd. In fact if you try to run the same command without the flag you will get this error:

INFO[0000] starting colima
INFO[0000] runtime: docker+k3s
FATA[0000] dependency check failed for docker: docker not found, run 'brew install docker' to install
Enter fullscreen mode Exit fullscreen mode

Why is that? In the shell.nix file we only have colima and kubernetes as dependencies. colima already ships with containerd, which is the same container runtime that docker uses under the hood. So since we are already using colima we don't need to download docker for the runtime.

What about the docker-cli? colima also ships with a docker-compatible cli to interact with containerd called nerdctl. We can execute the same docker cli commands like:

[nix-shell:~/colima-k8s-nix]$ colima nerdctl pull nginx
Enter fullscreen mode Exit fullscreen mode

For brevity and "developer experience" I created a local alias in shell.nix for colima nerdctl which is called docker:

[nix-shell:~/colima-k8s-nix]$ type docker
docker is aliased to `colima nerdctl'
Enter fullscreen mode Exit fullscreen mode

❗❗ Please note that this alias could clash with your global configuration if you have docker installed globally on your machine.

Advanced usage (optional)

This is by no means an exaustive list of how you can customize your builds, deployments and development environments with nix, but just some examples.

lorri + direnv

lorri is a nix-shell replacement for project development. Once you've downloaded and installed it you can initialize your project like this:

my-computer:~/colima-k8s-nix$ lorri init
Enter fullscreen mode Exit fullscreen mode

This command will also create an .envrc file in which you can configure your shell. Then you can run:

my-computer:~/colima-k8s-nix$ lorri deamon
Enter fullscreen mode Exit fullscreen mode

lorri's deamon gives you information about your shell, takes care of the garbage collection for you and "watches" your shell.nix files for changes, so you don't need to reload the shell if you add a dependency. Neat!

Moreover lorri works together with direnv.

One downside of using nix-shell is that you can only use bash inside of it. Instead when using lorri in the newly created .envrc file you can define other shells like zsh or fish, thanks to lorri's integration with direnv.

Environment isolation

Run with --pure

When you run the nix-shell, it inherits your global configuration by default. If you don't want the already installed packages to affect your development and productivity, you can run:

my-computer:~/colima-k8s-nix$ nix-shell --pure
Enter fullscreen mode Exit fullscreen mode

The pure flag gives you a clean slate. But in this case the nix-shell will only have installed the packages that are specified in shell.nix. So for example in a nix-shell --pure:

[nix-shell:~/colima-k8s-nix]$ ls 
bash: ls: command not found
Enter fullscreen mode Exit fullscreen mode

So if you will need ls or other commands that come with your Unix operating system, you will have to specify them in the shell.nix file together with the other packages:

[...]

  buildInputs = with pkgs.buildPackages; [
    ls
    colima
    kubectl
  ];

[...]
Enter fullscreen mode Exit fullscreen mode

Using a virtual machine

You can run a virtual machine (e.g. lima) from inside a nix-shell, exactly as you would do with a regular shell.

In this case lima will provide with the level of isolation that nix-shell lacks when we don't pass the pure flag. In this case the control is relinquished to the virtual machine, so e.g. lima can't write by default in the ~ folder, but provides an isolated folder in which it has both reading and writing rights.

Installing an OS inside the shell

Although this is a different approach with respect to using an already existing virtual machine, we can achieve similar results in terms of isolation and to which we can do inside the shell (for example combined with the pure flag).

It goes without saying, that you can download the NixOs linux distro within your nix-shell. But you can install other operating systems as well, like you would do in a container or virtual machine.

So as you can see, this is the power of nix, we started from a simple package management problem and incrementally we are building a virual machine in a declarative way. Simply amazing!

This is an example of how nix-shell allows you to create ephemeral environments. Check this video if you want to know more.

nix user profiles

Let's put aside the level of isolation that a virtual machine guarantees for a moment.

Another possibility to partially isolate in nix is using user profiles. If you have multiple users in your machine you can switch between them and also give root priviledges to any user for whatever specific action. In this case, without additional isolation though, you can affect existing users on your machine.

Improving the development experience

With a few tweaks we can improve our development experience.

  1. Automatic environments
  2. Improving reproducibility

Code repository

Article originally posted on github.

  1. I'm using this term loosely here, not necessarily related to nix-env which is a concept and tool of its own. ↩

Top comments (0)

Read next

samos123 profile image

Tutorial: Deploying Llama 3.1 405B on GKE Autopilot with 8 x A100 80GB

Sam Stoelinga -

dm8ry profile image

Setting Up MySQL on Kubernetes with Prometheus & Grafana Monitoring

Dmitry Romanoff -

dm8ry profile image

How to Set Up Prometheus Exporters for PostgreSQL and MongoDB in Kubernetes

Dmitry Romanoff -

notharshhaa profile image

Writing a Dockerfile: Beginners to Advanced

H A R S H H A A -