Running AssemblyScript in Kubernetes with Krustlet

Containers are fine an all but I really do wonder why we've adopted so much overhead to simply run code in other environments. I think of the most interesting things with WASM is the opportunities it presents outside of the browser. What would it look like to run platform agnostic code? The JVM tried but it ultimately didn't quite live up to the promise. Now we have a web-centric stab at the problem and it looks pretty promising. Krustlet lets us run WASM code in kubernetes but instead of separate heavy pods we can use a Wastime runtime.

WASM and WASI

WASM is Webassembly, a binary format for code execution that's natively supported on the web. Webassembly is pretty simple because it's a compile to format which means handwriting it isn't really a great idea (it's still possible via Webassembly text form [WAT]). It only has support for simple numeric types, and simple linear memory and arithmetic operations so it's a bit like a virtual CPU. The more difficult pieces are how the code interacts with system stuff like threads, schedulers, IO etc. WASM knows about none of it. WASI is a specification that allows the host to provide implementations of those sorts of things in a standard way. In this way we can do stuff like console log which would otherwise not be impossible without the browser parts. The main use of WASI is to allow WebAssembly to be more portable so it doesn't have to run in a specific runtime environment, just one that conforms to the spec. This means we can have truly portable code that's also pretty safe.

Prerequisites, Installing Stuff

Let's first do that hard part and install all the necessary things. I'll go through this again because this time we need to do it on something that isn't Windows because krustlet isn't supported well enough on Windows yet. I'm using Windows Subsystem of Linux (WSL) which you can install on a Windows PC. If you are using MacOS or Linux you shouldn't need anything extra.

I'm using Ubuntu with WSL2 (and you'll want to use WSL2 instead of v1). You can find instructions here: https://docs.microsoft.com/en-us/windows/wsl/install-win10. Once we have that we can setup Docker, Kubectl and Kind. There's already good instructions for that too: https://kubernetes.io/blog/2020/05/21/wsl-docker-kubernetes-on-the-windows-desktop/

Ok now whichever OS you started with we can now use kubectl, docker and kind. I specifically didn't use minikube (I initially tried) but it's much harder to setup on WSL and Kind's registry feature saves us a lot of headache.

If using WSL it's recommended you stay there and install and run commands on just that side. It will save you problems down the road. Mercifully vscode makes that pretty easy. Just open vscode from the WSL-side and everything should just work by default though you might need to re-install extensions.

Install Krustlet

Krustlet is still a bit early so you have to install it manually. First let's grab a release:

curl -o krustlet.tar.gz https://krustlet.blob.core.windows.net/releases/krustlet-v1.0.0-alpha.1-linux-amd64.tar.gz

To download it from the github release page (update the version as necessary). Then move and unpack it somewhere.

tar -xzf krustlet.tar.gz

/usr/local/bin/ is fine. Anyway once you unpack it you'll have a "readme", "license" and "krustlet-wasi". The latter is the executable you need. You'll also need to make sure it's in your path. WSL shares the path with Windows so you just need to add the folder in the Windows path for it to work.

If you run krustlet-wasi now you should get an error about not finding a default IP address for a node which means it's installed correctly.

An assemblyscript project

There's numerous ways to get a WASM output and you can use any way you'd like but for this I decided to use assemblyscript since it's should seem more familiar to a web audience. Specifically what we want is to write code and convert it to WASM, and of that the WASI sub-flavor.

Assemblyscript is a subset of typescript, or so they say. In reality it's typescript syntax but it's doing something very different when you build. Since Webassembly only deals with numbers AssemblyScript will require that everything have the right types as there's no dynamic runtime shenanigans. There's also lots of things and APIs that javascript usually has that aren't there like promises. So just keep in the back of your head, it looks similar but it's a different beast.

First we start a new node project npm init, and install @assemblyscript/loader (--save) and assemblyscript (--save-dev). Then we run npx asinit . which will setup the directory to be in the standard assemblyscript structure. This will give us a basic hello world add function (assembly/index.ts) and the tsconfig for code highlighting. It will also setup a build folder and a test folder (build and test respectively). In the root, we get the asconfig.json which has Assemblyscript specific config and the index.js which has the glue to import the WASM file so you can use it directly from a browser or node, or deno. We can compile everything with npm run asbuild which it added to the package.json scripts.

When you run asbuild it will dump things into the build folder. It will add the optimized and unoptimized versions of the WASM file, WAT files which are "human readable" output for webassembly, and the source maps that link them so you get a nicer debug experience in the browser dev tools. By default it expects the optimized file is the one you will be exporting.

Running the WASM file with Wasmtime

This part isn't required but will help us understand how our code actually executes.

Now we have a WASM file, we need a way to run it. We could import the index.js into a web application but we're really not trying to work in the browser for this, we want something close to a console app. For this we can use Wasmtime. Wasmtime is a WebAssembly runtime which means we can feed it WebAssembly and it'll execute it. It also conforms to the WASI spec so it can inject those extra syscall bits too.

You can install Wasmtime here: https://wasmtime.dev/.

Now we can run the module from the command line. wasmtime build/untouched.wasm --invoke add 2 2. The first argument is the wasm file. In this case the default add function is really meant to be a function, there is no concept of main so we need to tell Wasmtime what to run. That's the --invoke option, you give it the name of the function to run. After that are the arguments to the function. This will give you a warning that functions with argument and function that return are experimental but at least right now it should work and it will print out 4.

Adding WASI

The add function is a useful hello-world because it doesn't require anything. Let's add some actual functionality with WASI. Let's add as-wasi: npm install as-wasi --save.

//index.ts

import "wasi";
import { Console } from "as-wasi";

Console.log("Hello World!");

We don't need to use the function anymore as the script will be the default entry point. We just need to import "wasi" which I believe is just some basic setup stuff and then we can use "Console" from "as-wasi" which gives us console write ability. We can change the run command to reflect these changes: wasmtime build/untouched.wasm. Compile with asbuild. The module is certainly a lot bigger now. Then run and you should get "Hello World!". Nice!

Kind Registry

It seems that Krustlet works on the container registry level so we'll need one of those and it can't be dockerhub because it won't allow certain image types. I initially tried to do this with github container registry but ran into all sorts of authentication problems. The official tutorial uses Azure but that's an entire new can of worms. Plus, I didn't like uploading my code to a 3rd party service every time I wanted to test. Instead we'll use the registry built into Kind.

Kind provides a script that will setup a cluster with a registry and that's what I used: https://kind.sigs.k8s.io/docs/user/local-registry/

The only different is at the line that calls kind create cluster I give it a different name "krust-test" by adding --name krust-test to it.

Run that script to start the cluster.

WASM OCI

Next we need to get Wasm-to-OCI which is a tool that will convert the WASM module to an OCI container. You can download it here: https://github.com/engineerd/wasm-to-oci/releases. You can use wget on the link address to the one you want.

If Kind is already running, then we can push the image using wasm-to-oci: wasm-to-oci push build/optimized.wasm localhost:5000/as-test:latest. Where build/optimized.wasm is the WASM file and "as-test:latest" is the image name and version. The localhost:5000 part of the string as far as I can tell is necessary to let docker know it's going to the localhost:5000 registry.

Thankfully this setup spares us from dealing with Kubernetes secrets.

Krustlet

We can now create the yaml file for the pod resource:

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: wasm-test
spec:
  containers:
    - name: as-test
      image: localhost:5000/as-test:latest
      imagePullPolicy: Always
  tolerations:
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wasi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wasi"
      effect: "NoSchedule"

Notice that the image is pointed to my image. The main thing of interest here are the "tolerations". The "tolerations" tell what sort of node that the pod is allowed to run on. This is the information Krustlet uses to figure out if it needs to take over and run the pod.

In order for Krustlet to work we need to give it a bootstrap config. https://docs.krustlet.dev/howto/bootstrapping/. You just need the first part: bash <(curl https://raw.githubusercontent.com/krustlet/krustlet/main/scripts/bootstrap.sh) Yes it's an untrusted script but it's also easier for a tutorial, just hope nobody hacks it.

Once we have the bootstrap config we can run: KUBECONFIG=~/.krustlet/config/kubeconfig krustlet-wasi --node-ip 172.17.0.1 --bootstrap-file ~/.krustlet/config/bootstrap.conf. Note the the ip can change depending on what platform/VM you are using. 172.17.0.1 works for WSL but you can check the instructions here: https://docs.krustlet.dev/howto/krustlet-on-kind/. The environment variable needs to be set as that's where krustlet will write the kubeconfig it creates.

This will produce a warning that TLS certificate requires approval. Copy and paste the command in a new terminal. The command will look like kubectl certificate approve {name}-tls. Once approved then krustlet will continue on in the original terminal.

To see if it's working run kubectl get nodes -o wide. This will print all the nodes and there should be 2. One is the control-plane and will be named as such. The other will be missing a lot of data but the container runtime is mvp and it should have a version that is the same as your krustlet version.

Krustlet and the config generation part are highly temperamental as this stage. If anything goes wrong you may need to start over from a new Kind cluster because they get stuck with bad configuration. This can happen if cluster values change too.

Running a pod

Now the interesting part. We create the pod resource configuration we just need to run it with kubectl apply -f pod.yaml. If all goes well, when you run kubectl get pods you should see your pod with the status Exit0 meaning that it finished. If you want the pod to keep running then you need to wrap the console log in a loop or something, rebuild, repush and redeploy. I also found that krustlet can have an issue retrieving pod logs. If you get an error saying 127.17.0.1:3000 (the krustlet ip and port) retrieving container logs try doing curl -k {urlFromError} and it should work. In any case you should be able to see that the program printed "Hello World!"

What's next

I'll firstly acknowledge that this whole setup is rickety and not production ready. I ran into a lot of issues and not a lot of good error messages. I even crashed WSL. However even more than that I ended about where the official tutorials ended which is reading logs from the container because that's about all you really can do. You could write to a volume which might be a little more useful but you can't really do anything with sockets. This is because WASI doesn't have any interface for sockets. Without that we can't make a meaningful service.

However there is some cool ideas from DeisLabs around something called WAGI. This is, as they call it, just an experiment but it's essentially applying WASI to CGI (hence the name). If CGI was a bit before your time, it was a way in which web handlers were handled by CLI scripts. Reading posts from stdin and replying with stdout. There's a lot of reason we don't use that anymore but with WASI running in Kubernetes that becomes a lot more appealing and interesting again. I really think that this is a fantastic improvement on the container situation and I hope we can see more development in this space.