Accessing GitHub Action runners using Netbird

I've written previously about using WireGuard to get remote SSH access to GitHub Actions runners, something that can be really useful for troubleshooting build issues. Have a read over that blog post for the details, but in summary, we use a GitHub Action to provision a WireGuard tunnel between our machine and the runner. It's not the most straightforward solution, and a better alternative now exists which I'll run through here.

WireGuard is a modern VPN technology built into the Linux kernel, it implements the bare bones of a VPN, just enough to get a secure tunnel established between two devices. Everything else is left up to the reader to manage. In my WireGuard SSH GitHub Action, I provided some extras required which made it possible to SSH to a GitHub Actions runner over WireGuard. But it still requires quite a few manual steps to get it working.

Recently I've been looking at VPN solutions, specifically ones built around WireGuard, that take away some of the manual steps required to manage a large scale deployment. After building proof of concept solutions with several of these offers, I settled on NetBird.

NetBird provides a management layer, with a web UI, which handles all the necessary background management tasks needed when running more complex WireGuard based VPN solutions. It's BSD licensed so you can self-host or you can use their hosted offering, which is free for small deployments and works perfectly for accessing GitHub Actions runners. Let's see how using NetBird can simplify remote access to runners.

NetBird has the ability to use an identity provider, also known as an IDP, or through a secret key known as a setup key. I want this solution to be completely hands-off and automated, so I'm going to use setup keys. These are used by the NetBird client to automatically register new peers on-demand. They can be constrained using expiry dates, number of uses, or more usefully in this use case, they can also be used to make the peer ephemeral.

Setup keys can also automatically assign peers registered with the key to a group. Groups in NetBird allow fine-grained access control lists to be created, allowing traffic flows to be restricted. In this case, we've assigned the setup key to a new group called runners.

The key is set to be reusable, limited to the maximum 365 days and to be ephemeral. Peers registered using a setup key that's set to be ephemeral means the peers will be automatically cleaned up, and removed from Netbird if there's no activity for 10 minutes. This saves us the overhead of removing old runner peers.

⚠️ Store the setup key somewhere secure, it will only be displayed once at the time of creation.

We're now ready to start registering our GitHub Action runners with NetBird. But before we do we should register a client that we will use to connect to the runners. We'll use another setup key for this, but this setup key will be far more restrictive, only allowing a single use and automatically registering with another group called clients.

Once again, store the setup key somewhere secure, it will only be displayed once at the time of creation.

Now let's register our client device which we'll be using to connect to the GitHub Action runner. Go to https://app.netbird.io/install and download the client software for your device. Now let's use the client setup key to register our client:

paul@lightoak:~$ sudo netbird up --setup-key 0CE86986-6B44-4B44-8037-D587DCE86DFC
Connected

We can now see our client peer in the Netbird UI peers dashboard:

Clicking on the peer shows us further details:

The connection status can be checked at any time by running sudo netbird status:

paul@lightoak:~$ sudo netbird status
OS: linux/amd64
Daemon version: 0.29.3
CLI version: 0.29.3
Management: Connected
Signal: Connected
Relays: 3/3 Available
Nameservers: 0/0 Available
FQDN: lightoak.netbird.cloud
NetBird IP: 100.99.96.205/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 0/0 Connected

Here we can see that we're connected to both the management and signal backends, and there's 3 relays available. The Netbird website has further details on Netbirds Architecture.

We can also see that the 'Peers count' is 0/0, meaning we have no peers available for us to connect to. Let's fix that by adding in our GitHub Action runners.

Luckily, there's no need to write my own GitHub Action this time around as Alemiz112 has already done the hard work for us with the netbird-connect action.

Let's create a minimal workflow we can use as an example:

name: Netbird demo
on: [push]
jobs:
  demo:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Netbird Connect
        id: netbird
        uses: Alemiz112/netbird-connect@v1.0.1
        with:
          setup-key: ${{ secrets.NETBIRD_SETUP_KEY }}
      - name: Install public SSH key
        run: |
          mkdir ~/.ssh
          echo "${{ secrets.SSH_PUBLIC_KEY }}" > ~/.ssh/authorized_keys

The action uses the runners setup key we created earlier to authenticate to Netbird. This needs adding as a secret into the repository settings along with the public SSH key to use.

The NETBIRD_SETUP_KEY and SSH_PUBLIC_KEY values need adding as repository secrets, rather than as environment variables.

With everything in-place, and the workflow config pushed to GitHub, we can now see a successful run when viewing the workflow.

When viewing the list of peers in the Netbird console we can also see the peer for the runner.

The grey circle to the left of the peer indicates that the peer is offline. This is expected, as GitHub Action runners are ephemeral and the runner was destroyed after the workflow completed. As we used an ephemeral setup key to create the runner, the Netbird backend will automatically remove this peer after 10 minutes.

Now to validate we can actually connect to a runner we'll add a forced sleep period to the workflow:

      - name: Forced sleep
        run: sleep 30m

Now we can see that the peer is still online and available, highlighted by the green circle next to the peer name.

In the list of peers we can now get the private IP of the runner, in this case it's 100.99.231.31. All we need to do now is to use ssh to connect to the private IP over the Netbird provided tunnel:

paul@lightoak:~$ ssh -o "StrictHostKeyChecking no" runner@100.99.231.31
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.5.0-1025-azure x86_64)

runner@fv-az1149-775:~$   

In this setup it's safe to disable strict host key checks as we're connecting peer to peer over a private encrypted and pre-authenticated tunnel.

The solution could be further refined by using the built-in SSH support that Netbird provides. This isn't something I've explored as I've always had existing SSH key infrastructures to use instead, but it could simplify the setup further and is definitely worth exploring.

Netbird does a lot more, including the ability to have network ACLs to restrict the traffic that can flow between peers. This is a killer feature above all the other solutions and will be the topic of a future blog post.