Since Grady Booch introduced continuous integration (CI) in 1991, code changes can be tested and automatically integrated into a shared repository. Although CI/CD solutions have evolved quickly, the concept remains the same: you push, test, and deploy, ensuring the fast delivery of software updates.
In this post, you will discover how PostNL has designed fully serverless and centralized CI/CD pipelines in the AWS environment. These pipelines provide scalability, security, and convenience through services such as IAM and Amazon VPC.
Introduction
Nowadays, building your software on CI/CD pipelines is an easy task. Especially with tools like GitHub and Atlassian BitBucket, which offer hosted virtual machines to execute pipelines in a plug-and-play approach: you create a build file, define your build steps, and voila - your pipeline is running.
At PostNL, we use GitHub for collaboration and version control and we leverage the GitHub-hosted runners as our CI/CD tool, to run workflows for many use cases across the organization, like for instance deploying infrastructure and software, executing all kinds of tests and so on.
The AWS Center of Excellence (AWS CoE) team is dedicated to provisioning and establishing foundational infrastructure for PostNL development teams, to efficiently develop and deploy business-critical applications according to the best practices.
With that in mind, we have decided to design a solution that allows private communication between GitHub and AWS and supports architectures such as ARM64 in the pipelines, leveraging the Amazon CodeBuild feature that allows self-hosting GitHub action runners.
About GitHub-Hosted Runners
GitHub offers hosted virtual machines to run workflows. The Github-hosted runners are used by default across the PostNL organization.
Networking
By default, GitHub-hosted runners have access to the public internet. This means establishing communication with resources AWS, the traffic goes over the public internet. They are also shared across all GitHub customers.
However, things can get slightly complicated when a team needs, for example, to run integration tests against private resources inside an Amazon VPC, without having to expose these resources through a public Application Load Balancer or Amazon API Gateway.
Compute images
GitHub offers different types of runners for public and private repositories. Since all the repositories inside the PostNL Organization are private, teams are subject to use the supported runners for private repositories only. The list does not include, for example, the Linux Virtual Machine on ARM64 Architecture. This limits PostNL teams to executing builds using the Intel architecture only.
Note: According to GitHub documentation, the ARM64 Linux runner is in public preview and subject to change. It is only available for public repositories.
Why Self-Host GitHub Action Runners?
A self-hosted runner is a system that we deploy and manage to execute jobs from GitHub Actions. This approach offers more control of hardware, operating system, and software tools than GitHub-hosted runners provide to meet PostNL needs, for example:
- Create custom hardware configurations with processing power or memory to run larger jobs;
- Communicate with resources available on the internal network;
- Choose a CPU architecture not offered by GitHub-hosted runners;
Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud. This also means that by self-hosting GitHub runners we are responsible for the cost of maintaining the runner system.
PostNL CCoE Shared Services
As the PostNL CCoE (Cloud Center of Excellence), our mission is to drive innovation and business agility for the PostNL engineering community and enable them to build cloud-native solutions. We do this by delivering a Landing Zone and facilitating knowledge sharing for teams to build, innovate, and own cloud solutions while delivering maximum customer value.
One of the practice areas under the CCoE structure is called Shared Services, managed by the AWS CoE team. In practice, the Shared Services is an AWS account inside the PostNL organization in AWS, responsible for offering business and engineering functions to PostNL development teams.
Network setup
At PostNL we are using a Transit Gateway to enable private network communications. All AWS accounts of the development teams are connected to two different routing tables on the Transit Gateway: one for production accounts and one for non-production accounts.
By providing the accounts with CIDR ranges coming from a big supernet we could prevent network traffic between production and non-production accounts creating just one blackhole route in the routing table containing the supernet. Since we wanted all teams to be able to access our Shared Services Account this account is attached to its routing table providing access to all production and non-production accounts.
For example, one of the services we provide in the Shared Services Account is the DNS service. For DNS we are using both public and private hosted zones. Although the public-hosted zones are authoritative, the private-hosted zones need to be reached from the PostNL internal DNS server so we created DNS resolver endpoints in the Shared Services Account.
For all the development team accounts we create a private hosted zone and associate that zone with all the VPCs of its workload and with the VPC of the Shared Services Account so the entries in all the private hosted zones can be retrieved from the Shared Services Account.
Because the Shared Services is at the organization level and can seamlessly communicate with the PostNL network in the AWS ecosystem, we have decided to design the Self-Hosted Runner in the Shard Services account, making it possible to process jobs for multiple repositories in the organization, while keeping security and compliance in the centralized CCoE environment.
The PostNL Self-Hosted Runner
The PostNL Self-Hosted Runner leverages the Amazon CodeBuild feature that allows self-hosting GitHub action runners. This solution provides:
- A fully managed service to handle builds on ephemeral hosts.
- No need for managing the underlying infrastructure, fully serverless.
- Seamless integration with the PostNL internal network.
- Full control over the runner's image and compute type supported by CodeBuild
Connection between GitHub and AWS
As the solution is offered and managed by the PostNL CCoE in the Shared Services account, the centralized approach enables development teams to execute builds on AWS without having to manage the CodeBuild project or the connection between AWS and GitHub.
This is a one-time configuration made in the Shared Services AWS account, creating a GitHub App connection for GitHub for Amazon CodeBuild. This is necessary to allow CodeBuild to execute jobs in the GitHub PostNL organization.
One CodeBuild project per team
By creating one CodeBuild project for each team, we can achieve the least privileges by using ABAC (Attribute-Based Access Control) based on the repository prefix, where only the GitHub repositories belonging to that specific team can run builds on their specific CodeBuild project. This is achieved by using the Filter Group of the CodeBuild source:
Since CodeBuild allows to assignment of at least one security group to the project, having one security group per team also makes it more fine-grained.
CodeBuild supports 5,000 build projects within a single region.
Leveraging existing IAM role permissions
CodeBuild assumes the existing OIDC (Open ID Connect) role configured for that specific GitHub repository.
Internal network communication
The solution also enables development teams to execute end-to-end and integration tests against resources in the internal network, leveraging the Shared Services Account network setup.
This can be achieved by using Security Group Referencing towards the security group of the CodeBuild project, adding an extra layer of security.
Customizing image and compute type
By default, the Self-Hosted CodeBuild projects are created using the image type of Linux ARM64 (not supported by standard GitHub-Hosted Runners) and compute type Medium (8 GB memory, 4 vCPUs).
When running builds, development teams have the freedom to customize the image and compute type by using all types supported by CodeBuild:
runs-on:
- codebuild-<project-name>-${{ github.run_id }}-${{ github.run_attempt }}
- image:<environment-type>-<image-identifier>
- instance-size:<instance-size>
An example of how the CodeBuild project can be set up can be found here.
Next steps
The PostNL Self-Hosted Runner is currently in its first version, although it is already a reliable choice for running builds that require more power and connectivity across the organization, we want to evolve it into a more robust solution with a couple of ideas in mind:
- Limit the size of the runners to a list pre-defined by the CCoE to the team's use case, since running very large builds can lead to cost increases. We can achieve that by creating, for example, a reusable GitHub Workflow to share across the PostNL organization.
- Create our buildspec.yml file: since CodeBuild manages the build commands, we would have to manipulate the build steps (INSTALL, PRE_BUILD, and POST_BUILD) to install new packages/software needed for the builds at PostNL.
As CCoE, we want to be close to the development teams using the PostNL Self-Hosted Runner to gather feedback on how the solution can be improved.
Conclusion
The CCoE PostNL's implementation of Self-Hosted GitHub Action Runners using Amazon CodeBuild represents a significant advancement in our CI/CD pipeline capabilities.
By leveraging AWS's serverless infrastructure and integrating it with our internal network, we have achieved a scalable, secure, and efficient solution that meets the diverse needs of our development teams. This centralized approach not only enhances our ability to run complex builds and tests but also ensures compliance and security within the PostNL ecosystem.
Top comments (0)