How To Mount an Amazon Elastic File System on Amazon CodeBuild From Another VPC

#devops #serverless #aws #containers

With CodeBuild, depending on the compute and environment type you configure, the runner offers a pretty good amount of disk space. However, after the build is completed, you lose the data in its disk because the storage is ephemeral. Mounting an Amazon EFS (Elastic File System) in the CodeBuild runner on the fly might be useful if you need to persist data across builds, run integration tests if the file system is used across multiple services, or manage large files efficiently.

In this quick how-to guide, I will show you how to mount an EFS on a CodeBuild ephemeral host that lives in a different VPC.

Prerequisites

Before mounting an EFS in the CodeBuild ephemeral host, make sure you take the following steps:

Make sure you have a way to access your CodeBuild runner (for example, via Systems Manager). This guide uses a self-hosted GitHub Actions CodeBuild runner. In a previous article, I shared how to set up a CodeBuild runner to self-host GitHub Actions in a large organization environment. Here is an example using CDK to show how that can be configured.
The CodeBuild project must be configured inside a VPC and have a security group assigned to it.
Both VPCs must be connected either via peering or Transit Gateway.
It is important that both VPCs are located in the same AWS region.
The CodeBuild project must be configured to run in privileged mode.
Make sure your local environment has the AWS CLI properly configured.

Step one: Configuring the EFS

If you haven't created the EFS yet, let's use the CLI to create a simple file system with the configuration needed for mounting on CodeBuild.

Create a new security group. Later, you will assign this security group to the newly created EFS.

aws ec2 create-security-group \
--group-name efs-example-sg \
--description "SG for the EFS mount target" \
--vpc-id vpc-id-example \

Write down the security group ID.

Create a new inbound rule for the new security group, allowing the CodeBuild runner security group.

aws ec2 authorize-security-group-ingress \
--group-id ID of the security group created for Amazon EFS mount target \
--protocol tcp \
--port 2049 \
--source-group ID of the security group of the CodeBuild runner \

Create a new file system with the name "ExampleFileSystem"

aws efs create-file-system \
    --performance-mode generalPurpose \
    --throughput-mode bursting \
    --encrypted \
    --tags Key=Name,Value=ExampleFileSystem

Write down the file system ID.

Create a mount target. The mount target must live in the same AZ as the CodeBuild runner and inside a private subnet.

aws efs create-mount-target \
--file-system-id file-system-id \
--subnet-id private-subnet-id \
--security-group ID-of-the security-group-created-for-mount-target \

NOTE: If you manage the CodeBuild runner configuration, you can create/modify your CodeBuild project to have the EFS mounted by default on every build. This reduces the complexity of having to manually mount it on every build. If that is your case, you can follow this guide from the AWS documentation and skip the following steps.
If the CodeBuild runner is managed by a DevOps/Platform team and you don't have control of its configuration or need to manually mount the EFS on the fly for any reason, you can continue with this guide.

Step two: Mounting the EFS

Make sure you have the DNS name of your file system as you follow the steps in this section. You can construct this DNS name using the following generic form:

file-system-id.efs.aws-region.amazonaws.com

Now, inside the CodeBuild runner (assuming you have a way to access it, as mentioned before), install the EFS utils as in the example below. Make sure you add the desired region in the /etc/amazon/efs/efs-utils.conf file:

sudo yum install -y amazon-efs-utils
sudo sed -i 's/#region = us-east-1/region = $REGION/' /etc/amazon/efs/efs-utils.conf

With the EFS mount helper installed, you can now mount the file system, pointing it to its DNS name. Make sure you add the IP address of the EFS in /etc/hosts, mapping the mount target IP address to your EFS file system's hostname:

sudo mkdir efs
EFS_IP_ADDR=$(aws efs describe-mount-targets --file-system-id $FILE_SYSTEM_ID --region $REGION | jq -r '.MountTargets[0].IpAddress')
echo "${EFS_IP_ADDR} $FILE_SYSTEM_DNS_NAME" | sudo tee -a /etc/hosts
sudo mount -t efs -o tls,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $FILE_SISTEM_DNS_NAME efs/

Because both VPCs are connected via peering or a Transit Gateway, the CodeBuild runner can resolve the DNS name of the EFS.

Step three: Testing the EFS

To test the file system, you can run the following commands:

cd efs
sudo mkdir testing-efs
cd testing-efs
touch test.txt

Voila! Now, with you EFS mounted in the CodeBuild runner, this setup is particularly useful for sharing data across services or maintaining state between builds.

Is this a scenario you would use on your builds? Let me know in the comments!