Creating an AWS CI Pipeline: Demo and Instructions

Here we will use a simple CI pipeline to build a workflow for illustrating the procedure of how AWS develop tools works together for CI.

1. Code Repo: GitHub
   • can also use code repos that you like GitLab, or AWS CodeCommit, but CodeCommit is not support new user anymore.
2. Builder: CodeBuild
3. Dependency Repo: CodeArtifact
4. Output: S3 and AWS ECR
   • here we output both as binary and docker image

0. Code and Dependencies

For the code part we will use a simple C# console app DemoConsoleApp with a private library DemoLib which we made later.

The whole structure is like:

.
├── DemoConsoleApp
│   ├── buildspec.yml
│   ├── DemoConsoleApp.csproj
│   ├── Dockerfile
│   └── Program.cs
├── DemoLib
│   ├── Class1.cs
│   ├── DemoLib.csproj
│   ├── bin
│   ├── obj
│   └── output
|       └── DemoLib.1.0.1.nupkg
└── README.md

0.1 DemoLib

we use a class with a simple method to print out a line of words, and later we will use this lib as a dependency.

dotnet new console -n DemoLib
cd DemoLib

then edit the code generated:

namespace DemoLib;

public class Class1
{
    public void DemoMethod()
    {
        Console.WriteLine("Hallo from My Demo Lib Class!");
    }
}

0.2 DemoConsoleApp

Next we will create our main code:

dotnet new console -n DemoConsoleApp
cd DemoConsoleApp

and edit the main code here:

using DemoLib;

Class1 myClass = new Class1();
myClass.DemoMethod();

0.2.1 (Option) Refer lib to the code (for checking)

Now we have the lib and the code, we can reference the lib in the code:

$ dotnet add reference ../DemoLib/DemoLib.csproj 
Reference `..\DemoLib\DemoLib.csproj` added to the project.

And then run the code to see the result:

$ dotnet run
Hallo from My Demo Lib Class!

This shows the lib works well with the code, then let's package it so that we can upload to CodeArtifact later.

Notice here, after checking, we SHOULD change the DemoConsoleApp.csproj file reference to the name of file which we will upload to the CodeArtifact repo, otherwise it will keep the reference above and using the local reference, which will not be found during compiling. And also keep the version updated, here as later I will update the lib package in CodeArtifact so here you see I have change it to verion 1.0.1.

  <ItemGroup>
    <PackageReference Include="DemoLib" Version="1.0.1" />
  </ItemGroup>

0.3 Package Nuget lib

cd DemoLib
dotnet pack -c Release -o ./output
# generate the package inside ./output/DemoLib.1.0.0.nupkg

Now we have the lib package and the code we need (but we still need to fix the dependency later so that it can use the lib package we created here). Next we can start the pipeline build.

1. CodeArtifact

Let's prepare a CodeArtifact repo to store the dependencies we need when we compiling the code.

1.1 Create the Repo

we can use the AWS CodeArtifact panel directly, and here are two concepts should be setup:

1. setup Public upstream repositories
   • allow upstream checking packages and cache them for next use.
2. setup domain name

we can also use the comamnd line:

aws codeartifact create-domain --domain <Artifact_Domain_Name>
aws codeartifact create-repository --domain <Artifact_Domain_Name> --repository MyNugetRepo

1.2 Upload Lib package to Repo

# 1. connection
dotnet nuget add source "***" -n "<Artifact_Domain_Name>/MyNugetRepo"

# 2. authentication
aws codeartifact login --tool dotnet --domain <Artifact_Domain_Name> --domain-owner <AWS_ACCOUNT_ID> --repository MyNugetRepo

# 3. upload (remember domain/repo)
dotnet nuget push output/*.nupkg --source <Artifact_Domain_Name>/MyNugetRepo

Now we have the dependent lib package uploaded, which can be used later in our code. Before we use it, we need to fix the orginal reference and push it to GitHub repo.

As we said abve fix the DemoConsoleApp.csproj file.

  <ItemGroup>
    <PackageReference Include="DemoLib" Version="1.0.1" />
  </ItemGroup>

2. Prepare ECR or S3

When we build the code via CodeBuild, it will generate the binary file, we can either store it in a S3 bucket, or we can make it as an docker image then push it to a docker registry, here is ECR.
So we need to create a ECR repository first either by aws panel or command.

aws ecr create-repository --repository-name <Namespace_Name/ECR_Name> --region eu-central-1

Here for presentation, we will create both a S3 bucket and a ECR repository.

3. Source code

And we need to add the package and use the lib code we made above.

cd DemoConsoleApp
git init
git add .
# make a .gitignore to ignore the bin/ and obj/
git commit -m "..."
git remote add origin <git-hub-repo-url>
# fix the main branch name, normally github use main not master
git branch -M main
git push -u origin main

3.1 buildspec.yaml

And we also need to add a buildspec.yml file, so that the CodeBuild know how to deal with it.

version: 0.2 # Notice here should not be 0.1, it not support


phases:
  install:
    runtime-versions:
      dotnet: 6.0
    commands:
      - echo "Logging into CodeArtifact"
      - aws codeartifact login --tool dotnet --domain my-demo-nuget-repo --domain-owner <ACCOUNT_ID> --repository MyNugetRepo
      - echo "Restoring dependencies"
      - dotnet restore
  build:
    commands:
      - echo "Building the application"
      - dotnet build -c Release -o ./output
      - echo "Building Docker image"
      - docker build -t my-app:latest .
      - echo "Tagging and pushing Docker image to ECR"
      - aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com
      - docker tag my-app:latest <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/<Namespace_Name/ECR_Name>:my-app
      - docker push <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/<Namespace_Name/ECR_Name>:my-app

artifacts:
  files:
    - output/**/*  # Store all built .NET binaries
  discard-paths: no  # Keep folder structure

If we want to remove the compiled output, we can remove the artifacts part, and also remove the artifact part from the aws panel setting or commands.

Here we not only want to check the output but also to make it as a docker image, so we also need a Dockerfile here.

# Use lightweight .NET 6 runtime image
FROM mcr.microsoft.com/dotnet/runtime:6.0

WORKDIR /app

# Copy the prebuilt application from CodeBuild
COPY ./output ./

# Set entrypoint
ENTRYPOINT ["dotnet", "DemoConsoleApp.dll"]

Both files are put under the root directory. And the DemoConsoleApp directory structure becomes:

.
├── DemoConsoleApp.csproj
├── DemoConsoleApp.sln
├── Dockerfile
├── Program.cs
└── buildspec.yml

For ecr docker image related command, can check the official doc, here.

Here we use the simplest example, which is a separated Github repository. If you want, you can also try to only compile one project/folder inside a repo.

4. CodeBuild

4.1 create a GitHub connection

see here

4.2 Create a CodeBuild Project

• create a new role

Here we use EC2 instance, and create a new role called codebuild-my-console-app-role.

• add artifact as a s3 bucket

Also we need to assign the S3 bucket we want the output goes into. So we choose the bucket we created above as the destination for the output. And we can choose the packaging as zip, also we can leave the other as default.

4.2.1 Attach Policy to new Role

we need to attach new policy to this new created role codebuild-my-console-app-role, so that it can access the artifact. So here
we attach

• AWSCodeArtifactAdminAccess
• AmazonEC2ContainerRegistryPowerUser (for ECR)
  • Access denied for ecr:GetAuthorizationToken
• AWSCodeArtifactReadOnlyAccess (for CodeArtifact)
  • Access denied for codeartifact:GetAuthorizationToken to the new role.

4.3 Run the build

If the build fails, there will be detailed error messages, it's helpful to fix the error to following the messages.

5. Test

After build successed, we can check the S3 bucket and the ECR repo to see the new generated files.

And now we can either download the zip file (if you set the package as zip) from S3 bucket or pull down the docker image from the ECR for testing.

5.1 docker iamge pull down

aws ecr describe-images --repository-name <Namespace_Name/ECR_Name> --region eu-central-1

aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin <Account_ID>.dkr.ecr.eu-central-1.amazonaws.com

docker pull <Account_ID>.dkr.ecr.eu-central-1.amazonaws.com/<Namespace_Name/ECR_Name>:my-app

docker run -d <Account_ID>.dkr.ecr.eu-central-1.amazonaws.com/<Namespace_Name/ECR_Name>:my-app

Be sure checking your docker is on.

5.2 Download from S3 bucket

We can also check the result by download the compiled file from S3 bucket we created, and run the file, we should see the same sentence as the local compiled file's result.

# download
$ aws s3 cp s3://<S3_Bucket>/demo-code-build .                               
download: s3://<S3_Bucket>/demo-code-build to ./demo-code-build

# unpack
$ unzip demo-code-build
Archive:  demo-code-build
  inflating: output/DemoLib.dll      
  inflating: output/DemoConsoleApp   
  inflating: output/DemoConsoleApp.deps.json  
  inflating: output/DemoConsoleApp.runtimeconfig.json  
  inflating: output/DemoConsoleApp.dll  
  inflating: output/DemoConsoleApp.pdb 

$ cd output
$ ls
DemoConsoleApp                          DemoConsoleApp.dll                      DemoConsoleApp.runtimeconfig.json
DemoConsoleApp.deps.json                DemoConsoleApp.pdb                      DemoLib.dll

# check the result
$ dotnet DemoConsoleApp.dll 
Hallo from My Demo Lib Class!

Here we see as we expected.

demo-code-build is the name the compiled file name, as we use zip, we may better to add a suffix as demo-code-build.zip, it's better.

Summary

1. We created the code and reference the dependency, then test locally;
2. We uploaded the depenency package to CodeArtifact repository;
3. Then we fix the code config file to make it use the CodeArtifact repo dependency, also made the buildspec.yml and a Dockerfile, and push the code to GitHub;
4. We made a S3 bucket for output and a ECR repo for generated docker image;
5. Created a CodeBuild Project to make the compile work, and set the artifact as S3 bucket;
6. Run the CodeBuild and download the file for testing.