Kenn C for AWS Community Builders

Posted on Dec 27

AWS Best Practice of DevOps Agile Delivery for the Financial Services Industry

#aws #devops #agile #financial

The previous chapter 《AWS DevOps Agile Delivery of 16 Leadership Principles for the Financial Services Industry》shared how AWS DevOps pipelines can solve pain points in the financial services industry, and utilize the Amazon 16 Leadership Principles.

In this chapter, you will learn how to build an AWS DevOps pipeline:

AWS Services	Description
IAM	Identity and Access Management
EC2	Cloud-computing platform
Elastic IP address	Static IPv4 address designed for dynamic cloud computing
Route53	Cloud domain name system (DNS) service
CodeDeploy	Automate application deployments to Amazon EC2 instances
GitHub Actions	Easy to automate all your software workflows
Pricing Calculator	Create an estimate for the cost of your use

2.0 AWS DevOps Pipeline

2.1 Pre-requisites

2.1.1 Knowledge Pre-requirements

Create an EC2 server
Have a GitHub account and know basic Github Actions.
Know how to setup NGINX
Know basic AWS services, including EC2, CodeDepoly, IAM.

2.1.2 Project Requirements

First upload a simple static web project codedeploy.nginx.001 on Github, which includes:

Object	Location
index.html	./	Static Web Page
ic_alana_002_20241022_a.jpg	./icons	images on a static web page
appspec.yml	./	CodeDeploy code
application-stop.sh before-install.sh after-install.sh application-start.sh validate-service.sh	./scripts	CodeDeploy code
appspec.yml	./github/workflows	CodeDeploy code

Also, GitHub access tokens are needed to configure codeDeploy permissions.

Github -> Setting -> Developer Setting -> Tokens. Add a GitHub access token.

2.2 Creating IAM Roles

A good naming style is important because as the number of IAM roles grows, it can be confusing for developers.

AmazonSageMaker-ExecutionRole-20240805T101031
AmazonSagemakerCanvasBedrockRole-20240801T140683

{service}-{role}-{datetime}-{version}. AWS Bedrock and SageMaker auto-generated IAM naming style.

AWSCodeDeployService-EC2AccessCodeDeployRole-20241024T000000
AWSCodeDeployService-DepolyEC2Role-20241024T000000
AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000

This is the clear IAM naming style, so we will create three IAM roles for EC2, CodeDeploy, and GitHub Actions, respectively, following this official IAM naming style.

2.2.1 AWSCodeDeployService-EC2AccessCodeDeployRole-20241024T000000

Select EC2 on Use Case Tab。

AmazonEC2FullAccess
AmazonEC2RoleforAWSCodeDeploy
AmazonS3FullAccess
AmazonSSMManagedInstanceCore
AWSCodeDeployFullAccess
AWSCodeDeployRole

Add AmazonEC2, AmazonS3, and AWSCodeDeploy permissions.

2.2.2 AWSCodeDeployService-DepolyEC2Role-20241024T00000

Select CodeDeploy on Use Case Tab.

AWSCodeDeployFullAccess
AWSCodeDeployRole

Add AWSCodeDeploy permissions.

2.2.3 AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000

Select Access management -> Identity providers -> Add provider.

Used to listen to GitHub Actions.
Provider URL: token.actions.githubusercontent.com
Audience: sts.amazonaws.com
The GitHub Identity Provider then adds the AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000 role.

Select Assign Role -> Web identity -> GitHub organization.

AmazonS3FullAccess
AWSCodeDeployFullAccess

Add S3, AWSCodeDeploy permissions.

2.3 Create Amazon EC2

Fill in the name ec2.cheaper.001
Click Amazon Linux 2023 AMI
Click t3a.nano

Finally, click Launch instance to create EC2.

2.3.1 Associate Elastic IP address

Click on Elastic IPs
Click the Allocate Elastic IP Address button

Select the name ec2.paper.001 where EC2 has just been created
Select the default Private IP address
Click the Associate button

2.3.2 Amazon Route 53

Fill in the sub-domain name
Fill in the EC2's Private IP address
Click the save button

Successfully set up the static sub-domain name and IP address.

2.3.3 Add `AWS IAM` roles

Select Actions
Select Security
Select Modify IAM role

Add AWSCodeDeployService-EC2AccessCodeDeployRole-20241024T000000.

2.3.4 Install `CodeDeploy Agent` on `Amazon EC2`

Enter the Amazon EC2 terminal.

Select Connect button
Select EC2 Instance Connect tab
Select Connect button

Successfully log into the Amazon EC2 terminal.

sudo apt update
sudo yum install ruby
sudo apt install wget
cd /home/ec2-user
wget https://aws-codedeploy-us-east-2.s3.us-east-2.amazonaws.com/latest/instal
chmod +x ./install
sudo ./install auto

Install CodeDeploy Agent

Success, CodeDeploy Agent is running.

2.3.5 (Optional) Install `Git` on `Amazon EC2`

sudo yum install git-all
git clone https://{YOUR_GITHUB_SECRET_ID}@github.com/{YOUR_GITHUB_ORGANIZATION_NAME}/{YOUR_GITHUB_PROJECT_NAME}.git
git checkout .
git pull origin main
sudo chmod 777 -R PATH

Install git and pull the project to Amazon EC2.

2.3.6 (Optional) Install `NGINX`

sudo yum update
sudo yum install nginx -y
sudo service nginx start
sudo service nginx status

Install NGINX

sudo netstat -tunpl

Show Amazon EC2 listening ports. At this moment NGINX is on port :80.
The default home page of NGINX is in /var/www/html/index.html.

Ensure that Source and Destination are publicly accessible, set to 0.0.0.0/0.

2.3.7 Appspec.yml

Reference Articles:

Appspec.yml is used to indicate the codeDeploy procedure code.
Deployment is divided into 5 steps: (1) BeforeInstall -> (2) BeforeInstall -> (3) AfterInstall -> (4) ApplicationStart -> (5) ValidateService.

In the root directory, add ./appspec.yml.

version: 0.0
os: linux
files:
    - source: /
      destination: /usr/share/nginx/html
hooks:
    ApplicationStop:
    - location: scripts/application-stop.sh
      timeout: 300
      runas: root
    BeforeInstall:
    - location: scripts/before-install.sh
      timeout: 300
      runas: root
    AfterInstall:
    - location: scripts/after-install.sh
      timeout: 300
      runas: root
    ApplicationStart:
    - location: scripts/application-start.sh
      timeout: 300
      runas: root
    ValidateService:
    - location: scripts/validate-service.sh
      timeout: 300
      runas: root

Source is the root directory of the GitHub project.
Destination is the project pulled intoAmazon EC2.

In addition, a new ./scripts folder, in which there are 5 xxxxxxxx.sh respectively.

application-stop.sh
before-install.sh
after-install.sh
application-start.sh
validate-service.sh

There are 5 xxxxxxxx.sh in there, which are the 5 steps of codeDeploy.

(1) application-stop.sh

#!/bin/bash

Empty. There is no need to stop the application in this tutorial.

(2) before-install.sh

#!/bin/bash

Empty. There is no need to stop the application in this tutorial.

(3) after-install.sh

#!/bin/bash

sudo yum update
sudo yum install nginx -y

Install NGINX

(4) application-start.sh

#!/bin/bash

sudo service nginx start

restart NGINX

(5) validate-service.sh

#!/bin/bash

Empty. There is no need to stop the application in this tutorial.

2.3.8 Static Website Pages

Added ./icons folder, which shows the site image ic_alana_002_20241022_a.jpg.

Also, added index.html home page.

<html lang="en" data-bs-theme="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">
<script src="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.3.3/js/bootstrap.min.js" integrity="sha512-ykZ1QQr0Jy/4ZkvKuqWn4iF3lqPZyij9iRv6sGqLRdTPkY69YX6+7wvVGmsdBbiIfN/8OdsI7HABjvEok6ZopQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.3.3/css/bootstrap.min.css" integrity="sha512-jnSuA4Ss2PkkikSOLtYs8BlYIeeIK1h99ty4YfvRPAlzr377vr3CXDb7sb7eEEBYjDtcYj+AjBH3FLv5uSJuXg==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<title>Alana Lam</title>
</head>
<body>
<div class="container">
  <div class="row">
  <div class="col-12 mt-4 text-center">
  <h1>CodeDeploy + Github Actions + EC2</h1>
  <img src="./icons/ic_alana_002_20241022_a.jpg" class="mt-4 rounded-circle" alt="Alana Lam" width="200" height="200">
  <h5 class="mt-4">Alana Lam (AWS Builder Community Manager, Hong Kong)</h5>
  </div>
  </div>
</div>
</body>
<html>

A simple static site with text and images.

If you have completed “2.3.5 Install GIT” and “2.3.6 Install NGINX”, you can type EC2 EIP or the domain name in your browser, to see the Static Website Pages.

2.4 Create `AWS CodeDeploy`

2.4.1 Create the `AWS CodeDeploy` application

Fill in the application name test.codeDeploy.001
Select EC2/On-premises
Select Create application button

2.4.2 Create `AWS CodeDeploy` Deployment Group

Select Create deployment group button

Fill in the Deployment group name test.deploymentGroup.001
Select the IAM role, AWSCodeDeployService-DepolyEC2Role-20241024T000000
Remove Enable load balancing, because this is the simplest DevOps pipeline case, so there is no need for additional AWS services

2.4.3 Create `AWS CodeDeploy` Deployment

Go to test.deploymentGroup.001
Select Create deployment button
First, Select My application is stored in GitHub

Fill GitHub token name
Fill in the Repository name, codedeploy.nginx.001
Fill in Commit ID
Select Create deployment button

2.4.4 Successful run of `AWS CodeDeploy`

Successfully run AWS codeDeploy

2.5 Create `GitHub Actions`

Reference Articles:

Why I switched from AWS CodePipeline to GitHub Actions

2.5.1 Create `GitHub Actions` workflow

Click New workflow button
Select set up a workflow yourself link
After writing the GitHub Actions command, click the Commit changes button

2.5.2 Configurate `GitHub Actions` secrets and variables

Select Settings Tab
Select Secrets and variables -> Actions Tab
Select Secrets Tab

2.5.3 Add `GitHub Actions secrets` variables

Add a new secrets variable with name IAMROLE_GITHUB_ARN
The value is the ARN of the IAM role arn:aws:iam::{xxxxxxxxx}:role/AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000
Click the Add secret button

2.5.4 Add `GitHub Actions variables`

Select Variables Tab
Add four of Actions Variables
Select New repository variable button

Variables Name	Value	Description
AWS_REGION	us-east-1	The default region is `US East (N. Virginia)`
CODEDEPLOY_APPLICATION_NAME	test.codeDeploy.001	2.4.1 Create the `AWS CodeDeploy` application
CODEDEPLOY_DEPLOYMENT_GROUP_NAME	test.deploymentGroup.001	2.4.2 Create `AWS CodeDeploy` Deployment Group
IAMROLE_GITHUB_SESSION_NAME	AWSGitAssumeRoleWithAction	2.2.3 AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000

2.5.5 Write `GitHub Actions` Code

.github/workflows/main.yml

name: Deploy

on:
  workflow_dispatch: {}

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: Prod
    permissions:
      id-token: write
      contents: read
    steps:
    - name: Git clone the repository
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ secrets.IAMROLE_GITHUB_ARN }}
        role-session-name: ${{ vars.IAMROLE_GITHUB_SESSION_NAME }}
        aws-region: ${{ vars.AWS_REGION }}
    - run: |
        commit_hash=`git rev-parse HEAD`
        aws deploy create-deployment --application-name ${{ vars.CODEDEPLOY_APPLICATION_NAME }} --deployment-group-name ${{ vars.CODEDEPLOY_DEPLOYMENT_GROUP_NAME }} --github-location repository=${{ github.repository }},commitId=${{ github.sha }} --ignore-application-stop-failures

A basic version of the GitHub Actions Code.

2.5.6 Run GitHub Actions Code

Select Actions Tab
Select Deploy Tab
Select Run workflow button

2.5.7 Successfully running `GitHub Actions`

Successfully ran main.yml

4.0 Cost

Plan	USD
Monthly cost	$11.83
Total 12 months cost	$141.96

Overall, AWS's prices are quite competitive. The most important thing is that CodeDeploy is cheap, and the cost of using Amazon EC2 t4g.nano is very low, so AWS is a low-cost + efficient cloud service provider.

4.1 Detailed Estimate

Service	Monthly	First 12 months total (USD)
AWS CodeDeploy	$8.8	$105.6
Amazon EC2	$1.533	$18.4
Amazon Route 53	$0.4	$4.8
VPN Connection	$1.1	$13.2

5.0 Summary

GitHub Actions + CodeDepoly are powerful DevOps tools that fulfill the principle of “think big, take small steps” in a business environment.

To conclude, let's summarize the key points of this chapter:

5.1 Principles

The new “Macro Portfolio” system is to comply with the “Least Effort Principle”, which includes (1) agile development, and (2) agile deployment
The real issues were (1) the project took too long to deploy, and (2) automated deployment was not achieved
Success is due to the following: (1) Other departments want small features in small increments. (2) More simplicity means more understanding of the problem's root cause.

5.2 Action

Give the “Updated API Manual” to other departments to try before every Thursday
Simplicity is a good result of the Highest Standards because we performed (1) a “DIVE DEEP investigation” and (2) understanding the root cause of the problem

5.3 AWS DevOps

The development engineer commits the code via GitHub Push
GitHub Actions trigger workflows
IAMROLE_GITHUB_ARN authorizes access to AWS resources
GitHub Actions triggers AWS CodeDeploy
AWS CodeDeploy triggers deployment to Amazon EC2 instances
AWS CodeDeploy pulls Github resources and deploys to Amazon EC2 instances

5.4 `AWS IAM` （`CodeDeploy`, `EC2`, `Github`）

AWSCodeDeployService-EC2AccessCodeDeployRole-20241024T000000
AWSCodeDeployService-DepolyEC2Role-20241024T000000
AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000

5.5 `AWS CodeDeploy` （Appspec.yml）

BeforeInstall
BeforeInstall
AfterInstall
ApplicationStart
ValidateService

5.6 Cost

Monthly cost: $11.83 (USD)
Total 12 months cost: $141.96 (USD)

Postscript

On 14 December 2024, I attended the annual Amazon Greater China Community Gathering. I am very thankful to AWS for bringing me an unforgettable experience.

📷Shoot and 🎬Edit by Kenny Chan
Also, thanks to Smile (Lingxi) Lv - Developer Experience Advocacy Program Manager for supporting AWS Community Builder.

About Me

Kenny Chan, AWS Community Builder (Hong Kong), specialty of Fintech & Machine Learning

2.0 AWS DevOps Pipeline

2.1 Pre-requisites

2.1.1 Knowledge Pre-requirements

2.1.2 Project Requirements

2.2 Creating IAM Roles

2.2.1 AWSCodeDeployService-EC2AccessCodeDeployRole-20241024T000000

2.2.2 AWSCodeDeployService-DepolyEC2Role-20241024T00000

2.2.3 AWSCodeDeployService-GitAssumeRoleWithAction-20241024T000000

2.3 Create Amazon EC2

2.3.1 Associate Elastic IP address

2.3.2 Amazon Route 53

2.3.3 Add AWS IAM roles

2.3.4 Install CodeDeploy Agent on Amazon EC2

2.3.5 (Optional) Install Git on Amazon EC2

2.3.6 (Optional) Install NGINX

2.3.7 Appspec.yml

2.3.8 Static Website Pages

2.4 Create AWS CodeDeploy

2.4.1 Create the AWS CodeDeploy application

2.4.2 Create AWS CodeDeploy Deployment Group

2.4.3 Create AWS CodeDeploy Deployment

2.4.4 Successful run of AWS CodeDeploy

2.5 Create GitHub Actions

2.5.1 Create GitHub Actions workflow

2.5.2 Configurate GitHub Actions secrets and variables

2.5.3 Add GitHub Actions secrets variables

2.5.4 Add GitHub Actions variables

2.5.5 Write GitHub Actions Code

2.5.6 Run GitHub Actions Code

2.5.7 Successfully running GitHub Actions

4.0 Cost

4.1 Detailed Estimate

5.0 Summary

5.1 Principles

5.2 Action

5.3 AWS DevOps

5.4 AWS IAM （CodeDeploy, EC2, Github）

5.5 AWS CodeDeploy （Appspec.yml）

5.6 Cost

Postscript

About Me

Read next

Interview Questions on AWS Networking: VPC, Subnets, and Security Groups

Day 24: Thanks and Goodbye!

I would love to get some feedback on this.

Terraform: Steps to create complete AWS S3 module

2.3.3 Add `AWS IAM` roles

2.3.4 Install `CodeDeploy Agent` on `Amazon EC2`

2.3.5 (Optional) Install `Git` on `Amazon EC2`

2.3.6 (Optional) Install `NGINX`

2.4 Create `AWS CodeDeploy`

2.4.1 Create the `AWS CodeDeploy` application

2.4.2 Create `AWS CodeDeploy` Deployment Group

2.4.3 Create `AWS CodeDeploy` Deployment

2.4.4 Successful run of `AWS CodeDeploy`

2.5 Create `GitHub Actions`

2.5.1 Create `GitHub Actions` workflow

2.5.2 Configurate `GitHub Actions` secrets and variables

2.5.3 Add `GitHub Actions secrets` variables

2.5.4 Add `GitHub Actions variables`

2.5.5 Write `GitHub Actions` Code

2.5.7 Successfully running `GitHub Actions`

5.4 `AWS IAM` （`CodeDeploy`, `EC2`, `Github`）

5.5 `AWS CodeDeploy` （Appspec.yml）