Best Practices to Deploy AWS Resources in the world of EKS

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that makes it easy to deploy, scale, and manage containerized applications using Kubernetes. To enhance the integration between EKS and AWS services, AWS introduced AWS Controllers for Kubernetes (ACK). In this blog post, we'll dive into what ACK is, how it simplifies the management of AWS resources within EKS clusters, and how you can leverage it to supercharge your Kubernetes applications.

Understanding AWS Controllers for Kubernetes (ACK)

AWS Controllers for Kubernetes (ACK) is a powerful tool that extends Kubernetes' capabilities by allowing you to define and use AWS resources directly within your Kubernetes clusters. It achieves this by introducing custom resource definitions (CRDs) and controllers that understand how to interact with AWS services.

Suppose you want to provision an Amazon S3 bucket for storing static assets of your web application. With ACK, you can define an S3Bucket custom resource in your Kubernetes manifest like this:

apiVersion: s3.services.k8s.aws/v1alpha1
kind: S3Bucket
metadata:
  name: my-static-assets-bucket
spec:
  bucketName: my-static-assets-bucket
  acl: private

This declarative definition instructs ACK to create an S3 bucket named my-static-assets-bucket with private access control.

When using ACK, it's important to ensure that the necessary IAM roles and permissions are set up to allow the Kubernetes cluster to interact with AWS services. This includes providing IAM roles for the ACK service account and, if required, for the pods to interact with the AWS services.

1. Setting Up ACK in your EKS Cluster
   Setting up ACK involves deploying ACK controllers and CRDs using Kubernetes manifests. This can be done manually or using tools like Helm charts for easier management.
   
   helm repo add eks https://aws.github.io/eks-charts
helm upgrade -i aws-controllers s3-controller --namespace ack-system eks/aws-controllers
   

2. Managed AWS Resources with ACK
   ACK allows you to define and manage EC2 instances directly from Kubernetes manifests. This provides the flexibility to dynamically provision and scale compute resources. Lets define an EC2Instance custom resource to create an EC2 instance:
   
   apiVersion: ec2.services.k8s.aws/v1alpha1
kind: EC2Instance
metadata:
name: my-ec2-instance
spec:
instanceType: t2.micro
imageId: ami-12345678
keyName: my-key-pair
subnetId: subnet-12345678
   
   This YAML manifest instructs ACK to create an EC2 instance of type t2.micro using the specified AMI, key pair, and subnet.
   

3. Security and Best Practices with ACK
   To ensure secure interactions between ACK and AWS services, it's crucial to set up IAM roles and policies. These roles define the permissions required for ACK to manage AWS resources. IAM Roles for Service Accounts, or IRSA, is a system that automates the provisioning and rotation of IAM temporary credentials (called a Web Identity) that a Kubernetes ServiceAccount can use to call AWS APIs. The primary advantage of IRSA is that Kubernetes Pods which use the ServiceAccount associated with an IAM Role can have a reduced IAM permission footprint than the IAM Role in use for the Kubernetes EC2 worker node (known as the EC2 Instance Profile Role). This security concept is known as Least Privilege.

4. Create OIDC identity provider for EKS
   
   export EKS_CLUSTER_NAME=<eks cluster name>
export AWS_REGION=<aws region id>
eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --region $AWS_REGION --approve
   

• Create an IAM role and policy for your service account # Update the service name variables as needed SERVICE="s3" ``` AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") ACK_K8S_NAMESPACE=ack-system

ACK_K8S_SERVICE_ACCOUNT_NAME=ack-$SERVICE-controller

Create TRUST_RELATIONSHIP Policy

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_PROVIDER}:sub": "system:serviceaccount:${ACK_K8S_NAMESPACE}:${ACK_K8S_SERVICE_ACCOUNT_NAME}"
}
}
}
]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json

ACK_CONTROLLER_IAM_ROLE="ack-${SERVICE}-controller"
ACK_CONTROLLER_IAM_ROLE_DESCRIPTION="IRSA role for ACK ${SERVICE} controller deployment on EKS cluster using Helm charts"
aws iam create-role --role-name "${ACK_CONTROLLER_IAM_ROLE}" --assume-role-policy-document file://trust.json --description "${ACK_CONTROLLER_IAM_ROLE_DESCRIPTION}"
ACK_CONTROLLER_IAM_ROLE_ARN=$(aws iam get-role --role-name=$ACK_CONTROLLER_IAM_ROLE --query Role.Arn --output text)```

• Attach IAM policy to the IAM role
  
  aws iam put-role-policy \
--role-name "${ACK_CONTROLLER_IAM_ROLE}" \
--policy-name "ack-recommended-policy" \
--policy-document "$INLINE_POLICY"
  

• Attach an IAM role to a service account
  Verify that your service account exists using kubectl describe:
  
  kubectl describe serviceaccount/$ACK_K8S_SERVICE_ACCOUNT_NAME -n $ACK_K8S_NAMESPACE
  

Associate IAM Role to SA:

```# Annotate the service account with the ARN
export IRSA_ROLE_ARN=eks.amazonaws.com/role-arn=$ACK_CONTROLLER_IAM_ROLE_ARN
kubectl annotate serviceaccount -n $ACK_K8S_NAMESPACE $ACK_K8S_SERVICE_ACCOUNT_NAME $IRSA_ROLE_ARN

Note the deployment name for ACK service controller from following command

kubectl get deployments -n $ACK_K8S_NAMESPACE
kubectl -n $ACK_K8S_NAMESPACE rollout restart deployment ```

• Verify the association kubectl get pods -n $ACK_K8S_NAMESPACE kubectl describe pod -n $ACK_K8S_NAMESPACE <NAME> | grep "^\s*AWS_"

The output should show aws role arn.

Follow the principle of least privilege when creating IAM roles and policies for ACK. Only grant the necessary permissions required for ACK to manage specific AWS resources.

• Monitoring and Troubleshooting ACK Implementing effective logging and monitoring for ACK-managed resources is crucial for gaining visibility into performance and health. Configure CloudWatch Logs to collect logs from ACK-managed resources. You can set up CloudWatch Alarms to receive notifications based on specific metrics, ensuring proactive monitoring. Establish comprehensive logging and monitoring strategies, leveraging tools like CloudWatch, to gain insights into the behavior and performance of resources managed by ACK. Create dashboards to visualize important metrics.

By integrating AWS Controllers for Kubernetes (ACK) into your EKS clusters, you unlock a powerful set of tools for managing AWS resources directly from Kubernetes manifests. This seamless integration streamlines operations, enhances security, and opens up new possibilities for building resilient, scalable, and highly available applications. As you continue your Kubernetes journey, keep an eye on ACK updates and leverage its capabilities to further optimize your cloud-native workloads.