How to Deploy Azure Pipeline Agent on EKS with Auto Scaling!

To deploy self-hosted Azure DevOps agents on Amazon EKS with KEDA-based auto-scaling, follow this step-by-step guide.
requirements:

• External Secrets Operator ( https://external-secrets.io).
• KEDA ( https://keda.sh/docs/2.16/scalers/azure-pipelines/).

Step 1: Create the IAM Role and IAM Policy.

Here are the AWS CLI commands to IAM Role:
Replace
<ACCOUNT_ID> → Your AWS Account ID.
<AWS_REGION> → Your AWS region (e.g., us-east-1).
<OIDC_ID> → Your EKS OIDC ID (get it using the command below).

aws iam create-role --role-name AzureDevOpsEKSRole \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<NAMESPACE>:azure-devops-agent"
                    }
                }
            }
        ]
    }'

Here are the AWS CLI commands to create IAM policy:

aws iam create-policy \
    --policy-name ecr-secrets-policy \
    --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:GetAuthorizationToken",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:CreateRepository"
            ],
            "Resource": ["arn:aws:ecr:<AWS_REGION>:<ACCOUNT_ID>:repository/*"]
        },
        {
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": ["arn:aws:secretsmanager:<AWS_REGION>:<ACCOUNT_ID>:secret/*"]
        }
    ]
}'

Here are the AWS CLI commands to attach the policy to IAM Role:

# Attach policy to role
aws iam attach-role-policy \
    --role-name AzureDevOpsEKSRole \
    --policy-arn arn:aws:iam::<ACCOUNT_ID>:policy/ecr-secrets-policy

Step 2: Create Docker Image and push to AWS ECR!
Here are the AWS CLI commands to create docker image and push to AWS ECR:

aws ecr get-login-password --region <AWS_REGION> | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com
docker build -t azure-pipeline-agent .
docker tag azure-pipeline-agent:latest <ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/azure-pipeline-agent:latest
docker push <ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/azure-pipeline-agent:latest

Step 3: Create a Secret for the PAT Token
Generate a PAT (Personal Access Token) in Azure DevOps with Agent Pools (Read & Manage) permissions.

Step 4: Store the PAT Token in AWS Secrets Manager
Instead of storing the Azure DevOps Personal Access Token (PAT) directly in Kubernetes, we'll store it securely in AWS Secrets Manager and retrieve it dynamically.
Here are the AWS CLI commands to store it securely in AWS Secrets:

aws secretsmanager create-secret --name azure-pipelines-secret \
--secret-string '{
  "AZP_URL": "https://dev.azure.com/YOUR_ORG",
  "AZP_TOKEN": "your_personal_access_token",
  "AZP_AGENT_NAME": "AWS-EKS-Agent-Pool"
}'

Step 5: Create an Agent Pool in Azure DevOps

Go to Azure DevOps → Organization Settings → Agent Pools.
Click Add Pool.
Choose a name, e.g., AWS-EKS-Agent-Pool.
Select Self-hosted and click Create.

Step 6: Install Helm To EKS!

Clone https://github.com/alexg84/azure-pipeline-agent and navigate to Your Helm Chart Directory.
Create the azure-pipeline-agent Namespace and ensure the namespace exists before deploying the Helm chart:

kubectl create namespace azure-pipeline-agent
kubectl get namespaces

cd /azure-pipeline-agent/agent
helm dependency build
helm install azure-pipeline-agent . -n azure-pipeline-agent

Step 7: Check Helm Release Status
Run the following command to check if Helm successfully installed the chart:

helm list -n azure-pipeline-agent

If the deployment is successful, you should see an entry like this:

NAME                    NAMESPACE               REVISION    STATUS      CHART                 APP VERSION
azure-pipeline-agent    azure-pipeline-agent    1           deployed    my-chart-0.1.0        1.0.0

If the STATUS is failed, use the following command to inspect what went wrong:

helm status azure-pipeline-agent -n azure-pipeline-agent

Verify that the pods are running correctly:

kubectl get pods -n azure-pipeline-agent

Expected output (example):

NAME                                    READY   STATUS    RESTARTS   AGE
azure-pipeline-agent-7f5d8f6dfd-abc12   1/1     Running   0          2m

If the pods are not running, use:

kubectl describe pod <POD_NAME> -n azure-pipeline-agent

To check logs:

kubectl logs <POD_NAME> -n azure-pipeline-agent

If something is wrong, check Helm and Kubernetes events:

kubectl get events -n azure-pipeline-agent --sort-by=.metadata.creationTimestamp

If needed, delete the deployment and try again:

helm uninstall azure-pipeline-agent -n azure-pipeline-agent
helm install azure-pipeline-agent . -n azure-pipeline-agent