Automating ECR Token Renewal for Private Registry Pulls in Kubernetes Using a CronJob.

In Kubernetes, the authentication token for Amazon ECR (Elastic Container Registry) expires every 12 hours. This is due to ECR's use of the 'GetAuthorizationToken' API, which generates a base64-encoded token with a default validity period of 12 hours.

Why the Token Expires in 12 Hours:

1. Security: Short-lived tokens reduce the window of opportunity for unauthorized access if a token is compromised. Regularly rotating the token enhances security by ensuring that stale or leaked tokens are invalidated in a short time frame.
2. AWS Design: AWS designed the ECR token system to issue tokens with a 12-hour expiration as part of its security best practices. This time limit balances the need for frequent reauthentication with minimizing user disruption.

Handling Expiration in Kubernetes:

To maintain continuous access to ECR from within a Kubernetes cluster, it is common to automate the refresh process using a Kubernetes CronJob. This job would periodically refresh the authentication token and update the necessary secrets to ensure uninterrupted image pulls from ECR. This article walks through automating the process using a kubernetes CronJob to refresh ECR credentials.

Step 1: Create a Base Docker Image

You will need a Dockerfile that includes essential tools such as aws-cli, curl, wget, docker, and kubectl. This image will serve as the base for the CronJob to refresh the ECR secret.
Dockerfile

# create Dockerfile with this content
FROM alpine:latest
RUN apk --no-cache add aws-cli wget curl docker docker-compose \
    && wget https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kubectl \
    && mv kubectl /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl \
    && apk del wget

After creating the Dockerfile:

1. Build the image.
2. Push it to a public container repository (e.g., Docker Hub or Amazon ECR).

Step 2: Create an AWS Credentials Secret

In this step, create a Kubernetes secret that contains AWS credentials with permission to access ECR. Ensure that the AWS credentials have the necessary ECR permissions.

kubectl create secret generic aws-credentials \
  --from-literal=aws_access_key_id=<AWS_ACCESS_KEY_ID> \
  --from-literal=aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> \
  --namespace dev #replace with your namespace

Step 3: Create a Service Account

A dedicated Service Account will be used by CronJob to perform its tasks. This Service Account ensures proper security scoping for the refresh job.

cronjob-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cronjob-sa
  namespace: dev #replace with your namespace

Apply file
kubectl apply -f cronjob-service-account.yaml

Step 4: Create a Role with Secret Permissions

Create a Role that allows the Service Account to manage secrets, as it will need to update the ECR secret with new credentials.

cronjob-role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cronjob-role
  namespace: dev
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "delete"]

Apply the Role:
kubectl apply -f cronjob-role.yaml

Step 5: Bind the Role to the Service Account

Bind the Role to the Service Account using a RoleBinding to ensure the Service Account has the required permissions.

cronjob-rolebinding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cronjob-rolebinding
  namespace: dev
subjects:
- kind: ServiceAccount
  name: cronjob-sa
  namespace: dev
roleRef:
  kind: Role
  name: cronjob-role
  apiGroup: rbac.authorization.k8s.io

Apply the RoleBinding:
kubectl apply -f cronjob-rolebinding.yaml

Step 6: Create the Kubernetes CronJob

Now, set up a CronJob that will periodically refresh the ECR Docker registry credentials. In this example, the job runs every 11 hours.

The CronJob performs the following actions:

1. Retrieve the ECR token: Using the AWS CLI command aws ecr get-login-password, the CronJob retrieves the authentication token for the ECR registry.
2. Create/Update a Kubernetes secret: The token is then stored in a Kubernetes secret. This secret is used by Kubernetes to authenticate with the ECR registry whenever it needs to pull Docker images.

cronjob.yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-creds-refresh
  namespace: dev
spec:
  schedule: 0 */11 * * *
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: ecr-creds-refresh
              image: awsecr-kubectl:latest #Input the image from step 1
              command:
                - /bin/sh
                - '-c'
                - >-
                  aws --version 

                  aws ecr get-login-password --region <region>

                  echo "deleting imagepull secret..."

                  kubectl delete secret -n dev --ignore-not-found ${SECRET_NAME}

                  echo "recreating imagepull secret..."

                  kubectl create secret -n dev docker-registry ${SECRET_NAME} \

                  --docker-server=***.dkr.ecr.us-east-1.amazonaws.com \

                  --docker-username=AWS \

                  --docker-password="$(aws ecr get-login-password --region
                  us-east-1)" 
                  echo "secret recreated!!"
              env:
                - name: AWS_REGION
                  value: us-east-1
                - name: SECRET_NAME
                  value: aws-ecr # secret name
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      name: aws-credentials
                      key: aws_access_key_id
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: aws-credentials
                      key: aws_secret_access_key
              imagePullPolicy: IfNotPresent
          restartPolicy: OnFailure
          serviceAccountName: cronjob-sa
          serviceAccount: cronjob-sa

Deploy the CronJob:
kubectl apply -f cronjob.yaml

Purpose of the Secret:
Kubernetes uses this secret when pulling Docker images from ECR. Without this step, your cluster could lose access to ECR when the token expires, leading to failed deployments or pods unable to start due to image pull errors.

Conclusion

By automating the update of ECR Docker registry credentials with a Kubernetes CronJob, you can eliminate manual intervention and ensure your cluster always has valid credentials for pulling Docker images from ECR. This approach leverages Kubernetes-native tools like CronJob, RBAC, and Secrets to securely and efficiently manage credentials. The steps outlined, from creating a base Docker image to setting up the necessary roles and service accounts, provide a robust solution for maintaining a seamless container lifecycle in your Kubernetes environment. This automation not only improves security by regularly rotating credentials but also enhances operational efficiency, freeing your team to focus on more critical tasks.

Refrence
Amazon ECR AuthorizationData