The perfect combination for Azure Key Vault integration with AKS (External Secret Operator + Reloader + Workload Identity)

Hello everyone !

I will show a quick demonstration of how we can use this integration with simple steps. I believe this is one of the most secure and easy to use method to read secrets from Azure key vault. In particular, if you're using IaC tools and creating SQL servers, app registrations and so on, you're passing secrets to the Key Vault during creation and reading them in kubernetes. There are several other methods and possibilities. But we won't compare them in this article.

Let's get started

Essentially, these are the requirements for this example,

• Azure Kubernetes Service with Workload Identity enabled
• External Secret Operator
• Reloader

What are these tools ? Here are the official for deep dive links.

• https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster
• https://external-secrets.io/latest/
• https://github.com/stakater/Reloader

The flow will be like this,

to make it more secure, private endpoint and internal ci/cd agent should be used. I will be acting as an agent in the article.

All of this will be documented as code on this GitHub repo as well,

https://github.com/Learn4Ops/esowir

git clone https://github.com/Learn4Ops/esowir
cd esowir/infrastracture
az login --use-device-code
terraform init
terraform apply --auto-approve -var "subscription_id=xxxxx" -var "tenant_id=xxxxx"

Subscription id and tenant id are required

The key point is to manage the subject part for managed identity,

subject = "system:serviceaccount:testapp:external-secret-service-account"

And this managed Identity will need 'Get' secrets permission.

We will create service account and use the namespace as above.

Let's connect kubernetes and install required objects.

az aks get-credentials --resource-group aks-resource-group --name aks-cluster --overwrite-existing

We will work on these namespaces,

kubectl create ns external-secrets
kubectl create ns testapp

Helm Charts,

ESO,
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets

Reloader,
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader -n external-secrets

Key vault secrets should exist (terraform already created), 'test-password' , 'test-username'

Okey, now everything is ready to test the flow, with the following command we will install each kubernetes object, so let's check it first.

1. Create a service account to use as the workload identity
2. Create a CRD named SecretStore that will use the service account and connect to Azure Key Vault
3. Create CRD named ExternalSecret that will create kubernetes secret using SecretStoreRef
4. Create deployment which will use kubernetes secret

kubectl apply -f ../application

Check service account, If you are following don't forget to change related fields

kubectl get sa -n testapp

You can check SecretStore health, Detailed API guide: https://external-secrets.io/latest/api/secretstore/

kubectl get secretstore -n testapp

Also External Secret, Detailed API guide: https://external-secrets.io/latest/api/externalsecret/

kubectl get externalsecret -n testapp

Looks good, so secret should be created

kubectl get secret -n testapp

Then I will check the secrets values from the pod,

kubectl exec -it -n testapp testapp-deployment-65f7dc494c-xztjw -- sh
echo $TEST_USERNAME
or
echo $TEST_PASSWORD

Same as azure key vault values,

What about changing the secret? Reloader takes over here with the annotation adjustment,

After the creation of new value,

Then wait 30 seconds because the interval is set to this value.

Amazing ! I don't need to worry about new secrets for rollout deployments.

Let's clear the environment.
terraform destroy --auto-approve -var "subscription_id=xxxxx" -var "tenant_id=xxxxx"

I mentioned earlier that to make it more secure, you can use Microsoft's internal network, Kubernetes should access KeyVault through a private endpoint, and creating the secret can be managed on the Azure side with an agent deployed in the container or virtual machine.

Let's recap important advantages,
-> Workload identity is really great way to use Azure resources, in our case we didn't give any secret to use service account as identity and we filter related service account with namespace option.
-> External Secret Operator has a lot features, everything is documented on official site, not so complex to use and define configuration.
-> Reloader is very handy, if you have a lot of pods using secret and need to refresh this value, no worries, you just wait for new secret to restart pods automatically.