Injecting secrets from Vault into Helm charts with ArgoCD

Managing secrets in Kubernetes isn’t a trivial topic. As is usual with Kubernetes, there are always many ways to achieve the desired goal and it’s often a problem to choose the right one for our case. In this article, I will show you one of ways to use ArgoCD with the help of Vault Plugin to inject secrets into Helm Charts. So, as you can guess, this may be one of the best ways to inject secrets if you are already using ArgoCD and Vault in your project.

Prerequisites

The basic knowledge of Kubernetes and Helm is obvious here. For ArgoCD and Vault I will try to guide you step by step in this article.
We will use the CLI wherever possible, so I recommend you install all the following:

# Helm
brew install helm

# Vault
brew tap hashicorp/tap
brew install hashicorp/tap/vault

#ArgoCD
brew install argocd

For structure, we will create 2 namespaces. The first one - technical, will be for Vault and ArgoCD instances, in the second one, we will install target Helm charts with injected secrets.

# namespace for Vault & ArgoCD
kubectl create ns toolbox

# namespace for resoruces installed by ArgoCD
kubectl create ns sandbox

I also encourage you to install kubectx + kubens to navigate Kubernetes easily.

Vault installation

For the beginning select toolboox namespace

kubens toolbox

To install Vault we will use the official Helm chart provided by HashiCorp. For simplicity, install it in developer mode. In dev mode, Vault doesn't need to be initialized or unsealed, but remember, it's only for development or experimentation. Never, ever run a dev mode in production

helm install vault hashicorp/vault --set "server.dev.enabled=true"

Vault can be configured via HTTP API, UI, or CLI. To operate Vault from local CLI establish port forwarding to vault-0 Pod, and setup Vault server address for already installed Vault CLI.

# port forwarding in separate terminal window
kubectl port-forward -n toolbox vault-0 8200

# login into Vault. Use 'root' token to authenticate into Vault
export VAULT_ADDR=http://127.0.0.1:8200
vault login 

I also encourage you to explore browser UI interface of Vault. You have to use this same root token generated in dev mode.

Vault setup

Vault uses Secrets Engines to store, generate, or encrypt data. The basic Secret Engine for storing static secrets is Key-Value engine. Let’s create one sample secret that we’ll inject later into Helm Charts.

# enable kv-v2 engine in Vault
vault secrets enable kv-v2

# create kv-v2 secret with two keys
vault kv put kv-v2/demo user="secret_user" password="secret_password"

# create policy to enable reading above secret
vault policy write demo - <<EOF
path "kv-v2/data/demo" {
  capabilities = ["read"]
}
EOF

Now we need to create a role that will authenticate ArgoCD in Vault. We said that Vault has Secrets Engines component. Auth methods are another type of component in Vault but for assigning identity and a set of policies to user/app. As we are using Kubernetes platforms, we need to focus on Kubernetes Auth Method to configure Vault accesses. Let’s configure this auth method.

# enable Kubernetes Auth Method
vault auth enable kubernetes

# get Kubernetes host address
K8S_HOST="https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"

# get Service Account token from Vault Pod
SA_TOKEN=$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# get Service Account CA certificate from Vault Pod
SA_CERT=$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)

# configure Kubernetes Auth Method
vault write auth/kubernetes/config \
    token_reviewer_jwt=$SA_TOKEN \
    kubernetes_host=$K8S_HOST \
    kubernetes_ca_cert=$SA_CERT

# create authenticate Role for ArgoCD
vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-repo-server \
  bound_service_account_namespaces=toolbox \
  policies=demo \
  ttl=48h

That's all for now in Vault. Once you have created all components, you can try to find them in the browser interface http://localhost:8200/

ArgoCD & Vault Plugin Installation

Time for the main actor of this article - Argo CD Vault Plugin It will be responsible for injecting secrets from the Vault into Helm Charts. In addition to Helm Charts, this plugin can handle secret injections into pure Kubernetes manifests or Kustomize templates. Here we will focus only on Helm Charts. Different sources required different installations, which you can find in plugin documentation.

What makes plugin documentation less clear is that it can be installed in two ways:

• Installation via argocd-cm ConfigMap (old option, deprecated from version 2.6.0 of ArgoCD)
• Installation via a sidecar container (new option, supported from version 2.4.0 of ArgoCD)

Since the old option will be not supported in future releases, I will install the ArgoCD Vault Plugin using a sidecar container. In order to properly install and configure ArgoCD, we need to follow a few steps:

Before all make sure you are still in toolbox namespace where we want to place Vault, ArgoCD, and all stuff for Vault plugin.

kubens toolbox

1. Create k8s Secret with authorization configuration that Vault plugin will use.

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
type: Opaque
stringData:
  AVP_AUTH_TYPE: "k8s"
  AVP_K8S_ROLE: "argocd"
  AVP_TYPE: "vault"
  VAULT_ADDR: "http://vault.toolbox:8200"

Make sure you set the proper Vault address and role name.

1. Create k8s ConfigMap with Vault plugin configuration that will be mounted in the sidecar container, and overwrite default processing of Helm Charts on ArgoCD. Look carefully at this configuration file. Under init command you can see that we add Bitnami Helm repo and execute helm dependency build. It is required if Charts installed by you use dependencies charts. You can customize or get rid of it if your Charts haven’t any dependencies.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      init:
       command:
          - bash
          - "-c"
          - |
            helm repo add bitnami https://charts.bitnami.com/bitnami
            helm dependency build
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s toolbox:argocd-vault-plugin-credentials -
      lockRepo: false

1. Finally, we have to install ArgoCD from the official Helm Chart but with extra configuration that provides modifications required to install Vault plugin via sidecar container.

repoServer:
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.11.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}
    - name: tmp-dir
      emptyDir: {}

# If you face issue with ArgoCD CRDs installation, then uncomment below section to disable it
#crds:
#  install: false

Save that Helm values as argocd-helm-values.yaml and execute below commands:

# once againe make sure to use proper namespace
kubens toolbox

# install ArgoCD with provided vaules
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n toolbox -f argocd-helm-values.yaml

All of the above configurations you can find in dedicated GitHub repo

If all went well, you should see similar list of Pods in toolbox namespace. Note that argocd-repo-server has sidecar container avp-helm

Install your resources with secrets injection

Is the time for a final check of our setup and installation of our Helm Charts.

Firstly, let’s try to authorize against ArgoCD. To obtain admin user password execute the below command:

kubectl -n toolbox get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

As with Vault, with ArgoCD we will also be working partly with the CLI and partly web UI.

# port forwarding in separate terminal window
kubectl port-forward svc/argocd-server 8080:80

# authorize ArgoCD CLI
argocd login localhost:8080 --username admin --password $(kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

As our demo Chart we will use my debug Spring Boot application from GitHub repo. It’s simple web server that exposes a few debugging endpoints. Application has Helm templates and ArgoCD Application definition under /infra directory. To deploy this stack to k8s with Argo we need to apply ArgoCD Application CRD. Below full code sample, which you can also explore here.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo
spec:
  destination:
    namespace: sandbox
    server: https://kubernetes.default.svc
  project: default
  source:
    path: infra/helm
    repoURL: https://github.com/luafanti/spring-boot-debug-app
    targetRevision: main
    plugin:
      env:
        - name: HELM_VALUES
          value: |
            serviceAccount:
              create: true
            image:
              repository: luafanti/spring-boot-debug-app
              tag: main
              pullPolicy: IfNotPresent
            replicaCount: 1
            resources:
              memoryRequest: 256Mi
              memoryLimit: 512Mi
              cpuRequest: 500m
              cpuLimit: 1 
            probes:
              liveness:
                initialDelaySeconds: 15
                path: /actuator/health/liveness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
              readiness:
                initialDelaySeconds: 15
                path: /actuator/health/readiness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
            ports:
              http:
                name: http
                value: 8080
              management:
                name: management
                value: 8081
            envs:
              - name: VAULT_SECRET_USER
                value: <path:kv-v2/data/demo#user>
              - name: VAULT_SECRET_PASSWORD
                value: <path:kv-v2/data/demo#password>
            log:
              level:
                spring: "info"
                service: "info"
  syncPolicy: {}

You can see in line 54 & 56 placeholders with pattern <path:vault_secret_path#secret_key> where Vault Plugin will inject the actual value from Vault secret.
I also encourage you to compare this definition file with a definition without secret injection and without using Vault Plugin here. You should notice that source property is different when we use secrets injection. When we want to leverage on Vault plugin we need to define our Argo Application with source plugin and pass Helm Values using env HELM_VALUES

Let’s install this Argo Application and sync them.

# make sure you are in namespace where Argo has benn installed
kubens toolbox

# once you download soruce from GIT repo
kubectl apply -f infra/argocd/argocd-application-with-vault-secrets.yaml

# List ArgoCD applications
argocd app list

# Sync application
argocd app sync toolbox/demo

Once synchronization is finished, you should see beautiful green-full screen in ArgoCD UI

Verify if injection works.

# use port other than 8080 as the tunnel to Argo already uses this port
kubectl port-forward -n sandbox svc/demo-spring-debug-app 8090:8080

# check injected envs 'VAULT_SECRET_PASSWORD' 'VAULT_SECRET_USER' in debug app 
chrome http://localhost:8090/envs

One of the greatest things about the plugin is that if the value changes in Vault, ArgoCD will notice these changes and display OutOfSync status. Let's prove it.

# update secrets in Vault
vault kv put kv-v2/demo user="secret_user_new" password="secret_password_new"

# refresh application as well with target manifests cache
argocd app get toolbox/demo --hard-refresh

After Hard refresh you should see that your Argo Application back to status OutOfSync what is expected during Vault secret update. Thanks to this mechanism, you don't have to worry about losing control of keeping your secrets up to date.

Troubleshooting & possible problems

• make sure your Vault secrets don’t disappear from Vault. In this guide, we use Vault in dev mode so secrets are stored in-memory. After cluster reboot all Vault objects will disappear.
• if you would like to use different namespace names for Vault/ArgoCD etc. make sure you adjust your configuration files properly, especially HERE & HERE
• Sometimes if you install ArgoCD multiple times in your cluster you can face error related to CRDs. You can uncomment this section to resolve it.

Comments

[Alex Vaque]

Thank you for your tutorial.

I was following your tutorial and I added these 2 lines in argo helm chart , is it possible that you forgot these these 2 lines in the Argo Helm Chart ?

   - secretRef:
       name: argocd-vault-plugin-credentials

I was running env |grep AVP in the sidecar container and I got empty results but adding these two lines , I got the Vault credentials variables into this