Integrating HashiCorp Vault with Kubernetes for Secure Secrets Management

In the world of cloud-native applications, securing sensitive information like API tokens, database passwords, encryption keys, and certificates is critical. A breach in your secrets management could expose vulnerabilities that attackers can exploit. While Kubernetes has its own mechanism to store secrets, it lacks several key security features like encryption at rest (enabled only in recent versions) and strong access control. In default configurations, Kubernetes stores secrets in plaintext in etcd, which can be a security risk if access to etcd is compromised.

HashiCorp Vault steps in as a powerful tool designed for secrets management. Vault offers dynamic secrets generation, encryption-as-a-service, and tight access control mechanisms. One of its key features is its integration with Kubernetes, enabling applications to securely retrieve secrets and ensuring that sensitive data is never exposed unnecessarily. This integration provides an enhanced way of managing secrets with fine-grained access controls, dynamic secrets generation, and encrypted storage.

This article will guide you through the integration of HashiCorp Vault with Kubernetes for managing secrets securely, complete with detailed explanations and code snippets. By the end, you will have a Kubernetes setup where your application pods can securely retrieve secrets from Vault.

Prerequisites
Before starting, ensure you have the following prerequisites:

1. Kubernetes Cluster: A running Kubernetes cluster with kubectl installed and configured to interact with the cluster.
2. Helm: Helm is used to install and manage Vault within Kubernetes.
3. Vault CLI: Installed locally on your machine for interacting with Vault.
4. Basic Kubernetes Knowledge: Familiarity with concepts such as Pods, Service Accounts, RBAC (Role-Based Access Control), and Deployments.
5. Understanding of Secrets Management: A basic understanding of how Kubernetes handles secrets and why more advanced solutions, like Vault, are necessary.

Why Use HashiCorp Vault with Kubernetes?
Kubernetes secrets are a basic way to store and manage sensitive information, but they have limitations:

• Storage in etcd: By default, Kubernetes stores secrets in the etcd datastore in plaintext. This presents a potential risk because anyone who can access etcd could read your secrets.

• Limited Access Controls: While Kubernetes RBAC can be used to control access to secrets, it is often challenging to implement fine-grained controls.

• Lack of Dynamic Secrets: Kubernetes secrets are static and need to be manually rotated. Vault, on the other hand, can generate dynamic, short-lived secrets that are automatically revoked when no longer in use.

HashiCorp Vault addresses these issues by providing:

1. Encryption: All secrets stored in Vault are encrypted with AES-256, ensuring that even if someone accesses the storage layer, they cannot read the secrets.
2. Dynamic Secrets: Vault can generate secrets on demand, for example, creating a new database user with specific privileges that expire after a defined TTL (Time-to-Live).
3. Access Control: Vault provides a robust access control mechanism based on policies that define which secrets users or systems can access and what actions they can perform.

Step 1: Installing HashiCorp Vault in Kubernetes

Let’s start by deploying HashiCorp Vault inside the Kubernetes cluster. We will use Helm to deploy Vault in High Availability (HA) mode, ensuring fault tolerance and redundancy.

1. Add the HashiCorp Helm Repository First, you need to add the HashiCorp Helm repository, which contains the official Vault Helm charts.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

This command adds the HashiCorp repository to Helm and fetches the latest charts for deployment.

1. Install Vault Now, deploy Vault in HA mode. HA mode is recommended for production environments to ensure that Vault remains available even if one of the pods crashes.

helm install vault hashicorp/vault --set "server.ha.enabled=true"

This command installs Vault with high availability enabled, ensuring that the system remains operational even during node failures.

1. Verify the Installation To ensure that Vault is correctly installed and running, check the status of the pods by running:

kubectl get pods -l app.kubernetes.io/name=vault

You should see multiple Vault pods running, which confirms that Vault is operating in HA mode. Each pod will act as a separate Vault instance, and they work together to provide high availability.

1. Accessing Vault Locally Since Vault is deployed inside the Kubernetes cluster, you need to forward the Vault service to a local port to interact with it via the Vault CLI.

kubectl port-forward svc/vault 8200:8200

This will forward the Vault service to port 8200 on your local machine, allowing you to access Vault at http://localhost:8200.

1. Initialize and Unseal Vault Vault, by design, is sealed on first initialization. It needs to be initialized, and unseal keys need to be used to unseal it.

• Initialize Vault:

Run the following command to initialize Vault:

vault operator init

This command will output a set of unseal keys and a root token. The unseal keys are critical, as they are used to unseal Vault every time it restarts.

• Unseal Vault:

Use at least three unseal keys to unseal Vault. Run the command below for each key:

vault operator unseal <unseal-key>

Once unsealed, Vault is ready to start managing secrets.

Step 2: Configuring Kubernetes Authentication in Vault

For Kubernetes applications to securely access secrets in Vault, we need to configure Kubernetes authentication in Vault. This allows pods to authenticate using their service accounts.

1. Enable the Kubernetes Auth Method First, enable the Kubernetes authentication method in Vault.

vault auth enable kubernetes

This tells Vault to recognize Kubernetes as an authentication provider.

1. Configure Vault to Communicate with Kubernetes
   Vault needs to know how to interact with the Kubernetes API. You will need to provide Vault with the Kubernetes API server address, the CA certificate, and the JWT token used for authentication.

2. Get the Kubernetes API server URL:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'

1. Retrieve the Vault service account token:

kubectl get secret $(kubectl get serviceaccount vault -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

1. Configure Vault to use these credentials:

Now, configure Vault to authenticate against the Kubernetes cluster using the API server URL and service account token:

vault write auth/kubernetes/config \
  token_reviewer_jwt="<vault-service-account-token>" \
  kubernetes_host="<kubernetes-api-url>" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

1. Create a Kubernetes Role in Vault With the Kubernetes auth method enabled, create a role in Vault that maps a Kubernetes service account to Vault policies.

vault write auth/kubernetes/role/demo-role \
  bound_service_account_names=demo-sa \
  bound_service_account_namespaces=default \
  policies=demo-policy \
  ttl=24h

• bound_service_account_names: The name of the Kubernetes service account that will authenticate with Vault.

• bound_service_account_namespaces: The namespace where the service account exists.

• policies: Vault policies define what the service account is allowed to access.

• ttl: The time-to-live for tokens issued to the service account.

In this example, any pod running with the demo-sa service account in the default namespace can authenticate to Vault and access secrets as defined by the demo-policy.

Step 3: Storing and Managing Secrets in Vault

Now that Vault is set up to authenticate Kubernetes service accounts, let’s start storing secrets in Vault.

1. Enable the Key/Value Secrets Engine Vault uses the Key/Value (kv) secrets engine to store secrets. Enable it at a specific path, such as secret/, which is commonly used to store general secrets.

vault secrets enable -path=secret kv

1. Storing Secrets Once the kv secrets engine is enabled, you can store secrets in Vault. For instance, store a database password under the path secret/db-password.

vault kv put secret/db-password password="mypassword"

This stores a database password securely inside Vault. You can retrieve this secret later by querying Vault.

1. Creating Policies for Access Control To control who can access the stored secrets, create Vault policies. These policies dictate which paths can be accessed and what operations are allowed.

For example, create a policy that allows reading the db-password secret.

Create a file named demo-policy.hcl:

path "secret/data/db-password" {
  capabilities = ["read"]
}

Apply the policy in Vault:

vault policy write demo-policy demo-policy.hcl

This policy allows the service account tied to the demo-role to read the db-password secret.

Step 4: Injecting Secrets into Kubernetes Pods

Now that secrets are stored in Vault, the next step is to inject them into Kubernetes pods automatically. We’ll use the Vault Agent Injector to inject secrets as environment variables or files in the pods.

1. Install the Vault Agent Injector The Vault Agent Injector runs as a sidecar alongside your application container and retrieves secrets from Vault on behalf of the pod.

First, enable the Vault Injector:

kubectl apply -f https://raw.githubusercontent.com/hashicorp/vault-k8s/main/deploy/injector/vault-agent-injector-deployment.yaml

This deploys the Vault Agent Injector in your Kubernetes cluster.

1. Annotate Pods for Injection To inject secrets into a pod, annotate the pod definition with the necessary Vault annotations. Here’s an example of a pod definition that retrieves the db-password secret:

apiVersion: v1
kind: Pod
metadata:
  name: demo-app
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "demo-role"
    vault.hashicorp.com/secret-volume-path: "/vault/secrets"
    vault.hashicorp.com/secret-secret/data/db-password: "db-password.txt"
spec:
  serviceAccountName: demo-sa
  containers:
  - name: demo-container
    image: nginx
    volumeMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
  volumes:
  - name: vault-secrets
    emptyDir: {}

• vault.hashicorp.com/agent-inject: Enables the Vault Agent Injector.

• vault.hashicorp.com/role: Specifies the Vault role that should be used to retrieve the secrets.

• vault.hashicorp.com/secret-volume-path: Specifies the path where secrets will be mounted.

• vault.hashicorp.com/secret-secret/data/db-password: Specifies the path of the secret to retrieve.

1. Verifying Secrets Injection Once the pod is deployed, Vault will automatically inject the secrets into the pod, either as environment variables or as files (depending on your configuration).

To verify that the secret was injected, enter the running pod:

kubectl exec -it demo-app -- /bin/sh

Navigate to the /vault/secrets directory, and you should find a file named db-password.txt containing the secret:

cat /vault/secrets/db-password.txt

Step 5: Best Practices for Vault and Kubernetes Integration

When integrating Vault with Kubernetes, it is essential to follow best practices to ensure the security of your secrets management infrastructure.

1. Use Short-Lived, Dynamic Secrets Whenever possible, use Vault’s dynamic secrets feature to generate short-lived credentials. This minimizes the impact of credential leaks and reduces the need for manual rotation.

For example, Vault can dynamically generate database credentials with a limited TTL, ensuring that the credentials are automatically revoked after a certain period.

1. Enable Auto-Unseal in Production
   Unsealing Vault manually is not practical in production environments. Instead, configure auto-unseal using a trusted cloud provider's KMS (Key Management Service). This allows Vault to automatically unseal without manual intervention after restarts.

2. Regularly Rotate Secrets
   Make it a routine to rotate secrets regularly, even if they haven’t been compromised. Vault makes it easy to automate this process using its built-in secret rotation capabilities.

3. Use Least Privilege for Service Accounts
   Ensure that Kubernetes service accounts and Vault roles are assigned the least privilege necessary to access secrets. Avoid using wildcard roles or policies that grant broad access.

4. Secure Communication Between Vault and Kubernetes
   Always ensure that the communication between Vault and Kubernetes is secured using TLS. Vault should be configured to use encrypted communication channels to protect secrets during transmission.

Conclusion

Integrating HashiCorp Vault with Kubernetes provides a highly secure and scalable solution for managing secrets in a Kubernetes environment. By following the steps outlined in this article, you can securely store and manage sensitive data like API keys, database credentials, and encryption keys within Vault, and dynamically inject them into your Kubernetes pods.

Vault’s dynamic secrets generation, fine-grained access control, and strong encryption make it an ideal choice for enterprises looking to enhance their Kubernetes security posture. This integration not only simplifies secrets management but also ensures that sensitive data is protected, reducing the risk of security breaches.

By implementing this solution, you are better positioned to meet modern security standards and ensure compliance with best practices for managing secrets in a cloud-native infrastructure.