DEV Community

suin
suin

Posted on

Sharing Secrets Between Kubernetes Clusters Using external-secrets PushSecret

#kubernetes #security #devops #externalsecrets

In this article, I'll explain how to share secrets between Kubernetes clusters using the PushSecret feature of external-secrets. In multi-cluster environments, secret synchronization is one of the crucial operational challenges. By building a system that securely shares and automatically synchronizes secrets between clusters, we can improve both operational efficiency and security.

Desired Architecture

We'll implement the following architecture to automatically synchronize secrets from the source cluster to the target cluster:

Prerequisites

To follow along with this article, you'll need the following tools:

  • Docker
  • k3d
  • kubectl
  • Helm
  • kubectx (optional: for easier context switching)
  • kubectl-view-secret (optional: for viewing secret contents)

Environment Setup

We'll set up two clusters using k3d:

  • Source cluster: k3d-source-cluster
  • Target cluster: k3d-target-cluster

Creating a Shared Network

First, let's create a shared network to enable communication between clusters. This network will facilitate communication between the two Kubernetes clusters we'll create later.

docker network create shared-net --subnet 172.28.0.0/16 --gateway 172.28.0.1
Enter fullscreen mode Exit fullscreen mode

This command creates a network with the following characteristics:

  • Subnet: 172.28.0.0/16
  • Gateway: 172.28.0.1
  • Network name: shared-net

Setting Up the Source Cluster

The source cluster will provide the secrets. Create a source-cluster.yaml with the following content:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: source-cluster
servers: 1
agents: 1
image: docker.io/rancher/k3s:v1.30.0-k3s1
kubeAPI:
  host: 0.0.0.0
  hostIP: 127.0.0.1
  hostPort: "6443"
ports:
  - port: 8080:80
    nodeFilters:
      - loadbalancer
registries:
  create:
    name: registry.localhost
    host: 127.0.0.1
    hostPort: "15000"
network: shared-net
options:
  k3d:
    wait: true
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  k3s:
    extraArgs:
      # Different CIDR ranges to avoid overlap
      - arg: "--cluster-cidr=10.42.0.0/16"
        nodeFilters:
          - server:*
      - arg: "--service-cidr=10.43.0.0/16"
        nodeFilters:
          - server:*
Enter fullscreen mode Exit fullscreen mode

Key points of this configuration:

  1. Network Settings
    • KubeAPI: Bound to port 6443
    • Load balancer: Port mapping 8080:80
    • Network: Uses the shared network created earlier
  2. Registry Settings
    • Creates a local registry (name: registry.localhost)
    • Bound to port 15000
  3. CIDR Settings
    • Cluster CIDR: 10.42.0.0/16
    • Service CIDR: 10.43.0.0/16 (Important to avoid network overlaps between clusters)

Create the cluster with:

k3d cluster create --config source-cluster.yaml
Enter fullscreen mode Exit fullscreen mode

Setting Up the Target Cluster

The target cluster will receive the secrets. Create a target-cluster.yaml with different settings:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: target-cluster
servers: 1
agents: 1
image: docker.io/rancher/k3s:v1.30.0-k3s1
kubeAPI:
  host: 0.0.0.0
  hostIP: 127.0.0.1
  hostPort: "6444"  # Different from source cluster
ports:
  - port: 8081:80   # Different from source cluster
    nodeFilters:
      - loadbalancer
registries:
  use:
    - registry.localhost  # Use source cluster's registry
network: shared-net
options:
  k3d:
    wait: true
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  k3s:
    extraArgs:
      # Different CIDR ranges from source cluster
      - arg: "--cluster-cidr=10.44.0.0/16"
        nodeFilters:
          - server:*
      - arg: "--service-cidr=10.45.0.0/16"
        nodeFilters:
          - server:*
Enter fullscreen mode Exit fullscreen mode

Key differences from the source cluster:

  1. Different Ports
    • KubeAPI port: 6444 (source uses 6443)
    • Load balancer port: 8081:80 (source uses 8080:80)
    • CIDR ranges: 10.44.0.0/16 and 10.45.0.0/16
  2. Registry Setup
    • Uses the registry created by source cluster
    • Specified in the use section
  3. Network Settings
    • Uses the same shared network
    • Enables inter-cluster communication

Create the cluster:

k3d cluster create --config target-cluster.yaml
Enter fullscreen mode Exit fullscreen mode

Setting Up external-secrets

Now that our clusters are ready, let's set up external-secrets.

Installing on the Source Cluster 

# Switch to source cluster context
kubectl config use-context k3d-source-cluster

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace
Enter fullscreen mode Exit fullscreen mode

Configuring Target Cluster Authentication

Let's set up the authentication credentials needed for the source cluster to access the target cluster.

Getting Authentication Information

First, get the target cluster's client certificate information:

kubectl config view --raw

# Note down these values:
# - client-certificate-data
# - client-key-data
# - certificate-authority-data
Enter fullscreen mode Exit fullscreen mode

Setting Up Authentication

Create target-cluster-credentials.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: target-cluster-credentials
  namespace: default
type: Opaque
data:
  client-certificate-data: ... # Base64 encoded data from kubeconfig
  client-key-data: ... # Base64 encoded data from kubeconfig
Enter fullscreen mode Exit fullscreen mode

Apply the credentials to the source cluster:

# Switch to source cluster context
kubectl config use-context k3d-source-cluster

# Apply credentials
kubectl apply -f target-cluster-credentials.yaml

# Verify the created Secret
kubectl get secret target-cluster-credentials -o yaml
Enter fullscreen mode Exit fullscreen mode

Setting Up SecretStore

The SecretStore defines the backend store where external-secrets will store and retrieve secrets.

Create secret-store.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: target-cluster
  namespace: default
spec:
  provider:
    kubernetes:
      remoteNamespace: default  # Target cluster namespace
      server:
        url: https://k3d-target-cluster-server-0:6443  # Using k3d internal hostname
        caBundle: ... # certificate-authority-data from kubeconfig
      auth:
        cert:
          clientCert:
            name: target-cluster-credentials
            key: client-certificate-data
          clientKey:
            name: target-cluster-credentials
            key: client-key-data
Enter fullscreen mode Exit fullscreen mode

Apply and verify the SecretStore:

# Apply SecretStore
kubectl apply -f secret-store.yaml

# Check status
kubectl describe secretstore target-cluster

# Expected output should include:
Status:
  Conditions:
    Last Transition Time:  ...
    Message:               SecretStore validated
    Reason:               Valid
    Status:               True
    Type:                 Ready
Enter fullscreen mode Exit fullscreen mode

Setting Up and Testing PushSecret

Creating PushSecret

PushSecret automatically pushes secrets from the source cluster to the target cluster.

Create a Sample Secret (Source Cluster)

# Create sample secret
kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password=supersecret

# Verify created secret
kubectl get secret my-secret -o yaml
Enter fullscreen mode Exit fullscreen mode

Create PushSecret Definition

Create push-secret.yaml:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example
  namespace: default
spec:
  # Replace existing secrets in provider
  updatePolicy: Replace
  # Delete provider secret when PushSecret is deleted
  deletionPolicy: Delete
  # Resync interval
  refreshInterval: 10s
  # SecretStore to push secrets to
  secretStoreRefs:
    - name: target-cluster
      kind: SecretStore
  # Target Secret
  selector:
    secret:
      name: my-secret  # Source cluster Secret name
  data:
    - match:
        secretKey: username  # Source cluster Secret key
        remoteRef:
          remoteKey: my-secret-copy  # Target cluster Secret name
          property: username-copy    # Target cluster Secret key
    - match:
        secretKey: password  # Source cluster Secret key
        remoteRef:
          remoteKey: my-secret-copy  # Target cluster Secret name
          property: password-copy    # Target cluster Secret key
Enter fullscreen mode Exit fullscreen mode

Apply PushSecret:

kubectl apply -f push-secret.yaml

# Check status
kubectl describe pushsecret pushsecret-example
Enter fullscreen mode Exit fullscreen mode

Verifying Operation

Check Secret in Target Cluster

# Switch to target cluster
kubectl config use-context k3d-target-cluster

# Check secret
kubectl describe secret my-secret-copy

# Expected output:
Name:         my-secret-copy
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password-copy:  11 bytes
username-copy:  5 bytes
Enter fullscreen mode Exit fullscreen mode

Verify Secret Contents

# Using kubectl-view-secret plugin
kubectl-view-secret my-secret-copy --all

# Or directly using base64 decode
kubectl get secret my-secret-copy -o jsonpath='{.data.username-copy}' | base64 -d
kubectl get secret my-secret-copy -o jsonpath='{.data.password-copy}' | base64 -d
Enter fullscreen mode Exit fullscreen mode

Expected output:

password-copy='supersecret'
username-copy='admin'
Enter fullscreen mode Exit fullscreen mode

Test Automatic Synchronization

Update the secret in the source cluster and verify it's reflected in the target cluster:

# Switch to source cluster
kubectl config use-context k3d-source-cluster

# Update secret
kubectl create secret generic my-secret \
  --from-literal=username=newadmin \
  --from-literal=password=newsecret \
  --dry-run=client -o yaml | kubectl apply -f -

# Switch to target cluster and verify
kubectl config use-context k3d-target-cluster
kubectl-view-secret my-secret-copy --all
Enter fullscreen mode Exit fullscreen mode

Cleanup 

# Switch to source cluster
kubectl config use-context k3d-source-cluster

# Delete PushSecret
kubectl delete pushsecret pushsecret-example

# Verify secret deletion in target cluster
kubectl config use-context k3d-target-cluster
kubectl get secret my-secret-copy
# Should show: "Error from server (NotFound): secrets "my-secret-copy" not found"

# Delete clusters
k3d cluster delete --config source-cluster.yaml
k3d cluster delete --config target-cluster.yaml

# Delete shared network
docker network rm shared-net
Enter fullscreen mode Exit fullscreen mode

Conclusion

In this article, we've explored how to use external-secrets' PushSecret feature to share secrets between Kubernetes clusters. In production environments, this feature can significantly improve secret management efficiency in multi-cluster setups!

References

Top comments (0)

Read next

itechburner profile image

CCSP Study Guide: How to Prepare for the CCSP Exam

Steve Smith -

lakhera2015 profile image

Introduction to k8sgpt - Simplifying Kubernetes Troubleshooting - Part 1

Prashant Lakhera -

pentest_testing_corp profile image

Preventing XML External Entity (XXE) Injection in Laravel Applications

Pentest Testing Corp -

signadot profile image

Why Quick Fixes Fail: Rethinking Microservices Testing

Signadot -