Kubernetes Private Docker Registry With Distribution Registry

• NFS Server
  • Package Installations
  • Exports
• Persistent Volume Setup
• Certificate Setup
  • Certificate Signing
  • Secret Setup
• Load Balancer
• Ingress
• Registry Basic Setup
  • Ingress Setup
  • Testing
• Conclusion

When working with container images you can get them from a registry or use custom build ones. However, custom built ones need to be in a place where kubernetes recognizes them on all nodes. This is easy if you use something like Docker Hub but another solution will be required for custom built internal images. This article will look at a very simple container registery hosted on Kubernetes and persistent storage provided through NFS.

NFS Server

In order for this container setup to work on Kubernetes, there needs to be a permanent file storage that lives beyond pod lifetimes. This will be achieved through the Network File System (NFS) protocol. Support for this is available in most Linux kernels and the client/server packages are available on most major distributions.

Package Installations

This setup will require both an NFS server to be installed and NFS client tools available on all Kubernetes nodes. For this task I'll use a Raspberry Pi with a 128GB USB storage drive attached to it. First for the server installation:

$ sudo apt-get update
$ sudo apt-get install -y nfs-kernel-server

On non-NFS server nodes I'll install nfs-common so they're able to interact with NFS:

$ sudo apt-get update
$ sudo apt-get install -y nfs-common

Next I'll setup a directory on the USB drive specifically for container storage. I'll also ensure it's owned by nobody:nogroup which is what NFS will run under:

$ sudo mkdir /mnt/usbdrive/containers
$ sudo chown nobody:nogroup /mnt/usbdrive/containers

And finally an /etc/fstab entry:

UUID=d3401f62-06ba-4737-8e19-c35c5a688fdc /mnt/usbdrive ext4 defaults 0 2

The UUID I obtained via:

# blkid
<snip>
/dev/sda1: UUID="d3401f62-06ba-4737-8e19-c35c5a688fdc" BLOCK_SIZE="4096" TYPE="ext4"

This allows for more reliable drive targeting than using the standard /dev nodes.

Exports

Now exports will be needed for exposing directories to consumers of the NFS share. In this case it will be the docker registry container, though it could be used for other purposes as well. Given I'm the only one using the local network I'll allow the local network CIDR access:

/mnt/usbdrive/containers/ 192.168.1.0/24(rw,async,no_subtree_check)

• rw: Just making sure containers can be both read and written
• async: I ended up enabling this for performance purposes
• no_subtree_check: While I don't see there being much renames, I still chose this as an option to be on the safe side stability wise

I'll save the file and restart the server to enable the export changes:

$ sudo systemctl restart nfs-kernel-server
$ sudo systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2023-09-09 00:26:41 BST; 5s ago
    Process: 2318294 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 2318295 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
   Main PID: 2318295 (code=exited, status=0/SUCCESS)
        CPU: 16ms

Persistent Volume Setup

With NFS setup we can integrate it with Kubernetes as a PersistentVolume. The overall setup looks like this:

apiVersion: v1
kind: Namespace
metadata:
  name: private-registry
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-registry
  labels:
    k8s-app: private-registry
spec:
  storageClassName: nfs
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  mountOptions:
    - wsize=32768
    - rsize=3276
    - hard
    - rw
    - nolock
    - nointr
  nfs:
    path: /mnt/usbdrive/containers
    server: 192.168.1.93
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-registry
  namespace: private-registry
  labels:
    k8s-app: private-registry
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

First a namespace is created for our registry components. Then a PersistentVolume will describe the overall storage properties, including NFS connection. Be sure to replace <serverIP here> respectively. There's a number of mount options which are also performance related, as I found docker uploads become really unreliable/hang without them. Values given are mostly taken from a somewhat related GitHub issue. PersistentVolumeClaim takes a portion of the PersistentVolume for use and also ties it to the namespace. In this case it's exclusively going to be used for container purposes so the full capacity is allocated. Now to make a test pod to ensure everything is working properly:

apiVersion: v1
kind: Pod
metadata:
  name: nfs-test
  namespace: private-registry
spec:
  volumes:
    - name: nfs-test-storage
      persistentVolumeClaim:
        claimName: pvc-registry
  containers:
    - name: nfs-test-container
      image: alpine:latest
      command: ['sh', '-c', 'sleep 99d']
      volumeMounts:
        - mountPath: "/tmp/test"
          name: nfs-test-storage

Then apply and test that everything works:

$ kubectl apply -f nfs-test.yaml
pod/nfs-test created
$ kubectl exec -i -t -n private-registry nfs-test -- /bin/sh
/ # ls -lah /tmp/test
total 8K
drwxr-xr-x    2 nobody   nobody      4.0K Sep  8 22:11 .
drwxrwxrwt    1 root     root        4.0K Sep  9 00:37 ..
/ # touch /tmp/test/test.txt

To ensure NFS is completely working I'll check the USB drive backing the NFS server:

$ ls -lah /mnt/usbdrive/containers/
total 8.0K
drwxr-xr-x 2 nobody nogroup 4.0K Sep  9 01:38 .
drwxr-xr-x 4 root   root    4.0K Sep  8 23:11 ..
-rw-r--r-- 1 nobody nogroup    0 Sep  9 01:38 test.txt

Certificate Setup

For this use case I'll have the registry available over HTTPS. As far as the certificates go they can either be self-signed or from a certificate authority. In my case I just happen to have a certificate authority so I'll go ahead and use that to first create the registry certificate.

Certificate Signing

First an openssl configuration:

docker-registry.cnf

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
C = US
ST = Texas
L = Dallas
O = cwprogram
OU = cwprogram Cert Authorization
CN = registry-auth.cwprogram.com

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = registry.cwprogram.com
DNS.2 = registry.private-registry.svc.cluster.local

This will provide two separate DNS entries for the cert. The first is for viewing from the outside (resolved via /etc/hosts) and the other is for when Kubernetes pods will interact with it. The format of service DNS resolution is:

[service-name].[namespace].svc.cluster.local

Now for generation and signing of the cert:

$ openssl ecparam -genkey -name prime256v1 -out docker-registry.pem
$ openssl req -config ./docker-registry.cnf -key docker-registry.pem -new -sha256 -out ./intermediateCA/csr/docker-registry.csr
$ openssl ca -config ./openssl_intermediate.cnf -engine pkcs11 -keyform engine -extensions server_cert -extensions req_ext -extfile docker-registry.cnf -days 375 -notext -md sha256 -in ./intermediateCA/csr/docker-registry.csr -out ./docker-registry.crt

Given that I'm using an intermediate signing key (this step is not necessary for self signed) the final certificate needs to be a bundle of the server and the intermediate certificate:

$ cat docker-registry.crt ./intermediateCA/certs/intermediate.cert.pem > docker-registry.bundle.crt

On a final note you'll also need the Root CA cert or self-signed cert:

• Trusted by all nodes
• Trusted by container system trust store if you plan to build docker images within a container

You can check this page for a nice list of how to trust certs for various operating systems and Linux distributions.

Secret Setup

I'll go ahead and setup a TLS secret for the registry:

$ kubectl create secret tls docker-registry --cert=docker-registry.bundle.crt --key=docker-registry.pem -n private-registry

Kubernetes secrets have a Docker Registry (we'll be using this later) and TLS special storage that make it easy to store special secrets. The TLS one in particular excepts both the cert and key for us making the process easy. Now to verify our secrets:

$ kubectl describe secret -n private-registry
Name:         docker-registry
Namespace:    private-registry
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1779 bytes
tls.key:  302 bytes

kubectl is smart in showing the certificate information so that we know they're present without revealing the underlying secret data. I'll also store my Root CA as well to make it available for configuration later:

$ kubectl create secret generic root-cert --from-file=tls.crt=./cwprogram_root.crt -n private-registry

I'm not using the TLS secret type here because I'd rather not store my Root CA key in a less than ideal security lacking location. Instead I'll create a generic secret that stores the key tls.crt with the contents of my root certificate. I've also bound it to the private-registry namespace.

Load Balancer

One issue to deal with is that we need an IP address to resolve our custom host name to. The main way to handle this would be a LoadBalancer service type. Unfortunately, Kubernetes supported LoadBalancer backends tend to be cloud provider solutions like AWS ELB. Given that my cluster is hosted on bare metal this won't quite work out. To work around this issue I'll be utilizing MetalLB. Before beginning though I'll need to meet some prerequisites regarding kube-proxy:

$ kubectl edit configmap -n kube-system kube-proxy

The main things to change here is ensuring mode: "ipvs" and strictARP: true. Once that's done save and quit the edit session and then do a rolling update on kube-proxy:

$ kubectl rollout restart daemonset kube-proxy -n kube-system

This will ensure it picks up the new changes. Now it's time for the actual metallb installation:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.11/config/manifests/metallb-native.yaml

Immediately after doing this I'm going to get rid of a component called "speaker". This is meant to do some BGP protocol work to assign IPs, but I'm using Calico which handles that for us:

$ kubectl delete ds -n metallb-system speaker

Now the way that this will work is Calico and MetalLB will share some IP information. The main requirement is that the IP addresses we chose do not overlap with DHCP. Looking at my router DHCP starts at 192.168.1.64 and ends at 192.168.1.253. Using the CIDR 192.168.1.0/26 will get me to about that IP allocation wise. For this to work I'll need to adjust Calico's settings. This is based off the work by Edward Cooke's setup article. First I'll get a base YAML to hold the BGPConfig that Calico will use:

bgp-config.yaml

apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  asNumber: 65000
  nodeToNodeMeshEnabled: true
  serviceLoadBalancerIPs:

Then I'll explicitly declare each IP that's part of my CIDR to enable Source IP tracking later on:

for i in $(seq 1 62); do echo "  - cidr: 192.168.1.$i/32"; done >> bgp-config.yaml

With that complete I'll finish the setup:

$ kubectl apply -f bgp-config.yaml
$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 192.168.1.91 | node-to-node mesh | up    | 18:25:58 | Established |
| 192.168.1.93 | node-to-node mesh | up    | 18:25:58 | Established |
+--------------+-------------------+-------+----------+-------------+

I've also verified that the BGP routing with Calico isn't extremely broken by this change. Finally I'll setup the IP pool that MetalLB will use to assign to LoadBalancer type services:

ip-pools.yaml
``yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: service-pool
namespace: metallb-system
spec:
addresses:

• 192.168.1.0/26 `

This is a newer custom resource method that MetalLB uses which deprecates the previous config map and extends across the IPs I declared in my BGPConfig. Given that Calico is handling most everything the layout is pretty simple. All that's left is to apply it:


$ kubectl apply -f ip-pools.yaml


Ingress

In order to setup the registry I'll be utilizing an Ingress. However, out of the box there's nothing handling that ingress. An ingress provider needs to be installed for this purpose. I'll go ahead and use haproxy-ingress to meet this ned. The installation requires Helm, a Kubernetes package manager and templating system. Installation will be done through a dpkg repository:


$ curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
$ sudo apt-get install apt-transport-https --yes
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm


I'll add the repository pointing to haproxy-ingress so helm knows where to pull from:


$ helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts


Making this work with our setup requires a slight adjustment to the configuration, so create the following new file:

haproxy-ingress-values.yaml
yaml
controller:
hostNetwork: true
kind: DaemonSet
ingressClassResource:
enabled: true
service:
externalTrafficPolicy: "Local"
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule


The hostNetwork is the more essential option here as it ensures the resulting pods will be bound to the IPs of each node. This is useful for accessing the registry from local systems outside of kubernetes nodes. It's also setup as a DaemonSet to be on all nodes, with a toleration to include the control plane in that making it even more accessible from the outside. The "Local" external traffic policy keeps source IP information thanks to the serviceLoadBalancerIPs work that was done earlier. Now for the final installation:


$ helm install haproxy-ingress haproxy-ingress/haproxy-ingress \
--create-namespace --namespace ingress-controller \
--version 0.14.5 \
-f haproxy-ingress-values.yaml


After everything is spun up:


$ kubectl --namespace ingress-controller get services haproxy-ingress -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
haproxy-ingress LoadBalancer 10.104.252.216 192.168.1.0 80:30368/TCP,443:31766/TCP 8s app.kubernetes.io/instance=haproxy-ingress,app.kubernetes.io/name=haproxy-ingress


Haproxy ingress is now setup to start acting as an ingress backend for our services.

Registry Basic Setup

The registry itself is fairly simple and bundled as part of distribution. Note that if you're looking for a more scalable solution something like Harbor might be a good fit. First I'll define a deployment and service for the registry:

docker-registry.yaml
`yaml
apiVersion: v1
kind: Service
metadata:
name: registry
namespace: private-registry
spec:
selector:
k8s-app: docker-registry
ports:

• name: https protocol: TCP port: 4343 targetPort: 4343 selector: app: docker-registry type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: name: docker-registry namespace: private-registry labels: app: docker-registry spec: replicas: 3 selector: matchLabels: app: docker-registry strategy: type: Recreate template: metadata: labels: app: docker-registry spec: containers:
  • image: docker.io/registry:2 name: docker-registry env:
    • name: REGISTRY_HTTP_SECRET value: "[redacted]"
    • name: REGISTRY_HTTP_ADDR value: "0.0.0.0:4343"
    • name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY value: "/var/lib/registry"
    • name: REGISTRY_HTTP_TLS_CERTIFICATE value: "/certs/domain.crt"
    • name: REGISTRY_HTTP_TLS_KEY value: "/certs/domain.key"
    • name: REGISTRY_STORAGE_DELETE_ENABLED value: "true" ports:
      • containerPort: 4343 protocol: TCP volumeMounts:
      • name: registry-mount mountPath: "/var/lib/registry"
      • name: docker-certificate subPath: tls.crt mountPath: "/certs/domain.crt"
      • name: docker-certificate subPath: tls.key mountPath: "/certs/domain.key" volumes:
      • name: registry-mount persistentVolumeClaim: claimName: pvc-registry
      • name: docker-certificate secret: secretName: docker-registry `

First there's a Deployment resource with quite a lot happening. The NFS share is mounted to /var/lib/registry which is set as the directory where layer data and other information will go. I've setup 0.0.0.0:4343 as the bind address for the port exposure to work properly. The registry certificate and key are mounted to a path where they can be exposed to the proper environment variables. REGISTRY_HTTP_SECRET is simply a random string I set that is primarily for load balancing purposes (which haproxy is) and REGISTRY_STORAGE_DELETE_ENABLED ensures I can easily clean up images through the registry service. A service is then connected to it exposing the proper ports and of type ClusterIP. I chose this as it makes it easier to interact with inside the pod network. registry is chosen as the service name to match the DNS host on the SSL certificates. Now it's time to apply the changes:


$ kubectl apply -f docker-registry.yaml


Assuming everything is setup properly the pods should show up rather quickly.

Ingress Setup

Now all that's left is to create the ingress:

`yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: docker-registry-ingress
namespace: private-registry
annotations:
ingress.kubernetes.io/secure-backends: "true"
ingress.kubernetes.io/backend-protocol: "HTTPS"
haproxy.org/server-ssl: "true"
haproxy.org/server-ca: "private-registry/root-ca"
spec:
ingressClassName: haproxy
rules:
- host: registry.cwprogram.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: registry
port:
number: 4343
tls:

• secretName: docker-registry hosts:
  • registry.cwprogram.com `

First, ingressClassName is set to haproxy to ensure it's using the haproxy-ingress controllers. This also uses the HTTPS certificate for the registry so external sources see the properly verified one. The root-ca secret that was also made a bit ago is now referenced here. This is because it will be talking to the docker registry backend over an encrypted connection (that's set by backend-protocol and secure-backends) and needs a way to verify the cert. Anything that hits the ingress with the hostname registry.cwprogram.com will be directed to the registry service. Now it's time to apply:


$ kubectl apply -f registry-ingress.yaml


Now I'll put an /etc/hosts in all node's /etc/hosts that points to the haprox-ingress service's IP:


192.168.1.0 registry.cwprogram.com


Testing

It's time to validate with some testing. I'll pull an alpine image with arm64 (since my workers are Raspberry PIs), tag it, then attempt to upload:


$ docker pull --platform linux/arm64 alpine:latest
latest: Pulling from library/alpine
9fda8d8052c6: Pull complete
Digest: sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
$ docker tag alpine:latest registry.cwprogram.com/alpine:latest
$ docker push registry.cwprogram.com/alpine:latest
The push refers to repository [registry.cwprogram.com/alpine]
b2191e2be29d: Pushed
latest: digest: sha256:b312e4b0e2c665d634602411fcb7c2699ba748c36f59324457bc17de485f36f6 size: 528


I can also interact with the web API:


$ curl https://registry.cwprogram.com/v2/_catalog
{"repositories":["alpine","gitea/act_runner","jenkins-python-agent","ubuntu"]}


Now to make sure the actual images are working:

pullpod-test.yaml
yaml
apiVersion: v1
kind: Pod
metadata:
name: pulltest
namespace: private-registry
spec:
containers:
- name: pulltest
image: registry.cwprogram.com/alpine:latest
command: ['sh', '-c', 'sleep 99d']


A simple test run:


$ kubectl apply -f pullpod-test.yaml
pod/pulltest created
$ kubectl exec -n private-registry -i -t pulltest -- /bin/sh
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/aarch64/APKINDEX.tar.gz
v3.18.3-154-g9fca9473248 [https://dl-cdn.alpinelinux.org/alpine/v3.18/main]
v3.18.3-160-gd72f0485282 [https://dl-cdn.alpinelinux.org/alpine/v3.18/community]
OK: 19939 distinct packages available
/ #


So at this point you have a perfectly usable registry if you don't care about authentication. You can also see proper source IPs thanks to the serviceLoadBalancerIPs configuration we did:


192.168.1.60 - - [10/Sep/2023:23:33:46 +0000] "PUT /v2/alpine/manifests/latest HTTP/1.1" 201 0 "" "docker/24.0.5 go/go1.20.3 git-commit/24.0.5-0ubuntu1~22.04.1 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5 \\(linux\\))"
192.168.1.91 - - [10/Sep/2023:23:36:27 +0000] "HEAD /v2/ubuntu/manifests/latest HTTP/1.1" 200 529 "" "containerd/v1.7.3"


Conclusion

This includes a look at getting a private registry for Kubernetes setup using distribution's registry. Originally I thought of including authentication with this, but given that this is geared towards a local LAN setup authentication is not that much of a concern for most I'd imagine. I'll be looking at authentication in the next installment of this series instead. The fun I had with this including:

• Lots of hours spent trying to get MetalLB working just to realize it was because of using Calico for my Kubernetes networking was causing BGP conflicts
• Most of the complexity is because LoadBalancer services are dead simple if you're using some kind of managed service like GKE, forcing you into a BGP nightmare if you're doing bare metal
• Almost thought I lost my Yubikey pin when signing the certs...
• I really REALLY need to get a DNS server going... getting tired of updating host files (or maybe I'll just slap something together with an Ansible playbook)
• Networking used to be simple, now here I am dealing with CIDRs and BGP on my poor home network...
• Honestly I think it's a lot because development is very much steered by people with ridiculously complex network topologies where they actually know the ASN number of their router
• You won't believe how many browser tabs went into this
• NFS was actually not as bad as I thought