Fixing Expired Certificates In Kubernetes

By default, when you setup your Kubernetes cluster, the certificates expires after one year.

If it's been a while since you started up your Kubernetes cluster and you try to issue kubectl commands and notice connection refused errors.

The commands used requires 'root' privileged, so if you are login as non-root, you will need to prepend sudo before the commands.

Troubleshooting further, you noticed that your kubelet service is failing to start (systemctl status kubelet).

Upon checking logs related with kubelet (journalctl | grep kubelet), you noticed the belong error messages.

To verify, issue kubeadm certs check-expiration.

Bingo!

Let's manually renew certificates to fix our issues.

If your cluster has more than one control-plane node, be sure to run the following commands on all control-plane nodes in the cluster.

First, let's backup our certificates just in case we need them.

• cp -R /etc/kubernetes/pki /etc/kubernetes/pki.backup

Now let's renew our certificates with

• kubeadm certs renew all

Verify new certificate installed correctly.

• kubeadm certs check-expiration

Restart kubelet service

• systemctl restart kubelet

Once the renew process is complete, we will need to restart all the control plane pods. One way to restart is moving it's manifest file out and wait for for about 20 seconds before moving the file back in to the 'pki' folder. This will will recreate the Pod to use the new certificates.

Copy the administrator certificates

• sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
• sudo chown $(id -u):$(id -g) $HOME/.kube/config

This should get your cluster running again. You can verify with 'kubectl get pods' and not get an error.

Additional Things To Do If The Steps Above Doesn't Work

If for some reason the above doesn't work, you will need to do the following. Make a backup of /etc/kubernetes/pki/

• cp -R /etc/kubernetes/pki /etc/kubernetes/pki-backup

Delete the following files from /etc/kubernetes/pki

• rm apiserver.crt \ apiserver-etcd-client.key\ apiserver-kubelet-client.crt\ front-proxy-ca.crt\ front-proxy-client.crt\ front-proxy-client.key\ front-proxy-ca.key\ apiserver-kubelet-client.key\ apiserver.key\ apiserver-etcd-client.crt

Remove the following all .crt and .key files from /etc/kubernetes/pki/etcd

• rm /etc/kubernetes/pki/etcd/*.crt
• rm /etc/kubernetes/pki/etcd/*.key

Then create certs with

• kubeadm init phase certs all --apiserver-advertise-address <IP> -- substitute your cluster IP.

Backup\Move conf files from /etc/kubernetes

• mkdir conf-backup
• mv admin.conf\ controller-manager.conf\ kubelet.conf\ scheduler.conf /etc/kubernetes/conf-backup/

Create new conf files

• kubeadm init phase kubeconfig all

Finally, restart kubelet service or reboot system.

• systemctl restart kubelet or reboot