kubernets related logs & configurations

1. Logs

1. Components level logs
   • kubelet, kube-apiserver, kube-scheduler, kube-controller, etcd, kube-proxy ...
2. Pod level logs
   • generate by container

1.1 Log files

The base path for logs is /var/log/

Component	Log Path	Description
kubelet	/var/log/syslog	Kubelet logs if not using journald.
kube-apiserver	/var/log/kube-apiserver.log	API server logs (depends on setup).
kube-scheduler	/var/log/kube-scheduler.log	Scheduler logs (depends on setup).
kube-controller-manager	/var/log/kube-controller-manager.log	Controller manager logs.
Pod logs	/var/log/pods/ /var/log/containers/	Logs for all containers running on the node. Docker: /var/lib/docker/containers/<container-id>/<container-id>-json.log and containerd: /var/log/pods/<namespace>/<pod-name>/<container-name>.log
etcd	/var/log/etcd.log	etcd logs, if configured

1.2 CMD

kubelet logs:

grep kubelet /var/log/syslog
journalctl -u [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
# use describe/logs
kubectl describe pod <pod-name> -n kube-system
kubectl logs <pod-name> -c <container-name>
kubectl logs <pod-name> -n kube-system
# use crictl or docker (in case when docker is used) to check the logs and process
crictl ps
crictl logs
docker ps
docker logs

2. Config

2.1 Config Files

The basic path for kubernetes config file store in /etc/kubernetes/ folder, for kubelet normaly
stores at /var/lib/kubelet/config.yaml
Default Paths:

• /etc/kubernetes/: Central directory for most Kubernetes configuration files.
• /var/lib/kubelet/: Directory for kubelet-related data and configuration.
• /var/lib/etcd/: Directory for etcd data.
• /etc/systemd/system/: Directory for systemd service configuration files.

Component	Config Path	Description	Params
kubelet	/var/lib/kubelet/config.yaml	Main kubelet configuration file.
	/etc/kubernetes/kubelet.conf	Kubeconfig for API server communication.	--config: Points to the kubelet's configuration file (e.g., /var/lib/kubelet/config.yaml). --pod-manifest-path: Specifies the directory for static pod manifests (default: /etc/kubernetes/manifests). --kubeconfig: Path to the kubelet's kubeconfig file for API server communication (default: /etc/kubernetes/kubelet.conf).
kube-apiserver	/etc/kubernetes/manifests/kube-apiserver.yaml	Static pod manifest for API server.	--etcd-servers: Specifies the etcd endpoints. --secure-port: Port for secure communication (default: 6443). --service-cluster-ip-range: Range of IPs for cluster services. --tls-cert-file and --tls-private-key-file: Certificates for secure
kube-scheduler	/etc/kubernetes/manifests/kube-scheduler.yaml	Static pod manifest for scheduler.	--kubeconfig: Path to the kubeconfig file for the scheduler. --leader-elect: Ensures only one active scheduler in HA setups.
kube-controller-manager	/etc/kubernetes/manifests/kube-controller-manager.yaml	Static pod manifest.	--kubeconfig: Path to the kubeconfig file for the controller manager. --service-account-private-key-file: Key for signing service account tokens. --cluster-signing-cert-file and --cluster-signing-key-file: Certificates for signing.
kubeconfig files	/etc/kubernetes/admin.conf	Admin kubeconfig for kubectl.
	/etc/kubernetes/controller-manager.conf	Kubeconfig for controller manager.
	/etc/kubernetes/scheduler.conf	Kubeconfig for scheduler.
CNI plugins	/etc/cni/net.d/	Directory for CNI plugin configurations.
Static pods	/etc/kubernetes/manifests/	Directory for static pod definitions.
etcd	/etc/kubernetes/manifests/etcd.yaml	etcd configuration.	--data-dir: Directory for storing etcd data. --advertise-client-urls: URLs advertised to clients. --listen-client-urls: URLs where etcd listens for client requests.
kube-proxy	/var/lib/kube-proxy/config.conf	Often referenced in the kube-proxy pod or daemonset	--cluster-cidr: Range of IPs for the cluster network. --kubeconfig: Path to the kubeconfig file for API server communication.

2.1.1 /etc/kubernetes/manifests/ file

As kube-controller always monitor the manifests folder files, so we can directly edit in manifests file. But it's better to backup first.

## always make a backup
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.bk
## make the change
vim /etc/kubernetes/manifests/kube-apiserver.yaml
## wait till container restarts
watch crictl ps
## check for apiserver pod
k -n kube-system get pod

2.2 CMD

# View the active kubeconfig settings.
kubectl config view
# check the corresponding pods
kubectl describe pod <pod-name> -n kube-system
# use systemctl cat
systemctl cat [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
# can even use find
find / | grep kubeadm

By understanding these logs, configurations and their purposes, we can better manage and troubleshoot our Kubernetes cluster.