Making Kubernetes Operations Easy with kubectl Plugins

kubectl is powerful tool that allows you to perform literally any and all Kubernetes-related tasks. Regardless of whether you need to just list pods, debug node, manage RBAC or whatever else, kubectl can do it. Some of these common tasks however, can be quite clunky or might include many steps that might take considerable time to perform. In other cases the output of kubectl might not be exactly readable or might include lot of noise, which might be very annoying, especially if you're for example trying to debug something, in which case time is of essence. So, why waste time on repetitive, common, time consuming tasks when we can avoid it. Asking how? Well, let me introduce you to kubectl plugins!

What Plugins?

kubectl ships with limited set of core features which don't cover all the tasks that Kubernetes admins or users might need to perform. Therefore, to resolve this limitation, we're able to extend kubectl with plugins, which function as subcommands of kubectl itself. All of these plugins are standalone executables that can be written in any language, but considering that we're talking about Kubernetes tooling and ecosystem, most of them are obviously written in Go.

Now you might be thinking - "Where do I find all these plugins? And why not just use the standalone binaries without kubectl?" - Answer to both of these question is krew - a package manager for kubectl plugins as well as Kubernetes SIG that aims at solving the package management issue for kubectl.

krew being a package manager, helps with discovering, installing and updating all of our plugins, but to use it, we first need to install it, because well... krew itself is also a plugin. You can navigate to installation guide/script here to install it using your preferred method.

Now that we have krew, let's find and install some plugins!

~ $ kubectl krew search
NAME                            DESCRIPTION                                         INSTALLED
access-matrix                   Show an RBAC access matrix for server resources     no
advise-psp                      Suggests PodSecurityPolicies for cluster.           no
allctx                          Run commands on contexts in your kubeconfig         no
apparmor-manager                Manage AppArmor profiles for cluster.               no
...
~ $ kubectl krew search rbac-lookup
NAME         DESCRIPTION              INSTALLED
rbac-lookup  Reverse lookup for RBAC  no

~ $ kubectl krew info rbac-lookup
NAME: rbac-lookup
INDEX: default
URI: https://github.com/reactiveops/rbac-lookup/releases/download/v0.6.4/rbac-lookup_0.6.4_Linux_x86_64.tar.gz
SHA256: 9f6f63b2ee6f5420530b6aa589b4c8c4a4685bf87447286881d37afdd80a7fb2
VERSION: v0.6.4
HOMEPAGE: https://github.com/reactiveops/rbac-lookup
DESCRIPTION: 
Easily find roles and cluster roles attached to any user, service account, or group name in your Kubernetes cluster.

The above code show a couple of ways to search and get info about particular plugin(s). Besides searching using kubectl krew you can also use plugin index on krews website here. In addition to the information shown by kubectl krew, this also gives you link to source repository and number of GitHub stars for each plugin. So, when you find what you needed, you can just run kubectl krew install and start using it:

~ $ kubectl krew install rbac-lookup
Updated the local copy of plugin index.
Installing plugin: rbac-lookup
Installed plugin: rbac-lookup
\
 | Use this plugin:
 |  kubectl rbac-lookup
 | Documentation:
 |  https://github.com/reactiveops/rbac-lookup
/
WARNING: You installed plugin "rbac-lookup" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

~ $ kubectl rbac-lookup version
Version:0.6.4 Commit:3c86486a94e56e762b2a46d899e5f7b235c74ca8

Notice the warning in the above output - even though these plugins are listed in official plugin index, that does not guarantee that they are safe to use or that they actually do what they claim to be doing. You should treat all of these as any random, unverified script downloaded from internet.

Even though krew contains a lot of plugins, that does not mean that it's exhaustive list of every plugin available. So, in case you can't find plugin that solves your task/issue, you can also check other places. One such place would be awesome-kubectl-plugins repository, which has a couple extra plugins, or you can also just try Googling around.

Considering that these ones are not part of krew, to install them we would need to take manual approach which goes as follows:

~ $ kubectl krew search dig
# ... nothing relevant

~ $ git clone https://github.com/sysdiglabs/kubectl-dig.git && cd kubectl-dig
~ $ make build
~ $ cp _output/bin/kubectl-dig /home/martin/.krew/bin/kubectl-dig

~ $ kubectl dig
Deep kubernetes visibility.

Usage:
  dig
  dig [command]
...

As was mentioned earlier, these plugins are just scripts or binaries, therefore you can download them manually and just use them. If you want kubectl to recognize them as plugins, you additionally need to give them a name in format kubectl-plugin-name and put them somewhere in path. In the above example we installed dig plugin by downloading its sources, building the binary and moving it to krew directory which is in path. To check if kubectl found the newly installed plugin you can run kubectl plugin list.

Must Have's

There are quite a few plugins in the index (149 at the time of writing) and there's many more outside of the krew index, so to save you some time going through all of them, I made a list of the plugins that are in my opinion particularly useful. So, let's break it down by category starting with the most neglected area - the security:

• rakkess - known as access-matrix in krew is plugin for showing and reviewing access to kubernetes resources. This can be very useful when designing RBAC roles - you can for example run kubectl access-matrix --as other-user --namespace some-ns to verify that user or service account has desired access rights in specified namespace.

• kubesec - known as kubesec-scan in krew is plugin for scanning resources with https://kubesec.io/ scanner. When you run this plugin against your manifests, it will tell you recommended changes to improve security of your workloads. To view all rules that the scanner uses visit the above website.

• rbac-lookup - Similar to the first plugin we mentioned, this plugin also helps with RBAC in your cluster. This can be used to perform reverse lookup of roles, giving you a list of roles that user, service account or group has assigned. For example, to find roles bound to service account named my-sa you use the following - kubectl rbac-lookup my-sa --kind serviceaccount --output wide.

When debugging some critical issue, there's really no time to waste and there are some debugging plugins that can help speed up the process:

• ksniff - known as sniff is a tool for debugging and capturing networking data. It's able to attach to a pod and using tcpdump to forward networking data to your local Wireshark. This tool also works pretty well with tshark - the command-line version of Wireshark.

• dig - This plugin built by SysDig provides very nice terminal interface for exploring all kinds of node-level data - for example - ports, traces, running pods, page faults, etc. To see proper demo, checkout the video in dig repository here. This plugin however is not in krew and might also require some extra setup on your cluster nodes (see this issue).

There are also useful plugins that can help with day-to-day management of cluster and its resources:

• neat - possibly my favourite of all the plugins is neat which removes all the generated, redundant fields from YAML output of Kubernetes resources. If you're tired of scrolling through all the managedFields and other garbage, then definitely give this one a try.
• kube-capacity - known as resource-capacity in krew tries to provide better insight into cluster resource usage and utilization. It's essentially a kubectl top on steroids. It can show you resource utilization and consumption per namespace or pods, allows for node or pod label filtering, as well as sorting of output.
• kube-pug - is a plugin known as deprecations in krew. Every cluster needs to be upgraded sooner or later and at some point you will run into API deprecations and/or removals. Finding what's being deprecated can be long and error-prone process and this plugin tries to simplify that. All you need to do is run kubectl deprecations --k8s-version=v1.XX.X and you will get list of all the instances of API objects in cluster that will be deprecated or removed in the specified version.

Last and also the biggest category is power tools - there's a lot of tasks that can be complicated, cumbersome or require multiple repetitive steps to perform with vanilla kubectl, so let's make some of those simpler with these plugins:

• tree - Creating a single object in Kubernetes can trigger creation of many more dependent resources, whether it's just Deployment creating ReplicaSets or instance of an operator creating 20 different objects. This hierarchy can be difficult to navigate and kubectl tree can help with that by creating filesystem-like tree visualization of dependant resources.
• kubelogin - If you're using OIDC provider such as Google, Keycloak or Dex for authenticating to Kubernetes cluster, then this plugin also known as oidc-login in krew can help you avoid having to manually login into your cluster over and over again. When you setup this plugin, every time you attempt to run any kubectl command without having valid authentication token, oidc-login will automatically open your provider's login page and after successful authentication grabs the token and logs you into the cluster. To see video of workflow check out the repository here.
• kubectx - known as ctx in krew is probably the most popular of all the plugins. It allows you to easily switch between kubectl context and cluster namespaces with without having to deal with kubectl config.
• ketall - We all know that kubectl get all doesn't really give you all the resources. To really list all resources you can use ketall also known as get-all in krew. This plugin can just dump all the resources into your terminal as well as filter based on time, exclusions, label selectors or scopes (cluster or namespace).

Closing Thoughts

This is just a list of things that I find useful, so what works for me might not work for you and at the same time, there might a lot of plugins that I omitted, yet they can be super useful for you. So, go check out the krew index or awesome-kubectl-plugins repository for more. If you happen to find something cool, please share it, so others can benefit from it too.

With that said, there isn't a plugin for every use case, so if you can't find plugin that solves your issue, maybe you can build one and fill that gap (more info about that in docs). 😉

There are also other tools out there beyond kubectl plugins, which can improve your productivity and simplify Kubernetes operations. The most prominent one is k9s, so if plugins aren't enough and you want to grab a bigger hammer, then this might be a right tool for you.