Karpenter: The Better Autoscaling Solution for Kubernetes- Part 1

#kubernetes #containers #eks

If you're running Kubernetes, you're likely familiar with the standard cluster autoscaler. While it's a useful tool, it has its limitations. Enter Karpenter, an open-source autoscaling solution that offers many advantages over the standard cluster autoscaler. It is a flexible, high-performance Kubernetes cluster autoscaler built with AWS.

How Karpenter Works

Karpenter takes a different approach to autoscaling than the standard cluster autoscaler. Instead of adding or removing nodes based on demand, Karpenter provisions nodes based on application requirements. This means that it can optimize resource utilization and reduce costs.

Karpenter works by creating custom Kubernetes resources called "provisioners." Provisioners are used to define the resources that Karpenter should provision, such as nodes or virtual machines. When an application needs more resources, Karpenter checks the provisioners to see if any need to be created. If so, Karpenter will create the new resources and add them to the cluster.

Image Courtesy.

Comparing Karpenter to Cluster Autoscaler

Karpenter offers several advantages over the standard cluster autoscaler as detailed below.

Optimal Resource Utilization

One of the most significant advantages of Karpenter is its ability to optimize resource utilization. It can do this by automatically provisioning nodes based on application needs. This means that you can avoid overprovisioning, which can lead to wasted resources and increased costs.

Customizable Scaling

Karpenter offers the ability to customize scaling behaviors based on your specific needs. You can configure scaling based on metrics such as CPU or memory usage, or you can use your own custom metrics.

Cost Savings

Because Karpenter optimizes resource utilization, it can lead to significant cost savings. By avoiding overprovisioning, you can reduce the number of nodes required to run your applications, which can result in lower cloud bills.

Ease of Use

Karpenter is easy to use and deploy. It can be installed using Helm, and it integrates seamlessly with Kubernetes.

The standard cluster autoscaler can be more challenging to set up, and it may require more manual configuration.

How to Get Started with Karpenter

There are different ways to get started with Karpenter. This article will just highlight the steps. (Watch out for part 2 with a step-by-step guide on how to install and configure Karpenter) We will use Helm Chart to install Karpernter. Here are some steps before having it operational:

Create the KarpenterNode IAM Role - Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking.
Create the IAM role for Karpenter Controller - Associate the Kubernetes Service Account and the IAM role using IRSA
Update aws-auth ConfigMap - to allow the nodes that use the KarpenterRole IAM Role to join the cluster
Deploy Karpenter Helm Chart:

helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION}

Create a default Provisioner, (example)

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    intent: apps
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values: [nano, micro, small, medium, large]
  limits:
    resources:
      cpu: 1000
      memory: 1000Gi
  ttlSecondsAfterEmpty: 30
  ttlSecondsUntilExpired: 2592000
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  tags:
    KarpenerProvisionerName: "default"
    NodeType: "karpenter-workshop"
    IntentLabel: "apps"
EOF

Once you've installed Karpenter, you can begin using it to optimize your Kubernetes cluster's resource utilization and reduce costs.

Provisioner configuration

Karpenter configuration comes in the form of a Provisioner CRD (Custom Resource Definition). A single Karpenter provisioner is capable of handling many different pod shapes. You can play with it to suit whatever needs you have. For example

One can limit Karpenter to use either on-demand or spot instances, you can use the spot field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    type: karpenter
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
        # - key: karpenter.sh/capacity-type
    #   operator: In
    #   values: ["spot"]
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["c5.large", "m5.large", "m5.xlarge"]

In this example we're setting the karpenter.sh/capacity-typeto initially limit Karpenter to provisioning On-Demand instances, and karpenter.k8s.aws/instance-typeto limit to specific instance types.

One can also limit Karpenter to specific instance types, regions and zones you can use the instanceTypes field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha1
kind: Provisioner
metadata:
  name: example-provisioner
spec:
  constraints:
    - type: "awsec2"
      region: "us-west-2"
      zones:
        - "a"
      instanceTypes:
        - "t2.micro"

In this example, the instanceTypes field is set to t2.micro, which means that the provisioner will only use t2.micro instances. You can add additional instance types to the list if you want to allow for more flexibility.

Limitations

While Karpenter offers many advantages over the standard cluster autoscaler, it also has some limitations. Some of the key limitations include:

Limited support for custom metrics: While Karpenter does offer support for custom metrics, it is more limited than some other solutions. This can make it challenging to implement certain types of custom scaling behaviors.
Lack of integration with some cloud providers: Karpenter is designed to work with Kubernetes, but it may not integrate seamlessly with all cloud providers. This can make it more challenging to deploy Karpenter in certain environments.
Complexity: Karpenter is a powerful tool, but it can also be complex to configure and use. It may require more expertise and resources than some other scaling solutions.

Despite these limitations, Karpenter is still an excellent choice for many Kubernetes users. Its ability to optimize resource utilization, customize scaling behaviors, and reduce costs make it a compelling option for many organizations.

Conclusion

Karpenter is a powerful autoscaling solution that offers many advantages over the standard cluster autoscaler. With its ability to optimize resource utilization, customized scaling, and ease of use, Karpenter is a must-have tool for Kubernetes users. Follow the steps above to get started with Karpenter today!

Stay tuned for Part 2 of this blog, where we'll provide a step-by-step guide on how to install and configure Karpenter. Part 2 will be more technical, so if you're interested in getting started with Karpenter, be sure to check it out!

References

Follow me at LinkedIn and Twitter.