Taming FluxCD HelmReleases: The Kustomize Way approach

FluxCD is a powerful tool for managing deployments in Kubernetes using GitOps principles. While it offers a wide range of features, this post will explore scenarios where a simpler approach might be preferable, aligning with the #SimplerIsBetter philosophy.

NOTE: This article shares insights and perspectives gained through experience in various contexts and companies. Feel free to disagree, but with respect! 😉

HelmRelease, what is this?

HelmRelease is a custom resource provided by FluxCD, which provides users
a way to automatically install helm charts using FluxCD and its declarative system.

For example, if you want to install the PodInfo app, you have to declare
the following manifest:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  chart:
    spec:
      chart: podinfo
      version: '6.5.*'
      sourceRef:
        kind: HelmRepository
        name: podinfo
      interval: 5m
  releaseName: podinfo
  values: # part dedicated to the all `values` the chart accept
    replicaCount: 2

NOTE: For the sake of simplicity, I kept only relevant attributes, but I can say FluxCD offers a rich API to cover most of use cases.

The simplified workflow can be described like this:

• The HelmOperator verifies and retrieves the Helm chart from a remote source like GitHub, GitLab, Artifactory or an OCI registry (recommended)
• It then renders the chart using configuration from the HelmRelease object, which can include values provided in different ways (inlined, using ConfigMap or Secret)
• Finally, the HelmOperator applies the generated YAML manifests to the Kubernetes API to install or upgrade the application.

This approach can work for simple deployments, but as an operator, I've encountered several design drawbacks that I'd like to discuss. Let's see these issues!

NOTE Like every GitOps solution, FluxCD requires a connection to a GitRepository too. To keep the diagram simple, I've not materialized this item.

GitOps, aka "only source of truth"

The GitOps philosophy is built on top of 3 key principles:

• Declarative System: You define the desired state of your system (what you want) using declarative language (4GL). This approach focuses on the "what" instead of the "how," making your configuration easier to understand and maintain.
• System State Captured in a Git Repository: The desired state of your system is stored in a Git repository. This provides a central location for managing your infrastructure configurations, enabling version control, collaboration, and easy rollbacks if needed.
• Automatic Deployment system: Any changes pushed to the Git repository trigger an automated deployment process. This automates the process of translating your desired state into actual changes within your system, reducing manual intervention and the risk of errors.

The HelmRelease is a Declarative approach, where automation is managed by the controller. However, "desired state of your system is stored in a Git repository" and all it implies are not respected.

External Chart Dependencies: A Potential Weak Point

A core principle of building resilient systems is ensuring the availability of all their components. When using HelmRelease, we introduce an external dependency, the location where the Helm chart resides.

If communication with this external location fails (due to server downtime, network issues, etc.), you might not be able to install or update your application. This introduces a potential single point of failure (SPOF) in your deployment process. Additionally, the desired state of your cluster is not solely defined by your git repository; it also depends on all the charts your system downloads.

Limited Visibility into Helm Chart Content

Another purpose of this System State Captured in a Git Repository is its auditability. If you capture the complete state of a system in git, you can review it before an installation or upgrade. However, using HelmRelease introduces a layer of opacity.

While you declare the specific chart you want to use, you might not have a complete picture of what resources the chart will actually install in your cluster. It could potentially create various resources like ClusterRoles, NetworkPolicies, or DaemonSets. You can't say without… deploying it, running it locally or worth, reading the chart's source. 😞

Lack of Immutability in Helm Charts!

The problem we already have with container images applied the same way to helm charts. A chart produced and published at a specific date might be un-published or re-published with a different content. In those case, you expose your system to the two previous points again…

NOTE We can now store charts in OCI registries, with immutability feature. To leverage it, you have to complexify your system with digest pinning, because in security, we can't blindly trust a 3rd party 😇.

Chart customisation, another nightmare 👻

While Helm charts offer a convenient way to package deployments, maintaining them can be challenging due to their complexity. Kubernetes itself provides a wide range of configuration options, which can further complicate matters.

This complexity can lead to situations where the desired configuration isn't readily available within a chart. For example, you might want to add an annotation to a workload or modify taints and tolerations, but the chart may not offer built-in ways to do so.

To address this challenge, FluxCD introduced the concept of postRenderers (see documentation within HelmRelease resources. This feature leverages the Kustomize API to customize deployments after the initial Helm chart rendering.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  releaseName: podinfo
  chart: { … }
  values: { … }
  postRenderers:
    - kustomize:
        patches:
          - target:
              version: v1
              kind: Deployment
              name: metrics-server
            patch: |
              - op: add
                path: /metadata/labels/environment
                value: production              
        images:
          - name: docker.io/bitnami/metrics-server
            newName: docker.io/bitnami/metrics-server
            newTag: 0.4.1-debian-10-r54

The configuration for postRenderers is separate from the Helm chart result. This can make it difficult to understand the complete picture of how the final deployment will be configured, potentially leading to hidden errors. 🤯

FluxCD doesn't provide robust mechanisms to analyze the resources created after the combined rendering and post-rendering steps. This can make troubleshooting issues arising from these modifications cumbersome. 😞

Solution is simplicity 🚀!

We've discussed the challenges associated with relying solely on Helm charts within FluxCD deployments. These challenges can compromise the visibility, maintainability, and overall health of your GitOps workflow.

So, how can we achieve the ideal balance: a declarative system, automatic deployments, and a clear picture of your system state captured entirely within your Git repository?

Render the chart and store it into git for better auditability

The answer lies in a simple yet powerful sub-command – helm template (or helmfile template if you use Helmfile). This command allows you to locally render helm charts along with your desired values, generating the final deployment manifest files.

$ helm repo add podinfo https://stefanprodan.github.io/podinfo
"podinfo" has been added to your repositories
$ helm repo update podinfo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "podinfo" chart repository
Update Complete. ⎈Happy Helming!⎈
$ echo "replicaCount: 2" > values.yaml
$ helm template podinfo/podinfo -f values.yaml --version 6.6.3 > podinfo.yaml

And that's it! It was not so complicated 😇. We have a file called podinfo.yaml, located in /k8s/podinfo of our git repository.

This file is now yours, it can be read, analyzed and pushed to kubernetes. After this generation, your system is free from the external chart registry where the chart is located.

NOTE I recommend to never modify a file "generated" by another tool, because you will loose this modification during the next rendering. tldr; treat them as "read-only".

How to deploy generated files with FluxCD?

Instead of using HelmRelease, we're going to use Kustomization resources provided by FluxCD. It is way simpler than HelmRelease, because it just deploys manifests located in a specific location.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
spec:
  sourceRef:
    kind: GitRepository
    name: our-gitops-repository
  path: "/k8s/podinfo" # our location in our GitOps repo
  prune: true
  timeout: 1m

Because we use the GitRepository, called our-gitops-repository and eventually used by FluxCD itself, there is no extra dependency in our system.

Direct Customization with Kustomize

While HelmRelease offers postRenderers for some customizations, the Kustomization resource provides full access to the powerful Kustomize capabilities.
This allows for more granular and flexible control over your manifests. Here's how to achieve a similar customization as the previous postRenderer example using a kustomization.yaml file:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - podinfo.yaml # your generated file from the previous step

patches:
- target:
    version: v1
    kind: Deployment
    name: metrics-server
  patch: |
    - op: add
      path: /metadata/labels/environment
      value: production              

images:
- name: docker.io/bitnami/metrics-server
  newName: docker.io/bitnami/metrics-server
  newTag: 0.4.1-debian-10-r54

kustomize offers a wider range of functionalities compared to postRenderers. You can manipulate resources using features like namePrefix, labels, replacements, components… The list is too long to be detailed here 😇.

As a bonus point, you can run kustomize build /k8s/podinfo/ and see the complete result of the generation before any interaction with FluxCD.

Enjoy reviews and audit with rich diff!

One of the significant advantages of managing your full Kubernetes state with GitOps is the ability to leverage Git's powerful version control capabilities for reviewing and auditing deployments.

From an operator perspective, there is nothing better than a clear and detailed diff views during tool upgrade:

Obviously, your IDE will be your best friend to understand what happened, with clear context and details of changes:

NOTE Upgrade can be automated using tools like renovate or dependabot

Conclusion

While Helm charts offer a convenient way to package deployments, maintaining them in a FluxCD workflow can introduce challenges related to transparency, maintainability, and control.

This article explored the limitations of HelmReleases and presented helm template … as a more powerful and flexible alternative, leveraging FluxCD Kustomize resource. Using Kustomize directly within your git repository, you gain greater control, visibility, and the benefits of Git version control for reviewing and auditing changes.

Ultimately, by adopting a Kustomize-based approach within your FluxCD workflows, you can achieve a more declarative, transparent, and auditable approach to managing your Kubernetes deployments.

Comments

[Jan Christoph Ebersbach]

Thank you for the article. I implemented and used is approach for a while for the very same reasons you brought forward. However, I concluded that too much custom tooling is required to make it work. Everyone using the gitops configuration also has to be aware of the mandatory rerender. Furthermore, deploying charts in different clusters using kustomize overlays suddenly becomes tricky as multiple rendered versions of the same chart need to be kept around.

I ended up going back to the drawing board and created instead tooling that would allow me to render the helm chart from HelmRelease and HelmRepository files using the exact configuration I have in my git repository. The tooling helps with linting and also makes debugging and comparing versions relatively easy.

The main thing I wish helm would do is combine repository and chart reference in one URL that also includes a hash of the chart, e.g. https://charts.example.org/index.yaml?chart=test&version=1.2.3&hash=abcdefg. This would simplify the installation - no more fiddling with repositories - and it would provide integrity guarantees so I don't have to worry about people manipulating chart data anymore. Until this level of simplicity is achieved, I do hope that helm will add an integrity check so that chart data can be verified.

[Jan Christoph Ebersbach]

There's one more thing: I'd love fluxcd to natively support kustomize's HelmChartInflationGenerator kubectl.docs.kubernetes.io/referen...
This would also make it easy to locally render and lint the chart. Furthermore, the chart data is being downloaded and cached by kustomize in the git repository, so external manipulation of charts becomes more difficult.

Here's the link to the corresponding github issue: github.com/fluxcd/kustomize-contro...

[Alec Hothan]

Good presentation about templating helm charts.
We’ve been using flux and helmrelease quite a lot in past few years, and did not feel any of the inconveniences described here were a problem for us, and we do upgrade helm charts quite frequently.
This is an interesting idea but here are some of the potential limitations with this templates approach…
The resulting rendered files can be quite large and complex and diffs across consecutive version of helm chart can be difficult to apprehend.
Helm chart values often provide a much needed simplified and higher level view of the configuration of the helm chart, and complex deployment use configuration variables (flux substitution variables) to control the behavior of a helm chart through the values section of a helmrelease. This can be extremely difficult to achieve with pre-templates files. Simple example is an enable field in values that includes conditionally certain manifest (eg a configmap or a service …) can no longer provide that flexibility after templating (at this time it is either enabled or disabled)