JuiceFS CSI: Smooth Upgrades of Mount Pods and Implementation Details

JuiceFS CSI Driver introduced smooth upgrades for Mount Pods in v0.25.0, aimed at minimizing service disruptions during upgrades. The two methods for smooth upgrades are binary upgrades, which update the client binary without recreating the Pod, and Pod recreate upgrades, which rebuild the Mount Pod to allow for configuration updates.

This article will discuss why we introduced smooth upgrades for Mount Pods, how CSI implements smooth upgrades, and how to trigger upgrades through JuiceFS CSI Dashboard or kubectl. These improvements enable more efficient and less disruptive updates to the JuiceFS cluster.

Why we introduced smooth upgrades for Mount Pods

When there’s a need to upgrade Mount Pods within a JuiceFS cluster, the current recommended method is to update the configuration and then re-mount application Pods through a rolling upgrade. However, this requires service restarts, which can be disruptive.

For cases where the usage pattern is well understood, such as when there’s no data being written, you can manually recreate the Mount Pod. This involves updating the configuration, manually deleting the existing Mount Pod, and waiting for it to recreate while relying on CSI’s automatic mount recovery feature to restore the mount point in the application Pod. However, this approach has several challenges:

• Complex operation: Users need to manually find the Mount Pod associated with an application Pod using kubectl.
• Service disruption: CSI waits for the old Mount Pod to fully terminate before creating a new one. This results in a period of unavailability for the service between the deletion of the old Mount Pod and the startup of the new Mount Pod.
• Impact on I/O operations: The automatic mount recovery in CSI relies on rebinding the application mount point. This leads to I/O errors even after recovery.

To address these issues, JuiceFS CSI Driver introduced smooth upgrades for Mount Pods in v0.25.0, allowing upgrades without disrupting services.

Benefits of smooth upgrades over disruptive methods:

• Easy to use: In JuiceFS CSI Dashboard, the associated Mount Pod can be easily located from the application Pod’s page, and a smooth upgrade can be initiated with a button click.
• No service interruptions: Smooth upgrades allow for parameter and configuration modifications of Mount Pods without affecting services.

How CSI implements smooth upgrades for Mount Pods

JuiceFS CSI supports two types of smooth upgrades:

• Binary upgrades
• Pod recreate upgrades

Binary upgrades

Binary upgrades do not recreate the Mount Pod but instead update the client binary within the Mount Pod. This method relies on JuiceFS’ built-in daemon process. The minimum version requirement for this upgrade is 1.2.0 (Community Edition) or 5.0.0 (Enterprise Edition).

The figure below shows the binary upgrade process:

Steps in a binary upgrade:

1. After initiating a smooth upgrade, CSI Node starts a job with the new image and copies the JuiceFS client binary into the Mount Pod.
2. Once the job completes, CSI Node sends a SIGHUP signal to the Mount Pod.
3. After the Mount Pod receives the SIGHUP signal, it saves the current I/O state to a temporary file and exits its service process.
4. After the Mount Pod’s service process exits, the daemon restarts the service process with the new binary.
5. After the new service process starts, it reads the intermediate state file and continues processing the previous requests.
6. The new service process obtains the current FUSE file descriptor (fd) from the daemon.

This binary upgrade applies to situations where only the client needs to be upgraded. However, after the upgrade, the Pod's YAML still shows the old image because the Pod has not been rebuilt. The advantage of this type of upgrade is its speed and low risk, while the disadvantage is that it cannot update other configurations of the Mount Pod.

Pod recreate upgrades

In a Pod recreate upgrade, the Mount Pod is rebuilt for smooth upgrading. This method relies on JuiceFS’ smooth upgrade feature. The minimum version requirement for the Mount Pod is 1.2.1 (Community Edition) or 5.1.0 (Enterprise Edition).

The figure below shows the Pod recreate upgrade workflow:

Steps in a Pod recreate upgrade:

1. After the Mount Pod starts, it sends its FUSE file descriptor to CSI Node, which maintains a relationship table for each Mount Pod’s FUSE file descriptors.
2. When a smooth upgrade is triggered, CSI Node pulls the new image in advance by launching an empty Job with the same configuration as the new Mount Pod.
3. Once the Job completes, CSI Node sends a SIGHUP signal to the old Mount Pod.
4. After the old Mount Pod receives the SIGHUP signal, it saves its I/O state to a temporary file and exits, entering a “complete” state.
5. Based on ConfigMap settings, CSI Node creates the new Mount Pod.
6. The new Mount Pod reads the saved state and resumes processing prior requests.
7. The new Mount Pod retrieves the FUSE file descriptor from CSI Node.

File descriptors are passed between the Mount Pod and CSI Node via a Unix domain socket. If a FUSE request is incomplete during the upgrade, it is forcibly interrupted, so it’s recommended to perform upgrades during low-load periods.

The smooth upgrade process resembles client upgrades on a host server. The main difference is that CSI Node sends the SIGHUP signal to the old service process and obtains the FUSE file descriptor from CSI Node after the new service process starts. This is because after the Mount Pod is rebuilt, the daemon inside cannot send the SIGHUP signal to the old daemon, nor can it pass the file descriptor through the Unix domain socket. Therefore, in a Kubernetes environment, these tasks are handled by CSI Node.

This upgrade method, which involves recreating the Pod, has the disadvantage of a higher risk of errors during the recreating process, especially in complex cluster environments. However, the advantage is that it allows the Mount Pod’s other configurations to be updated, and the Pod's YAML will show the new image.

How to trigger a smooth upgrade

Smooth upgrades can be triggered via the CSI Dashboard or the kubectl plugin.

Trigger via the Dashboard

1. In JuiceFS CSI Dashboard, update the new image version for the Mount Pod via Configuration.

2.In the Mount Pod details page, there are two upgrade buttons. One is for a Pod recreate upgrade and the other one is for a binary upgrade:

• In the Pod recreate upgrade, the Mount Pod is rebuilt, allowing updates to the image, mount parameters, and Pod resources. The minimum required version for the Mount Pod is 1.2.1 (Community Edition) or 5.1.0 (Enterprise Edition).
• In the binary upgrade, the Mount Pod is not rebuilt. Instead, only its binary is upgraded, and no other configurations are changed. After the upgrade, the Pod YAML still displays the original image. The minimum version required for the Mount Pod is 1.2.0 (Community Edition) or 5.0.0 (Enterprise Edition).

3.Click the upgrade button to trigger a smooth upgrade of the Mount Pod.

4.After triggering the upgrade, the entire process is visible. Once completed, the page will automatically redirect to the details page for the new Mount Pod.

Trigger the upgrade with the kubectl plugin

1.Use kubectl to update the image version for the Mount Pod in the CSI ConfigMap configuration.

apiVersion: v1
data:
   config.yaml: |
      mountPodPatch:
         - ceMountImage: juicedata/mount:ce-v1.2.0
           eeMountImage: juicedata/mount:ee-5.1.1-ca439c2
kind: ConfigMap

2.To initiate a smooth upgrade of the Mount Pod, use the JuiceFS kubectl plugin:

# Pod recreate upgrade
kubectl jfs upgrade juicefs-kube-node-1-pvc-52382ebb-f22a-4b7d-a2c6-1aa5ac3b26af-ebngyg --recreate
# Binary upgrade
kubectl jfs upgrade juicefs-kube-node-1-pvc-52382ebb-f22a-4b7d-a2c6-1aa5ac3b26af-ebngyg

Summary

Considering the drawbacks of the existing disruptive upgrade method, JuiceFS CSI Driver introduced smooth upgrades for the Mount Pod in version v0.25.0. CSI now provides two smooth upgrade options: binary upgrades and Pod recreate upgrades.

• Binary upgrades have low risk but do not support updating other Mount Pod configurations.
• Pod recreate upgrades have a risk of failure in complex cluster environments but allow updates to additional Mount Pod configurations. For example, resource allocations can be dynamically adjusted based on the Mount Pod's actual resource usage.

You can choose the most suitable upgrade method based on your needs and are encouraged to perform upgrades during periods of low workload.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.