In the evolving world of infrastructure-as-code (IaC), tools like OpenTofu are pushing boundaries, enabling developers to efficiently manage and deploy infrastructure. The OpenTofu team has been on a roll with new features to address some of the longest running complaints in the Terraform community.
Two recent standout features are encrypted state and provider iteration. Both are intriguing and deserve a closer examination to understand their potential impact (and limitations) in real-world scenarios.
In this article I'll show how to maintain infrastructure bootstrap code and its state in git without the need for a third party vault, cloud storage, or additional secret sprawl. I lay out fully working examples of how this might be done both with standard Terraform and also via OpenTofu's encrypted state feature.
The example project I'll be covering deploys a couple of local kind clusters with ArgoCD installed. It then creates and pushes a ssh public/private key pair as Kubernetes local secrets.
Working Example - Part 1 (Terraform)
To explore this further I'll start with a deployment done entirely via Terraform.
NOTE To follow along with things you should clone this repo locally and run in any local bash/zsh shell with docker running. Further configuration information can be found in the project readme.
PROJECT: tofu-exploration
BRANCH: main
The main branch includes manifests for deploying 2 kind clusters side by side in the infrastructure/environments/local folder. The state is stored for each component as separate Terraform state files in the ./secrets folder. This folder is then targeted with sops to encrypt contents within.
# Bring cluster1 and cluster2 up
task deploy:all
# Here you should review secrets and other state stuff in ./secrets.
# Don't commit this to git yet!
After this has completed you should have a handful of files in the local ./secrets folder including:
- Kubernetes configuration files with full rights to your created clusters
- Additional per-cluster public and private keys
- Infrastructure and per-cluster state files with all applied Terraform (including the generated ssh private keys and other sensitive information)
Encrypting Local State
Both plan and state files are inherently plain text. We can encrypt the state files easily enough though. To start you will need some private key that is kept locally. I've chosen age keys with sops. You could use PGP or anything that sops supports.
task | grep sops # Show a list of our convenience tasks
task sops:show # Show all the variables setup for the tasks
task sops:age:keygen # Generate a local age key
task sops:init # Initialize this project repo with your public age key
task encrypt:all # Encrypt every file in the ./secrets folder
You can now review the secrets files and see that they have all been encrypted. Binary looking files like ssh keys will be converted to JSON format with the information required to decrypt them baked into the metadata (obviously minus our private age key).
With the age private key in ~/.config/sops/age/keys.txt and all secrets files are encrypted you can now safely commit your changes to git.
When you need to decrypt and run terraform operations again:
task decrypt:all
NOTE You can and should use pre-commit hooks to prevent accidentally committing your secrets!
Clean Up
To remove the clusters and clean up your work in preparation for opentofu run this:
# Tear it down
task destroy:all
task clean
Working Example - Part 2 (OpenTofu)
I created, then updated the tofu-encryption branch from main.
PROJECT: tofu-exploration
BRANCH: tofu-encryption
This is the same deployment is done using opentofu's encrypted state instead of sops. First big update is that we are changing the binary used in our main Taskfile.yml definition to tofu.
NOTE I did try to use the VSCode plugin for OpenTofu but it was not very helpful for the more recent features (like the encryption block).
State/Plan Encryption
As per the docs we can encrypt state and plan data with native opentofu.
This can be enabled via the TF_ENCRYPTION environment variable or in the terraform block. The way this works is that you define a method which can optionally contain key providers or other configuration for encryption. The key providers and methods available are not so large currently but it is still enough to get along.
Vault Transit Support is not available if vault is running beyond 1.14 (the license change). It is experimental for openbao otherwise.
Anyway, the methods are assigned to the state and/or plan terraform definitions as either the primary or backup encryption types.
You can infer that your entry point for secret zero in a local file based state encryption will be that passphrase. We need to use something greater than 16 characters and private. The age private key can be used for this easily enough by setting the TF_VAR_state_passphrase variable I created just for this purpose.
Important! Ensure you have your local age key pair created with task sops:age:keygen (existing key will always be preserved).
With this in place I updated the local Taskfile.yml manifest to automatically source the private key value into that environment variable so it could be used as the encryption passkey in the relevant terraform block. The result is something like this:
variable "state_passphrase" {
type = string
description = "value of the passphrase used to encrypt the state file"
validation {
condition = length(var.state_passphrase) >= 16
error_message = "The passphrase must be at least 16 characters long."
}
}
terraform {
required_version = ">= 1.9.0"
encryption {
## Step 1: Add the desired key provider:
key_provider "pbkdf2" "mykey" {
passphrase = var.state_passphrase
}
## Step 2: Set up your encryption method:
method "aes_gcm" "passphrase" {
keys = key_provider.pbkdf2.mykey
}
method "unencrypted" "insecure" {}
state {
# enforced = true
method = method.aes_gcm.passphrase
fallback {
method = method.unencrypted.insecure
}
}
plan {
# enforced = true
method = method.aes_gcm.passphrase
fallback {
method = method.unencrypted.insecure
}
}
}
required_providers {
kind = {
source = "tehcyx/kind"
version = "0.7.0"
}
}
backend "local" {
path = "../../../secrets/local/infrastructure_tfstate.json"
}
}
If we run the deployment with no further changes then it automatically encrypts the terraform state files when we deploy via task deploy:all.
The SSH keys I was generating and encrypting via sops before are not covered in this case. But that data is sourced from our state so we simply start ignoring them via .gitignore knowing we can always recreate them later.
Interesting: Because the kind provider I used doesn't track the local config file resource when it gets created, I needed to make changes to isolate the kubeconfig files to their own generated file resources instead.
With this in place we should be able to push state up to your git repo directly after any kind of state altering task has been done, clone it later to another machine with the same age private key, and run through the deployment lifecycle again seamlessly.
Impressions
I'm really happy with how fluid encrypted state works and will definitely be using it for some personal projects. Remember to keep all your secrets in state when doing this, be extra careful of what you commit, and of course protect/backup that private age key.
Top comments (0)