Over the course of my brief time with Terraform, I've come to appreciate its power and relative ease of use over other Infrastructure as Code (IaC) tools. As an IaC noob, I have had to overcome many challenges that novices have stumbled over.
One such block I stumbled over involved applying the same configuration to different environments, i.e. dev, prod. Under certain conditions, when you apply the same configuration to multiple environments Terraform first destroys the previous resources before applying the changes to the new environment.
This can happen when you use variables as part of a resource name. Terraform sees changes to resource names as a reason to destroy a resource. These types of changes are destructive because you can't modify a resource's name in AWS. Instead, you have to delete and recreate the resource. Terraform is simply following the AWS rules.
In my case, I was naming my resources with the environment name appended to the resource name via a string interpolation template like user_${var.environment} This would give me names like "user_dev" or "user_prod". Terraform picked up the change in name and marked the IAM user resource for destruction when I ran the configuration for different environments.
I came to understand how Terraform stores this information in "state". I learned how Terraform Workspaces could be used to overcome this behavior. In this post I am going to describe what I learned and will demonstrate one method you can leverage using Workspaces to build your infrastructure in AWS for multiple environments.
A State of Destruction
Before diving into Workspaces, it's important to set some background about how Terraform saves configuration state.
Terraform stores data about a configuration in a local file named "terraform.tfstate". Terraform stores data like the values from outputs, dependencies, secrets, and AWS IDs of the resources it creates. As a side note, sensitive data is stored in plain text in the state file. You should protect those values at all costs. I'll do a future post talking about securing state data and state management.
Terraform uses state during subsequent apply operations to compare the previous plan to the current one and determine any deltas. Examining the deltas, Terraform knows whether or not it can update a resource, or if it must destroy the resource in order to apply the change.
When I was applying my configuration to multiple environments using variables, Terraform was capturing all changes to a single state file. So when the user name changed from user_dev to user_prod, Terraform saw that as a change to the user_dev resource and marked it for destruction. In actuality, what I wanted it do was create a new account named user_prod and not modify the existing user_dev account.
What I needed was a way to store state for each environment separately. This would allow me to apply my configuration to each environment independent of the others, and avoid the destruction of resources created in other environments. This is where Workspaces enter the picture.
Enter the Workspace
Terraform uses the concept of a Workspace for isolating state. When you initialize a configuration, Terraform creates a default Workspace named, "default". Configuration plans, and other data, are stored in the Workspace.
You can have multiple named Workspaces in Terraform. Each Workspace maintains its own state. This allows for multiple states to be associated with a single configuration. Knowing this, I set up one Workspace for each environment. Now I could apply the same configuration to different environments without affecting any other environments!
To create and manage a Workspace, use the terraform workspace set of commands as shown in the table below.
| Command | Description |
|---|---|
terraform workspace new <workspace name> |
creates and switches to the newly created workspace |
terraform workspace select <workspace name> |
switches to the workspace |
terraform workspace list |
lists workspaces and highlights the active one |
Let's use the following simple configuration to see Workspaces in action. The configuration below creates a read-only policy for all S3 buckets. It defines one variable named "environment", whose default value is "dev". The variable value is appended to the policy name to create a unique policy per environment.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.27"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "default"
region = "us-east-2"
}
variable "environment" {
type = string
default = "dev"
}
resource "aws_iam_policy" "s3_policy" {
name = "s3_policy_for_${var.environment}"
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : "s3:GetObject",
"Resource" : "*"
}]
})
}
Apply this configuration using the following command:
terraform apply
This creates the policy in AWS and names it "s3_policy_for_dev". Let's run it again, but this time provide a different value for the environment variable, like so:
terraform apply -var "environment=prod"
Our intention is to create a new policy named "s3_policy_for_prod", but as you can see from the command's output, Terraform sees the name change and decides it needs to replace the "s3_policy_for_dev" policy and create a new policy.
Let's resolve this by creating a Workspace for the dev environment. Run the following command to create a Workspace named "dev"
terraform workspace new dev
Reviewing the output you notice Terraform created and switched to a new Workspace named "dev". More importantly the output states, "...Terraform will not see any existing state for this configuration". Our changes to the dev environment will now be stored within the dev Workspace.
Remember what I said earlier about Workspaces having their own state? The dev Workspace is unaware of the previous apply command we executed. If you run the terraform plan command in the dev Workspace, the output will show you that Terraform is going to attempt to create a resource named "s3_policy_for_dev". This will fail upon execution because there is already a resource using that name in AWS.
Using the AWS portal, go ahead and delete the "s3_policy_for_dev" policy from AWS, then run the terraform apply command from the dev Workspace. This will synchronize the Workspace state with the resources in AWS.
Now, let's create the Workspace for our production configuration:
terraform workspace new prod
Create the policy for the production environment:
terraform apply -var "environment=prod"
This time, the Terraform output shows one new resource will be created and no other changes will be made. No resources will be destroyed because the separate state file doesn't know about the resources created in the dev Workspace.
Review your policies in AWS via the portal. You should see two policies, one named "s3_policy_for_dev" and the other named "s3_policy_for_prod".
In addition, if you review your local folder structure where your Terraform files are located, you'll see a folder named terraform.tfstate.d. Expand that folder to find subfolders, one for the dev Workspace and one for prod. Within each subfolder you will find the local state file associated with its parent Workspace.
Issue the terraform workspace list command to show all of the available Workspaces.
The Workspace marked with the asterisk is the currently selected Workspace. Issue the following command to switch back to the dev Workspace:
terraform workspace select dev
Conclusion
Terraform Workspaces help you to navigate issues that may arise from having a single state store. Isolating state within a Workspace provides additional options for multi-environment deployments. I recommend you use Workspaces in scenarios like these or perhaps segmenting your deployments by categories, i.e. networking, security, auditing, etc. Hit me up in the comments and let me know how you apply Workspaces to your configurations...and stay tuned for a post discussing state management.
Top comments (1)
Great article, thanks for sharing your experience.