DEV Community

Max Ivanov
Max Ivanov

Posted on • Edited on • Originally published at maxivanov.io

How to move resources and modules in Terragrunt

#terraform #terragrunt #refactoring #devops

If you're reading this, probably you manage your infrastructure with Terraform.
Or even better, to keep resource definitions DRY you use Terragrunt.

One day you realize one of the Terragrunt modules you maintain became too large.
Plan and apply operations are slow.
It's hard to navigate within the module and it's easy to make a mistake.
You want to refactor the module and extract some resources to a separate module.

Or maybe you decide to change how your modules are organized on the file system.
You want to rename some folders, add or remove nested directories - whatever is required to reflect the infrastructure change.

What happens if you just move a resource between files or a module between folders?
terragrunt plan will report resources to be destroyed in the old location and resources to create in the new location.

Probably not something you want as there are no changes in the managed infrastructure really.
Read below on how to migrate individual resources and/or whole modules in Terragrunt without having to recreate them.

I assume you're using remote state backend but the process is very similar when local state files are used.

Prerequisites

Scenarios below were tested with

  • Terraform v0.14.5
  • Terragrunt version v0.27.1

Terragrunt module == Terraform state file

Each module in Terragrunt represents a group of Terraform resources.
Each module is tracked in its own Terraform state file.

Thus moving resources and modules in Terragrunt boils down to keeping Terraform state files in sync.
When you move entire Terragrunt module, Terraform state file must be moved too (unfortunately it doesn't happen automatically).
When you move resources between modules, source and target state files have to be updated accordingly.

Important: steps below will not create/destroy any of the provisioned resources.
You only alter Terraform state.
If you see changes in the terragrunt plan (except for outputs maybe), something is wrong and you should review the steps that got you there.

Use case: move/rename Terragrunt module folder

Let's say we have an api module definition (Terraform files) in the infrastructure-modules folder.
There's an instance of that module in the infrastructure-live/staging, also named api.

There will be new APIs deployed soon and to not confuse them we want to rename that instance to api-analytics.
If you want to move the module around in the directory hierarchy, the steps will be the same.

Before:

infrastructure-modules/
├── api/
    ├── main.tf

infrastructure-live/
├── staging/
    ├── terragrunt.hcl # shared config
    ├── api/
        ├── terragrunt.hcl # module config
Enter fullscreen mode Exit fullscreen mode

After:

infrastructure-modules/
├── api/
    ├── main.tf

infrastructure-live/
├── staging/
    ├── terragrunt.hcl
    ├── api-analytics/ # module folder renamed
        ├── terragrunt.hcl
Enter fullscreen mode Exit fullscreen mode

1. Backup source state:

# infrastructure-live/staging/api/
terragrunt state pull > /var/app/staging-api-backup.tfstate
Enter fullscreen mode Exit fullscreen mode

Always make a backup and copy it to a safe place before you proceed.
Yes, Terraform creates backup files with every terraform state * command but:
a) It won't backup source state if it's in a remote backend
b) I managed to lose a backup file created by Terraform in an ephemeral Docker container

2. Move/rename the module folder:

# infrastructure-live/staging/
mv api api-analytics
Enter fullscreen mode Exit fullscreen mode

3. Install provider plugins and initialize empty state in the new module location:

# infrastructure-live/staging/api-analytics/
terragrunt init
Enter fullscreen mode Exit fullscreen mode

4. Restore state at new remote location from the backup:

# infrastructure-live/staging/api-analytics/
terragrunt state push /var/app/staging-api-backup.tfstate
Enter fullscreen mode Exit fullscreen mode

5. Make sure there are no changes:

# infrastructure-live/staging/api-analytics/
terragrunt plan
Enter fullscreen mode Exit fullscreen mode

Note the original state file was never removed from the remote backend.
You may go ahead and remove it manually (AFAIK there's no Terraform command to remove the entire remote state file).

Alternatively, instead of pulling the remote state file, you could move the resources from the source state to a local file and restore from that file.

I found this little trick to iterate Terraform resources here.
It will move all resources, one by one, to the target state file:

terragrunt state list 2>/dev/null | xargs -n1 -I{} terragrunt state mv -state-out=/var/app/moved-resources.tfstate {} {}
Enter fullscreen mode Exit fullscreen mode

But then the outputs are still kept in the source state file and it's never removed... So I find it easier to not bother with moving individual resources here.
Migrate the entire state and remove the source state file from the backend manually.

Use case: move individual resources between modules

Let's say we have an api module definition (Terraform files) in the infrastructure-modules folder.
It configures all resources required to run an API: storage, network, database, compute.
It started small but over time became hard to maintain.
There's an instance of that module in the infrastructure-live/staging, also named api.

We want to take a part of that module, e.g. database and extract it to its own module database.
Database depends on the network module which is still in the api module.
We will need to export the network ID in the api module and import it in the database module.

Before:

infrastructure-modules/
├── api/
    ├── main.tf # has way too many resources; we want to refactor it

infrastructure-live/
├── staging/
    ├── terragrunt.hcl # shared config
    ├── api/
        ├── terragrunt.hcl # module config
Enter fullscreen mode Exit fullscreen mode

After:

infrastructure-modules/
├── api/
    ├── main.tf # still holds storage, network, compute resources
    ├── outputs.tf # exports network ID
├── database/
    ├── main.tf # extracted database resources
    ├── variables.tf # to import network ID

infrastructure-live/
├── staging/
    ├── terragrunt.hcl
    ├── api/
        ├── terragrunt.hcl
    ├── database/
        ├── terragrunt.hcl
Enter fullscreen mode Exit fullscreen mode

1. Backup source state:

# infrastructure-live/staging/api/
terragrunt state pull > /var/app/staging-api-backup.tfstate
Enter fullscreen mode Exit fullscreen mode

2. Add outputs to the original module so that extracted module can use them:

# infrastructure-modules/api/outputs.tf
output "network_id" {
  value = vnet.my_vnet.network_id
}
Enter fullscreen mode Exit fullscreen mode 
# infrastructure-live/staging/api/
terragrunt apply
Enter fullscreen mode Exit fullscreen mode

3. Create Terraform module for the database:

Move relevant resources from infrastructure-modules/api/main.tf to infrastructure-modules/database/main.tf

# infrastructure-modules/database/main.tf
resource "database" "my_db" {
  ...
}
Enter fullscreen mode Exit fullscreen mode 
# infrastructure-modules/database/variables.tf
variable "network_id" {
  type = string
  description = "Network ID"
}
Enter fullscreen mode Exit fullscreen mode

4. Create Terragrunt module for the database and declare its dependencies and inputs:

# infrastructure-live/staging/database/terragrunt.hcl
include {
  path = find_in_parent_folders()
}

dependency "api" {
  config_path = "../api"
}

terraform {
  source = "../../modules//database"
}

inputs = {
  network_id = dependency.api.outputs.network_id
}
Enter fullscreen mode Exit fullscreen mode

5. Initialize the database module and create (empty) local state file:

# infrastructure-live/staging/database/
terragrunt init
terragrunt state pull > /var/app/database.tfstate
Enter fullscreen mode Exit fullscreen mode

6. Move resources one by one from api's remote state to database's local state.

Make sure to use absolute paths, otherwise generated state file will end up somewhere in the Terragrunt cache directory:

# infrastructure-live/staging/api/
terragrunt state mv -state-out=/var/app/database.tfstate database.my_db database.my_db
terragrunt state mv -state-out=/var/app/database.tfstate database_firewall_rule.my_rule database_firewall_rule.my_rule
...
Enter fullscreen mode Exit fullscreen mode

7. Push database local state to the remote backend:

# infrastructure-live/staging/database/
terragrunt state push /var/app/database.tfstate
Enter fullscreen mode Exit fullscreen mode

8. Verify there are no changes:

# infrastructure-live/staging/database/
terragrunt plan
Enter fullscreen mode Exit fullscreen mode 
# infrastructure-live/staging/api/
terragrunt plan
Enter fullscreen mode Exit fullscreen mode

If you want to migrate resources to an existing module, steps will be the same.
Make sure to backup target state too in this case!

References:

...

Moving Terraform state is no fun but hopefully this tutorial helps to make it a bit less painful.

Make backups of the source and target state before you start a migration.
Review Terraform plan output carefully when you're done refactoring to make sure there are no unexpected changes.

Top comments (0)

Read next

mike-vincent profile image

IBM Buys HashiCorp's Terraform and Vault: The End of Free?

Mike Vincent -

badlesh_mishra_ profile image

Maximize Your Coding Efficiency with These Sublime Text Plugins 🖥️

Badlesh Mishra -

anshul_kichara profile image

Restoring a Backup Stored in S3 to an EC2 Instance Using XtraBackup

Anshul Kichara -

danial profile image

Mastering Amazon Web Services: Tips for Managing Your AWS Email Address

Danial Ranjha -