Remote State Storage in AWS Using Terraform

Managing cloud infrastructure efficiently is a crucial challenge for developers and DevOps engineers. Terraform, an Infrastructure as Code (IaC) tool, allows us to define and provision infrastructure using code. However, one of the biggest hurdles when using Terraform is state management.

By default, Terraform stores its state in a local file (terraform.tfstate). While this works for small projects, it causes major issues when working in teams, scaling infrastructure, or ensuring security. Issues such as state conflicts, loss of state, security risks, and lack of versioning make local state storage unreliable.

A better approach is to use Remote State Storage with AWS S3 and DynamoDB. S3 provides a centralized, versioned storage location for Terraform state, while DynamoDB enables state locking, preventing multiple users from modifying the infrastructure at the same time.

In this blog, I’ll share my experience of setting up remote state storage in AWS, the problems it solved, and a step-by-step guide to implementing it. 🚀

I am Atri Ghosh, a second year engineering student from Bangalore.
This is my first blog, hope you like it.

Let's begin from the basics.
What is Infrastructure as Code (IaC)?

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure using code instead of manual processes. This approach enables automation, scalability, and consistency in deploying cloud resources. Terraform is one of the most popular IaC tools, allowing developers to define infrastructure using a declarative language.

What is Terraform?

Terraform is an open-source tool developed by HashiCorp that enables the provisioning of infrastructure across multiple cloud providers. It uses a configuration language called HashiCorp Configuration Language (HCL) to define infrastructure as code. With Terraform, you can create, update, and manage infrastructure efficiently.

Understanding terraform.tfstate

The terraform.tfstate file maintains a mapping between Terraform configurations and real-world infrastructure resources.It allows Terraform to track resources that have been created, modified, or deleted.The file is stored in JSON format and contains metadata about managed resources.

How Terraform Uses terraform.tfstate

a. Tracking Resources-When you run terraform apply, Terraform updates the terraform.tfstate file with the latest state of deployed resources.
b. Avoiding Unnecessary Changes-Before applying changes, Terraform compares the current state (from terraform.tfstate) with the desired state (from .tf files).If no changes are required, Terraform does nothing.
c. Managing Dependencies-Terraform knows which resources depend on others and ensures proper order when applying updates.

Where is terraform.tfstate Stored?

• Local State File (Default)
  By default, terraform.tfstate is stored locally in the project directory.
  Example: ./terraform.tfstate

• Remote State Storage (Recommended for Teams)
  For team collaboration and preventing conflicts, Terraform allows storing state remotely, such as in:
  a. AWS S3 (with optional DynamoDB locking)
  b. Terraform Cloud/Enterprise
  c. Google Cloud Storage (GCS)
  d. Azure Blob Storage

Best Practices for terraform.tfstate

• Always use remote state storage for teams to prevent conflicts.
• Use state locking (e.g., AWS DynamoDB) to avoid multiple users modifying the same state.
• Do not edit terraform.tfstate manually unless absolutely necessary.
• Regularly back up terraform.tfstate to prevent accidental loss.
• Use Terraform workspaces for managing multiple environments (e.g., dev, staging, production).

Terraform setup diagram

This document serves as a personal guide, explaining everything in remote state storage and how to use S3 and DynamoDB to accomplish it.

Why Remote State? The Problem with Local State

Terraform tracks resources using state files (terraform.tfstate). By default, it stores this file locally, which creates problems like:

• Collaboration Issues – If multiple engineers work on the same Terraform project, they will have separate local state files.
  This leads to conflicts and inconsistencies because Terraform won’t know what others have changed.
  Solution: Remote state storage ensures that everyone reads from and updates the same state file.

• State Loss – If the local state file is deleted, corrupted, or lost, Terraform loses track of the deployed infrastructure.
  There’s no built-in backup mechanism.
  Solution: Storing state in S3 with versioning allows you to restore previous versions if something goes wrong.

• Security Risks – The state file contains sensitive information, such as:
  AWS access keys
  Database credentials
  IP addresses
  Storing this locally can lead to accidental leaks or unauthorized access.
  Solution: Remote state storage in S3 with encryption ensures the state file is stored securely.

• No Locking Mechanism (Concurrency Issues) - If two engineers apply Terraform changes at the same time, it can lead to race conditions where both modify the same resource.
  Terraform doesn’t have a built-in mechanism to prevent simultaneous updates.
  Solution: DynamoDB state locking prevents multiple people from modifying the Terraform state at once.

Setting Up Remote State Storage in AWS

1.Prerequisites
Before starting, ensure you have:

• An AWS Free Tier account.
• AWS CLI installed (aws configure set up with your credentials).
• Terraform installed (terraform -v).

2.Create an S3 Bucket for Storing State
We’ll create an S3 bucket to store the terraform.tfstate file.

Run the following command:

aws s3 mb s3://my-terraform-state-bucket --region us-east-1

Enable versioning to keep history:

aws s3api put-bucket-versioning --bucket my-terraform-state-bucket --versioning-configuration Status=Enabled

Why versioning? If a state file gets corrupted or deleted, you can restore a previous version.

3.Create a DynamoDB Table for State Locking

DynamoDB prevents multiple people from modifying the state at the same time (called state locking).

aws dynamodb create-table \
    --table-name terraform-state-lock \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST

Why state locking? Prevents race conditions when multiple users apply Terraform changes.

4.Create Terraform Configuration Files
Terraform configuration to use the remote backend.

Clone the Terraform project from the repository:

git clone https://github.com/iam-veeramalla/write_your_first_terraform_project.git
cd write_your_first_terraform_project/aws/remote_state

Modify main.tf and output.tf to replace any sensitive information.

main.tf

terraform {
  backend "s3" {
    bucket         = "your-actual-bucket-name"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
  }
}

terraform {
  required_version = ">= 0.12"
}

provider "aws" {
  region = "us-east-1"
}

data "aws_caller_identity" "current" {}

locals {
  account_id = data.aws_caller_identity.current.account_id
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "${local.account_id}-terraform-states"
  versioning {
    enabled = true
  }
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_dynamodb_table" "terraform_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

output.tf

output "s3_bucket_name" {
  value       = aws_s3_bucket.terraform_state.id
  description = "The NAME of the S3 bucket"
}

output "s3_bucket_arn" {
  value       = aws_s3_bucket.terraform_state.arn
  description = "The ARN of the S3 bucket"
}

output "s3_bucket_region" {
  value       = aws_s3_bucket.terraform_state.region
  description = "The REGION of the S3 bucket"
}

output "dynamodb_table_name" {
  value       = aws_dynamodb_table.terraform_lock.name
  description = "The NAME of the DynamoDB table"
}

output "dynamodb_table_arn" {
  value       = aws_dynamodb_table.terraform_lock.arn
  description = "The ARN of the DynamoDB table"
}

5.Initialize and Apply Terraform
Run:

terraform init

This will connect Terraform to the S3 backend.
Then:

terraform plan
terraform apply

Terraform will store the state file in S3 instead of locally.

Things to remember:

• Reconfigure AWS CLI using aws configure.
• Ensure the AWS access key is still active (delete old keys and create a new one).
• Make sure the DynamoDB table exists and isn’t manually locked.

Cleaning Up (Avoid Charges!)

When you're done, destroy the infrastructure:

terraform destroy

Then delete the S3 bucket and DynamoDB table:

aws s3 rb s3://my-terraform-state-bucket --force
aws dynamodb delete-table --table-name terraform-state-lock

Conclusion

Switching to remote state storage in AWS completely changed how I work with Terraform. No more local state file worries, no more accidental state loss, and seamless collaboration! 🚀.
Would love to hear your thoughts—have you faced any challenges with Terraform state management? Drop a comment! 👇.