Sebastian Korfmann

Posted on Mar 12, 2020 • Edited on May 19, 2020 • Originally published at skorfmann.com

Introducing Terrastack: Polyglot Terraform supercharged by the CDK

#terraform #aws #cdk #typescript

Note: This was published on my personal blog as well

Terrastack enables you to keep using Terraform as engine, while defining your resources in actual programming languages such as Typescript, Python, Java or C# - with more to come (perhaps Ruby?).

This is made possible by the Cloud Development Kit (CDK) and jsii for generating the polyglot libraries. While the major use-case for the CDK is generating Cloudformation configuration as YAML, it's capable of generating pretty much any configuration.

The actual idea of going forward with Terrastack was inspired by yet another project which leverages the CDK / jsii combo to generate Kubernetes configuration: cdk8s. When I saw that project, I eventually realized that the CDK could be useful for more scenarios than just generating Cloudformation YAML.

A first quick prototype was up and running in roughly two evenings. While not all types were properly generated, it was able to generate a simple but working Terraform configuration.

The general workflow is:

Generate a JSON schema for a given Terraform provider (e.g. AWS or Google Cloud) with builtin Terraform command
Use this schema as input for a Typescript generator, to generate classes for each Resource and Data type of that provider
Compile polyglot libraries from these generated Typescript classes via jsii
Define your Cloud resources in your prefered language
Generate Terraform compatible JSON and use it as you would any other HCL based Terraform project

When looking at these steps, that's pretty much what the CDK team does for Cloudformation. Since Terraform and Cloudformation are both declarative, it's conceptually pretty close. This makes Terraform a perfect contender for getting "CDKified".

Similar to the CDK, that's how a simple Terrastack looks like right now

Background

Finishing up an on premise -> AWS migration utilising Terraform, here are the pain points we've hit on a regular basis over the last two years:

Refactoring is hard - Unless you love digging around in JSON state files
Lack of good dependency management
Most of the product teams, don't actually want to learn a new syntax like HCL.
Terraform Modules are a primitive way to share code. Want to dynamically adapt its definition? Good luck with that.
Code Distribution becomes complex in larger organisations
Ensuring certain configuration standards (think of Tagging, Object Storage / VM configurations)
Git Ops with small components to separate state is more complex than it should be
Established Software Engineering practices like unit / integration tests are hard to implement
HCL / Terraform integration in code editors such as VS Code is poor and mostly broke for us with Terraform 0.12

Terrastack tackles all of the above by leveraging existing technologies:

The Javascript ecosystem
Cloud Development Kit (CDK)
jsii for polyglot libraries

The engine of Terraform and its providers are quite good overall. From my point of view, most of the issues are stemming from the rather static and declarative nature of Terraform.

With Terrastack we can start to treat "raw" Terraform configuration as an "assembly" artifact which we can generate in a predictable way. This enables us to tackle most of the pain points above:

Use existing, well established dependency management like NPM or Yarn
Leverage mono repo / build tooling in the respective programming language
Build upon existing testing frameworks for unit and integration tests
Enjoy intellisense of your favourite language in whatever editor you're using right now
Create commonly shared packages of cloud solutions for internal, or even public usage.
Share code which is aimed at the AWS CDK (e.g. Tags, IAM Policies, ECS Task Definitions, Kubernetes config?)

I'm still thinking about the refactoring part. My current conclusion is, that there are mainly two things are getting in the way of proper refactorings:

Static naming of resources
Terraform modules aka namespaces

I strongly believe, that we can come up with a better concept in Terrastack. But that's for another blog post.

Summary

From the initial very rough prototype, it took about a week to release a bit more polished prototype on Github.

Please check it out and give it a try. I would love to get feedback!

TerraStackIO / terrastack

This project is archived, but the idea of Terrastack lives on in the Terraform CDK. - https://github.com/hashicorp/terraform-cdk

This is the first post of a series regarding Terrastack. In the next post, I'll dive a bit deeper into the internals of the CDK itself and what I've learned about its mechanics so far.