Integrating Infracost with Azure DevOps to Improve Terraform Cost Predictability

According to a recent report from GMinsights, more and more enterprises and government institutions are moving to the cloud and adopting automation technologies like Continuous Integration (CI) and Continuous Deployment (CD) to make the Software Development Life Cycle more agile. This shift is driving a surge in the Infrastructure as Code (IaC) methodology, as it enables the management of cloud-based services using human-readable files, making the automated workflow easier to manage. Consequently, the Infrastructure as Code (IaC) market size in the Asia-Pacific-Japan region is projected to witness a growth rate of nearly 28% through 2030.

Source: https://www.gminsights.com

However, with great power comes great responsibility, and cloud costs can escalate rapidly. Therefore, it’s critical to have visibility into the expense that we are going to face upfront, which can prevent unexpected and potentially significant charges. This is where Infracost comes into play. In this article, I will explain what Infracost is, how it works, and how you can integrate it with your Azure DevOps and GitHub pipelines to get a clear and readable cost report in each Pull Request targeting the deployment branch.

What’s Infracost?

Infracost is an open-source tool developed using the Go language that creates estimates about the cost of cloud infrastructure before it’s deployed. It supports IaC tools like Terraform and Pulumi and is compatible with major cloud service providers AWS, Google Cloud, and Azure.

Pricing

Infracost offers three pricing tiers:

• Free tier: This is targeted at engineers, offering basic features like cost breakdowns, CI/CD integrations, and up to 1,000 runs per month.
• Cloud tier: This costs $50 per license per month and adds features like dashboards, automatic budget checks, and team management.
• Infracost Enterprise: This is designed for larger enterprises with complex setups and extends the features of the Cloud tier, adding tools like GitHub Enterprise Server & GitLab Enterprise apps and dedicated enterprise support.

How does it work?

Infracost uses the directory containing the IaC files, or the provisioning plan, to identify the cloud resources that you’re planning to provision or modify. It then queries its Cloud Pricing API for up-to-date pricing information for those resources and calculates the cost of each resource based on its price and quantity. Finally, Infracost aggregates the costs and generates a report that provides a detailed breakdown of the estimated costs. This report can be output in various formats, such as a table in the terminal, a comment in a PR.

Self-Hosted Cloud Pricing API

By default, the Infracost CLI is configured to connect with the Cloud Pricing API hosted by Infracost for fetching pricing information. However, there might be situations where organizations prefer not to grant external access due to various considerations, such as security or policy restrictions. To continue using the service, Infracost allows you to host the Cloud Pricing API within your own environment and then update the endpoint to target your own hosted instance of the Cloud Pricing API instead of https://pricing.api.infracost.io. However, to keep the pricing information up-to-date, you must periodically download the database dump containing pricing details from Infracost’s API and update your local PostgreSQL database.

Configure Your Azure DevOps Pull Request to run Infracost as Build Validation

Let’s imagine that our organization is using the GitHub flow as branching strategy, meaning we will have the following branches:

• main
• features/xxxxx The main branch won’t accept direct commits and will require developers to open Pull Requests before being able to merge their code with the main branch. To enforce this, we can use Azure DevOps branch protection policies.

Install the Infracost Tasks extension

Infracost has developed a set of Azure DevOps tasks that have mapped the CLI instructions and will make the configuration and usage of the Infracost CLI easier. To utilize these tasks, we will need to install the Infracost extension.

• Browse to your organization settings and click Browse Marketplace.

• Search for Infracost Tasks, and then click Get it free.

Finally, select your organization and click the Install button.

Store Your Infracost API in Azure DevOps as Secret

To run Infracost, you’ll need to configure the Infracost instance running on the Azure DevOps agent with the API key related to your Infracost account. However, because this is sensitive information, we need to mark it as a secret so its value won’t be visible to anyone.

• Browse and log in to Infracost.
• Click Org Settings and then copy the API key.

• Browse to your Azure DevOps project, select Library from the Pipelines section, and click the Variable group option.

Create a new variable (in this example I called it INFRACOST_API_KEY, but you can choose the name that you prefer). Paste the value of the Infracost API key and click the lock button to mark it as a secret.

In this example, I stored the App registration credentials, Azure, and remote backend information in the same variable group.

Create a Build Validation Pipeline

The initial step taken by the validation pipeline is add a reference to the group that we have previously created.

variables:
- group: Infracost

Next, we’ll check out the repository and install the Infracost CLI on the agent. This will be configured to use the Infracost API key that we’ve previously secured within the Azure DevOps variable groups.

- checkout: self

- task: InfracostSetup@2
  displayName: Setup Infracost
  inputs:
    apiKey: '$(INFRACOST_API_KEY)'
    currency: 'USD'

Next, we will checkout the source branch of the pull request, and generate an Infracost cost estimate baseline.

- bash: |
    branch=$(System.PullRequest.SourceBranch)
    git fetch origin $branch
    git checkout ${branch#refs/heads/}
  displayName: Checkout target branch

- bash: |
    infracost breakdown --path=$(Agent.BuildDirectory) \
                        --format=json \
                        --out-file=$(Agent.TempDirectory)/infracost-base.json
  displayName: Generate Infracost cost estimate baseline

Next, we will switch branch to the target branch, generate an Infracost differential which showcases the variation in monthly costs between the source and target branches, and store this data in a JSON file.

- bash: |
    branch=$(System.PullRequest.TargetBranch)
    git fetch origin $branch
    git checkout ${branch#refs/heads/}
  displayName: Checkout target branch

- bash: |
    infracost diff --path=$(Agent.BuildDirectory) \
                   --format=json \
                   --compare-to=$(Agent.TempDirectory)/infracost-base.json \
                   --out-file=$(Agent.TempDirectory)/infracost.json
  displayName: Generate Infracost diff

Lastly, we’ll post a comment to the Pull Request with the report.

- bash: |
     infracost comment azure-repos --path=$(Agent.TempDirectory)/infracost.json \
                                   --azure-access-token=$(System.AccessToken) \
                                   --pull-request=$(System.PullRequest.PullRequestId) \
                                   --repo-url=$(Build.Repository.Uri) \
                                   --behavior=update
  displayName: Post Infracost comment

Allowing the Pipeline to Access the Variable Group

Even if we’ve instructed the pipeline to use the variable group we previously created, it still doesn’t have permission to do so. The first time we trigger the pipeline, it won’t start until we review and grant access to the variable group.

Instead of triggering the pipeline and then reviewing the permissions, we can set them up beforehand:

• Browse to the variable group we previously created.
• Click the Pipeline Permissions button on the top navbar.
• Click the + button and select the pipeline that can access the variables stored in the group.

Build Service Permissions

To allow the validation pipeline to posts comment with the cost estimate in the pull request, we need to enable the Build Service to post pull request comments, which is not set by default. To enable this, we need to:

• Click Project settings and select Repositories.
• Select the Security tab and select the [project name] Build Service ([org name]).
• Finally, enable the Contribute to pull request option.

Build Validation

To complete the workflow, we need to create a new build validation branch protection policy on the main branch. This will automatically trigger the pipeline every time there is a Pull request targeting the protected branch.

• Expand the Repos section and select Branches.
• Click the icon with the three vertical dots in the row of the branch you want to protect and click Branch Policies.

• Click the plus button next to Build Validation, select the pipeline we previously created.

• Click Save.

Testing

Now that everything is in place, it’s time to check that everything is working as expected. To do so, create a new branch and updated the Terraform configuration. Then, open a new Pull Request with the new branch as the source and the main branch as target. As soon as the Pull Request is created and an agent is available, the validation pipeline is triggered.

After its completion, we’ll see a new comment marked as closed (so it won’t affect other policies) in the Pull Request. The comment will have a simple table with the cost changes and an output where you can see the result coming directly from Infracost.

References

• GMInsights IaC market analysis: https://www.gminsights.com/industry-analysis/infrastructure-as-code-market
• Infracost source code: https://github.com/infracost/infracost
• Infracost cloud pricing API source code: https://github.com/infracost/cloud-pricing-api
• Official Infracost Azure DevOps task: https://marketplace.visualstudio.com/items?itemName=Infracost.infracost-tasks