Hacking Go-TFE and Fetching All Workspaces in Terraform Enterprise: A Journey Through Pagination

Terraform Enterprise (TFE) is a powerful platform for organizations using Terraform at scale. It provides collaboration, governance, and self-service workflows for infrastructure automation. As organizations grow, managing infrastructure becomes increasingly complex, often requiring the ability to programmatically interact with TFE using its robust API.

Recently, I found myself in a situation where I needed to fetch all workspaces for a given organization in TFE. Workspaces, as you may know, are the fundamental units in Terraform where runs occur. Each workspace holds the Terraform state file, which tracks infrastructure resources. For automation and reporting, I needed to pull in the full list of workspaces, but I quickly ran into a seemingly simple yet frustrating problem: pagination.

The Goal: Get All Workspaces

Our goal was clear — we wanted to retrieve every single workspace associated with a particular organization, without worrying about pagination limits. Whether for generating reports, validating configurations, or orchestrating CI/CD workflows, having a full view of all workspaces was crucial.

The go-tfe library by HashiCorp provides an elegant way to interact with TFE’s API. The most straightforward way to list workspaces looks something like this:

workSpaces, err := tfeClient.Workspaces.List(ctx, orgName, nil)

However, there was a catch.

The Pain Point: Pagination Woes

When we ran this code, we noticed something odd — it only returned 20 workspaces. After digging into the documentation and the source code of the go-tfe library, it became clear that the API defaults to a page size of 20. Even when we explicitly set the page size to the maximum limit of 100, it still only gave us one page of results:

workSpaces, err := tfeClient.Workspaces.List(ctx, orgName, &tfe.WorkspaceListOptions{
    ListOptions: tfe.ListOptions{
        PageNumber: 1,
        PageSize:   100,
    },
})

The issue was evident: the API uses pagination. This means that if your organization has more than 100 workspaces (which is not uncommon for larger teams), you have to make multiple requests to get all of them.

But we didn’t want to be bound by pagination. We wanted all the workspaces in one go — a clean, consolidated list.

The Solution: Handling Pagination Manually

To work around the pagination limit, the only solution was to manually paginate through the results until all workspaces were fetched. We crafted the following solution:

func fetchAllWorkspaces(ctx context.Context, client *tfe.Client, orgName string) ([]*tfe.Workspace, error) {
    var allWorkspaces []*tfe.Workspace
    page := 1
    pageSize := 100 // Max allowed page size

    for {
        opts := &tfe.WorkspaceListOptions{
            ListOptions: tfe.ListOptions{
                PageNumber: page,
                PageSize:   pageSize,
            },
        }

        workspaces, err := client.Workspaces.List(ctx, orgName, opts)
        if err != nil {
            return nil, err
        }

        allWorkspaces = append(allWorkspaces, workspaces.Items...)

        if len(workspaces.Items) < pageSize {
            break // No more pages left
        }

        page++
    }

    return allWorkspaces, nil
}

Breaking Down the Solution

Let’s walk through the logic step by step:

1. Initialization: We start with an empty slice to hold all workspaces and set the page number to 1.
2. Pagination Loop: Inside the for loop, we make requests using the ListOptions, setting both the page number and the maximum page size.
3. Appending Results: After each request, we append the returned workspaces to our allWorkspaces slice.
4. Exit Condition: If the number of workspaces in the current response is less than the page size (meaning we’ve reached the last page), we break out of the loop.
5. Result: Finally, we return the full list of workspaces.

Why This Matters

Fetching all workspaces might seem like a trivial task at first glance, but it highlights a crucial aspect of working with APIs — understanding and handling pagination correctly.

When dealing with APIs that support pagination:

• Always check default page sizes.
• Understand the max page size allowed.
• Implement proper loop termination to avoid infinite loops.
• Consider API rate limits when making multiple requests.

In our case, the solution was not only about fetching data but about ensuring the stability and reliability of our automation pipelines.

Conclusion

In the end, handling pagination in TFE’s API required us to step back, understand the underlying mechanics, and build a robust way to gather all the data we needed. The final solution is now part of our infrastructure tooling, allowing us to work seamlessly with Terraform Enterprise’s workspaces.

So next time you hit a wall with an API and pagination, remember — it’s not a bug; it’s a feature. Master it, and your automation game will only grow stronger.

Happy coding!

terraform #golang #go-tfe #workspaces #fetching #coding #solution #server #cicd