Step Toward Better Tenant Provisioning

In a multi-tenant SaaS application, you often need to manage resources that are tenant-specific. Whether it's a tenant-specific role, isolated DynamoDB tables, or per-tenant Cognito user pools, you need to have a way to deploy and update these resources across your application. In this blog, I'll show you how I have approached this problem.

There are three components of managing tenant-specific resources; creating new resources as a tenant is added, updating resources as your application needs changes, and deleting resources when a tenant is deleted.

The Old Way

In the past, I would use a model that looks something like the image above to create and delete resources. When a new tenant is added to the system an event is sent out and that would trigger a Lambda function. That Lambda function would use the SDK to create the necessary resources. Similarly, a delete would send out an event that would trigger a Lambda function that would use the SDK to delete tenant-specific resources.

There are some problems with this approach.

First, I like to use CDK, specifically L2 constructs, for all of my infrastructure. The SDK is very different, so there is a cognitive cost of using the SDK. You often need to remember more details, and the structure is very different.

Second, there isn't a place to go see all the resources associated with a tenant. This isn't a big deal when you have one resource per tenant, but as that grows it's nice to be able to go to a single spot to see everything that belongs to that tenant.

Third, while creating new resources when a tenant is created, and deleting them when a tenant is deleted, is pretty straightforward, updating is not. There isn't an event that you can use to trigger the update; at least not one that is a system event. Updates, unlike creates and deletes, are deploy time changes. As the application changes, you need to update the infrastructure of your tenants. That's a very different process than creating and deleting. Updating requires code that is aware of the current state and understands how to go from one to the other. It requires code to handle rolling back into a previous state when something goes wrong. Updates are complicated, and anytime I can remove complicated code I'm going to do it.

A Better Approach

Hearing others have the same problem, I wanted to find a solution to make things easier. I knew I wanted CDK and CloudFormation to be a part of the solution, and my thoughts quickly went to Step Functions. Could there be an answer there?

Here is what I figured out.

It starts with CDK. I create a Stack in CDK that holds all the resources for a tenant. This stack includes the use of CfnParameter to pass in the identifier of the tenant. Any resources that you need to create are added to this stack. The code looks something like this.

export class TenantTemplateStack extends Stack {
    constructor(scope: Construct, id: string, props: StackProps) {
        super(scope, id, props);

        const parameterTenantId = new CfnParameter(this, 'TenantId', {
            type: 'String',
            description: 'The ID of the tenant',
        });

        const tenantId = parameterTenantId.valueAsString;

        // any resources you want to provision per tenant go here
    }
}

The stack needs to be synthesized and made, so in our tenant management stack I include an S3 bucket where the template will be deployed, I synthesize the above template, and I deploy the synthesized template to the bucket. The keys to this are the use of the BootstraplessSynthesizer in the template stack and the Stage that I'll use to synthesize it. This creates a sort of CDK Inception, where your "cdk.out" will have your synthesized stack(s) and each stack will have another synthesized stack for your tenant template. Accessing the assembly of the stage allows us to grab the output and push it to S3 using the BucketDeploy construct.

export class TenantManagement extends Construct {
    public readonly templateBucketName: string;

    public readonly templateBucketKey: string;

    constructor(scope: Construct, id: string, props: TenantManagementProps) {
        super(scope, id);

        const stack = Stack.of(this);

        const templateBucket = new Bucket(this, 'TemplateBucket', {
            blockPublicAccess: {
                blockPublicAcls: true,
                blockPublicPolicy: true,
                ignorePublicAcls: true,
                restrictPublicBuckets: true,
            },
            objectOwnership: ObjectOwnership.OBJECT_WRITER,
            encryption: BucketEncryption.S3_MANAGED,
            enforceSSL: true,
            publicReadAccess: false,
            versioned: false,
            // important so that updates can be trigger based on this event
            eventBridgeEnabled: true,
        });

        const stage = new Stage(this, 'SynthStage');

        new TenantTemplateStack(stage, 'TenantTemplate', {
            // this allows the synthesis to generate a template without resolving CDK values like account and region
            synthesizer: new BootstraplessSynthesizer(),
        });

        // synthesize the template stack
        const assembly = stage.synth();

        // the stage only has one stack, so it's safe to grab index zero here to get the path of the output
        const templateFullPath = assembly.stacks[0].templateFullPath;

        // the bucket deployment construct will copy the resources in the specified path to S3
        new BucketDeployment(this, 'EachTenantStackDeployment', {
            destinationBucket: templateBucket,
            sources: [Source.asset(dirname(templateFullPath))],
        });
    }
}

At this point, I have a template that is being created that can be used for each of our tenants, but I still need to run the template at the right times. I'll create a few Step Functions to do this.

I'll start with the create because it's really hard to test doing an update and delete if you don't first create it. :) The create is triggered in the same way our Lambda function that was making SDK calls was triggered; via a tenant-created event sent to EventBridge.

The flow looks a bit like this:

The detail of the Step Function looks like this:

The Step Function is actually rather simple. It makes a call to CreateStack using the template I uploaded to the S3 bucket. After calling CreateStack it calls DescribeStack in a loop, checking to see that it has completed and failing if the stack fails. This way I can add metric alarms to notify the team if there are failures.

Next, I'll do the delete. Like the create, the delete is triggered from a tenant-deleted event sent to EventBridge. This runs a Step Function that looks a bit like this:

This one is a bit more complicated than the create because it first checks to see that the template is in a state that allows it to be deleted. This way you don't end up with errors if you try to delete a stack while it's in the process of being updated.

Finally, the update.

This Step Function is started whenever the template is updated in the S3 bucket. This means that when the deployment of our tenant management sends the updated template to the bucket this Step Function will run, which will automatically update all of your tenant's stacks. The State Machine looks a bit overwhelming, but when broken down into its parts it's pretty easy to understand.

The first part of this is just getting a list of tenants. I am storing our tenants in a DynamoDB table so I can query the data from there. DynamoDB uses paging, so I have to have some logic to loop over the data and call back into DynamoDB to get the next page.

The next bit is checking to see if the tenant's stack can be updated. This is important because a tenant may be in the process of being created when you deploy a new template. If you don't do this loop the update will fail and the new tenant won't get the updates.

Lastly, the UpdateStack call, and subsequent looping, looks just like our create logic adjusted for an update.

Conclusion

When you are building a multi-tenant SaaS app it's important to have a strategy for managing any tenant-specific resources you may have. Using Step Functions with the CDK is a great way to manage those updates. With this approach I get to continue to use CDK to model our resources, I have one place to go to see all the resources for a tenant (the CloudFormation stack for that tenant), and the complexities of updating and rolling back changes are managed by CloudFormation.

Comments

[Roman]

An easier way for some uses cases would be to use stacksets. Then all you need is to provision an AWS account. Then a stackset gets deployed automatically upon account creation.

You get complete isolation for free.

[Jason Wadsworth]

There are certainly some use cases where StackSets would work well. Especially if each tenant is in its own AWS account. In my experience that’s not how most SaaS applications are built. While there is a place for a multi account strategy, I don’t think an account per tenant is generally the best approach. As is often the case, there are always unique situations in which it may be.

[Roman]

In my experience that’s not how most SaaS applications are built.

In my experience having tenant stacks is also not how applications are built.

I don’t think an account per tenant is generally the best approach

Why not?

AWS recommends account-level isolation as a one of the tenant isolation strategy.

CloudFormation has a quota for the number of stacks, which is 2,000.

While StackSet allows 100,000 stack instances per StackSet. If that's not enough you can create a new StackSet and get another 100,000.

There's also a much higher limit on the number of AWS accounts; you can go into 10s or 1000s.

It is also much easier to implement. You only need to call an AWS API to create a new AWS account. Then StackSet will trigger immediately and provision your solution for the new account.

[Jason Wadsworth]

AWS recommends account isolation as an option. They have an entire team dedicated to SaaS (the SaaS factory). There are many ways to do tenant isolation. A per account approach increases operational complexity. It has benefits, as do other approaches. No solution is the right solution for all situations. I’ve built quite a few SaaS applications and I personally prefer to treat account level isolation as a last resort. Clearly you’ve had a different experience. My post is intended to support those who are not doing account level isolation.

[Martin Muller 🇩🇪🇧🇷🇵🇹]

This article is great. But I still lack a bit seeing the value in using Step Functions over just like a Lambda. Can you help me with that :) ?

[Jason Wadsworth]

If you were to try to use a Lambda in place of Step Functions you’d have to have the function running the entire time the CloudFormation is running. That’s not only not cost effective but might not work if the stack runs for more than 15 minutes.