Dealing with CDK Custom Resources and failures.

With the AwsCustomResource Constrcut CDK provides an easy way to execute AWS SDK calls during CloudFormation deployments. Common use cases are e.g. fetching values from Parameter Store or executing Lambda Functions.

How does it work?

To perform SDK calls CloudFormation deploys a singleton Lambda Function including the JavaScript SDK. This function will be executed during CloudFormation life cycle events (CREATE, UPDATE and DELETE) and perform the SDK calls against the AWS APIs. The response JSON of the singleton Lambda is stored in S3 and will be read from CloudFormation.

What about failures?

As mentioned in the title this post is about failures. Because at this point I was a little irritated when executing Lambda Functions this way.

Let's say we have a Lambda Function called "MigrationLambda" to do some database operation (e.g. data migrations) during our deployment. Therefore we can use AwsCustomResource to execute our MigrationLambda on every UPDATE event of our CloudFormation Stack.

new custom_resources.AwsCustomResource(this, 'CustomResource', {
  onUpdate: {
    service: 'Lambda',
    action: 'invoke',
    parameters: {
      FunctionName: migrationLambda,
      InvocationType: 'Event',
      Payload: JSON.stringify({
        someKey: 'someValue'
      }),
    },
  },
});

So far - so good.
But what happens if the MigrationLambda fails for some reason… ? Nothing. Our CloudFormation deployment will be successful.

Ok, but why?

As mentioned in the beginning the singleton Lambda Function is responsible for the AWS SDK calls. In other words it is just a "carrier" between CloudFormation and the AWS APIs.

That means for the state of the CustomResource (SUCCESS or FAILED) it is only relevant if the API call itself was successful or not. So if for example the MigrationLambda would not exits any more and the API call returns an error our whole deployment would fail.

Solution

But what if we want the deployment to fail? In our example we want to notice if the database migration was not successful.

You can achieve this by using a custom Provider instead auf the singleton Lambda Function which comes out of the box.

Provider Framework vs. Lambda directly

There are two ways to use a Lambda Function as custom Provider. You can use the Provider Framework or a Lambda directly. I also was a bit confused at this point at the beginning, because the way you handle the response depends on the variant you choose.

The main difference here is that the Provider Framework takes care about sending your response to the S3 Bucket. In other words you simply can return data or throw an error in your Lambda Function. When using a Lambda Function directly you have to perform a PUT request against the pre-sigend S3 URL by yourself. AWS recommends using the Provider Framework ("[...] unless you have good reasons not to.")!

Anyway, with both variants we have full control of what happens with our deployment depending on the custom resource Lambda.

// CDK

const providerLambda = new aws_lambda_nodejs.NodejsFunction(
  this,
  "ProviderLambda",
  {
    entry: path.join(__dirname, "index.ts"),
  }
);

const provider = new custom_resources.Provider(this, "Provider", {
  onEventHandler: providerLambda,
});

new CustomResource(this, "CustomResource", {
  // Using the Provider Framework
  serviceToken: provider.serviceToken,

  // Alternative: Using Lambda directly
  // serviceToken. providerLambda.functionArn,

  properties: {
    timestamp: new Date().toISOString(),
  },
});

Lambda Function

export const handler = async (
  event: AWSLambda.CloudFormationCustomResourceEvent,
  _context: AWSLambda.Context
) => {
  // your business logic here ...

  // Variant 1: Using the Provider Framework
  // Success case
  const responseObject: AWSLambda.CloudFormationCustomResourceResponse = {
    Status: "SUCCESS",
    PhysicalResourceId: "SomeCustomResourceId",
    StackId: event.StackId,
    RequestId: event.RequestId,
    LogicalResourceId: event.LogicalResourceId,
    Data: {
      key: "some custom value",
    },
  };
  return responseObject;

  // Error case
  throw new Error("Your error message");



  // Varinat 2: Using Lambda directly
  // Success case
  const responseObject: AWSLambda.CloudFormationCustomResourceResponse = {
    Status: "SUCCESS",
    PhysicalResourceId: "SomeCustomResourceId",
    StackId: event.StackId,
    RequestId: event.RequestId,
    LogicalResourceId: event.LogicalResourceId,
    Data: {
      key: "some custom value",
    },
  };
  await fetch(event.ResponseURL, {
    body: JSON.stringify(responseObject),
    method: "PUT",
  });

  // Error case
  const responseObject: AWSLambda.CloudFormationCustomResourceResponse = {
    Status: "FAILED",
    Reason: "Your error message",
    PhysicalResourceId: "SomeCustomResourceId",
    StackId: event.StackId,
    RequestId: event.RequestId,
    LogicalResourceId: event.LogicalResourceId,
  };
  await fetch(event.ResponseURL, {
    body: JSON.stringify(responseObject),
    method: "PUT",
  });
};

Points to consider

UPDATE event

By default the CustomResource will no be executed when updating the CloudFormation stack, even if the code of the Provider changes. This only happens if properties change between deployments. Therefore we can set a timestamp to achieve the required behaviour (see CDK example above).

Life cycle events in generell

Our Provider will be executed on every life cycle event. So we have to take care which events are relevant inside our Lambda code. Maybe you only want it to do something on CREATE but nut on UPDATE and DELETE etc.

---

Suggestions or feedback

If you got any kind of feedback, suggestions or ideas - feel free and write a comment below this article. There is always space for improvement!