⛳ AWS CDK 101 🏊 - Building Constructs and Simple counter store in dynamodb

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If incase missed the previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted previous post at 🔗 dev to @aravindvcyber

In this article, let us introduce a new construct which would help us in tracking the invocation of the simple lambda function which we have created in our last article above and let us name this event counter.

Here we will also use dynamodb table to setup a simple counting store besides a general overview about constructs.

Why dynamodb in our solution ❓

Now when the lambda gets invoked receiving events, it may be vital for us to track the number of events coming into our system through the resource path in our endpoint. We will keep track of this count in dynamodb table by means of defining a new construct, which we develop.

This can be re-used for various use case, when we can generalize the construct and take for further vertical integration in our design.

New construct named Event-Counter 🎯

Let us create a new file constructs/event-counter.ts and include the below code block

export interface EventCounterProps {

  backend: lambda.IFunction,
  tableName: string,
  partitionKeyName: string,

}

export class EventCounter extends Construct {

  public readonly handler: lambda.Function;

  public readonly table: dynamodb.Table;

  constructor(scope: Construct, id: string, props: EventCounterProps) {
    super(scope, id);
    const {tableName, partitionKeyName, backend} = props;

    const Counters = new dynamodb.Table(this, tableName, {
        partitionKey: { name: partitionKeyName, type: dynamodb.AttributeType.STRING },
    });

   Counters.applyRemovalPolicy(RemovalPolicy.DESTROY);

   const eventCounterFn = new lambda.Function(this, 'EventCounterHandler', {
        runtime: lambda.Runtime.NODEJS_14_X,
        handler: 'event-counter.counter',
        code: lambda.Code.fromAsset('lambda'),
        environment: {
            BACKEND_FUNCTION_NAME: backend.functionName,
            EVENT_COUNTER_TABLE_NAME: Counters.tableName
        },
        logRetention: logs.RetentionDays.ONE_MONTH,
    });

    this.handler = eventCounterFn;

  }
}

The code above does the below resource provisioning in it.

• New DynamoDB table with path as the partition key.
• New Lambda function which is bound to the lambda/event-counter.counter code.
• We have wired the Lambda’s environment variables to the functionName and tableName of our resources also provisioned above.

Discovering resources at runtime 📜

You’ll notice that this code relies on two environment variables:

• EVENT_COUNTER_TABLE_NAME is the name of the DynamoDB table to use for storage.
• BACKEND_FUNCTION_NAME is the name of the downstream AWS Lambda function to invoke immediately after the counter is incremented.

Since the actual name of the table and the downstream function will only be decided when we deploy our app, we need to wire up these values from our construct code inside our stack. This is how we make this to be reusable in every other implementations.

The late bounded values means that if you print their values during synthesis, you will get a “TOKEN”, which is how the CDK represents these late-bound values. You should treat tokens as opaque strings. This means you can concatenate them together for example, but don’t be tempted to parse them in your code.

You may also notice we have included or set an removal policy as shown below. It is only to drop and create every time we destroy and redeploy, so that we make not let orphaned resources in development environment, which is also consuming dedicated provisioned capacity units for read and write.

Counters.applyRemovalPolicy(RemovalPolicy.DESTROY);

Here the dynamo db by default takes a provisioned capacity of 5 read and write units.

Lambda function for the new construct 📜

Let us define the lambda/event-counter.ts as follows.

const { DynamoDB, Lambda } = require('aws-sdk');
exports.counter = async function(event:any) {

  const message = event.body;
  console.log("Initial request:", JSON.stringify(message, undefined, 2));

  const dynamo = new DynamoDB();
  const lambda = new Lambda();

  await dynamo.updateItem({
    TableName: process.env.EVENT_COUNTER_TABLE_NAME,
    Key: { 'Counter Name': { S: 'SimpleEventsReceived' } },
    UpdateExpression: "SET hits = if_not_exists(hits, :start) + :inc",
    ExpressionAttributeValues: {
        ':inc': { N: '1' },
        ':start': { N: '0' },
    },
    ReturnValues: "UPDATED_NEW",
  }).promise();

  const resp = await lambda.invoke({
    FunctionName: process.env.BACKEND_FUNCTION_NAME,
    Payload: JSON.stringify(message)
  }).promise();

  console.log('Backend response:', resp);
  return JSON.parse(resp.Payload);
};

In the above code, you can quickly notice that we have also applied the same environment values to certain elements in the lambda function

Basically before invoking the required lambda function this function, increments the counter asynchronously using the provided table name.

Dynamodb updateItem logic to perform a counter increment operation 🔧

Here in the dynamodb updateItem command, we did actually hard coded the key and column values, but let us manage it in the later sections.

await dynamo.updateItem({
    TableName: process.env.EVENT_COUNTER_TABLE_NAME,
    Key: { 'Counter Name': { S: 'SimpleEventsReceived' } },
    UpdateExpression: "SET hits = if_not_exists(hits, :start) + :inc",
    ExpressionAttributeValues: {
        ':inc': { N: '1' },
        ':start': { N: '0' },
    },
    ReturnValues: "UPDATED_NEW",

  }).promise();

In a summary,the above block of code update a record with key SimpleEventsReceived by means of increment starting from 0.

Invoking the backend processing function 🔨

And towards the end we have invoked the actual backend function immediately with a simpler payload message.

const resp = await lambda.invoke({
    FunctionName: process.env.BACKEND_FUNCTION_NAME,
    Payload: JSON.stringify(message)
  }).promise();

Wiring construct into our stack ✏️

Now let us configure the new construct into our common event stack by including the import appropriately.

import { EventCounter } from '../constructs/event-counter';

 const eventCounter = new EventCounter(this, 'eventEntryCounter', {
      backend: eventEntry,
      tableName: 'Event Counters',
      partitionKeyName: 'Counter Name'
});

Thus we have modified the props data members with the necessary tableName, partitionKey for our use case.

 const eventGateway = new apigw.LambdaRestApi(this, 'EventEndpoint', {
      handler: eventCounter.handler,
      proxy: false,
      deployOptions: {
        accessLogDestination: new apigw.LogGroupLogDestination(eventGatewayALG),
        accessLogFormat: apigw.AccessLogFormat.jsonWithStandardFields(),
    }
 });

You can find from the above code we have overwritten our previous api gateway implementation by changing the handler function.

Adding logic to enable access logging in api gateway 🍌

Additionally, we have added some access logging to enable capture of the api requests fired against the deployed api. This does not include the test invocation we perform from the api gateway section in aws console by adding the deployOptions with the newly setup access log eventGatewayALG.

const eventGatewayALG = new logs.LogGroup(this, "Event Gateway Access Log Group", {
      retention: logs.RetentionDays.ONE_MONTH
});

eventGatewayALG.applyRemovalPolicy(RemovalPolicy.DESTROY);

Enable RetentionDays for logs for any resources we provision 🍇

Always make sure you specify some value for RetentionDays for retention, because we will always ignore the logs created and when we deploy too much serverless resources, though it is available due to too much logs we wont find much value in the old logs. This applies to any log for an aws resource and it is considered to be a good practice.

Override the method specification in api gateway 📋

Now one more important setup is we have to override the method for the api gateway method to use the eventCounter handler not the backend handler.

You may miss below setup by then the counter will not be running and you won't you find any logs for counter lambda, whereas the backend lambda will be firing as usual.

const eventHandler: apigw.LambdaIntegration = new apigw.LambdaIntegration(eventCounter.handler);
const event = eventGateway.root.addResource('event');

const eventMethod: apigw.Method = event.addMethod('POST', eventHandler, {  
    apiKeyRequired: true,
});

Couple of AccessDeniedExceptions ⌚

Let us deploy this and let us do a test, do expect some errors.

Now it is time to make use of the log groups created 🚂

Find the access log and identify whether the request is received by the gateway

Now we have to check the latest event counter lambda logs. And we are able to identify that dynamodb not writable by the event counter lambda function as follows.

Handler function needs access to read/write to dynamodb table 🚑

{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::************:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: dynamodb:UpdateItem on resource: arn:aws:dynamodb:ap-south-1:************:table/CommonEventStack-eventEntryCounterEventCounters821717DD-1A9Y14K4FSFW0",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::********:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: dynamodb:UpdateItem on resource: arn:aws:dynamodb:ap-south-1:************:table/CommonEventStack-eventEntryCounterEventCounters821717DD-1A9Y14K4FSFW0",
    "time": "2022-03-15T06:49:17.940Z",
    "requestId": "6S65HTNO6PRBAHHMIEHT9ESLH3VV4KQNSO5AEMVJF66Q9ASUAAJG",
    "statusCode": 400,
    "retryable": false,
    "retryDelay": 2.253276876174415,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::************:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: dynamodb:UpdateItem on resource: arn:aws:dynamodb:ap-south-1:************:table/CommonEventStack-eventEntryCounterEventCounters821717DD-1A9Y14K4FSFW0",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"
    ]
}

Let us add the below code into the lambda to fix this by granting the read and write access.

    Counters.grantReadWriteData(this.handler);

IAM Statement Changes
┌───┬───────────────────────────────────────────────────────────┬────────┬───────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────┬───────────┐
│   │ Resource                                                  │ Effect │ Action                                                    │ Principal                                                 │ Condition │
├───┼───────────────────────────────────────────────────────────┼────────┼───────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┼───────────┤
│ + │ ${eventEntryCounter/Event Counters.Arn}                   │ Allow  │ dynamodb:BatchGetItem                                     │ AWS:${eventEntryCounter/EventCounterHandler/ServiceRole}  │           │
│   │                                                           │        │ dynamodb:BatchWriteItem                                   │                                                           │           │
│   │                                                           │        │ dynamodb:ConditionCheckItem                               │                                                           │           │
│   │                                                           │        │ dynamodb:DeleteItem                                       │                                                           │           │
│   │                                                           │        │ dynamodb:GetItem                                          │                                                           │           │
│   │                                                           │        │ dynamodb:GetRecords                                       │                                                           │           │
│   │                                                           │        │ dynamodb:GetShardIterator                                 │                                                           │           │
│   │                                                           │        │ dynamodb:PutItem                                          │                                                           │           │
│   │                                                           │        │ dynamodb:Query                                            │                                                           │           │
│   │                                                           │        │ dynamodb:Scan                                             │                                                           │           │
│   │                                                           │        │ dynamodb:UpdateItem                                       │                                                           │           │
└───┴───────────────────────────────────────────────────────────┴────────┴───────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┴───────────┘

Handler function needs access to invoke backend lambda function 🔔

Let us try now again after deploying this, whether does that helps?

Yet another internal error, let us go back to the logs to check what happened this time.

We find have found the in the same log group for event counter lambda.

It could be understood that this time event counter lambda needs invocation access on the backend lambda we supplied to perform to perform lambda:InvokeFunction.

{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::************:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:ap-south-1:************:function:CommonEventStack-EventEntryHandler0826D724-tLR8gIzQkfyH because no identity-based policy allows the lambda:InvokeFunction action",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::************:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:ap-south-1:************:function:CommonEventStack-EventEntryHandler0826D724-tLR8gIzQkfyH because no identity-based policy allows the lambda:InvokeFunction action",
    "time": "2022-03-15T07:05:52.998Z",
    "requestId": "d47a8b75-1d6b-4d4d-a642-2a41a3996813",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 16.153069169276037,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::************:assumed-role/CommonEventStack-eventEntryCounterEventCounterHand-CPRUDNNBDNZG/CommonEventStack-eventEntryCounterEventCounterHand-KxRg8fa00D2I is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:ap-south-1:************:function:CommonEventStack-EventEntryHandler0826D724-tLR8gIzQkfyH because no identity-based policy allows the lambda:InvokeFunction action",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

Let us provide us provide it as follows and deploy quickly.

 // grant the lambda role invoke permissions to the downstream function
    props.backend.grantInvoke(this.handler);

We get the message to approve the IAM policy changes.

IAM Statement Changes
┌───┬──────────────────────────┬────────┬───────────────────────┬──────────────────────────────────────────────────────────┬───────────┐
│   │ Resource                 │ Effect │ Action                │ Principal                                                │ Condition │
├───┼──────────────────────────┼────────┼───────────────────────┼──────────────────────────────────────────────────────────┼───────────┤
│ + │ ${EventEntryHandler.Arn} │ Allow  │ lambda:InvokeFunction │ AWS:${eventEntryCounter/EventCounterHandler/ServiceRole} │           │
└───┴──────────────────────────┴────────┴───────────────────────┴──────────────────────────────────────────────────────────┴───────────┘

Finally this worked 🎊

Do note the fact that we made 3 request with only one successful request and it is tracked in our counter.

Add more counters based on the backend function processing results 🔮

Let us do one more thing here and try to parse the response from the backend lambda and set its own counter value as well.

  const result = JSON.parse(resp.Payload);

    console.log("Backend process result code: ", result.statusCode);



    if(result.statusCode == 200)
    {

      await dynamo.updateItem({
      TableName: process.env.EVENT_COUNTER_TABLE_NAME,
      Key: { 'Counter Name': { S: 'SimpleEventsProcessedSuccessfully' } },
      UpdateExpression: "SET hits = if_not_exists(hits, :start) + :inc",
      ExpressionAttributeValues: {
        ':inc': { N: '1' },
        ':start': { N: '0' },
      },
      ReturnValues: "UPDATED_NEW",
      }).promise();
    }

    return result;

And again, it is successful and we did managed to get counter value updated as well.

  let resp = { Payload: ''};
  try{
    resp = await lambda.invoke({
    FunctionName: process.env.BACKEND_FUNCTION_NAME,
    Payload: JSON.stringify(message)
  }).promise();

  }
  catch(err){
    console.log(JSON.stringify(err.message));
    resp = {
      "Payload": JSON.stringify(err),
    };

    await dynamo.updateItem({
      TableName: process.env.EVENT_COUNTER_TABLE_NAME,
      Key: { 'Counter Name': { S: `SimpleEventsProcessingErrored-${err.code}` } },
      UpdateExpression: "SET hits = if_not_exists(hits, :start) + :inc",
      ExpressionAttributeValues: {
        ':inc': { N: '1' },
        ':start': { N: '0' },
      },
      ReturnValues: "UPDATED_NEW",
    }).promise();
  }

Let us trigger the same error, we received have simulated last time once again and find the results.

So now we are able to get the success and failure counts successfully and specifically the exception type is also taken into account.

Counters construct summary 🚧

This basically means that whenever our endpoint is hit, API Gateway will route the request to our event counter handler, which will log the hit and relay it over to the event receiving backend function. Then, the responses will be relayed back in the reverse order all the way to the user. At the same time we are processing the response from the backend server and also able to register the success and failure count of the invocation as well.

By now we are able to create a new construct and include that into our stack. In a similar way, can share the construct and include them in other projects.

Using third-party constructs in your stack ✅

Likewise let us see one third party publicly available construct cdk-dynamo-table-viewer.

First we have to install this third party construct.

npm i --save cdk-dynamo-table-viewer

import it into your cdk stack as follows.

import { TableViewer } from 'cdk-dynamo-table-viewer'
}

Simply initialize this by creating a new object of this construct as follows as per our requirement.

const tblViewer = new TableViewer(this, 'EventHitsCounter-', {
      title: 'Event Counters from Dynamodb',
      table: eventCounter.table,
    });

CommonEventStack.EventHitsCounterViewerEndpoint6D4FD8C2 = https://**********.execute-api.ap-south-1.amazonaws.com/prod/

When you deploy this you can find a new endpoint which will provide the response similar to the below.

Applying sorting to this viewer results 📎

I did a small change to change the sorting of the table items as follows.

const tblViewer = new TableViewer(this, 'EventHitsCounter-', {
      title: 'Event Counters from Dynamodb',
      table: eventCounter.table,
      sortBy: '-hits'
    });

GitHub Link for cdk-dynamo-table-viewer

• The endpoint will be available (as an deploy-time value) under viewer.endpoint. It will also be exported as in stack output.
• Paging is not supported. This means that only the first 1MB of items will be displayed.

Conclusion 🔵

I have used this only for the purpose of the demo, you can choose similar constructs and try to learn from its implementation first and then you can start doing similar re-usable constructs for your project needs and even you can also try to use genuine open-source ones based on the trust level. Or you can start contributing to similar open-source construct for your learning and development portfolios as well.

We will add more connections to this api gateway and lambda stack and make it more usable in the upcoming articles stay subscribed.

⏭ We have our next article in serverless, do check out

aws-cdk-101-jest-testing-with-a-tdd-approach-for-our-construct-2aeh

🎉 Thanks for supporting! 🙏

Would be really great if you like to ☕ Buy Me a Coffee, to help boost my efforts.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber