🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.
If in case missed my previous article, do find it with the below links.
🔁 Original previous post at 🔗 Dev Post
🔁 Reposted the previous post at 🔗 dev to @aravindvcyber
Also, we have started to develop an open source project which we would be using to play around with refracting the architecture as well as learn CDK stuff at the same time we will provide something useful for our community. Find more about this discussed in the article below.
🔁 Original project post at 🔗 Dev Post
🔁 Reposted project post at 🔗 dev to @aravindvcyber
Cross Account sendMessage 🍡
Earlier in our article, we have seen how to use custom Eventbridge and SQS by configuring an event rule and target which shifts the messages to the sqs queue and extended the same to remote stacks as well. Now let us make one more addition to our stack by retrieving the dlq messages from the remote stack across regions to our processor region.
Original post at 🔗 Dev Post
Reposted at 🔗 dev to @aravindvcyber
To start with we will be first discussing how to start polling the messages from the dlq using a lambda processor.
Before that let us set up a lambda layer that will have our external dependencies necessary for logging and monitoring.
export const generateLayerVersion = (
scope: Construct,
layerName: string,
props: Partial<LayerVersion>
): LayerVersion => {
return new LayerVersion(scope, layerName, {
...defaultLayerProps,
code: Code.fromAsset(join(__dirname, "..", "layers", layerName)),
...props,
});
};
const powertoolsSDK = generateLayerVersion(this, "powertoolsSDK", {});
exportOutput(this, "powertoolsSDKArn", powertoolsSDK.layerVersionArn);
Lambda processor definition 🪴
Here you can find the definition of the lambda function which will be used to poll messages from dlq and push to SNS topic.
const failedMessageAggregator = new Function(
this,
"failedMessageAggregator",
{
code: Code.fromAsset("dist/lambda/failed-message-aggregator"),
handler: "failed-message-aggregator.handler",
...commonLambdaProps,
functionName: "failedMessageAggregator",
layers: [powertoolsSDK],
environment: {
TOPIC_ARN: remoteStackEventTargetDlqSns.topicArn,
TZ: config.get("timeZone"),
LOCALE: config.get("locale"),
},
}
);
failedMessageAggregator.applyRemovalPolicy(RemovalPolicy.DESTROY);
Lambda handler code 🌷
The full and latest code should be found in the git hub repo below.
class Lambda implements LambdaInterface {
@tracer.captureMethod()
private async processSQSRecord (rec: SQSRecord) {
logger.info("Fetching DLQ message:", {rec});
const params: PublishInput = {
Message: rec.body,
Subject: "Forwarding event message to SNS topic",
TopicArn: process.env.TOPIC_ARN,
};
const snsResult: PublishResponse = await sns.publish(params).promise();
logger.info("Success", { params, snsResult });
}
public async handler(event: SQSEvent) {
try {
await Promise.all(
event.Records.map(async (rec: SQSRecord) => {
await this.processSQSRecord(rec);
})
);
return {
statusCode: 200,
headers: { "Content-Type": "text/json" },
body: {
EventsReceived: [...event.Records].length,
},
};
} catch (error) {
logger.error("Error", { error });
return {
statusCode: 400,
headers: { "Content-Type": "text/json" },
body: {
EventsReceived: [...event.Records].length,
Error: error
},
};
}
};
}
Event Source mapping DLQ to lambda 🌳
Here we will map the remote dlq to trigger the lambda which we have built above.
failedMessageAggregator.addEventSource(
new SqsEventSource(remoteStackEventTargetDlq.queue, {
batchSize: 10,
maxBatchingWindow: Duration.seconds(20),
})
);
SNS topic to push to subscribers 🦚
This topic will be used to receive messages from the lambda and push into relevant subscriber channels. Here we will subscribe this to common dlq in the processor stack.
const remoteStackEventTargetDlqSns = new Topic(
this,
"remoteStackEventTargetDlqSns",
{
displayName: "remoteStackEventTargetDlqSns",
topicName: "remoteStackEventTargetDlqSns",
}
);
remoteStackEventTargetDlqSns.applyRemovalPolicy(RemovalPolicy.DESTROY);
exportOutput(
this,
"remoteStackEventTargetDlqSnsArn",
remoteStackEventTargetDlqSns.topicArn
);
Granting access to lambda to Send Message 🍋
Now will be grant access to the lambda function to send messages as the producer.
remoteStackEventTargetDlqSns.grantPublish(failedMessageAggregator);
Two-way handshake to link SNS to SQS 🥬
With regards to sns and sqs in different account it is essential to set up the two-way handshake for this there have to be two actions allowed one at each end.
-
sns:Subscribein remote topic -
sqs:SendMessagein consumer queue (subscriber)
Remote stack configurations
Granting access to processor account to subscribe
Here we will be granting access to processor account resources to subscribe to this topic as follows.
remoteStackEventTargetDlqSns.addToResourcePolicy(
new PolicyStatement({
sid: "Cross Account Access to subscribe",
effect: Effect.ALLOW,
principals: [new AccountPrincipal(targetAccount)],
actions: ["sns:Subscribe"],
resources: [remoteStackEventTargetDlqSns.topicArn],
})
);
Processor stack configurations 🏝️
remoteAccounts.map((account: string) => {
remoteRegions.map((region: string) => {
// Here we will be adding the reference and the subscription
});
});
Referencing to the remote topic
In the processor stack, we will be getting the reference to the relevant topics as follows.
const remoteStackEventTargetDlqSns = Topic.fromTopicArn(
this,
`remoteStackEventTargetDlqSns-${region}-${account}`,
`arn:aws:sns:${region}:${account}:remoteStackEventTargetDlqSns`
);
Subscribing to the remote topic
Here we will be subscribing to the processor region dlq to receive the messages from the remote region SNS topic as follows.
Note it is highly recommended to subscribe from the consumer stack so that the subscription gets auto-confirmed, else there will be another confirmation step you may need to do from the console or confirmation message to do that yourself.
const subProps: SqsSubscriptionProps = {
rawMessageDelivery: true,
};
remoteStackEventTargetDlqSns.addSubscription(
new aws_sns_subscriptions.SqsSubscription(
stackEventTargetDlq.queue,
subProps
)
);
The above subscription setup from the processor stack also grants the sqs:SendMessage implicitly while the subscription is created.
Conclusion ⛲
With this approach just like how we pooled the remote cfn events to a common event bridge across regions and accounts, we are also able to get the remote dlq events to a common dlq. These messages in dlq can be inspected without switching to another region or account, which the maintainer doesn't have any access.
This will be extremely useful when you build similar event-driven solutions.
We will be talking about more similar engineering concepts as we refactor and refine the event forwarder project. Keep following for similar posts on engineering with IaC primarily using AWS CDK and Serverless.
Also, feel free to contribute to the progress of the below solution with your comments, and issues, maybe you can also do a pr if you feel it can help our community.
🔁 Original project post at 🔗 Dev Post
🔁 Reposted project post at 🔗 dev to @aravindvcyber
⏭ We have our next article in serverless and IaC, do check out
https://dev.to/aws-builders/aws-cdk-101-sam-local-to-test-and-debug-lambda-function-1afj
🎉 Thanks for supporting! 🙏
Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.
🔁 Original post at 🔗 Dev Post
🔁 Reposted at 🔗 dev to @aravindvcyber
Top comments (0)