Any good business runs with goals and key performance indicators (KPIs) that are monitored and managed. These can be user signups, or Subscriptions sale trend, or customer churn. Any Cloud-born or cloud-native business is no exception to this.
This post provides how to guide for creating business metrics using Amazon CloudWatch and AWS Lambda services.
Readers, familiar with EMF format for CloudWatch, can jump to the Implementation section.
Introduction
Before we dive deeper, let's have a tweet size introduction to AWS CloudWatch and AWS Lambda service.
Amazon CloudWatch helps to observe and monitor AWS resources and applications in the cloud and on premises.
AWS Lambda is a serverless, event-driven compute service that lets you run code for any type of application or backend service without provisioning or managing servers. AWS Lambda helps to focus on delivering customer value.
Any AWS service publish metrics in Amazon CloudWatch. Amazon CloudWatch can also publish your own application specific metrics. These metrics are called as custom metrics. The number of third-party API calls, or the count of status codes returned by an API, etc are a few examples. Based on these metrics, you can build your own dashboards or alert notifications etc.
Ways to add Amazon CloudWatch custom metrics
PutMetricData API
Amazon CloudWatch provides an API through which you can put the custom metrics. But, sending the API request is a synchronous process. For high-volume applications, you have to batch up the requests for CloudWatch API – "PutMetricData" API. Otherwise, your requests will be throttled by CloudWatch API. CloudWatch API usage for adding custom metrics also incurs cost.
Metric Log Filters
Metric log filters are another way to create CloudWatch custom metrics. The Metric log filters can search and filter data points needed to create metrics from CloudWatch log groups. It's a better option as compared to using CloudWatch API since it's asynchronous. The drawback of this method is, every time you need to create a new metric, you have to create the metric log filters.
EMF For CloudWatch
A third option is using Amazon CloudWatch specification for Embedded Metric Format (EMF). It's again an asynchronous way to create custom metrics. But with EMF format, there are no incurring charges for customer. There are no prerequisites as previous method of log filters.
Implementation
This walkthrough assumes familiarity with AWS Lambda and AWS CloudWatch services. It also assumes that you can deploy AWS lambda in your own AWS account.
The implementation uses AWS Lambda and Amazon CloudWatch. But, you can use Amazon CloudWatch EMF format with any other AWS compute services such as Amazon ECS, Amazon EKS and AWS Fargate.
Let's create AWS Lambda function to log custom metrics. In our example, below, the lambda function orderservice
represents a fictitious order service. In this service, we would be creating metrics for average order value amount trend.
- For order service lambda function, first create a directory and initialize npm.
mkdir orderservice && cd orderservice && npm init -y
- Now, install aws-embedded-metrics library in the
orderservice
project directory.aws-embedded-metrics
library handles logging custom metrics in Amazon CloudWatch EMF format.
npm install aws-embedded-metrics
This will create a node_modules
directory with the required dependencies.
- Once you have installed the dependencies, create a new file in the project directory and name it
index.js
. Inindex.js
, export the function as shown below.
const { metricScope } = require("aws-embedded-metrics");
const orderMetrics = metricScope(metrics =>
async () => {
// ...
});
exports.handler = orderMetrics;
- Now, within the function, you have to create your own metric namespace and select the dimensions for the metrics. For high cardinality values, you can use property rather than dimensions. So, complete
orderMetrics
function is as follows
const { metricScope } = require('aws-embedded-metrics');
const orderMetrics = metricScope(metrics => async event => {
//define the namespace for custom metrics
metrics.setNamespace('OrderManagement');
//set the dimentions
metrics.putDimensions({ Service: 'OrderService', CustomerId:'26102022' });
// write the metric value to cloudwatch
metrics.putMetric('Avg. Order Value',Math.floor(Math.random() * 100) , 'None');
// add context as properties to build cloudwatch log insights queries
metrics.setProperty('AccountId', '123456789012');
metrics.setProperty('RequestId','45b7d3b0-ca99-44a2-93ad-05e29c9d40e6');
metrics.setProperty('OrderId', '7b125931-7898-4eab-bf16-78b8dc87707d');
metrics.setProperty('Payload', {
sampleTime: new Date(),
customerId: 26102022,
});
});
exports.handler = orderMetrics;
Note that, putMetric
parameter value is randomly generated for demo purposes. You have to replace it with your own metric value.
- Now, you need to compress all the files in the project directory and upload the ZIP file to Lambda console.
zip -r ../orderservice.zip .
After uploading the
orderservice.zip
to lambda function, you can test the lambda with test event. Since, we have randomized theputMetric
input value, we are expected to see the logs having the custom metric. There can be a couple of seconds of lag for reflecting values in Amazon CloudWatch metrics page and dashboard.To simulate real world traffic, you have to invoke orderservice lambda several times. You can use lambda function's
Test
feature with sample event to simulate invocations.Now, you will see the
ordermanagement
metric as shown below under CloudWatch Metrics.
- Clicking on the
OrderManagement
metric, you can seeCustomerId,Service,ServiceType and ServiceName
metric. After small touch ups on the graph, the final output of custom metric looks as below.
- You can consider this business metric of
Avg. Order Value
as any other Amazon CloudWatch metric. Furthermore, you can create CloudWatch alerts based on CloudWatch custom metrics, for example – set up an alert when we receive an order of more than USD 50 million etc.
The same business metrics can be utilized to create CloudWatch dashboards to better serve your customers.
In summary, EMF format for Amazon CloudWatch is a great feature which can be utilized for creating the custom business metrics. Many customers have enjoyed up to 65% of cost reduction with using EMF format for Amazon CloudWatch, among other benefits. What you are waiting for? Now Go Build!
Top comments (0)