Server Performance report with the help of AWS CloudWatch Iog Insights

#aws #cloudwatch #log #tooling

This article focuses on Creating Performance report using Aws CloudWatch Log Insights serverless technology.

Aws CloudWatch is a logging service for AWS and any deployed application. All logs are stored and accessed via a two-level hierarchy named as log group and streams. These logs are stored at CloudWatch as system files and folders separately for different regions i.e., they are saved via the AWS vicinity in which they have been captured and accessed accordingly. So, there is no global view of a log and it is region based. All those log can be formatted as JSON, CLF (Common log Format) or any other desired format. CLF is used by me to create a report in AWS CloudWatch log Insights.

CloudWatch Logs Insights
CloudWatch Logs Insights is an interactive log query tool to visualize and analyse log data. Under this system Queries can filter and graphs or table can be created for aggregate log data to visualize and publish it to CloudWatch dashboard.

CloudWatch Dashboard
AWS CloudWatch dashboard are customizable pages that can be modify to monitor resources from single location.

How to extract data and plot graphs using CloudWatch Log Insights.

Initially we must have a clear visualization of our requirement and its purpose. In my project I have to extract data for Average, Min and Max latency of each APIs and SQL query with respect to the number of request received on a single server during defined interval of time and plot graph for the same. The project also required to show error/exception message and it's count from server with respect to it's API and show the number of request made, successfully executed and error/exception of each APIs.
In order to get above the requirement, it is need to added the custom log to backend servers as detailed below :

    //Backend service name.
    String serviceName="ContentService";

    //Logs as API hits.
    public void ApiStartLog(String apiName){
        logger.info("service {} {} START Api ",serviceName,apiName);
    }

    //Logs as API successfully executed with its latency. 
    public void ApiEndLog(String apiName,long latency){
        logger.info("service {} {} END Api latency {} ms.",serviceName,apiName,latency);
    }

    //Logs as API have some exception.
    public void ApiExceptionLog(String apiName,String error){
        logger.error("service {} {} EXCEPTION : {}",serviceName,apiName,error);
    }

    //Logs as SQL Query hits.
    public void QueryStartLog(String queryName){
        logger.info("service {} {} START Query ",serviceName,queryName);
    }

    //Logs as SQL Query ends with its latency.
    public void QueryEndLog(String queryName,long latency){
        logger.info("service {} {} END Query latency {} ms.",serviceName,queryName,latency);
    }

In above log, the text like "service","Api","Query" are added to filter all logs and classify Api's log and Query's log separately in the result. In the above log the text "START","END","EXCEPTION" are added to separate the log in respect of their status. The dynamic values like Api / Query name, latency and serviceName are also added in above log. The serviceName is added to differentiate other backend services.

Now we are ready to go to CloudWatch service and open Logs Insights.
Now click on "Select log group(s)" and select log group from dropdown menu.
Now click "Run Query" to check the latest log of all selected log groups.
After expansion of logs, the log will be distributed in Field and Value as key-value pair in dictionary.
@message field contains the log data, we required to plot graphs and table.
In CloudWatch Logs Query Syntax, write a query to filter log with text "END", "latency" in @message field. After that will get parse the @message by mapping the text with text query as shown in image.

fields @message
| filter @message like "END"
| filter @message like "latency"
|parse @message "service * * * * latency * " as @service,@request,@status,@type,@latency

Then add stats query, to analyze latency of Api/Query.

fields @message
| filter @message like "END"
| filter @message like "latency"
|parse @message "service * * * * latency * " as @service,@request,@status,@type,@latency
|stats avg(@latency) as avgg,max(@latency),min(@latency),count(@request) by @request,@type
|sort avgg desc

Now sort the result with average latency.

Here, check the graph of this table by clicking on "Visualization" and selecting "bar graph".

Now we have a graph, showing the Average, min and max latency of each Api/Query of backend service.

Even we can create a table for the Error/Exception log also by having this query.

filter (
    @message like " EXCEPTION :" or @message like " ERROR :")
|parse @message "service * * * : *" as @service,@function,@type,@msg
|stats count(*) as Count by @service,@type,@function,@msg
|sort Count desc