In the past few years, the DevOps reports have shown how high-performance teams use monitoring to quickly detect errors, reduce downtime and costs.
Monitoring a complex application is a significant engineering endeavor in and of itself, but thanks to effort of cloud providers, developers can instrument their system with few clicks or configurations.
When it comes to your system health, monitoring your dependencies is just as important as monitoring your system. If your system is experiencing issues, your system dashboards and alarms should highlight if one of your dependency is having a bad time.
In the rest of the article I show a simple pattern to monitor your dependencies. The examples use CloudWatch Embedded Metric Format and golang but the underlying idea can be applied to other cloud platforms or languages.
What’s EMF?
CloudWatch Embedded Metric Format allows to automatically generate metrics from logs. As a developer you need to log the metrics as JSON to stdout and Cloud Watch takes care of publishing the metric on your behalf.
You can use one of the AWS client libraries for NodeJS, Python or Java to build and log an embedded metric structure. Sadly, at the moment AWS has not published a client library for Golang which means you need to create the JSON object yourself following the schema described in the specification. It’s relatively straightforward and not surprisingly a few community maintained projects do exactly that, for example aws-embedded-metrics-golang.
I find EMF very convenient but it’s usage is not strictly necessary to monitor your dependencies. You can publish the metric yourself if that’s what you want to do.
Instrumenting the client
The main idea is to instrument the client you use to call your service. Regardless of the language or the library you use, it’s almost certain that the client you use to call your dependencies allows to plug in custom logic. You can leverage it to log the EMF metric.
For example, assuming you use a golang http client, you can implement the RoundTripper interface:
The code implements the RoundTripper interface with the logic to log details about the http calls. In the example, we log the Duration and the result of the call (client error or server error ). Once I deployed the code (in my case to Lambda) and triggered it a few times, we can see the metrics published to CloudWatch:
Instrumenting a different client is very similar. For example, here we instrument the golang AWS SDK to log calls made to other AWS services:
Conclusions
With AWS CloudWatch you can easily set up monitoring for your AWS resources and applications. EMF is a very convenient way to plug custom metrics in your system. As I mentioned, the pattern is quite generic and you can easily port it in other languages or even other monitoring systems.
If you develop on AWS take also a look at AWS X-Ray. With AWS X-Ray you can visualize and analyze the health, performance, and availability of your applications in one location, and see an end-to-end view of requests as they travel through your application.
Follow me on Twitter to get new posts in your feed.
Credit for the cover image to Luke Chesser on Unsplash.
Top comments (0)