mydeveloperplanet

Posted on Mar 10, 2021 • Originally published at mydeveloperplanet.com

How to Monitor a Spring Boot App

#java #devops #tutorial

In this blog you will learn how to monitor a Spring Boot application. You will make use of Spring Actuator, Micrometer, Prometheus and Grafana. Seems a lot of work, but this is easier as you might think!

1. Introduction

When an application runs in production (but also your other environments), it is wise to monitor its health. You want to make sure that everything is running without any problems and the only way to know this, is to measure the health of your application. When something goes wrong, you hopefully will be notified before your customer notices the problem and maybe you can solve the problem before your customer notices anything. In this post, you will create a sample Spring Boot application which you can monitor with the help of Spring Actuator, Micrometer, Prometheus and Grafana. This is visualized in the overview below, where Spring Actuator and Micrometer are part of the Spring Boot App.

The purpose of the different components is explained briefly:

Spring Actuator: supplies several endpoints in order to monitor and interact with your application. See Spring Boot Actuator in Spring Boot 2.0 for more information.
Micrometer: an application metrics facade that supports numerous monitoring systems, Spring Boot Actuator provides support for it.
Prometheus: a timeseries database in order to collect the metrics.
Grafana: a dashboard for displaying the metrics.

Every component will be covered in the next sections. The code used in this post can be found at GitHub.

2. Create Sample App

First thing to do is to create a sample application which can be monitored. Go to Spring Initializr, add dependency Spring Boot Actuator, Prometheus and Spring Web. The sample application will be a Spring MVC application with two dummy endpoints.

Create a RestController with the two endpoints. The endpoints only return a simple String.

@RestController
public class MetricsController {

    @GetMapping("/endPoint1")
    public String endPoint1() {
        return "Metrics for endPoint1";
    }

    @GetMapping("/endPoint2")
    public String endPoint2() {
        return "Metrics for endPoint2";
    }

}

Start the application:

$ mvn spring-boot:run

Verify the endpoints are working:

$ curl http://localhost:8080/endPoint1
Metrics for endPoint1
$ curl http://localhost:8080/endPoint2
Metrics for endPoint2

Verify the Spring Actuator endpoint. The endpoint returns the information in json. In order to format the response so that it is readable, you can pipe the output of the actuator endpoint to mjson.

$ curl http://localhost:8080/actuator | python -mjson.tool
...
{
   "_links":{
      "self":{
         "href":"http://localhost:8080/actuator",
         "templated":false
      },
      "health":{
         "href":"http://localhost:8080/actuator/health",
         "templated":false
      },
      "health-path":{
         "href":"http://localhost:8080/actuator/health/{*path}",
         "templated":true
      },
      "info":{
         "href":"http://localhost:8080/actuator/info",
         "templated":false
      }
   }
}

By default, the above information is available. Much more information can be provided by Spring Actuator, but you need to enable this. In order to enable the Prometheus endpoint, you need to add the following line into the application.properties file.

management.endpoints.web.exposure.include=health,info,prometheus

Restart the application and retrieve the data from the Prometheus endpoint. A large bunch of metrics are returned and available. Only a small part of the output is displayed because it is a really long list. The information which is available at this endpoint, will be used by Prometheus.

$ curl http://localhost:8080/actuator/prometheus
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",} 2.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",} 0.009
...

As mentioned before, Micrometer is also needed. Micrometer can be seen as SLF4J, but then for metrics. Spring Boot Actuator provides autoconfiguration for Micrometer. The only thing you need to do is to have a dependency on micrometer-registry-{system} in your runtime classpath and that is exactly what we did by adding the prometheus dependency when creating the Spring Boot app.

The metrics Actuator endpoint can also be accessed when you add it to the application.properties file.

management.endpoints.web.exposure.include=health,info,metrics,prometheus

Restart the application and retrieve the data from the metrics endpoint.

$ curl http://localhost:8080/actuator/metrics | python -mjson.tool
...
{
    "names": [
        "http.server.requests",
        "jvm.buffer.count",
        "jvm.buffer.memory.used",
...

Each individual metric can be retrieved by adding it to the URL. E.g. the http.server.requests parameter can be retrieved as follows:

$ curl http://localhost:8080/actuator/metrics/http.server.requests | python -mjson.tool
...
{
    "name": "http.server.requests",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 3.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.08918682
        },
...

3. Add Prometheus

Prometheus is an open source monitoring system of the Cloud Native Computing Foundation. Since you have an endpoint in your application which provides the metrics for Prometheus, you can now configure Prometheus to monitor your Spring Boot application. The Spring documentation for doing so can be found here.

There are several ways to install Prometheus as described in the installation section of the Prometheus documentation. In this section, you will run Prometheus inside a Docker container.

You need to create a configuration prometheus.yml file with a basic configuration to add to the Docker container. The minimal properties are:

scrape_interval: how often Prometheus polls the metrics endpoint of your application
job_name: just a name for the polling job
metrics_path: the path to the URL where the metrics can be accessed
targets: the hostname and port number. Replace HOST with the IP address of your host machine

global:
  scrape_interval:     15s

scrape_configs:
  - job_name: 'myspringmetricsplanet'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['HOST:8080']

If you have difficulties finding out your IP address on Linux, you can use the following command:

$ ip -f inet -o addr show docker0 | awk '{print $4}' | cut -d '/' -f 1

Start the docker container and bind-mount the local prometheus.yml file to the one in the docker container. The above prometheus.yml file can be found in the git repository in directory prometheus.

$ docker run \
    -p 9090:9090 \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

After successful startup of the Docker container, first verify whether Prometheus is able to gather the data via url http://localhost:9090/targets.

It seems that Prometheus is not able to access the Spring Boot application running on the host. An error context deadline exceeded is mentioned.

This error can be solved by adding the Docker container to your host network which will enable Prometheus to access the URL. Therefore, add --network host as a parameter. Also remove the port mapping as this has no effect when --network is being used. Finally, give your container a name, this will make it easier to start and stop the container. The -d parameter will run the container in detached mode.

$ docker run \
    --name prometheus \
    --network host \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    -d \
    prom/prometheus

Verify again the Prometheus targets URL, the state indicates UP which means the prerequisite of accessing the metrics endpoint is fullfilled now.

It is now possible to display the Prometheus metrics. Navigate to http://localhost:9090/graph, enter http_server_requests_seconds_max in the search box and click the Execute button. Access a couple of times the endPoint1 URL in order to generate some traffic. This parameter will give you the maximum execution time during a time period of a request.

4. Add Grafana

The last component to add is Grafana. Although Prometheus is able to display the metrics, Grafana will allow you to show the metrics in a more fancy dashboard. Grafana also supports several ways for installing it, but you will run it in a Docker container, just like you did with Prometheus.

$ docker run --name grafana -d -p 3000:3000 grafana/grafana

Navigate to URL http://localhost:3000/, the URL where Grafana is accessible.

The default username/password is admin/admin. After clicking the Log in button, you need to change the default password. Google Chrome will also warn you about the default username/password.

Next thing to do, is to add a Data Source. Click in the left sidebar the Configuration icon and select Data Sources.

Click the Add data source button.

Prometheus is at the top of the list, select Prometheus.

Fill the URL where Prometheus can be accessed, set HTTP Access to Browser and click the Save & Test button at the bottom of the page.

When everything is OK, a green notification banner is shown indicating that the data source is working.

Now it is time to create a dashboard. You can create one of your own, but there are also several dashboards available which you can use. A popular one for displaying the Spring Boot metrics is the JVM dashboard.

In the left sidebar, click the + sign and choose Import.

Enter the URL https://grafana.com/grafana/dashboards/4701 where the JVM dashboard can be found and click the Load button.

Enter a meaningful name for the dashboard (e.g. MySpringMonitoringPlanet), select Prometheus as Data Source and click the Import button.

At this moment, you have a cool first Grafana dashboard at your disposal. Do not forget to scroll down, there are more metrics than shown in the screenshot. The default range is set to 24 hours, this is maybe a bit large when you just started the application. You can change the range in the top right corner. Change it to f.e. Last 30 minutes.

It is also possible to add a custom panel to the dashboard. At the top of the dashboard, click the Add panel icon.

Click Add new panel.

In the Metrics field, you enter http_server_requests_seconds_max, in the Panel title field in the right sidebar, you can enter a name for your panel.

Finally, click the Apply button at the top right corner and your panel is added to the dashboard. Do not forget to save the dashboard by means of the Save dashboard icon next to the Add panel icon.

Set some load to the application and see what happens to the metrics on the dashboard.

$ watch -n 5 curl http://localhost:8080/endPoint1
$ watch -n 10 curl http://localhost:8080/endPoint2

5. Conclusion

In this post, you have learnt how you can set up some basic monitoring for a Spring Boot application. It is necessary to use a combination of Spring Actuator, Micrometer, Prometheus and Grafana, but these are quite easy to set up and configure. This is of course just a starting point, but from here on, it is possible to expand and configure more specific metrics for your application.