Server Monitoring with Prometheus and Grafana setup in Docker and Portainer

I-Introduction
Portainer is a free Docker Container management tool with compact size and intuitive management interface, simple to deploy and use, allowing users to easily manage Docker host or Swarm cluster. This tool works on a container deployed on Docker Engine.

Grafana is a leading time-series, an open-source platform for visualization and monitoring. It allows you to query, visualize, set alerts, and understand metrics no matter where they are stored. You can create amazing dashboards in Grafana to visualize and monitor the metrics.

Prometheus is an open-source time-series monitoring system for machine-centric and highly dynamic service-oriented architectures. It can literally monitor everything. It integrates with Grafana very smoothly as Grafana also offers Prometheus as one of its data sources.

Prometheus Architecture: At its core, Prometheus has a main component called Prometheus Server, responsible for the actual monitoring work.

The Prometheus server consists of:

• Time Series Database that stores all the metric data like current CPU usage, memory usage etc.
• Data Retrieval Worker is responsible for all the data pulling activities from applications, services, servers etc. and pushing them into the database.
• HTTP Server API meant to accept queries for the stored data. The Server API is used to display the data in a dashboard or a Web UI.

II-Install Portainer with Docker Swarm on Linux
1. Introduction
Portainer can be deployed on top of any K8s, Docker or Swarm environment. It works seamlessly in the cloud, on prem and at the edge to give you a consolidated view of all your containers.
Portainer consists of two elements, the Portainer Server and the Portainer Agent. Both elements run as lightweight Docker containers on a Docker engine. This document will help you deploy the Portainer Server and Agent containers on your Linux environment. To add a new Linux Swarm environment to an existing Portainer Server installation, please refer to the Portainer Agent installation instructions.
To get started, you will need:

• The latest version of Docker installed and working
• Swarm mode enabled and working, including the overlay network for the swarm service communication
• sudo access on the manager node of your swarm cluster
• By default, Portainer will expose the UI over port 9443 and expose a TCP tunnel server over port 8000.
• The manager and worker nodes must be able to communicate with each other over port 9001.

2. Deployment
Portainer can be directly deployed as a service in your Docker cluster. Note that this method will automatically deploy a single instance of the Portainer Server, and deploy the Portainer Agent as a global service on every node in your cluster.
First, retrieve the stack YML manifest:

curl -L https://downloads.portainer.io/portainer-agent-stack.yml \
    -o portainer-agent-stack.yml

Then use the downloaded YML manifest to deploy your stack:

docker stack deploy -c portainer-agent-stack.yml portainer

Portainer Server and the Agents have now been installed. You can check to see whether the Portainer Server and Agent containers have started by running docker ps:

root@manager01:~# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS              PORTS                NAMES
59ee466f6b15   portainer/agent:2.11.1          "./agent"                About a minute ago   Up About a minute                        portainer_agent.xbb8k6r7j1tk9gozjku7e43wr.5sa6b3e8cl6hyu0snlt387sgv
2db7dd4bfba0   portainer/portainer-ce:2.11.1   "/portainer -H tcp:/…"   About a minute ago   Up About a minute   8000/tcp, 9443/tcp   portainer_portainer.1.gpuvu3pqmt1m19zxfo44v7izx

3. Logging In
Now that the installation is complete, you can log into your Portainer Server instance by opening a web browser and going to:

https://localhost:9443

Accessing the Portainer dashboard page, at the first interface after successful setup, the user will see the information of the successfully connected endpoint: the number of stacks, the number of containers, the number of volumes, the number of images and one Docker and host information.
...readmore...

III-Server Monitoring with Prometheus and Grafana setup in Docker and Portainer
Ref
1. Deploy Prometheus and Grafana

Create a new stack and define or paste the content of your docker-compose file in a box web editor.
image1
We need to deploy Prometheus and Grafana so the full content of docker-compose will look like this:

version: '3'

volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - /etc/prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

2. Configure Prometheus
Configuring Prometheus to monitor itself

$sudo mkdir /etc/prometheus

sudo vim /etc/prometheus/prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  # external_labels:
  #  monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

3. Third-Party Exporters
Node_exporter is one of the exports that help prometheus collect metrics of the machine to be monitored and installed on the target machine (the machine being monitored).
cAdvisor is to analyze usage, performance, and many other metrics from Container applications, providing users with an overview of all running containers.

At stack monitoring we need to add the content config of cadvisor and node_exporter to the docker-compose file then update the stack.

  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    # ports:
    #   - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped

In the Prometheus config file we need to add node-exporter and cadvisor to the scrape configuration as follows:

 # Example job for node_exporter
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

  # Example job for cadvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

4. Visualize data with Grafana
In the Grafana page we built above at localhost:3000
Here we log in with the user and password as admin.

At the Home homepage, we need to add the data source, here we will add the data source from Prometheus because what we need to do is visualize on the Grafana dashboard to receive the data poured in from Prometheus.

At the setting we need to reconfigure the HTTP tab at the URL is http://prometheus:9090

After connecting your data source, at Home we create a new dashboard => Add an empty panel.

The Metrics browser helps us query the data to visualize.

5. Import Grafana Dashboards
In the Grafana homepage https://grafana.com at the Dashboards tab we will see there are many dashboards. In this article we will learn about node exporters and cadvisor.

Get node exporter full by copying the ID and pasting it into the dashboards server we built.

Get a Cadvisor exporter by copying the ID and pasting it into the dashboards server we built.

IV-Alertmanager with Slack
1.Introduction

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
Readmore...

Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉

Comments

[githubmame]

Hi, first of all great post. THX
Secondly as a first information, I am doing my first steps with Grafana/Prometheus.
I did follow your steps and could creat my first dahboards.
Unfortunately I do not see any data flowing into the dashboards.
I believe I missed some basic steps to get data. I have set the datasource to prometheus.
Job and Host have the default entry NONE. Is this my missed configuration? And if so, where I can find these missed configuration to be added?
THX for support.
Cheers mame

[Adeleke Adeniji]

Great guide, would you be kind enough to do a follow up on Prometheus Alert Manager?

Thank you.

[Dalen Ward]

Excellent guide, exactly what I needed. Thank you!