DEV Community

Cover image for How to set up a system monitoring for linux server using node exporter, prometheus & grafana
coder7475
coder7475

Posted on

How to set up a system monitoring for linux server using node exporter, prometheus & grafana

To set up system monitoring for a Linux VPS using Node Exporter, Prometheus, and Grafana Cloud Dashboard, follow these steps:

1. Install and Configure Node Exporter

Node Exporter collects system metrics from the Linux VPS.

  1. Download Node Exporter:
wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode
  1. Extract and Move the Binary:
tar -xvf node_exporter-1.9.0.linux-amd64.tar.gz
sudo mv node_exporter-1.9.0.linux-amd64/node_exporter /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode
  1. Create a System User:
sudo useradd --system --no-create-home --shell /bin/false node_exporter
Enter fullscreen mode Exit fullscreen mode
  1. Set Up Node Exporter as a Service: Create the service file /etc/systemd/system/node_exporter.service:
sudo vim /etc/systemd/system/node_exporter.service
Enter fullscreen mode Exit fullscreen mode

Add the following content:

[Unit]
Description=Node Exporter
Documentation=https://prometheus.io/docs/guides/node-exporter/
After=network.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple

ExecStart=/usr/local/bin/node_exporter

# Security Setting
ProtectHome=true 
NoNewPrivileges=true
ProtectSystem=strict 
PrivateTmp=true 
PrivateDevices=true 
ProtectKernelTunables=true 
ProtectKernelModules=true 
ProtectControlGroups=true 
RestrictNamespaces=true 
RestrictRealtime=true 
RestrictSUIDSGID=true 

# Resource limits 
CPUQuota=10% 
MemoryLimit=50M 
LimitNOFILE=4096
LimitNPROC=4096
LimitCORE=100M 

# Restart configuration 
Restart=always 
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
  1. Start and Enable Node Exporter:
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter
Enter fullscreen mode Exit fullscreen mode
  1. Verify Node Exporter: Ensure it is running on port 9100:
ss -aplnt | grep 9100
Enter fullscreen mode Exit fullscreen mode

Ensure it is exporting metrices:

curl http://localhost:9100/metrics
Enter fullscreen mode Exit fullscreen mode

2. Install and Configure Prometheus

Prometheus scrapes metrics from Node Exporter.

  1. Download Prometheus:
wget https://github.com/prometheus/prometheus/releases/download/v3.2.0/prometheus-3.2.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode
  1. Extract and Move Files:
# Extract 
tar -xvf prometheus-3.2.0.linux-amd64.tar.gz

# Move Prometheus Binaries to System Path
sudo mv prometheus-3.2.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-3.2.0.linux-amd64/promtool /usr/local/bin/

# Create Configuration and Data Directories
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mkdir -p /var/lib/prometheus/data
sudo mkdir -p /var/lib/prometheus/query_log

# Move Configuration File
sudo mv prometheus-3.2.0.linux-amd64/prometheus.yml /etc/prometheus/
Enter fullscreen mode Exit fullscreen mode
  1. Edit the Configuration File (/etc/prometheus/prometheus.yml): Run the command below to open the file:
sudo vim /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Add the Node Exporter's target:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 5s
    scrape_timeout: 5s
Enter fullscreen mode Exit fullscreen mode
  1. Set Up Prometheus as a Service: Create a system user:
sudo useradd -rs /bin/false prometheus
Enter fullscreen mode Exit fullscreen mode

Set required file permission:

sudo chown prometheus:prometheus /etc/prometheus 
sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chmod -R 755 /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Create /etc/systemd/system/prometheus.service:

sudo vim /etc/systemd/system/prometheus.service
Enter fullscreen mode Exit fullscreen mode

Add this content:

[Unit]
Description=Prometheus Monitoring System
Documentation=https://prometheus.io/docs/prometheus/latest/installation/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
WorkingDirectory=/var/lib/prometheus

ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/data \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --query.lookback-delta=5m \
    --enable-feature=memory-snapshot-on-shutdown

# Security Settings
ProtectHome=true
NoNewPrivileges=true
ProtectSystem=false
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

# Resource Limits
CPUQuota=10%
MemoryLimit=500M
LimitNOFILE=65535
LimitNPROC=4096
LimitCORE=infinity

# Restart Configuration
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode
  1. Start and Enable Prometheus:
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode
  1. Verify Prometheus:

Run the command below to verify its running :

ss -aplnt | grep 9090
curl localhost:9090/metrics
Enter fullscreen mode Exit fullscreen mode

3. Set Up Grafana Cloud Dashboard

Grafana visualizes metrics collected by Prometheus.

  1. Sign Up for Grafana Cloud:
    • Create an account on Grafana Cloud (if you don’t already have one).
    • Obtain your organization’s unique Prometheus remote write URL and API key.
  2. Configure Prometheus for Remote Write: Edit /etc/prometheus/prometheus.yml to include the remote write configuration:
remote_write:
  - url: "https://prometheus-blocks-prod-us-central1.grafana.net/api/v1/write"
    basic_auth:
      username: "<your-grafana-cloud-instance-id>"
      password: "<your-api-key>"
Enter fullscreen mode Exit fullscreen mode
  1. Restart Prometheus: Apply the changes by restarting Prometheus:
sudo systemctl restart prometheus
Enter fullscreen mode Exit fullscreen mode
  1. Import Dashboards in Grafana Cloud:

4. Verify Setup

  • Access Grafana Cloud at https://<your-grafana-instance>.grafana.net.
  • View metrics such as CPU usage, memory consumption, disk I/O, etc., in real-time on your imported dashboards.

This setup provides comprehensive monitoring for your Linux VPS with minimal manual intervention123.

References


  1. https://www.liquidweb.com/blog/install-prometheus-node-exporter-on-linux-almalinux/ 

  2. https://www.idevopz.com/step-by-step-guide-to-install-prometheus-nodeexporter-grafana-on-a-ubuntu-20-04/ 

  3. https://betterstack.com/community/guides/monitoring/visualize-prometheus-metrics-grafana/ 

Top comments (0)