Monitoring Linux Servers with Prometheus and Grafana

Prometheus and Grafana together form the industry-standard open-source monitoring stack. Prometheus collects and stores metrics, while Grafana provides beautiful dashboards for visualization. This guide covers setting up comprehensive monitoring for your Linux infrastructure.

Architecture Overview

The monitoring stack consists of Prometheus (metrics collection and storage), Node Exporter (Linux metrics), Alertmanager (alerting), and Grafana (visualization). Prometheus scrapes metrics from exporters at configured intervals.

Installing Prometheus

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus

# Create systemd service
sudo tee /etc/systemd/system/prometheus.service > /dev/null << EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now prometheus

Installing Node Exporter

# Download Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now node_exporter

Prometheus Configuration

# /opt/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: 
        - 'localhost:9100'
        - 'server2:9100'
        - 'server3:9100'

Installing Grafana

# Debian/Ubuntu
sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana

# Start Grafana
sudo systemctl enable --now grafana-server

# Access at http://localhost:3000
# Default credentials: admin/admin

Setting Up Dashboards

In Grafana, add Prometheus as a data source (http://localhost:9090), then import dashboard ID 1860 for Node Exporter metrics. This provides CPU, memory, disk, and network visualizations out of the box.

Essential PromQL Queries

# CPU usage percentage
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk usage percentage
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100

# Network traffic
rate(node_network_receive_bytes_total[5m])

Setting Up Alerts

# alert_rules.yml
groups:
  - name: node_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"

Conclusion

Prometheus and Grafana provide enterprise-grade monitoring for free. Start with basic metrics, then expand with additional exporters for databases, web servers, and applications as your monitoring needs grow.