Grafana + Prometheus: metrics and dashboards

Monitoring stack via Docker Compose

version: '3.9'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
    volumes:
      - grafana-data:/var/lib/grafana

  node-exporter:
    image: prom/node-exporter:latest
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro

volumes:
  prometheus-data:
  grafana-data:

prometheus.yml: metrics collection configuration

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'nginx'
    static_configs:
      - targets: ['nginx-exporter:9113']

Useful PromQL queries

# CPU utilization
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# RAM usage %
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk free
node_filesystem_avail_bytes{mountpoint="/"} / 1024 / 1024 / 1024

Diagnosing common errors

The Grafana dashboard shows No data even though the Prometheus container is running — most often Prometheus cannot reach the exporter, or Grafana itself points at the wrong datasource.

# In the Prometheus UI (http://YOUR_IP:9090/targets)
node (0/1 up)
Get "http://node-exporter:9100/metrics": dial tcp: lookup node-exporter on 127.0.0.11:53: no such host

This error means the containers cannot see each other by service name — usually because they are declared in separate docker-compose.yml files without a shared network. All three services — prometheus, grafana, and node-exporter — must be in the same compose file or attached to the same external networks.

The error context deadline exceeded in the Prometheus logs during a scrape means the exporter answers slower than scrape_timeout (equal to scrape_interval by default). This is usually a symptom of an overloaded server, or an exporter that itself waits on a slow data source — for example mysqld-exporter when the database is unreachable.

Comparing metric collection methods

When this stack is overkill

• A single small VPS with one site — deploying three containers and learning PromQL just for basic CPU and RAM charts is overkill; Netdata with a ready dashboard out of the box is enough.
• You only need external site availability checks, not internal server metrics — that is a job for uptime monitors like UptimeRobot, not Prometheus.
• The team is not ready to maintain a metrics store — without tuning retention and TSDB resources, Prometheus on an active project quickly eats disk space and needs ongoing attention.

How much disk Prometheus eats

By default Prometheus keeps data for 15 days (--storage.tsdb.retention.time=15d). A practical size estimate is roughly 1-2 bytes per sample after compression; at a 15-second scrape interval with a couple hundred time series on a small three-service stack that is usually 200-500 MB over two weeks, but it grows non-linearly as the number of exporters and labels increases.

services:
  prometheus:
    image: prom/prometheus:latest
    command:
      - '--storage.tsdb.retention.time=15d'
      - '--storage.tsdb.retention.size=2GB'
      - '--config.file=/etc/prometheus/prometheus.yml'

The retention.size setting is a safety net against disk overflow: Prometheus will delete old data blocks earlier than retention.time if the total size exceeds the limit, instead of failing with an out-of-space error.

Verifying the result

1. Open http://YOUR_IP:9090/targets — every job must show status UP, not DOWN.
2. Open Grafana at http://YOUR_IP:3000, add a Prometheus datasource at http://prometheus:9090 (the compose service name, not localhost), and import a dashboard by ID.
3. Run the test PromQL query up — it must return 1 for every working exporter.
4. Confirm data survives a container restart: docker compose restart prometheus grafana, after which old graphs must remain thanks to the named volumes.

If you just need a quick resource overview without Docker and PromQL, see the article on Netdata. The stack itself is convenient to deploy with Docker Compose if you are not yet familiar with that tool.