Monitoring Stack: Prometheus, Grafana, DCGM, Loki for AI Servers

An AI server nobody is watching is a server silently throttling, leaking VRAM, accumulating ECC errors, or being OOM-killed at 03:00 — and you find out three days later when somebody asks why their fine-tune produced garbage. Most of the worst failure modes on a GPU box are invisible from the application: thermal throttle does not log to stderr, ECC corrections do not crash anything immediately, the OOM killer eats a process and the orchestrator restarts it cleanly. Without metrics on the hardware itself, you do not see any of this until it has already cost a week.

The standard sysadmin toolkit (htop, df, uptime, syslog) does not cover it. The GPU is the most expensive component in the chassis and the most failure-prone, and it has its own telemetry stack that needs explicit plumbing. This article is the opinionated build for that stack — Prometheus, Grafana, DCGM-exporter, node_exporter, Loki, and Alertmanager — packaged as a single docker-compose deployment we run on every K-AI server we ship.

The audience is somebody standing up a 4-GPU or 8-GPU AI server who can write a docker-compose file and wants the answer to "what should I actually monitor and at what threshold should I alarm."

The standard stack in 2026

Five components do almost all the work. Everything else is a bolt-on.

Total well under 0.5% of one CPU core on a 96-core EPYC and roughly 700 MB RAM. On a 256 GB / 8-GPU server this is a rounding error. The argument "monitoring steals cycles from training" stopped being true around 2019.

The one architectural decision worth making upfront: run the stack on a separate management VM or a small box, not on the GPU server itself. A dedicated mini-PC, a NUC, an EPYC service node, or a VM on the lab hypervisor — anything that is not the GPU box. Two reasons: when the GPU server crashes (out-of-memory kernel panic, PSU trip, thermal shutdown) you still want the metrics history to diagnose what happened, and you do not want a runaway training job competing with Prometheus for memory and triggering its own alert. DCGM-exporter and node_exporter run on the GPU host (they have to — they read local devices); Prometheus, Grafana, Loki, and Alertmanager run on the mgmt VM and scrape inward.

DCGM-exporter — the one thing you cannot skip

NVIDIA's Data Center GPU Manager (DCGM) is the supported, authoritative source for GPU telemetry. The dcgm-exporter container exposes DCGM metrics in Prometheus format on port 9400. It is the single most important component in the stack, and it is the thing nvidia-smi polling will never replace.

Install via the NGC container (nvcr.io/nvidia/k8s/dcgm-exporter, currently 4.x as of mid-2026) or the upstream Helm chart for Kubernetes. On bare Docker, one docker run --gpus all --rm is enough; the daemon then publishes ~80 metrics on :9400/metrics. The ones that actually matter, ranked by how often they catch real problems:

The throttle counters are the killer feature. DCGM_FI_DEV_THERMAL_VIOLATION is a monotonic counter of nanoseconds the GPU spent throttled. Compute its rate over a 5-minute window and you have an exact answer to "is my server thermally limited right now." Without it you are guessing from temperature alone, which lies — a 4090 will throttle at 83 °C on a sustained workload regardless of what the temperature panel shows.

Two known DCGM-exporter quirks worth knowing in 2026: some XID codes (notably XID 62) are not always surfaced through DCGM_FI_DEV_XID_ERRORS, and the metric is a gauge of the last XID seen — so it can fail to reset after a recovery without restarting the exporter. The mitigation is to also watch the kernel ring buffer via Loki for the literal NVRM: Xid string (more on that below). Belt and suspenders.

For pure consumer cards (RTX 4090, 5090) some DCGM data-center features are partial — MIG is not present, NVLink is absent on Blackwell consumer parts, certain ECC counters return zero. DCGM still works and reports what is exposed; the lightweight community alternative nvidia_gpu_exporter (which scrapes nvidia-smi) is a viable fallback if you do not want DCGM's dependency chain. For Pro 6000 Blackwell, L40, and L4 — anything in the data-center / pro line — use DCGM, not the wrapper.

node_exporter — the system half

GPU metrics tell only half the story. node_exporter covers the rest:

The "system RAM exhausted, OOM killer takes vLLM, container restarts cleanly" failure mode is caught here, not by DCGM. Watch node_memory_MemAvailable_bytes and alarm when it drops below 5% of total. On Linux the killer fires near 0% by default, but by then your process is dead.

Prometheus — config, retention, sizing

Prometheus is the time-series database and scrape orchestrator. The defaults are reasonable; the two settings worth changing on day one are scrape interval and retention.

A 15-second scrape interval is the right starting point for an AI server: fast enough to catch a thermal spike before it becomes a sustained throttle, slow enough that the time-series cost stays modest. Retention defaults to 15 days; 30 days is a better number for a build-guide context where you may compare today's behaviour to last month's tuning change.

Storage budget at 15 s scrape, ~80 DCGM metrics × N GPUs + ~400 node_exporter metrics + ~50 vLLM metrics: roughly 1.5–2 GB per week, so 30 days lands at 6–10 GB on an 8-GPU box. Set both retention by time and by size as a guardrail:

command:
  - "--storage.tsdb.retention.time=30d"
  - "--storage.tsdb.retention.size=20GB"

Either limit triggers compaction; whichever is hit first wins. A 100 GB allocation gives you over a year of headroom without thinking.

Grafana dashboards — start with the prebuilt

Do not build dashboards from scratch. The community has done it.

Import 12239 and 1860 first. They cover ~90% of what you actually want to look at and have been refined over years. Build your own only after a month, when you know which panels you actually open. The minor modifications worth making for K-AI hardware are: setting the GPU temperature threshold panels to Blackwell-appropriate ranges (5090 throttle near 87 °C, RTX Pro 6000 closer to 90 °C), and adding a per-PCIe-slot panel showing DCGM_FI_DEV_PCIE_LINK_GEN and DCGM_FI_DEV_PCIE_LINK_WIDTH so you can see at a glance if a riser has dropped from x16 to x8.

Loki — logs that explain what the metrics show

Metrics tell you what changed; logs tell you why. Loki is Grafana's log database; Promtail is the shipper that tails files and sends them. Point Promtail at:

• /var/log/syslog and journalctl -k — kernel OOM messages, NVIDIA driver complaints, XID dumps
• /var/log/nvidia-installer.log — driver install state
• vLLM container stdout (via the Docker JSON log driver)
• Application logs from anything the customer runs

The single highest-value query in Grafana's Explore tab is:

{job="syslog"} |= "NVRM:"

This surfaces every NVIDIA driver message in the kernel ring buffer — XID errors, fan failures, fallen-off-the-bus events, GSP RPC timeouts. Pair it with the DCGM_FI_DEV_XID_ERRORS Prometheus alert and you have both the structured metric and the unstructured detail in one pane.

We do not ship logs off-host by default. Loki keeps them locally with 30-day retention; the on-call SSH-tunnels into Grafana when needed. Customers wanting centralised logs across multiple servers can point Loki at S3 or run a regional instance — that is a deployment question, not an architecture one.

vLLM and SGLang metrics — instrument the app, not just the metal

DCGM tells you the GPU is busy. It does not tell you whether inference requests return in 200 ms or 2 s. For that you instrument the serving layer.

vLLM exposes /metrics natively on the same port as the OpenAI API. With default settings the endpoint at :8000/metrics publishes Prometheus metrics with the vllm: prefix (which becomes vllm_ after Prometheus scrape). The ones worth scraping:

The combination vllm:num_requests_waiting > 0 and DCGM_FI_DEV_GPU_UTIL < 90% means the queue is backing up while the GPU is idle — usually a tokenizer or scheduler bottleneck, not a compute one. You only see that with both metrics in the same dashboard, which is the whole point of unifying app and hardware telemetry in one Prometheus.

NVIDIA NIM exposes the same vLLM metrics under /v1/metrics without renaming them, so your existing vLLM dashboards and alert rules port over to a NIM-served deployment unchanged. SGLang publishes a similar set on its own port (default 30000); llama.cpp's HTTP server exposes a smaller subset. Triton publishes its own taxonomy. Whatever you serve, scrape the application — not just the hardware.

Alertmanager rules — the actual ones we ship

Dashboards are pretty. Alerts are useful. The rules below are the ones that have caught real problems on real customer hardware.

groups:
  - name: gpu
    interval: 30s
    rules:
      - alert: GPUTempCritical
        expr: DCGM_FI_DEV_GPU_TEMP > 87
        for: 30s
        labels: { severity: critical }
        annotations:
          summary: "GPU {{ $labels.gpu }} thermal critical ({{ $value }} °C)"

      - alert: GPUThermalThrottling
        expr: rate(DCGM_FI_DEV_THERMAL_VIOLATION[5m]) > 0
        for: 1m
        labels: { severity: warning }

      - alert: GPUECCUncorrectable
        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0
        labels: { severity: critical }
        annotations:
          summary: "GPU {{ $labels.gpu }} uncorrectable ECC — schedule replacement"

      - alert: GPUXIDError
        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        labels: { severity: critical }

      - alert: GPUPowerEnvelopeExceeded
        expr: DCGM_FI_DEV_POWER_USAGE > 590  # 5090 nominal 575 W; alarm above sustained ceiling
        for: 5m
        labels: { severity: warning }

      - alert: GPUIdleDuringWork
        expr: DCGM_FI_DEV_GPU_UTIL < 5 and DCGM_FI_DEV_FB_USED > 1024
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "GPU {{ $labels.gpu }} idle with VRAM allocated — likely hung"

  - name: system
    interval: 30s
    rules:
      - alert: HostMemoryLow
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.05
        for: 2m
        labels: { severity: critical }

      - alert: OOMKillerFired
        expr: increase(node_vmstat_oom_kill[5m]) > 0
        labels: { severity: critical }

      - alert: ContainerRestartLoop
        expr: rate(container_start_time_seconds[15m]) > 3
        for: 10m
        labels: { severity: warning }

  - name: serving
    interval: 30s
    rules:
      - alert: vLLMQueueBacklog
        expr: vllm:num_requests_waiting > 10
        for: 5m
        labels: { severity: warning }

      - alert: vLLMHighLatency
        expr: histogram_quantile(0.95, rate(vllm:e2e_request_latency_seconds_bucket[5m])) > 10
        for: 5m
        labels: { severity: warning }

Opinionated calls in those rules:

• 87 °C critical for GPU temperature. A 5090 throttles in the high 80s on most boards. If your room cannot keep cards under 80 °C at sustained load you have an HVAC problem, not a software one.
• Any uncorrectable ECC pages the on-call. A single double-bit ECC event invalidates whatever was in that VRAM page — a training step is wrong, an inference result is wrong. The card needs replacement, not a reboot.
• GPUIdleDuringWork is the deadlock detector. > 1 GiB allocated and < 5% util for 10 minutes means something is stuck. Catches CUDA-side hangs that the application happily ignores.
• OOM killer alert is the most-missed rule in any deployment. Linux quietly killing your training job and Docker restarting it cleanly is the failure mode that wastes the most engineer-hours per year. Make it loud.
• Container restart loop catches NIM and vLLM restart cycles that look healthy from outside (the container is "running") but are actually crashing on every model load.

Chassis and storage — IPMI and SMART

DCGM stops at the GPU. node_exporter stops at the OS. The rest of the chassis — fans, PSUs, ambient temperature, storage health — needs two more exporters.

ipmi_exporter (prometheus-community) talks to the BMC via IPMI/RMCP and surfaces chassis-level telemetry: per-fan RPM, PSU input voltage and current, system event log entries, ambient inlet temperature, BMC watchdog state. On a Supermicro or Bone64c chassis with a working BMC, this is half a day of work and catches things DCGM literally cannot see — a failing PSU on the dual-PSU 8-GPU build shows up as a voltage sag in the ipmi sensor data minutes before a GPU pops to XID 79. Run it on the host (it needs /dev/ipmi0) or remotely with stored BMC credentials using the multi-target exporter pattern.

smartctl_exporter (or the older smart_exporter) reads NVMe and SATA SMART attributes: media wear-out indicator, available spare, temperature, error counts. AI workloads are brutal on NVMe — dataset loaders, checkpoint writes, and HuggingFace caches push enterprise wear targets in months on consumer-grade drives. The metric to alarm on is nvme_available_spare dropping below 20% and nvme_percentage_used (the wear indicator) climbing above 80%. Drive failure is the second most common hardware fault on a busy AI server after the riser/PSU class.

A complete K-AI monitoring deployment includes both. The marginal cost is minutes; the value the first time a fan fails silently or a Samsung 990 Pro hits its DWPD ceiling is hours.

A real docker-compose for the mgmt VM

What we actually deploy on the management box. Adjust host.docker.internal (or use the GPU server's hostname/IP) to point Prometheus at the GPU host's exporter ports.

version: "3.8"

networks:
  monitoring: { driver: bridge }

volumes:
  prometheus_data:
  grafana_data:
  loki_data:

services:
  prometheus:
    image: prom/prometheus:v3.0.0
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--storage.tsdb.retention.size=20GB"
      - "--web.enable-lifecycle"
    ports: ["9090:9090"]
    networks: [monitoring]

  grafana:
    image: grafana/grafana:11.4.0
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_pw
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    secrets: [grafana_pw]
    ports: ["3000:3000"]
    networks: [monitoring]

  alertmanager:
    image: prom/alertmanager:v0.28.0
    restart: unless-stopped
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports: ["9093:9093"]
    networks: [monitoring]

  loki:
    image: grafana/loki:3.3.0
    restart: unless-stopped
    command: -config.file=/etc/loki/loki.yml
    volumes:
      - ./loki.yml:/etc/loki/loki.yml:ro
      - loki_data:/loki
    ports: ["3100:3100"]
    networks: [monitoring]

secrets:
  grafana_pw: { file: ./secrets/grafana_pw.txt }

On the GPU host itself (separate compose, on the GPU server):

services:
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:4.5.1-4.8.0-ubuntu22.04
    restart: unless-stopped
    runtime: nvidia
    environment: [NVIDIA_VISIBLE_DEVICES=all]
    cap_add: [SYS_ADMIN]
    ports: ["9400:9400"]

  node-exporter:
    image: prom/node-exporter:v1.9.0
    restart: unless-stopped
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
    ports: ["9100:9100"]

  ipmi-exporter:
    image: prometheuscommunity/ipmi-exporter:v1.10.0
    restart: unless-stopped
    privileged: true
    volumes: ["/dev/ipmi0:/dev/ipmi0"]
    ports: ["9290:9290"]

  smartctl-exporter:
    image: prometheuscommunity/smartctl-exporter:v0.13.0
    restart: unless-stopped
    privileged: true
    ports: ["9633:9633"]

  promtail:
    image: grafana/promtail:3.3.0
    restart: unless-stopped
    volumes:
      - /var/log:/var/log:ro
      - ./promtail.yml:/etc/promtail/promtail.yml:ro
    command: -config.file=/etc/promtail/promtail.yml

The Prometheus scrape config on the mgmt VM:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets: ["k-ai-01.lan:9400", "k-ai-02.lan:9400"]

  - job_name: node
    static_configs:
      - targets: ["k-ai-01.lan:9100", "k-ai-02.lan:9100"]

  - job_name: ipmi
    static_configs:
      - targets: ["k-ai-01.lan:9290", "k-ai-02.lan:9290"]

  - job_name: smart
    static_configs:
      - targets: ["k-ai-01.lan:9633", "k-ai-02.lan:9633"]

  - job_name: vllm
    metrics_path: /metrics
    static_configs:
      - targets: ["k-ai-01.lan:8000"]

That is a working stack for a one- to three-server lab. Past four or five servers, switch the Prometheus scrape config to file-based service discovery or — if you are on Kubernetes — to the GPU Operator's bundled DCGM-exporter Helm chart and Prometheus Operator's ServiceMonitor CRDs.

OpenTelemetry, Pixie, eBPF — the 2026 picture

A note on the moving pieces, because the question always comes up.

OpenTelemetry is the CNCF standard for observability instrumentation, and as of early 2026 all three signal types (metrics, traces, logs) are stable. The OTel Collector pipeline is replacing vendor-specific agents in many organisations as the universal telemetry router. For an AI server, the pragmatic answer in 2026 is: keep Prometheus for metrics (DCGM-exporter and node_exporter speak Prometheus natively, and PromQL is what your alert rules are in), and add an OTel Collector if and when you need distributed tracing across an inference-serving graph (request → API gateway → vLLM → tool call → vector DB → response). For a single-server install with one model behind nginx, OTel adds cost without value. The 71% of organisations that "use both" use Prometheus for the metals and OTel for app traces — a sane split.

Pixie is a Kubernetes-native observability tool that uses eBPF to gather kernel-level metrics, traces, and logs without code changes. The 2026 development worth tracking is the eBPF-on-GPU work (bpftime, eGPU, NVIDIA's open kernel modules) that extends eBPF instrumentation into GPU kernels via runtime PTX injection. This is research-grade today and not part of any production stack we ship. For now, DCGM is the supported answer.

Continuous profiling (Grafana Phlare, Pyroscope) is a third bolt-on worth knowing about. For inference workloads where you control the framework code (custom vLLM patches, tokenizer optimisations) it tells you which functions burn the CPU. For pure model serving with stock containers, it is overkill.

What breaks (monitoring edition)

Predictable failure modes of the stack itself, ranked by how often they bite:

• Prometheus disk fills up. Default retention is 15 days; we set 30 with a 20 GB ceiling. Past 60 days on a busy box, plan for tens of GB. Set retention by both time and size.
• DCGM-exporter loses GPUs after a driver upgrade. Symptom: all GPU panels go blank. Fix: restart the container; re-run nvidia-ctk runtime configure if the runtime moved.
• Alertmanager not configured for the receiver you actually use. People stand up Prometheus and Grafana, forget to point Alertmanager at email or Slack, and discover six months later that no alert ever fired. Test by deliberately tripping GPUTempCritical with a stress workload, or use amtool alert add to fire a synthetic alert on day one.
• Cardinality explosion from per-request labels. Do not label vLLM metrics with user_id or request_id. Prometheus is not designed for that — you will OOM the database.
• Grafana password leak via compose env. Use Docker secrets, not GF_SECURITY_ADMIN_PASSWORD in the environment block. docker inspect and logs leak environment values.
• XID gauge that does not reset. As noted, the DCGM-exporter XID gauge can stick at the last seen value after recovery. Combine the metric with Loki's NVRM: syslog tail to avoid false confidence.

The honest take

Most labs install Grafana once, build a dashboard the team admires for a week, and never look at it again. The dashboard is a satisfying artifact; it is not what catches your problems. Alerts are what catch your problems. Set up Alertmanager with a real receiver (email, Slack, PagerDuty) and a small set of high-precision rules — GPU temp, ECC double-bit, XID, OOM killer, container restart loop — and tune them until they fire only when something is actually wrong. Do this first. Then build dashboards for the post-incident debugging story.

The other half of the honest take: at single-server Kentino scale, you will get an alert from this stack maybe once a month, and most months it will be a false positive you tune out. That is the system working. The value is the alert that fires the day a riser starts cooking, two days before the card pops to XID 79, when there is still time to re-seat a cable instead of RMA a GPU. That single event pays for the entire monitoring effort, and it pays for it many times over.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.