←Back to Blog

June 9, 2026

Docker Swarm Monitoring with Prometheus and Grafana: The Stack, the Alerts and What to Watch (2026)

A deployable monitoring stack for Docker Swarm: Prometheus with service discovery, cAdvisor and node-exporter on every node, the Docker daemon’s own metrics, Grafana provisioned from a config, the seven alerts that matter, and where AI-assisted summaries help.

Docker Swarm Monitoring with Prometheus and Grafana: The Stack, the Alerts and What to Watch (2026)

Monitoring a swarm is easier than monitoring most things, because the swarm already knows where everything is. A global service puts one exporter on every node, an overlay network gives Prometheus a DNS name that resolves to all of them, and the daemon can publish its own metrics. This is the stack we run, as a file you can deploy, followed by the alerts worth waking up for and a short, honest section on what AI adds.

The stack file

Four services: Prometheus on a manager with a local volume, cAdvisor and node-exporter on every node, Grafana wherever, all on one overlay network that nothing else joins. Configuration comes in as Swarm configs so the stack file is the whole deployment.

Terminal
services:
  prometheus:
    image: prom/prometheus:v3.5.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d
    deploy:
      placement:
        constraints: [node.role == manager]
      resources:
        limits: { memory: 2G }
    configs:
      - source: prometheus_yml
        target: /etc/prometheus/prometheus.yml
      - source: alert_rules
        target: /etc/prometheus/alerts.yml
    volumes:
      - prometheus_data:/prometheus
    networks: [monitoring]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    command: ['--docker_only=true', '--housekeeping_interval=15s']
    deploy:
      mode: global
      resources:
        limits: { memory: 256M }
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks: [monitoring]

  node-exporter:
    image: prom/node-exporter:v1.9.1
    command: ['--path.rootfs=/host']
    deploy:
      mode: global
      resources:
        limits: { memory: 128M }
    volumes:
      - /:/host:ro,rslave
    networks: [monitoring]

  grafana:
    image: grafana/grafana:12.1.0
    deploy:
      placement:
        constraints: [node.labels.grafana == true]
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_SERVER_ROOT_URL: https://grafana.example.com
    secrets: [grafana_admin]
    configs:
      - source: grafana_datasource
        target: /etc/grafana/provisioning/datasources/prometheus.yml
    volumes:
      - grafana_data:/var/lib/grafana
    networks: [monitoring, traefik-public]
    # publish through your reverse proxy; do not expose 3000 on the mesh

volumes:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    driver: overlay
  traefik-public:
    external: true

configs:
  prometheus_yml:
    file: ./prometheus.yml
  alert_rules:
    file: ./alerts.yml
  grafana_datasource:
    file: ./grafana-datasource.yml

secrets:
  grafana_admin:
    external: true

Pin the image tags to what you tested; the ones above are the current releases at the time of writing. Prometheus and Grafana are pinned to a node because their data is a local volume that does not follow a task; the HA database guide explains why that is the right call rather than a distributed filesystem.

Discovery: tasks.<service>

prometheus.yml needs no list of nodes. On an overlay network, tasks.cadvisor resolves to the IP of every cAdvisor task, so a scrape target that names it scrapes every node, and a node that joins tomorrow is scraped tomorrow:

Terminal
global:
  scrape_interval: 15s

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: cadvisor
    dns_sd_configs:
      - names: ['tasks.cadvisor']
        type: A
        port: 8080

  - job_name: node-exporter
    dns_sd_configs:
      - names: ['tasks.node-exporter']
        type: A
        port: 9100

  - job_name: docker-daemon
    dns_sd_configs:
      - names: ['tasks.node-exporter'] # one entry per node; the daemon listens on the host
        type: A
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: __address__
        replacement: '${1}:9323'

The third job scrapes the Docker daemon itself, which publishes task counts, engine state and Swarm manager metrics once daemon.json on each node carries "metrics-addr": "0.0.0.0:9323" and the daemon is restarted. Bind it to the node's private address rather than 0.0.0.0 if the interface is reachable from outside the swarm; the metrics are not secret, but they are not for the internet either.

Grafana's data source is provisioned from a file, so a fresh Grafana comes up already pointed at Prometheus:

Terminal
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

Deploy with docker stack deploy -c monitoring-stack.yml monitoring after creating the admin secret and labelling the Grafana node.

The seven alerts that page

Everything else is a dashboard. These go in alerts.yml:

Terminal
groups:
  - name: swarm
    rules:
      - alert: NodeDown
        expr: up{job="node-exporter"} == 0
        for: 3m
        labels: { severity: page }
        annotations: { summary: 'node-exporter on {{ $labels.instance }} unreachable for 3 minutes' }

      - alert: DiskFilling
        expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"}) < 0.2
        for: 10m
        labels: { severity: page }
        annotations: { summary: 'root filesystem on {{ $labels.instance }} under 20% free' }

      - alert: TaskRestarting
        expr: changes(container_start_time_seconds{container_label_com_docker_swarm_service_name!=""}[1h]) > 3
        for: 0m
        labels: { severity: page }
        annotations:
          {
            summary: '{{ $labels.container_label_com_docker_swarm_service_name }} restarted more than 3 times in an hour',
          }

      - alert: ContainerNearMemoryLimit
        expr: container_memory_working_set_bytes{container_label_com_docker_swarm_service_name!=""} / container_spec_memory_limit_bytes{container_label_com_docker_swarm_service_name!=""} > 0.9
        for: 5m
        labels: { severity: warn }
        annotations:
          {
            summary: '{{ $labels.container_label_com_docker_swarm_service_name }} above 90% of its memory limit',
          }

      - alert: SwarmManagerQuorumAtRisk
        expr: swarm_manager_nodes{state="ready"} < swarm_manager_nodes{state="ready"} offset 30m
        for: 5m
        labels: { severity: page }
        annotations: { summary: 'fewer manager nodes ready than 30 minutes ago' }

      - alert: NodeMemoryPressure
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
        for: 10m
        labels: { severity: warn }
        annotations: { summary: '{{ $labels.instance }} above 90% memory for 10 minutes' }

      - alert: HighLoad
        expr: node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 2
        for: 15m
        labels: { severity: warn }
        annotations: { summary: '{{ $labels.instance }} load is twice its core count' }

Two of these deserve a note. TaskRestarting reads cAdvisor's container start time with the Swarm service label cAdvisor attaches, which is what turns "a container restarted" into "this service is crash-looping". SwarmManagerQuorumAtRisk reads the daemon's own swarm_manager_nodes gauge, which is why the daemon job is worth the daemon.json change: the manager count is the one number a swarm cannot afford to lose track of, and only the daemon knows it.

The eighth alert is a service below its desired replica count. cAdvisor cannot see desired state, so it needs either the daemon metrics on a manager, or a small exporter that reads docker service ls; several community exporters do this, and SwarmCLI shows it live in the services view, which is where most teams look first anyway.

Route them with Alertmanager, or with Grafana's built-in alerting reading the same rules; both work, and the choice is which UI you want to manage silences in.

Dashboards

Three, and no more until you need them:

  • Nodes: CPU, memory, disk and network per node from node-exporter, with the node's Swarm role in the legend.
  • Services: memory and CPU per service from cAdvisor, grouped by container_label_com_docker_swarm_service_name, and restarts per service over the last day.
  • Swarm: manager count, node count by state, tasks by state, from the daemon job.

Grafana's community dashboard library has starting points for node-exporter and cAdvisor; import one, then delete two thirds of its panels. A dashboard nobody reads is a dashboard.

Logs

Metrics say that something is wrong; logs say what. Loki with a Promtail or Alloy agent as a global service, reading /var/lib/docker/containers, gives Grafana one place for both, with the Swarm service name as a label. It is a second stack and a second retention decision, so add it once the metrics stack has been up for a month and you know what you are missing. Until then, docker service logs and SwarmCLI's log view across replicas cover the incident.

Where AI helps, and where it does not

An alert that fires at 3 a.m. with a summary line is better than one without, and a language model is good at writing that line: give it the alert, the last fifty log lines of the service and the last three deploy events, and ask for a one-paragraph account of what probably happened and what to check first. A webhook receiver that does this and posts the paragraph beside the alert is a hundred lines of code, and with Ollama on the swarm nothing leaves your network.

What it is not good at is deciding thresholds or restarting things. Anomaly detection on a swarm of a dozen services produces more noise than the seven rules above, because the workloads are few enough that a person already knows what normal looks like. And an automated remediation that restarts a service on an alert is a loop waiting to happen; the restart belongs in restart_policy and update_config, where Swarm already does it deliberately. Use the model to explain, and let the scheduler heal.

Common concerns, answered

How do I monitor Docker Swarm with Prometheus? Run cAdvisor and node-exporter as global services so every node has one, put them and Prometheus on an overlay network, and point Prometheus at tasks.cadvisor and tasks.node-exporter with dns_sd_configs, which Swarm resolves to every task IP. Add the Docker daemon’s own metrics with metrics-addr in daemon.json.

What should I alert on in a Docker Swarm? A node down, a disk over 80 percent, a task restarting repeatedly, a container near its memory limit, manager quorum at risk, a node under memory pressure, and sustained high load. Those are the seven rules this article ships; a service below its desired replica count is the eighth, and it needs an exporter that can see desired state. Everything else is a dashboard.

Can Grafana discover Docker Swarm services? Grafana reads from Prometheus, and Prometheus does the discovery: the tasks.<service> DNS name on an overlay network returns every task of that service, so a global exporter is scraped on every node without listing nodes anywhere.

What to watch on day one

  • up for every job equals the node count, or a node is not scraped.
  • The daemon job returns swarm_manager_nodes; if it does not, metrics-addr is missing on that node.
  • Disk on every node, because the first thing a monitoring stack does is fill a disk with metrics, and 30 days of retention on a busy swarm is a few gigabytes.

From there, the best-practices checklist has the operational rules the alerts above assume, and the ten most common issues are what the alerts will mostly be telling you about.