GPUs on Docker Swarm: NVIDIA Container Toolkit, Generic Resources and Stack Files

Running a model server across a few GPU machines is one of the most common reasons people pick Docker Swarm in 2026: the whole control plane fits in a few dozen megabytes, so the VRAM and the CPU go to the model. The scheduling part is not obvious, because Swarm's GPU support does not use the devices block Docker Compose users know. It uses generic resources: a node says what it has, a service says what it needs, and the scheduler matches them. This guide sets that up end to end and shows how to verify it.

1. The runtime: NVIDIA Container Toolkit on every GPU node

Install the driver and the NVIDIA Container Toolkit on each node that has a GPU, then let it configure Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker info | grep -i runtimes

The last line must list nvidia. A quick test on the node itself, outside Swarm:

docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If that prints your GPUs, the node is ready. If it does not, nothing below will help; fix the driver and the toolkit first.

2. Advertise the GPUs and make nvidia the default runtime

A Swarm service cannot ask for --gpus; that flag belongs to docker run. Instead, the node advertises each GPU as a generic resource, and the daemon uses the NVIDIA runtime for every container so the device is exposed when the scheduler places a task there. Both go into /etc/docker/daemon.json on each GPU node:

{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }
  },
  "node-generic-resources": [
    "NVIDIA-GPU=GPU-1f2d3c4b-0000-0000-0000-000000000001",
    "NVIDIA-GPU=GPU-1f2d3c4b-0000-0000-0000-000000000002"
  ]
}

nvidia-smi -L prints the UUIDs. One entry per GPU, all under the same resource name (NVIDIA-GPU here; the name is yours to choose, but it must match the reservation exactly). Restart Docker and confirm the node advertises them:

sudo systemctl restart docker
docker node inspect <node> --format '{{json .Description.Resources.GenericResources}}'

For a node advertising two GPUs by UUID, the output is a list of named resources:

[
  { "NamedResourceSpec": { "Kind": "NVIDIA-GPU", "Value": "GPU-1f2d3c4b-0000-0000-0000-000000000001" } },
  { "NamedResourceSpec": { "Kind": "NVIDIA-GPU", "Value": "GPU-1f2d3c4b-0000-0000-0000-000000000002" } }
]

An empty list or null means the daemon did not read the setting: a JSON error in daemon.json, or no restart.

Then label the node so you can also constrain placement by hand:

docker node update --label-add gpu=true <node>

3. Reserve a GPU from a service

On the command line:

docker service create --name ollama \
  --generic-resource "NVIDIA-GPU=1" \
  --constraint 'node.labels.gpu == true' \
  --mount type=volume,source=ollama,target=/root/.ollama \
  --publish 11434:11434 \
  ollama/ollama

Or in a stack file, which is how you will actually run it:

services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.gpu == true
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
                kind: 'NVIDIA-GPU'
                value: 1
    volumes:
      - ollama:/root/.ollama
    ports:
      - '11434:11434'
    networks:
      - ai-net

volumes:
  ollama:

networks:
  ai-net:
    driver: overlay
    attachable: true

The scheduler now places the task only on a node with an unreserved advertised GPU, marks that GPU as taken, and the NVIDIA runtime exposes it to the container. Two replicas need two advertised GPUs somewhere in the swarm; a value: 2 reservation needs one node with two free.

4. What about deploy.resources.reservations.devices?

The Compose Specification has a devices block under reservations, with driver: nvidia, count and capabilities: [gpu], and it is what Docker Compose honours on a single host. It is not what Docker's Swarm documentation describes for services, and whether docker stack deploy on your engine passes it through depends on the version. If you use it, verify rather than assume:

docker service inspect <service> --format '{{json .Spec.TaskTemplate.Resources}}'

If the reservation is absent from that output, the block was dropped and the task will be scheduled anywhere, GPU or not. Generic resources are the route that works on every engine that runs Swarm mode.

5. Verify the task got the device

From the task's container:

docker exec $(docker ps -q -f name=ollama) nvidia-smi

Or, in SwarmCLI, open a shell into the task and run nvidia-smi there. For a model server, the second check is the model actually loading onto the GPU. After a first request, ollama ps inside the container should report the processor as GPU:

NAME             ID              SIZE      PROCESSOR    UNTIL
llama3.1:8b      46e0c10c039e    6.2 GB    100% GPU     4 minutes from now

100% CPU in that column, with the task running on a GPU node, means the device was not exposed to the container: check default-runtime on that node.

6. Troubleshooting

Common concerns, answered

Does Docker Swarm support GPUs? Yes, through generic resources: each node advertises its GPUs in daemon.json and a service reserves one with --generic-resource or the generic_resources key in a stack file. The scheduler then places the task only on a node with a free GPU, and the NVIDIA runtime exposes it to the container.

Why does my GPU service stay Pending on Docker Swarm? Almost always because no node advertises the resource the service reserved: node-generic-resources is missing or misspelled in daemon.json, the daemon was not restarted, or the value name does not match the reservation exactly. docker node inspect shows what a node advertises.

Can I use deploy.resources.reservations.devices with docker stack deploy? That block is Compose Specification syntax that Docker Compose honours on a single host. Whether docker stack deploy honours it depends on the engine version, so verify with docker service inspect after deploying. The route Docker documents for Swarm is generic resources.

7. Putting it to work

With scheduling in place, the local AI models guide covers Ollama, Open WebUI and vLLM as stacks, and the ollama chart in the SwarmCLI charts repository deploys the server in one command. Give the GPU nodes a label, keep the model volumes local to them, and put the chat UI on the CPU nodes so a model load never competes with the interface. The Definitive Docker Swarm Guide covers placement and update mechanics in general.