←Back to Blog

May 8, 2026

How to Run Local AI Models on Docker Swarm in 2026 (Ollama, OpenWebUI, vLLM)

Ollama on Docker Swarm, plus Open WebUI and vLLM: how to run local AI models across a swarm with GPU placement, high availability and SwarmCLI, without a Kubernetes control plane taking the VRAM.

How to Run Local AI Models on Docker Swarm in 2026 (Ollama, OpenWebUI, vLLM)

Running large language models (LLMs) locally gives you full privacy, zero recurring API costs, and full control. In 2026, Docker Swarm remains one of the simplest and most reliable ways for homelabs, small teams, and even light production to run these models with high availability and easy scaling.

This guide shows you how to deploy Ollama, OpenWebUI, and vLLM on a Docker Swarm cluster in 2026. Whether you are running on high-end NVIDIA servers or a cluster of Apple Silicon Macs, you'll find the right configuration here.

Hardware Compatibility & GPU Support

Before we dive in, it’s important to understand how Docker Swarm interacts with your hardware.

PlatformGPU AccelerationBest Use Case
Linux + NVIDIA✅ Native (NVIDIA Container Toolkit)Production, heavy inference, large models.
macOS (Intel/M-Series)❌ CPU Only (in Docker VM)Testing, development, small models.
Windows + NVIDIA✅ WSL2 (Experimental in Swarm)Power-user workstations.

[!IMPORTANT] A Note for Mac Users: Docker Desktop on macOS runs inside a Linux virtual machine. Currently, this VM cannot access the Mac's GPU (Metal) for Ollama or vLLM. If you are following this on a Mac, you must use the CPU-mode configurations provided in this guide.

GPU reservations on Docker Swarm: devices versus generic resources

The stack files below reserve a GPU with the Compose Specification's deploy.resources.reservations.devices block, which is what Docker Compose honours on a single host:

Terminal
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

count and device_ids are mutually exclusive; count: all takes every matching device. Whether docker stack deploy honours that block depends on your engine version, so check the result with docker service inspect <service> --format '{{json .Spec.TaskTemplate.Resources}}' after deploying. The route Docker documents for Swarm scheduling is generic resources: each node advertises what it has, and the service reserves it.

On every GPU node, install the NVIDIA Container Toolkit, make nvidia the default runtime, and advertise the GPUs by UUID (nvidia-smi -L prints them) in /etc/docker/daemon.json:

Terminal
{
  "default-runtime": "nvidia",
  "runtimes": { "nvidia": { "path": "nvidia-container-runtime" } },
  "node-generic-resources": ["NVIDIA-GPU=GPU-1f2d3c4b-0000-0000-0000-000000000001"]
}

Then reserve one from the service, either on the command line or in the stack file:

Terminal
docker service create --name ollama --generic-resource "NVIDIA-GPU=1" ollama/ollama
Terminal
deploy:
  resources:
    reservations:
      generic_resources:
        - discrete_resource_spec:
            kind: 'NVIDIA-GPU'
            value: 1

Swarm then schedules the task only on a node with a free advertised GPU and the runtime exposes it to the container. Combine it with a placement constraint on a node label such as node.labels.gpu == true for anything you want pinned to particular hardware.

Why Docker Swarm for Local AI in 2026?

Kubernetes is powerful but often overkill for local or small-scale AI workloads. Docker Swarm offers:

  • Simplicity — Use familiar docker stack deploy commands.
  • Low overhead — Ideal for homelabs or small clusters (3–8 nodes).
  • Native Docker Compose compatibility — Easy migration from single-node setups.
  • Built-in load balancing and service discovery — Great for serving multiple AI models.
  • Resilience — Automatic failover for your chat interface and inference services.

Many users now prefer Swarm for local AI because it strikes the perfect balance between simplicity and reliability. Check out our comparison between SwarmCLI, Portainer, and native Docker to see why it's the preferred lightweight choice for modern clusters. With tools like SwarmCLI, managing these workloads becomes even more intuitive, providing real-time visibility into your GPU nodes and service health.

Prerequisites

  • A Docker Swarm cluster (3+ nodes recommended for HA; single node also works).
  • NVIDIA GPUs with drivers installed and NVIDIA Container Toolkit on every node.
  • At least 16GB RAM (32GB+ recommended for larger models).
  • Adequate storage for models (models can be 5–150+ GB each).
  • Basic familiarity with Docker Compose/stack files.

Enable Swarm (if not already done):

Terminal
docker swarm init
# On worker nodes:
docker swarm join --token <token> <manager-ip>:2377

Part 1: Deploying Ollama on Docker Swarm

Ollama is the easiest way to run popular models like Llama 3.2, Mistral, Gemma, and Phi-3.

Basic Ollama Stack File (ollama-stack.yml)

Terminal
version: '3.9'

services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      mode: global # One instance per node
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - '11434:11434'
    networks:
      - ai-network
    restart: unless-stopped

volumes:
  ollama_data:

networks:
  ai-network:
    driver: overlay
    attachable: true

Deploy it:

For Linux with NVIDIA GPUs, use the exact file above. For macOS or CPU-only Linux, you must remove the reservations block (lines 55-60) from the YAML file, otherwise the deployment will fail with an "invalid device" error.

Terminal
# Using standard docker
docker stack deploy -c ollama-stack.yml ollama

# Or verify node health first with SwarmCLI
swarmcli   # the tasks view shows each replica, its node and its last error

Pull models (run on any manager):

Terminal
docker exec -it <ollama-container> ollama pull llama3.2
docker exec -it <ollama-container> ollama pull mistral

Tip: For better performance, pin a specific version (e.g., ollama/ollama:0.5.x) instead of latest.

Part 2: OpenWebUI – The Beautiful Chat Interface

The production version of this section, with Traefik, secrets, pinned volumes and a second model server, is Open WebUI with Ollama on Docker Swarm.

OpenWebUI gives you a ChatGPT-like frontend with RAG, voice input, image understanding, and multi-user support.

Terminal
version: '3.9'

services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      mode: global
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ollama_data:/root/.ollama
    networks:
      - ai-network

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    deploy:
      replicas: 2 # Scale as needed
      resources:
        limits:
          memory: 4G
    ports:
      - '3000:8080'
    environment:
      - OLLAMA_BASE_URLS=http://ollama:11434
      - WEBUI_AUTH=true
    volumes:
      - openwebui_data:/app/backend/data
    networks:
      - ai-network
    depends_on:
      - ollama

volumes:
  ollama_data:
  openwebui_data:

networks:
  ai-network:
    driver: overlay
    attachable: true

Deploy with:

Terminal
docker stack deploy -c openwebui-stack.yml ai

You will see the services in SwarmCLI

Terminal
swarmcli   # the tasks view shows each replica, its node and its last error

[!TIP] Mac Tip: Just like with standalone Ollama, if you are running on a Mac, remove the resources: reservations: block from the ollama service in your openwebui-stack.yml.

Access the UI at http://your-swarm-ip:3000.

[!TIP] SwarmCLI Pro Tip: Use swarmcli service inspect ai_openwebui to quickly verify the replica status and ensure your environment variables are correctly propagated across the cluster. For a full list of service commands, see our CLI Documentation.

Part 3: High-Performance Inference with vLLM

Challenges on Swarm: vLLM needs --ipc=host or large shared memory for multi-GPU. Use constraints to pin services to GPU-rich nodes.

[!WARNING] vLLM on Mac: vLLM is highly optimized for NVIDIA GPUs. While it can run on CPU, it is not recommended for Mac Docker setups. Use Ollama instead for a much better experience on macOS.

vLLM Stack Example

Terminal
version: '3.9'

services:
  vllm:
    image: vllm/vllm-openai:latest
    deploy:
      replicas: 1 # Start with 1, scale later
      placement:
        constraints:
          - node.labels.gpu == high # Label your powerful nodes
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - --model meta-llama/Llama-3.1-8B-Instruct
      - --gpu-memory-utilization 0.85
      - --max-model-len 8192
    ports:
      - '8000:8000'
    volumes:
      - hf_cache:/root/.cache/huggingface
    networks:
      - ai-network
    ipc: host # Best effort; alternatives exist

volumes:
  hf_cache:

networks:
  ai-network:
    driver: overlay

Note on IPC: Swarm has limited support for --ipc=host. You can use a large tmpfs mount to /dev/shm as a workaround for tensor parallelism.

Part 4: Production Best Practices in 2026

Special: Apple Silicon (M1/M2/M3) Performance Tips

If you are running this stack on a Mac Swarm for development:

  1. Allocate More RAM: LLMs are memory-hungry. Go to Docker Desktop Settings > Resources and allocate at least 8GB-12GB of RAM to the VM.
  2. Use Quantized Models: Stick to 4-bit or 5-bit quantization (e.g., llama3.2:3b-instruct-q4_K_M). They run significantly faster on CPUs.
  3. Keep it Small: Models like Phi-3 Mini or Gemma 2B will feel much snappier in CPU mode than a full Llama 70B.

1. Load Balancing & High Availability

Add Traefik or Nginx as a reverse proxy in front of your services.

2. Model Management & Caching

  • Use shared volumes or GlusterFS/Ceph for model storage across nodes.
  • Pre-pull popular models on all nodes.

3. Monitoring & Observability

Deploy Prometheus + Grafana + Node Exporter + NVIDIA DCGM for GPU metrics. SwarmCLI provides a built-in TUI that gives you an instant overview of your cluster's resource consumption, making it easier to spot nodes that are under-utilized or over-burdened by heavy LLM inference.

4. Security

  • Enable authentication in OpenWebUI.
  • Use Swarm secrets for API keys.
  • Expose services only via a secure proxy. For enterprise environments, we recommend using SwarmCLI Business Edition's RBAC Proxy for automated mTLS and centralized access control (see the Proxy Setup Guide).

5. Scaling Strategy

  • Ollama: Global mode (one per node).
  • OpenWebUI: Scale replicas based on users.
  • vLLM: Add more replicas on GPU nodes with placement constraints.

Common Issues & Troubleshooting (2026)

  • "Invalid device reservation" error — This happens on Mac or Linux systems without NVIDIA drivers. Remove the reservations block from your stack file.
  • GPU not detected — Verify NVIDIA Container Toolkit and resource reservations.
  • Model download slow — Use a shared cache volume.
  • High latency — Try quantization (Q4_K_M, Q5_K_M) and adjust context length.
  • Swarm networking issues — Ensure services are on the same overlay network.

Advanced: Multi-Model Setup & RAG

You can run multiple specialized models (coding, general chat, vision) and route requests intelligently using a lightweight gateway or OpenWebUI’s built-in features.

Conclusion: Your Private AI Platform in 2026

With Docker Swarm and SwarmCLI, you can build a robust, private AI infrastructure that’s easy to maintain and scales with your needs. Whether you’re a homelabber running Llama 3.2 or a small team serving internal tools with vLLM, this stack delivers excellent performance without Kubernetes complexity.

Next Steps:

  1. Start with the Ollama + OpenWebUI stack.
  2. Add monitoring.
  3. Experiment with vLLM for higher throughput.