GitOps for One GPU Box, Without Kubernetes: A SwarmCLI CD User Story

Guest post by Peter, infrastructure engineer. The setup described here runs SwarmCLI CD 1.3.0 in its open-source (Apache-2.0) -oss build, with charts rendered by swarmcli 1.14.0.

We're a young company, and one rule we set ourselves early on is that production changes go through Git. Our platform runs on Kubernetes with Argo CD, where that's easy to keep. "I'll just SSH in and fix it" isn't supposed to be how things change.

So when we decided to host our own LLM, I had a problem. The hardware was a single RTX 4090, passed through into a VM. The model was a quantised Qwen, about 126 GB on disk. (Yes, that's a lot more than 24 GB of VRAM. How it fits is a post of its own; this one is about how it gets deployed.) The job itself was simple: serve an OpenAI- and Anthropic-compatible API behind TLS and basic auth, so our platform could call it like any other service.

A Kubernetes cluster for one GPU was never really on the table. That's a control plane, a CNI, the NVIDIA GPU operator and another Argo CD instance, all so that one container can run on one card. We wrote it down as a decision: Argo CD stays with Kubernetes, and we don't bring in Kubernetes just to manage an appliance.

The other obvious option was a cron job running git pull && docker stack deploy. Most of us have seen that script somewhere. It works fine until someone changes a service by hand during an incident and nobody notices for three months. It also has no idea whether the thing it just deployed actually came up.

Docker Swarm already covered most of what we needed. It schedules services, stores secrets, rolls back failed updates, and can reserve a GPU much like it reserves memory. What was missing was the Argo CD part: something that watches Git and makes the machine match it. The SwarmCLI team had told me about SwarmCLI CD, and it turned out to be exactly that piece.

The setup

Everything the controller runs on the box is described in one file, applications.yaml, in a Git repository. It has two applications: a Traefik edge that handles TLS and auth, and the inference server. The inference entry looks like this:

- name: gpu-freetoken
  source:
    repoURL: https://<git-host>/<org>/<inference-repo>.git
    revision: main
    chart: { path: charts/freetoken, values: [deployments/<target>/values/freetoken.yaml] }
  allow:
    hostPaths: [/srv/llm/models, /srv/llm/cache]
    networks: [gpu-ingress]
  driftDetection: live
  syncPolicy: { automated: true, interval: 3m, wait: true, timeout: 40m }

If you've used Argo CD you can probably read that without the docs. SwarmCLI CD polls the repo every three minutes, renders the chart and brings the stack in line with it. A few of those fields turned out to matter more than I expected.

allow lists what the application may touch on the host, in this case two model directories and one network. Anything else the chart reaches for outside its own stack is refused at deploy time. The edge has its own list, and it's the only application allowed near the Docker socket, because Traefik needs it for service discovery:

- name: gpu-edge
  allow:
    hostPaths: [/var/run/docker.sock]
    networks: [gpu-ingress]
    configs: [gpu-edge-dynamic-<sha256-prefix>]
    secrets:
      - gpu-tls-certificate-<sha256-prefix>
      - gpu-tls-private-key-<sha256-prefix>
      - gpu-basic-auth-<sha256-prefix>

I like that this boundary sits in the same file a reviewer is already reading.

driftDetection: live compares the services that are actually running with what's in Git, not only with the last commit the controller deployed. A hand edit on the host would therefore show up as drift.

wait and timeout mean a sync only counts as done once the service is healthy. For this model that takes a while, as we found out on the first day.

The charts are SwarmCLI charts, which are basically Helm charts for Swarm stacks: templates plus values, with swarmcli charts lint and swarmcli charts template for CI. Our production values for the inference server come down to one digest-pinned image line. The image sits in a private registry, and SwarmCLI CD pulls it with a credential stored as a Swarm secret, so nobody has to run docker login on the host.

The GPU itself is a Swarm generic resource. The node advertises the card by its UUID in daemon.json, with nvidia as the default runtime and swarm-resource = "DOCKER_RESOURCE_GPU" in the NVIDIA runtime's config, and the inference chart reserves it the way it reserves memory:

deploy:
  replicas: 1
  placement:
    constraints: [node.labels.gpu.inference == true]
  resources:
    limits: { memory: 96G }
    reservations:
      memory: 80G
      generic_resources:
        - discrete_resource_spec: { kind: GPU, value: 1 }

Because nvidia is the default runtime, every other service (the controller, Traefik and the auth proxy) sets NVIDIA_VISIBLE_DEVICES=void. A second GPU reservation stays Pending instead of silently sharing the card.

The first day

We switched it on one Tuesday in September, at about half past five in the evening. The controller came up, found both applications out of sync with Git, and started deploying them. The inference service was created straight away. The edge failed with this:

config 'dynamic': file: reads a path on the controller's own filesystem, not the chart's; a rendered manifest is a string, so there is no chart directory for the path to resolve against — declare the config external: and have an operator create it on the swarm

Our Traefik chart loaded its dynamic config through a Compose file: reference. That works when you run docker stack deploy from a checkout, because the file is sitting right there. A controller that deploys rendered output has no checkout for the path to point into. I'd have worked that out eventually, but the message told me what was wrong, why, and what to do instead, which is more than most error messages manage.

The fix was to create the config outside the chart, named after a hash of its contents, and reference it as external:. Just under twenty-five minutes after the failure, the controller picked up the fix, and three minutes later the edge was healthy.

The model was still loading at that point. Before it serves anything, the server checks the hash of each of the model's 218 files and then loads the weights, which takes about 35 minutes from a cold start. The controller waited through all of it and only marked the application healthy at eleven minutes past six, once the model could actually answer. We had given it a 40-minute timeout, which is closer to 35 than I'm comfortable with, so that's on our list.

That same evening our platform talked to the endpoint through the new edge for the first time. TLS verified, anonymous requests got a 401, and the first completion came back.

Day two

Since then, changes to what runs on the box go through pull requests. A new inference image is a one-line change. With only one GPU, the old container has to stop before the new one can start, so an update means the model is offline for its 35-minute load. Swarm is set to roll back if the new version fails, and the controller waits for the whole thing and reports what happened.

Secrets need a bit more thought. Swarm secrets can't be edited, so we name each one after a hash of its contents. On 23 September we added a new API client. Ansible staged the new secret on the host, and the pull request swapped the old secret name for the new one. SwarmCLI CD picked it up on its next poll and rolled the edge. The old secret is still there, so undoing the change is a git revert.

Testing the controller's code in CI

This part only works because SwarmCLI CD is written as an ordinary Go module rather than shipped only as a binary.

Our CI imports the controller's own packages and runs our rendered charts through them:

• the function the controller uses to turn a chart into Swarm service specs, to check that the GPU reservation, the stop-first update order and the Traefik middleware label survive the conversion;
• the live drift comparison, to check that deleting the GPU reservation from a running service is detected as drift, and that restoring it converges cleanly;
• the allow-list check, to make sure a chart that mounts the Docker socket into the inference container is rejected by the same code that runs on the host.

We also run four of SwarmCLI CD's own upstream reconcile tests. None of it needs a GPU, a model download or even a Docker daemon. The test client panics if anything tries to talk to a real Docker API, so nothing can pass by accident. Here is the socket check, trimmed:

// Embed a nil API so any unexpected live API call panics instead of returning a fake success.
type offlineAPI struct{ client.APIClient }

func TestInferenceCannotBindDockerSocket(t *testing.T) {
    raw := `version: "3.8"
services:
  unsafe:
    image: alpine:3
    volumes: [/var/run/docker.sock:/var/run/docker.sock]
`
    _, err := compose.ConvertUnresolved(ctx, raw, "unsafe", offlineAPI{},
        application.Allow{HostPaths: []string{"/srv/llm/models", "/srv/llm/cache"}})
    if err == nil {
        t.Fatal("controller accepted forbidden socket bind")
    }
}

What we haven't done yet

Drift correction and rollback are configured, and tested in CI against the controller's own code, but we haven't yet broken the live box on purpose to watch them work. That's next, together with a full rebuild rehearsal. Sync failures currently go to the log and nowhere else, so alerting is on the list too.

SwarmCLI CD talks to Docker through the socket, which is effectively root on the host. We keep it on its own network with no published ports, but it's worth knowing going in. We're also due to upgrade to SwarmCLI CD 1.4, mostly for the new console and live log streaming.

Would I do it again?

Yes. For context, here are some numbers from our own hardware (so please don't read them as a benchmark). We get about 25 tokens a second once warm, around 3 seconds to the first token, and a 128K context window. The controller runs in 512 MB and one CPU, and we don't maintain any deploy scripts for the workloads.

What I care about most, though, is that the GPU box now works like the rest of our infrastructure. When I want to know what's running on it, or why something changed, I look in Git.

SwarmCLI CD is open source under Apache-2.0. The code, the releases and a getting-started guide are on GitHub, and local AI models on Docker Swarm covers running the model servers themselves as stacks.