Building a GitOps Application Provisioning Pipeline
Every time OpenHost provisions a tenant application, several things have to happen in order: a manifest has to exist, it has to be committed somewhere durable, and a cluster has to be told to converge on it. Doing that inline, inside the request that asked for it, is the easiest way to build a system that looks correct in a demo and falls over in production.
Why the request/response cycle is the wrong place for this
An HTTP request has a lifetime measured in seconds and a single, synchronous failure mode: the connection drops, and the client is left not knowing whether anything happened at all. Provisioning work — generating manifests, committing to Git, waiting on a scheduler — doesn't fit that shape. It needs to survive restarts, retries, and slow dependencies without losing track of what it was doing.
The queue is what turns "hope this works" into "we know exactly what state this job is in."
The pipeline
The API's only job on a provisioning request is to validate it and write desired state to MongoDB. From there, a BullMQ job carries the work forward:
FluxCD is the last hop, and the most important one: it doesn't just apply the manifest once, it continuously reconciles the cluster against the Git repository. If a pod is deleted, a config drifts, or a node is replaced, Flux converges the cluster back to the declared state without anyone re-running a deploy script.
What this bought us
Failed jobs retry automatically with backoff instead of silently disappearing. Git history gives every deployment a diff and a rollback path. And because status is pushed over Socket.IO as the job progresses, users see "queued → provisioning → reconciling → healthy" in real time instead of staring at a spinner.
The tradeoff is complexity: a queue, a worker, and a reconciler are three more things that can fail compared to one script. In practice, they fail in ways that are visible and recoverable — which is the whole point.