The plan was the usual three extra environments: staging, UAT and development, each one a single machine running the stack in Docker. Production on its own, and three boxes sitting behind it for testing.
That is a normal setup and it works. It also stopped making sense, and the reason is about how the work itself changed.
With agents doing a large share of the development, branches stopped being something you finish quickly on your own. A feature branch is now a place where several things get attempted in parallel, and the questions people ask about it changed. Does this behave right in a real browser. Does the queue pick up the job. Does the migration run against actual data. Those questions want a running stack attached to the branch, not a shared machine that reflects whatever the last person deployed to it.
Once that is the requirement, a staging tier is the wrong shape. Three permanent machines are three copies of a stack to maintain in parallel, and they still only show you whatever was deployed to them last. A stack per branch gives you exactly the thing that was missing, and it costs nothing when nobody is using it.
So the plan became one permanent environment and an ephemeral stack for every branch we are working on.
What one preview looks like
A branch gets its own Kubernetes namespace, preview-<slug>, on a spot node pool that scales from zero. Inside it:
- The four application services: web, GraphQL, the Java write API, and the agent service.
- Postgres, single pod, no persistent volume.
- Redis, single pod.
- Typesense, single pod.
- Temporal in its local development mode.
- Sockudo for websockets, with its own public ingress.
- Prometheus, namespace scoped, six hour retention.
- MinIO for object storage.
That is about ten pods per branch. Nothing in there is highly available and nothing is meant to be. The whole namespace is disposable, and the database is seeded fresh on every deploy, so there is no state worth preserving between teardowns.
The URL shape is <app>-<branch-slug>.domain.com. The wildcard certificate only covers one label, so the app name and the branch slug have to collapse into a single label. That constraint is not obvious until you try to put the app name at a second level and the certificate does not match.
The isolation problem nobody thinks about first
The easy part is running a copy of each service. The hard part is the shared infrastructure, because production runs exactly one of each of those, and pointing previews at them is a bad idea for reasons that differ per service.
Redis is the clearest example. The GraphQL read API caches responses in Redis, and the rules in the repository say every write must invalidate the cache explicitly. If previews share production’s Redis, a preview branch doing a test write invalidates real cache entries, and a preview branch’s half finished feature writes cache entries that production then serves to real users.
The same argument applies to Typesense, where a preview branch could reindex or pollute the search collection, and to Temporal, where a preview workflow could collide with a production workflow’s identity.
So every one of those runs per branch. The decision that saved the most work was choosing the lightweight versions: single pod, no persistence, and for Temporal specifically its own local development mode rather than a cluster that needs an external database.
Prometheus turned out to be worth running per branch for a different reason. In preview it is scoped to a namespaced role that can only see its own namespace, so it is strictly less privileged than the production instance, which uses a cluster wide role. And it is genuinely useful, because the thing you want on a branch before merging is the OOM kill and the error rate you did not notice in local development.
Secrets, split into three categories
This is the part I expected to be tedious and that turned out to be the most interesting design decision.
Not all secrets are the same kind of secret, and treating them uniformly causes problems in both directions.
Boot time secrets. The app key, the auth secret, the JWT secret. The application cannot start without them, but nothing outside the namespace needs to trust them. A random value is exactly as good as production’s real one. So these are generated per branch, and made stable across upgrades with a lookup so a helm upgrade does not rotate the key underneath a running pod.
Internal service secrets. A mutual auth token that web, the API, GraphQL and the agent share inside the namespace. Also random per branch, also generated, because nothing external trusts it.
Real external credentials. SSO machine to machine, a legacy product API key, OpenAI keys for embeddings, agent calls and speech to text. A random value cannot fake these, because they authenticate against real systems.
For that third category there were two options, both bad. Reusing production’s real credentials means any preview branch can make real calls billed to us and attributed to production. Generating a per branch development tier credential is not possible, because no such tier exists. One of the integrations has no non production instance at all.
What we landed on is a third path: a separate secrets manager entry, with its own IAM role scoped only to the preview path, shared across all preview branches as one common value. A branch can authenticate. The credential is structurally isolated from production. There is exactly one place to rotate it.
The bug this design nearly shipped is worth mentioning. The preview deploy workflow pointed its base image at the same mutable latest tag that production’s deploy workflow pushes. Every preview build was silently overwriting the base layer production’s next deploy would build on top of. The fix was separate ECR repositories for all the preview images, each with its own short lived cleanup policy. Six repositories, one line of reasoning, and a failure mode that would have been extremely confusing to debug from a production incident.
Careful with what previews are allowed to send
Two more decisions that came out of asking “what could this accidentally do”.
Preview environments cannot send outbound mail. The mail credential is simply not wired into the preview values, so the service is configured with no way to reach a mail provider. The same goes for Google and Slack OAuth login, and for error reporting to the production project. Rather than crash looping on missing configuration, those integrations are absent, which is the correct state for a throwaway stack.
The other one is login. Single sign on in preview would mean a preview branch talking to a real identity provider, which is exactly the production attributed external call we are trying to avoid. So previews force email login with a dedicated flag, and that flag is separate from the one the end to end test suite uses. Reusing a test scoped flag for a different caller is the kind of shortcut that becomes a bug the day the test flag grows unrelated behaviour.
The lifecycle
Previews are created deliberately rather than automatically.
A plain push does not create a preview from scratch. A first job checks whether the namespace exists, and if it does not, everything downstream skips with a message telling you to run the workflow manually. That was a change from the original design, where any push created a stack. Small work in progress commits should not spin up ten pods.
Once a preview exists, ordinary pushes keep updating it. That is the common case and it stays automatic.
Teardown happens three ways. Deleting the branch triggers it natively, which lines up with our branch workflow tooling that already removes the GitHub branch when a feature finishes. There is also a manual trigger, for freeing resources without deleting the branch. And a daily job deletes any preview whose last activity label is older than five days. The reaper only removes the running stack, never the branch, so a push brings it back.
What it costs
The node pool is spot c6a.large instances, mixed family with capacity optimised allocation, scaling from zero to a configurable maximum. The stack sits behind a taint so nothing else lands on those nodes.
Per branch the requests come to roughly 1.3 vCPU and 2.3 GiB across those ten pods. At the ceiling we sized for, fifteen concurrent branches, that is around eleven to thirteen nodes, which works out to roughly $140 to $165 a month. At five or six concurrent branches it is closer to $50 to $60. When nothing is open it is zero.
The comparison that matters is not against the idle floor of three permanent machines. It is against what ten to fifteen permanent environments would have cost as the team grew, which is a fixed bill regardless of whether anyone is using them.
Two failure modes to know about
The first is the node pool. If previews exceed its capacity, the deploy still reports success. Helm succeeds, the manifests apply, and the pods sit pending forever with no error surfaced to the developer. The symptom is a URL that loads nothing, and the diagnosis is kubectl get pods in that namespace. Silent, but at least it is deterministic.
The second is scale from zero. When the pool drops to zero nodes, cluster autoscaler can only scale it up if the autoscaling group carries the node template tags describing the labels and taints new nodes will have. Without those tags it cannot tell that an empty group could satisfy the pending pods, and it simply never scales.