Multi-region cold standby at $5/month
TL;DR — We redesigned our multi-region disaster recovery from a textbook hot-standby model to a cold-standby model and dropped per-tenant DR cost from ~$78/month to ~$5–15/month. The trade-off is ~10–15 minutes of RTO instead of ~5. For a single-customer pilot running on a managed cloud, that’s the right call. Here’s the math, the runbook excerpt, and the things this design can’t do.
If you’ve read any production-database playbook from the last decade, the hot-standby DR model is the default answer: in every region you might ever fail over to, keep a continuously-running follower replica. It tails the writer’s log in near-real-time. When the primary region goes down, you promote the follower, flip DNS, and the customer sees a 5-minute outage instead of a 5-hour one.
It is the right answer when you have many customers, real revenue per minute, and a contractual RTO measured in minutes.
It is the wrong answer when you have one pilot customer, the engine has been live for six weeks, and the standby host costs more per month than the active customer pays you.
The hot-standby model we started with
The first version of multi-region DR, shipped a few weeks ago, ran the textbook pattern:
- Per enrolled tenant, one dedicated host in the DR region, running the engine binary as a follower replica.
- The follower replicated the writer’s committed changes via cross-region encrypted object storage (replicated snapshots and incremental state).
- A failover health check, with a 90-second detection window, watched the primary’s
/healthendpoint and flipped a DNS failover record when the primary went silent. - On promotion, the operator promoted the follower to writer and traffic resumed inside 5 minutes.
This worked. It passed the chaos drill. It was the design we’d recommend to a Series A customer with five-nines aspirations.
It cost about $78/month per enrolled tenant, per region of failover coverage. Most of that was the follower host itself: a steady-state mid-size compute instance, sitting in the DR region, running the binary, consuming bytes from the replicated log, and serving exactly zero customer traffic.
Across a single canary tenant, that’s ~$78/month. Across ten tenants, $780/month. Across one hundred, $7,800/month — a real line item on the cloud bill, for a feature that fires once a year, maybe.
Why this is wrong for a single-customer pilot
The right way to think about DR cost is “cost per outage-second avoided.” The hot-standby model buys you ~5 minutes of RTO. The cold-standby model — the one we shipped this week — buys ~10–15 minutes. The delta is 5–10 extra minutes of customer impact per regional outage.
For a single pilot customer who hasn’t paid us a dollar yet, on an infrastructure that’s seen one (zero) regional outage in its lifetime, paying $78/month every month for the 5-minute RTO instead of the 15-minute RTO is the wrong shape of bet. We’re insuring against an event that hasn’t happened, for a customer who hasn’t paid us, at a steady-state cost that compounds with every tenant we onboard.
The bet that is the right shape: keep all the cheap scaffolding (pre-allocated routable IP, DNS failover record, role, firewall rules, alarms, the replicated-change storage) so a future failover is a one-line operator command. Drop the standby host. Conjure it at failover time from the pre-baked machine image we already keep in the DR region.
That’s the cold-standby refactor.
What we kept hot, what we made cold
The bill of materials, after the refactor:
| Component | State | Why |
|---|---|---|
| Cross-region change replication | Hot | This is the RPO. Drop it and we lose data on failover. |
| Pre-allocated routable IP in the DR region | Hot | DNS points at this IP throughout. Don’t want to reallocate at failover time — DNS would have to wait for propagation twice. |
| DNS failover record | Hot | Pre-wired with the failover IP as the secondary answer. Promotion just trips the health check. |
| Pre-baked machine image | Hot | Already in the DR region, tagged identically to the primary. Conjures the host in ~3 minutes. |
| Role, firewall rules, networking | Hot | All pre-allocated. Trivially cheap to keep around. |
| Health check, alarm, paging topic | Hot | Same as before. The DR-region health check is what trips DNS. |
| The standby host itself | Cold | This is the line item we cut. ~$58–115/month per tenant per region. |
The economics:
- Steady-state cost per enrolled tenant: ~$5–15/month for the scaffolding (routable IP, replication transit, alarm, health check, certificate storage).
- Failover-only cost: $78/month-equivalent (prorated to the actual hours the conjured host runs).
- Time to first byte served from DR on failover: 2–3 minutes of host launch + 60–90 seconds of bootstrap + 30 seconds for the engine to bind + DNS propagation. End-to-end RTO ~10–15 minutes with normal on-call response time.
For a single canary tenant, we’re now paying ~$5/month for DR coverage instead of ~$78/month. That’s the headline.
The operator runbook, excerpted
The failover procedure, after the refactor, has one new load-bearing step at the front: spin up the host. The shape of it:
# Add the tenant to the active-DR list. If multiple tenants are
# already in failover, APPEND — don't replace.
activate-dr --tenant <tenant-id> --active <tenant-id>The active-DR list is the switch. In steady state it’s empty: the DR-side host is provisioned per entry in that list, which evaluates to zero hosts and costs zero dollars. At failover time, the operator appends the tenant id, applies the change, and the host materializes from the pre-baked image, attaches to the pre-allocated IP, and starts up as a follower replica against the replicated change stream.
About 2–3 minutes for the host to launch, 60–90 seconds for the bootstrap script, and 30 seconds for the engine to bind to its HTTPS listener. The DNS failover record has already been pointing at the routable IP throughout — the DNS layer doesn’t need to change anything at promotion time. As soon as the engine answers /health, traffic flows.
Promotion is the second step: promote the follower to writer. That’s another ~30 seconds.
End-to-end RTO from the moment the primary region goes silent: ~10–15 minutes with normal on-call response, ~5 minutes if the operator was already at the terminal.
RPO is unchanged
The cold-standby refactor doesn’t change RPO. Our recovery point is still bounded by the lag of the cross-region change replication, and it assumes off-box change shipping is enabled for the tenant - it is opt-in and in preview, and without it the recovery point falls back to the daily storage snapshot — typically ≤60 seconds during normal traffic, with outliers up to a few minutes during multi-region congestion in the underlying infrastructure.
A customer on a high-write-rate workload at the moment of a regional outage should still expect MB-scale data loss within the RPO window. That’s a property of replicated object-storage-backed change shipping, not of whether the standby host is hot or cold. Dropping the host doesn’t make replication slower.
What the refactor can’t do
Three things, in order of importance:
- Sub-5-minute RTO is off the table. The host needs to boot. That’s 3 minutes minimum, on the best day, with everything pre-warmed. If your contractual SLA is “minutes” measured in the low single digits, cold standby is the wrong design and we’d need to go back to hot.
- Concurrent failover of many tenants takes longer. If ten tenants all need DR hosts at once, the apply runs ten host launches in parallel — but per-account provisioning rate limits mean the actual provisioning isn’t ten-at-once. We’ve sized the runbook for one tenant at a time. For an event where many tenants fail over together, the procedure would need to batch.
- No protection against a DR-region failure-domain outage at the same moment. v1 of the DR design already had this gap: the DR host lives in a single failure domain of the DR region. If that domain outages at the same instant as the primary region, the DR is down too. Probability is roughly 10^-7 per hour; we haven’t budgeted to fix it. The hot-vs-cold refactor doesn’t change this property.
The first one is the load-bearing trade. We named it explicitly in the runbook header, so a 3am on-call operator doesn’t get surprised: “the standby host is conjured at failover time. Saves ~$78/month per enrolled tenant per region. RTO is ~10–15 min instead of ~5 min. The new load-bearing first step at failover time is to spin up the host.”
When to flip back to hot
The refactor isn’t permanent. The switch to flip back to hot standby is one line: set the active-DR list to the full enrollment list, and the host gets pre-allocated again, paying the $78/month every month from then on.
The criteria we’d use to flip a specific tenant from cold to hot:
- Contractual RTO ≤ 5 minutes.
- Sustained write volume where the RPO outlier scenario (several minutes of replication lag during a busy minute) would cost more than $78/month in customer trust.
- A history of regional outages in the primary region — empirical not theoretical.
Until any of those is true for a given tenant, cold is the right shape. The DR design supports both modes; the decision is per-tenant, not global.
The math, one more time
The cost-per-outage-second-avoided math, written out:
- Hot standby: ~$78/month × 12 months = $936/year, for ~5-minute RTO.
- Cold standby: ~$5/month × 12 months = $60/year, for ~15-minute RTO.
If a regional outage happens once per year:
- Hot standby costs $936 to avoid ~10 minutes of customer impact = $93.60/minute avoided.
- Cold standby costs $60 to take the full ~15 minutes = $4/minute “saved” by not paying.
For a paying enterprise customer where $93.60/minute is cheap insurance, hot is right. For a single pilot canary, $4/minute saved is the right shape of bet — especially given how few regional outages there actually are in a year.
This is the discipline of pricing infrastructure against the customer paying for it, not against the customer you wish you had. Today’s tenants get cold. Tomorrow’s might get hot. The DR design supports both with a one-line change.
Try it
If you’re building on OriginChainDB, this DR posture is what your tenant inherits today: cold standby in a second region, cross-region change replication on, DNS failover wired, 10–15 minute RTO documented. Create a free database and the same posture applies to your tenant from day one. When you graduate to a tier that needs hot standby, the same module flips on with one variable.