HA snapshot bootstrap: the cutover gap

OriginChainDB Team · May 5, 2026

ha replication architecture durability failover

TL;DR - Most database failovers silently drop writes that landed while a new replica was being seeded. OriginChainDB ships a snapshot-based bootstrap: a new replica streams a consistent point-in-time snapshot, then catches up on every write committed after that point, before it is allowed to take over. That closes the seeding gap. It does not make failover lossless - replication to the standby is asynchronous, so an abrupt loss of the primary can still cost the most recent acknowledged writes. Both halves of that are worth saying out loud.

The problem with naive failover

The typical “high-availability” Postgres or MySQL setup looks like this: one writer, one or more streaming replicas, a vote-based failover that promotes a replica when the writer goes silent. It works, mostly. Until it doesn’t.

The failure mode that catches everyone is the cutover gap. A new follower joins the cluster. It needs to catch up to the current state. The naive approach is to copy the writer’s data files, then start streaming changes from the point the snapshot was taken at. Sounds reasonable. The problem is that “copying data files while writes are happening” is itself an operation on a moving target, and the point you stamp is always approximate.

When that follower gets promoted later, it serves traffic from a state that’s missing the writes that landed in the gap between snapshot start and snapshot end. Customers see “I just placed an order - where is it?” and the team spends three hours diffing replicas to figure out what got dropped.

This is the bug we refused to ship.

What OriginChainDB does instead

OriginChainDB’s follower bootstrap is a two-phase exchange between the writer and the joining follower:

Snapshot phase. The writer freezes its current commit position (call it W₀), takes a consistent snapshot of every shape - rows, vectors, indexes, the lot - and streams it to the follower. The follower stores this as its starting point.
Catch-up phase. The follower then asks the writer for every write committed between W₀ and the writer’s current position, applies them in order, and continues replicating live changes after that.

The key invariant: a follower is not allowed to vote in failover or accept reads until both phases have completed and its applied position is within a small bounded distance of the writer’s. There is no “promoted with stale state” path.

This means a freshly-joined follower can take over from the writer the same millisecond it finishes catching up, without the hole that a naive file copy leaves behind.

Why this is hard in practice

The hard part isn’t the algorithm - every database textbook describes something like it. The hard part is making the snapshot phase consistent across many shapes (rows + vectors + secondary indexes + commit position + sequence values) without taking a global lock that stalls the writer.

Our trick is the same substrate that handles atomic multi-shape writes: every write, no matter the shape, advances a single monotonic commit position. A “snapshot at position W₀” means: materialize the state as of W₀ and ship it. Because the materialization is a deterministic function of the committed history up to W₀, two followers that bootstrap from the same W₀ produce bit-identical state.

That last property - bit-identical materialization - is what lets followers agree on snapshot bytes during a write-heavy cluster join.

The chaos drill

Architecture is one thing; verified behavior is another. So we run the drill:

Spin up a 3-node cluster.
Start a writer that’s executing 10k writes/second across rows + vectors + indexes.
Snapshot-bootstrap a fresh follower mid-write.
Once the follower applies position == writer position, kill the writer.
Trigger failover. The follower is promoted.
Diff the promoted follower’s state against the committed writes the application reported success on.

The pass condition is that the bootstrap leaves no hole: every write the follower was told about before it declared itself caught up must be present after promotion. The drill ran on 2026-04-30 against the production build and passed end-to-end. Note what the drill does not prove - it kills the writer only once the follower has caught up. Kill it while the follower is behind and you lose the difference, which is exactly the asynchronous-replication tradeoff described below.

What this gives you

No seeding gap: the window a naive file-copy bootstrap leaves open - writes that land between snapshot start and snapshot end - is closed by construction.
Snapshot-bootstrap a follower against a hot writer without freezing the writer or pausing application traffic.
Self-healing replicas: a follower that fell too far behind is automatically re-snapshotted instead of replaying terabytes of change history.

What it does not give you

Being straight about the boundary:

This is not zero-data-loss failover. Replication to the standby is asynchronous: the primary streams committed changes continuously but does not wait for the standby before answering you. If the primary dies abruptly, the most recent acknowledged writes can be lost. How much is at risk depends on how far behind the standby had fallen.
What a 200 does guarantee is local durability: the write was flushed to durable storage on the primary before you got your response, so it survives a process crash or a host restart of that instance.
Cross-region replication. The writer and followers all live in one private network for now. Cross-region is on the roadmap after multi-writer lands.

The practical advice that follows: make your writes idempotent, and retry through a failover.

FAQ

What is snapshot bootstrap?

Snapshot bootstrap is the procedure a new database replica uses to catch up to the current writer’s state. In OriginChainDB it has two phases - copy a consistent snapshot at commit position W₀, then apply every write committed after W₀ - and the replica is gated from voting in failover until both phases complete.

How is this different from streaming replication?

Streaming replication assumes the replica already has a consistent base state. Snapshot bootstrap is what gets you that base state without freezing the writer. The two work together: bootstrap once, stream forever after.

Does the writer block during snapshot bootstrap?

No. The snapshot is materialized from the committed history up to position W₀, while the writer continues taking writes at positions W₀+1, W₀+2, … in parallel. The follower replays the gap during the catch-up phase.

How long does failover take?

The cutover itself is fast. The slow part is deciding the writer is really gone: the old writer’s claim has to expire and a grace window has to pass before the standby is eligible, and both of those are configured rather than fixed. Automatic promotion is off by default, and when it is enabled it refuses to promote a standby that is not fully caught up. That is the standard tradeoff in any automatic-failover system, and we would rather be slow than promote over a live primary.

What if all replicas are behind?

Failover requires enough replicas to be within a bounded distance of the writer. If everyone’s far behind, failover refuses and pages a human. Better to alert than to silently promote a stale follower.