Start free
← All posts

RAG at 10M documents: version skew

OriginChainDB · May 26, 2026
rag vector-search hybrid-search engineering

In a RAG stack built on four databases, a single chunk’s embedding, BM25 posting and row text can end up committed at three different versions, because nothing spans the four systems transactionally. The Stage 7 faithfulness check then grades the answer against three versions of the same chunk and returns a number that no longer means what you think it means. On a single-substrate engine the skew is not reachable, because every shape of the chunk commits or none does.

A widely-shared Medium piece walks through a production RAG architecture for 10 million documents. The 10 stages - chunk, embed, hybrid retrieve, ANN plus rerank, confidence gate, constrained generate, cite, verify, semantic cache, trace - are the right blueprint. The article gets the retrieval math right, the confidence thresholds right, the faithfulness check right. We have nothing to add to that part.

What it doesn’t discuss is the failure mode the stack itself introduces. Almost every published RAG architecture at this scale assumes four separate systems:

Four databases. Every document write must succeed against all four. Every document update must invalidate all four. And nothing in those four systems is jointly transactional with the others.

This is fine when you have 10,000 documents and writes are infrequent. It becomes a class of silent bug at 10 million.

The bug

A user updates a regulatory filing. Your ingest pipeline re-chunks it, re-embeds each chunk, and writes:

  1. Pinecone gets the new embedding for chunk doc-471:12.
  2. Elasticsearch gets the new BM25 posting for the same chunk.
  3. Postgres gets the new row text and version timestamp.

Three writes, three systems. They do not commit atomically. There is no shared transaction. There is no shared commit boundary.

In the normal case nothing breaks - the three writes finish in milliseconds and the user moves on. But the moment one fails, retries, or even just lands out of order under load, you have an internally inconsistent corpus. A query against that corpus can see:

Your retriever picks the chunk. Your reranker scores it. Your LLM generates an answer using the v1 text. Your Stage 7 faithfulness checker - the one that verifies every assertion in the answer is grounded in the retrieved chunks - sees the v3 vector tag, the v2 BM25 posting, and the v1 text. The faithfulness math runs against three versions of “the same chunk” and produces a number that no longer means what you think it means.

That number then drives a decision: do we show the answer, do we re-run with a stricter prompt, do we escalate to a human? You are making a quality-gate decision based on a corpus that is, briefly, lying to you.

In our experience this happens enough that it is the dominant source of unexplained hallucination reports at scale. It is never on the dashboard because no system reports the skew - each individual database is healthy. It is never in the logs because the writes all succeeded. It surfaces as an LLM that “sometimes makes things up”, and the postmortem inevitably ends with someone saying “we tightened the prompt”.

The fix is not at the application layer

The natural reaction is to add cross-system reconciliation: write Postgres first, then Pinecone, then Elasticsearch, all behind a saga, with a worker that detects skew and re-syncs. We have seen teams build elaborate machinery for this. It works. Then it breaks under load. Then the worker itself becomes a thing to monitor. Then somebody starts skipping a step “because it is slow” and three weeks later you have skew again.

The fix is not at the application layer because the consistency boundary belongs to the engine, not the app. If your vectors, postings, and rows live in different systems addressed by different bearers and persisted separately, no amount of application code can give you atomic cross-shape writes. You can get eventual consistency, but Stage 7 of the canonical RAG pipeline asks for point-in-time consistency.

What we built

OriginChainDB is a single-substrate database where SQL rows, vector indexes, full-text indexes, and graph edges share one managed k/v store and one commit boundary. The write path for a document chunk is a single atomic operation that touches every shape at once:

await oc.transaction([
 { shape: "rows", table, id, set: row_payload },
 { shape: "vec", table, id, set: embedding_payload },
 { shape: "fts", table, field, id, set: posting_payload },
]);

One atomic write — all shapes or none. After a crash either every shape applies or none do. There is no path through the system where the vector lands and the row doesn’t, or where the BM25 posting commits and the vector doesn’t.

The Stage 7 faithfulness check then runs against a corpus that is internally consistent by construction. The version-skew failure mode doesn’t appear in postmortems because it isn’t reachable.

What the application still owns

A single-substrate engine is necessary but not sufficient for great RAG. The article’s other nine stages still apply, and they are still your code:

What you stop owning is the coordination problem. The atomic-write guarantee lets the verifier compute a number that actually means something.

What the code looks like

Ingest, in TypeScript with the SDK:

for (let i = 0; i < chunks.length; i++) {
 const id = `${docId}:${i}`;
 await oc.sql(
 `INSERT INTO chunks (id, source, page, text, created_at)
 VALUES (?, ?, ?, ?, ?)`,
 [id, source, chunks[i].page, chunks[i].text, Date.now],
 );
 await oc.vectorPut("chunks", {
 id, embedding: emb.data[i].embedding, dim: 1536,
 metric: "cosine",
 metadata: { source, page: chunks[i].page },
 });
 await oc.ftsIndex("chunks", "text", { doc_id: id, text: chunks[i].text });
}

Read, in parallel:

const [vecHits, ftsHits] = await Promise.all([
 oc.vectorTopk("chunks", { query: qVec, k: 20, dim: 1536, metric: "cosine" }),
 oc.ftsSearch("chunks", "text", { q: question, mode: "bm25", k: 20 }),
]);
const top = reciprocalRankFusion(vecHits, ftsHits).slice(0, 5);
const rows = await oc.sql(
 `SELECT id, source, page, text FROM chunks WHERE id IN (?, ?, ?, ?, ?)`,
 top.map(h => h.id),
);

The fuller walkthrough - including the semantic-cache pattern, configuration guidance for corpus sizes from 100K to 10M chunks, and the boundary diagram - lives in the RAG docs page.

On scale, honestly

The Medium article uses 10 million documents as its target. On OriginChainDB, 10 million 1536-dimensional chunks is roughly:

That is a mid-size configuration - a single dedicated instance with enough RAM to keep the HNSW graph hot. HNSW recall@10 above 0.95 holds at that scale; query latency stays in the tens of milliseconds before the reranker. Beyond 50 million chunks you are into multi-writer sharding territory, which is on our depth-first roadmap to 1.0 but not GA yet. Talk to us if you are already there.

The blueprint, with three fewer databases

The Medium article’s blueprint works. Removing three of its vendors (Pinecone, Elasticsearch, Redis-vector) removes the cross-system version skew with them; the 10-stage architecture and the retrieval math are unchanged.

If you are building RAG at any scale where “we sometimes hallucinate” is a thing you are tracking, the question worth asking is not “what is my reranker doing” - it is “what is my corpus saying when I read it three different ways at the same instant”.

← All posts Subscribe to RSS →