Start free
← All posts

Fuzzing a database: 850,000 probes/day

OriginChainDB engineering · Jun 6, 2026
fuzzing reliability canary engineering qa

TL;DR — A dedicated host in our managed cloud fires ~10,000 random API probes at the engine every minute, 24/7. That’s ~850,000 probes/day, ~310M/year. Every probe is recorded; every failure pages on-call within two minutes. Here’s what’s actually in the loop, how we assert “no bug” without ground-truth, and the surface we don’t yet cover.

You can’t unit-test your way to production confidence. Unit tests prove a function does what you wrote it to do. They don’t prove the HTTP layer above it stays well-formed under unexpected input, the auth wall holds against omitted headers, or the planner doesn’t 500 on a SQL string a customer types by accident.

The thing that does prove those properties is a fuzzer that runs forever against a real engine.

We built one. It runs as a continuous loop on a dedicated host in our managed infrastructure, hammering the engine over its public HTTPS endpoint with about 10,000 randomly-generated probes per iteration, sleeping 60 seconds, and starting again. The loop has been running uninterrupted, on a real tenant, with real auth, for weeks.

The numbers

The loop is a small script, supervised so it restarts on failure with a start limit (5 restarts per 60 seconds) so a runaway can’t take the host down. The loop body is short:

while true; do
  LOOP_COUNTER=$((LOOP_COUNTER + 1))
  SEED=$(( $(date +%s%N) ^ LOOP_COUNTER ))

  fuzz run \
    --base-url "$BASE_URL" \
    --bearer "$BEARER" \
    --tenant "$TENANT" \
    --iterations 10000 \
    --workers 16 \
    --seed "$SEED" \
    --report "$REPORT_PATH"

  FAILURES=$(jq -r '.summary.failed // 0' "$REPORT_PATH")
  publish_metric \
    --metric-name "FuzzFailures" \
    --value "$FAILURES" || true

  sleep 60
done

The metric publish is wrapped in || true so a monitoring-side outage never stops the canary. The fuzzer’s own report file is the source of truth for what failed; the metric is just the cheap pointer for the alarm.

The 14 generator families

The fuzzer doesn’t randomly mash bytes. It generates probes by family — each family knows the shape of one engine endpoint and randomizes within that shape. The bag of families is round-robin’d with random jitter so a long run gets balanced coverage.

The current rotation:

A probe carries an HTTP verb, a path, an optional JSON body, the generator family that produced it, and an intent string (“malformed SQL”, “missing PK”) that surfaces on failure so triage doesn’t reverse-engineer the generator. The full probe is serialized into the report file so a replay command re-issues the exact probe against any environment, including a local engine for diagnosis.

How we assert “no bug” without ground-truth

This is the part most people skip when describing a fuzzer. You can’t write assert_eq!(actual, expected) against a random probe — you don’t know what the expected output is. What you do know is the invariants the endpoint must maintain regardless of input.

The assertion rules, in order:

  1. No 5xx. Ever. A 500 means the engine panicked or returned an unhandled error. Every 5xx is a bug. Generated probes that are deliberately malformed should return 400, not 500. This is the single highest-signal assertion in the loop.
  2. The response body is parseable JSON. Even error responses. If the engine returns Content-Type: application/json and the body fails serde_json::from_slice, that’s a bug — it means a code path is emitting raw format! output where it should be emitting structured errors.
  3. The auth wall holds. Probes with bearer_omitted: true must come back 401. Not 403, not 500, not 200. A single 200 on a no-auth probe is the kind of bug that ends careers; the canary fails immediately and pages.
  4. The status code is within the expected band for the family. A Health probe returning 503 is OK (the engine is degraded). A Health probe returning 400 is a bug — health endpoints don’t take input. Each family carries its own band.

Generated probes that are deliberately wrong are expected to fail with 4xx — that’s not a failure of the engine, it’s a successful auth/parse/validate gate. The assertion rules distinguish “the engine correctly rejected garbage” from “the engine accidentally accepted garbage” from “the engine crashed on garbage.” Only the last two count as failures.

The p99 number, honestly

The current loop reports p99 around 115 ms across the full probe mix. That number is dominated by the heavier endpoints — vector search and SQL — not the cheap /health probes. The fixed-shape probes anchor the median to the low single-digit ms range; the p99 is where the planner work, the index lookups, and the natural-language parse actually show up.

We publish p99 and p99.9 into the per-iteration report. We watch both. The day p99 jumps to 200ms without a deployment, something has slipped in the planner — same dashboard you’d build to watch a real customer workload, except the canary’s “customer” is a 10,000-probe-a-minute random adversary.

What we don’t cover

Three surfaces the canary doesn’t touch, in order of how much we care:

These are real gaps. We don’t pretend they aren’t.

The CI counterpart

The same fuzz binary runs as a daily job in CI, against the same canary tenant, from outside our managed cloud. That’s the second opinion when the in-tenant loop’s metric drifts. If both the in-tenant loop and the external CI run agree, we trust the result. If they disagree, the disagreement is itself the signal — usually a routing or auth path that behaves differently from inside the cloud vs from outside.

The CI workflow uses a credentials secret keyed off the same bearer the canary uses; rotating one rotates the other.

Why the canary lives next to the engine

It would be tempting to run the canary from outside the cloud entirely — that’s what the daily CI run does. We run the in-tenant loop next to the engine for one reason: latency. From inside the same region, the canary’s p99 number reflects the engine’s p99, not the engine plus public-internet network jitter. Putting the canary on the same network the engine serves from gives us a clean signal that a regression is in the planner or the index code, not in the network edge or load balancer.

That’s load-bearing for the alarm threshold. A p99 jump on a remote canary could be a network anomaly. A p99 jump on a co-located canary, with the cheap fixed-shape probes still anchored to low-single-digit ms, is the engine.

What the canary catches

In the months it’s been running, the canary has caught:

None of these were caught by unit tests. They were caught by the loop hammering shapes the test suite hadn’t thought to write.

Try it on your own engine

If you’re evaluating OriginChainDB and want to know what “stable enough to run an agent against” looks like, the canary is the operational answer. Create a free database, point your own load at the engine, and watch the same metric the canary is publishing. The graph shape is the data.

One endpoint. Random probes. Forever. No 5xx.

← All posts Subscribe to RSS →