From a 20-second dashboard to 300 ms

OriginChainDB · May 13, 2026

infrastructure performance platform operations

The dashboard’s cold-path latency was a credential fetch, not the database. Reading each tenant’s bearer token with a remote command to the instance cost 4-15 seconds; moving that read to a managed parameter store cut it to ~150 ms, and the cold page load from 20 seconds to ~300 ms.

A customer mentioned in a thread that the schema-list page on their OriginChainDB dashboard took “like around 20 seconds.” Not a feature request: a complaint. The page renders four schemas. Each is a hundred-byte manifest. The engine serving them, behind a TLS connection, responds to GET /v1/tenants/.../schemas in ~350 ms end-to-end including the TLS handshake.

So where do the other ~19 seconds go? They went into how the dashboard backend authenticated to the tenant engine on the user’s behalf.

The architecture and the problem

OriginChainDB’s customer dashboard lives at app.originchain.ai. The backend serving that dashboard does cookie-auth against the user. When the user opens a schema page, the backend translates the request into a tenant-bearer call against the customer’s dedicated engine instance. The tenant bearer is the customer’s secret; it is written onto the instance at provision time.

The backend needs that bearer to make the proxy call. It doesn’t have it in memory. It has to ask the tenant box for it. The original implementation issued a remote command to the instance to read the credential file off disk and return its contents, then polled for the result:

1. issue a remote "read this credential" command to the instance
2. wait for the command to be dispatched
3. the instance runs it and returns the credential on stdout
4. poll for completion every 500 ms
5. once complete, read the result back

The remote-command API we used is beautifully versatile and operationally awful for a request that’s on the hot path of a page load. The wire flow involves spawning a helper process (~1 s cold start), a multi-second control round-trip to dispatch the command, a hop to the agent on the target instance, and then repeated polling - each poll paying the same cold-start cost again.

Cold path: 4–15 s typical, with a tail at 20 s. The result was cached in-process for 5 minutes, so most page loads were fast. But every cache miss - first load of the day, first load after a backend restart, first load after the 5-min idle - paid the full price.

When the user hits a dashboard page on a Monday morning, that’s the cache miss. Hence “20 seconds.”

Why the original design picked a remote command

Hindsight makes this look silly, but at the time it was reasonable:

The credential file was the source of truth for the bearer. Other operational code already used the same remote-command mechanism for things like log-tailing and one-shot diagnostic queries. Reusing the pattern felt clean.
The bearer was already on the box (written at boot during provisioning). Storing it somewhere else felt like duplicate state.
Permissions were already set up for the remote-command path. Adding another permission (parameter-store read) felt like work.

These are individually fine reasons. The sum of them, weighted by the fact that the call happens on every dashboard request after a 5-minute gap, was a bad trade.

The fix

Three things:

1. Write the bearer to a managed parameter store at provision time

Per tenant, encrypted at rest, under a per-instance path. The provision flow already mints the bearer and writes it to the instance; adding a parameter-store put alongside is a few lines in the instance-create handler:

let bearer = generate_bearer;
let bearer_hash = password::hash(&bearer)?;

//... existing DB insert + provisioning trigger...

bearer_store::put(instance_id, &bearer).await?;

The decrypt permission was already granted to the backend’s role; the put/get permission on the per-tenant path is a fresh statement.

2. Replace the bearer fetch with a direct call to the parameter store

pub async fn get(instance_id: Uuid) -> AppResult<Option<String>> {
 let client = param_client.await?;
 let name = format!("/oc/tenant-bearers/{}", instance_id);
 match client.get_parameter
.name(&name)
.with_decryption(true)
.send.await
 {
 Ok(out) => Ok(out.parameter.and_then(|p| p.value).map(String::from)),
 Err(err) if matches!(err.as_service_error,
 Some(GetParameterError::ParameterNotFound(_))) => Ok(None),
 Err(e) => Err(AppError::Internal(format!("param store: {e}"))),
 }
}

The cold call: one signed HTTPS GET, decrypt server-side, response in ~150 ms. The client is a process-wide singleton so the connection pool warms once and every subsequent call reuses connections.

3. Backwards-compatible fallback for unmigrated instances

The catch: existing tenants didn’t have their bearer in the parameter store yet. We could have run a one-shot migration script, but the correctness story is cleaner if the backend handles it automatically:

// Try the parameter store first. ~150 ms cold.
if let Some(bearer) = bearer_store::get(instance_id).await? {
 return Ok(cache_and_return(bearer));
}

// Fall through to the legacy remote-command read of the credential.
let bearer = fetch_tenant_bearer_via_remote_command(ctx).await?;

// Opportunistically backfill the parameter store. Next call lands fast.
if let Err(e) = bearer_store::put(instance_id, &bearer).await {
 warn!(error = %e, "bearer parameter-store backfill failed");
}
Ok(cache_and_return(bearer))

Three properties:

New tenants always land in the parameter store via the provision flow. Fast path from day one.
Existing tenants pay the legacy slow path ONCE - the very next call is fast.
The backfill is best-effort. If permissions are misconfigured or the store is briefly unavailable, the tenant still works; we just stay on the slow path until the issue is fixed.

We also pre-seeded the live demo tenant’s bearer directly so the first dashboard load after the deploy was fast, not 15 seconds. Self-bootstrap, when you can.

Bumping the in-process cache TTL

The legacy slow path was so painful that we’d capped the in-process cache at 5 minutes - the cost of a cache miss was so high that we wanted to refresh aggressively in case the bearer had rotated externally.

With the new path at ~150 ms, the cost of a miss is small. We bumped the cache TTL to 1 hour. The bearer rotates only on an explicit rotate-bearer call (which invalidates the cache locally) or rare operator intervention; 1 hour catches an out-of-band rotation soon enough without paying needless parameter-store calls.

Live numbers, end-to-end

After deploying:

Path	Before	After
Cache miss (1st load / 1h idle)	4–15 s	~300 ms
Cache hit (tab switch)	<1 ms	<1 ms
Tail (p99 of cache miss)	20+ s	~500 ms

The 300 ms vs 150 ms gap is the rest of the dashboard request: DB lookup of the user’s org + instance ownership check, the proxy call to the tenant engine, the JSON serialise. The parameter-store call itself is the smallest part of the new latency budget.

p99 dropped from “20 seconds” to “500 ms” - a 40× improvement on the worst case, ~100× on the mean. The hot path was already fast; the fix was about the cold path.

What this is NOT

A rotation system. The bearer can still rotate via the existing POST /v1/instances/:id/rotate-bearer endpoint; this fix doesn’t add automatic rotation on a schedule. That’s a separate compliance feature.
A trust-store. The parameter store uses an account-level encryption key. We didn’t move to per-tenant keys here; the trade-off is “blast radius if the key is compromised” vs “ops complexity at 100 tenants times $1/month.” The blast radius is the same as it was before; the perf is way better.
A free-lunch claim. The parameter store has rate limits. For our load - a few requests per dashboard page, behind an in-process cache - we’ll never hit them. A tenant with 50,000 simultaneous dashboard users on a cache-cold restart would; not our problem today.

Why this is worth blogging

Most database vendor blog posts are about the hot path: the smart index, the magic protocol, the new compression scheme. Cold paths shape product perception just as much. A 20-second first load makes your dashboard feel broken; a 300 ms first load makes it feel snappy even if the rest of the system is the same.

If you’re building a SaaS control plane and reaching for a remote-shell call because “it’s already there,” consider the alternative. A managed parameter store is the right primitive for any sub-kilobyte per-tenant secret, and the migration is a fallback plus a backfill.

Try it

Open the schema page on your tenant. It’ll feel faster than it used to.