11. HNSW health, and rebuilding a damaged index

what this does

GET /v1/tenants/:t/vector/:table/hnsw-health reports whether a table's HNSW graph can actually be walked: how many live vectors a search can reach, how many disconnected components the graph has, how many nodes nothing points at, and a one-line verdict.

POST /v1/tenants/:t/vector/:table/rebuild-hnsw rebuilds the graph from the vectors already in the store. No embedding is rewritten and nothing is re-ingested.

when to use it

• Recall is worse than it should be and widening the search beam does not help - that pattern is a graph problem, not a tuning problem.
• You inherited a table and do not know which distance metric its index was built for.
• Before a fleet-wide reindex, to find which tables actually need one instead of rebuilding everything.

the request

curl -G "https://$OC_HOST/v1/tenants/$OC_TENANT/vector/shop.products/hnsw-health" \
  -H "Authorization: Bearer $OC_TOKEN" \
  --data-urlencode "self_recall=512"

# No typed SDK helper for this route yet - call it directly.
import httpx

r = httpx.get(
    f"{OC_HOST}/v1/tenants/{OC_TENANT}/vector/shop.products/hnsw-health",
    headers={"Authorization": f"Bearer {OC_TOKEN}"},
    params={"self_recall": 512},
)
r.raise_for_status()
health = r.json()
print(health["verdict"])
print(health["self_recall"]["self_recall"])

// No typed SDK helper for this route yet - call it directly.
const res = await fetch(
  `${OC_HOST}/v1/tenants/${OC_TENANT}/vector/shop.products/hnsw-health?self_recall=512`,
  {
    method: "GET",
    headers: {
      Authorization: `Bearer ${OC_TOKEN}`,
    },
  },
);
const health = await res.json();
console.log(health.verdict, health.shattered);

// No typed SDK helper for this route yet - call it directly.
url := fmt.Sprintf("%s/v1/tenants/%s/vector/shop.products/hnsw-health?self_recall=512", host, tenant)
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)

what you get back

{
  "table":                "shop.products",
  "present":              true,
  "nodes":                41892,
  "live":                 41892,
  "tombstoned":           0,
  "reachable_live":       41892,
  "reachable_fraction":   1.0,
  "components":           1,
  "zero_indegree_live":   0,
  "indegree_percentiles": [1, 4, 9, 16, 27, 48, 91],  // min p1 p10 p50 p90 p99 max
  "mean_outdegree":       15.8,
  "shattered":            false,
  "damage":               [],
  "build_metric":         "cosine",
  "stored_vectors":       41892,
  "unindexed_records":    0,
  "verdict":              "healthy: 41892 of 41892 live vectors retrievable, ...",
  "self_recall": {
    "metric":      "cosine",
    "applicable":  true,
    "probes":      512,
    "hits":        512,
    "self_recall": 1.0,
    "misses":      [],
    "verdict":     "healthy: all 512 probed vectors retrieve themselves ..."
  }
  /* plus per-layer shape, norm deciles and advisories */
}

self_recall is present only when you ask for it. Everything else comes back on every call.

query parameters

how it works

The topology half walks the graph and counts. It catches the failure it was built for - a graph that reaches a small fraction of its own nodes, where no amount of beam widening recovers the rest - and it reports shattered: true with the reasons in damage.

The topology half is structurally blind to one thing: an index built for one distance metric and queried under another. That graph is perfectly connected and reports every reachability field as healthy. ?self_recall=N asks the other question, and needs no ground truth to do it - under cosine, L2 or Manhattan a vector is its own exact nearest neighbour, so any miss is a defect. Under dot product that is not true, and the probe says so with "applicable": false rather than returning a number that would read as a failure.

That makes the probe the tool for an unstamped index. Run it once per candidate metric; the one the graph was really built under is the one that comes back at 1.0. Then pass that metric to the rebuild, which records it beside the graph so nothing has to guess again.

rebuilding the graph

curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/vector/shop.products/rebuild-hnsw" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "metric":  "cosine",
    "dry_run": true
  }'

# No typed SDK helper for this route yet - call it directly.
import httpx

r = httpx.post(
    f"{OC_HOST}/v1/tenants/{OC_TENANT}/vector/shop.products/rebuild-hnsw",
    headers={"Authorization": f"Bearer {OC_TOKEN}"},
    json={"metric": "cosine", "dry_run": True},
    timeout=None,
)
r.raise_for_status()
plan = r.json()
print(plan["vectors"], plan["estimated_build_secs"], plan["estimated_peak_bytes"])

// No typed SDK helper for this route yet - call it directly.
const res = await fetch(
  `${OC_HOST}/v1/tenants/${OC_TENANT}/vector/shop.products/rebuild-hnsw`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${OC_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ metric: "cosine", dry_run: true }),
  },
);
const plan = await res.json();
console.log(plan.vectors, plan.estimated_build_secs);

// No typed SDK helper for this route yet - call it directly.
payload, _ := json.Marshal(map[string]any{
    "metric":  "cosine",
    "dry_run": true,
})
url := fmt.Sprintf("%s/v1/tenants/%s/vector/shop.products/rebuild-hnsw", host, tenant)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)

what the rebuild returns

{
  "tenant":                "…",
  "table":                 "shop.products",
  "rebuilt":               false,           // false on a dry run
  "metric":                "cosine",
  "build_metric_before":   null,            // null on an index built before the stamp
  "dim":                   768,
  "vectors":               41892,
  "estimated_peak_bytes":  137070624,
  "estimated_build_secs":  55,
  "health_before":         { /* the same report as GET hnsw-health */ },
  "health_after":          null,            // null on a dry run
  "elapsed_ms":            412
}

Drop dry_run to run it for real: rebuilt becomes true and health_after carries the post-rebuild report, so one response shows you the before and the after.

rebuild fields