OriginChainDB docs
docs · vector · IVF

IVF vector index.

IVF partitions your vectors into cells by their nearest centroid. At query time, the engine picks the nprobe closest cells to the query vector and scans only those. The result is the path to 10M+ vectors per tenant on the standard configurations, and a 1M-vector IVF build in 80 s.

The IVF pipeline.

  1. Train — k-means picks nlist centroids on a training sample. sqrt(N) is a reasonable starting point for nlist.
  2. Assign — every vector is assigned to its nearest centroid. Bulk-load runs this in parallel: 1M vectors in 80 s.
  3. Query — the query vector is scored against the centroids; only the nprobe closest cells are scanned.

IVF is not declared in the manifest.

A schema manifest has no index, nlist or quantization key. You choose IVF at runtime, with the install-centroids call below — nlist is one of its arguments, not a schema field. Quantization is a separate install call on the same collection, so IVF and quantization are chosen independently.

What the manifest can declare is the collection itself — how wide its vectors are and which distance function they are indexed under. That block is optional, and it is a single [vector] table: 

# manifest.toml — a 768-dim collection. Nothing here selects an index.
[vector]
dim      = 768        # required
distance = "cosine"   # cosine | l2 | dot | manhattan | l1

Every field, and what happens when a request contradicts the declared distance, is in the schema reference. Registration ignores keys it does not recognise, so an invented one registers with a 200 and silently does nothing — worth knowing before you improvise a field name.

Install centroids (admin).

Training is an HTTP admin call. Run it once when the corpus reaches training scale; re-run only if the data distribution shifts materially.

POST /v1/tenants/:t/vector/:table/install-centroids 
curl -X POST "https://acme.db.originchain.ai/v1/tenants/$T/vector/embeddings/install-centroids" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "nlist":        1024,
    "training_set": "sample",
    "iters":        25
  }'

The nprobe knob.

nprobe is the recall / latency dial. Higher = better recall, more cells scanned. The recall figures below are the engine's own illustrative curve, measured on a 1,000-vector planted-cluster corpus at 16 partitions — your numbers will move with corpus geometry.

nproberecall@10latency band
1 ~0.30 Cheapest scan, lowest p99.
2 ~0.55 Still cheap, recall climbing fast.
4 ~0.85 The default on a 16-partition table.
8 ~0.95 Recall-critical reads.
POST /v1/tenants/:t/vector/:table/topk 
curl -X POST "https://acme.db.originchain.ai/v1/tenants/$T/vector/embeddings/topk" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query":  [/* 768 floats */],
    "k":      10,
    "nprobe": 4
  }'

When to pick IVF over HNSW.

  • Working set is above ~5M vectors and HNSW memory is starting to bind.
  • You want to combine with PQ (see IVF-PQ) for the 64–768× memory story.
  • Workload is bulk-load heavy. The IVF bulk-load path lands 1M vectors in 80 s.
  • Recall budget can tolerate the nprobe trade — nprobe=4, the default on a 16-partition table, reaches recall@10 of about 0.85 on that corpus.