12. Build a compressed IVF-PQ index

what this does

POST /v1/tenants/:t/vector/:table/create-ivf-pq-index builds a product-quantized IVF index over the vectors already in the table: it trains the coarse cells and the quantizer together, installs both, and sets the query-time policy - all from one named preset.

You pick a profile, not a set of knobs. The engine resolves the cell count from the corpus size and the quantizer width from the vector dimension, and reports back exactly what it chose.

when to use it

• The corpus no longer fits comfortably in memory as raw float vectors and you want the index to carry compressed codes instead.
• You are on IVF already and want the smaller footprint that quantized codes buy.
• You want a repeatable index build: pass a seed and the same corpus produces the same index.

the request

curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/vector/shop.products/create-ivf-pq-index" \
  -H "Authorization: Bearer $OC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "preset": "high_recall",
    "seed":   42
  }'

# No typed SDK helper for this route yet - call it directly.
import httpx

r = httpx.post(
    f"{OC_HOST}/v1/tenants/{OC_TENANT}/vector/shop.products/create-ivf-pq-index",
    headers={"Authorization": f"Bearer {OC_TOKEN}"},
    json={"preset": "high_recall", "seed": 42},
    timeout=None,
)
r.raise_for_status()
built = r.json()
print(built["partitions"], built["pq_m"], built["keep_raw"])

// No typed SDK helper for this route yet - call it directly.
const res = await fetch(
  `${OC_HOST}/v1/tenants/${OC_TENANT}/vector/shop.products/create-ivf-pq-index`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${OC_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ preset: "high_recall", seed: 42 }),
  },
);
const built = await res.json();
console.log(built.partitions, built.pq_m);

// No typed SDK helper for this route yet - call it directly.
payload, _ := json.Marshal(map[string]any{
    "preset": "high_recall",
    "seed":   42,
})
url := fmt.Sprintf("%s/v1/tenants/%s/vector/shop.products/create-ivf-pq-index", host, tenant)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)

what you get back

{
  "trained":              true,
  "installed":            true,
  "preset":               "high_recall",
  "partitions":           1024,
  "pq_m":                 64,
  "pq_bits":              8,
  "keep_raw":             true,
  "raw_vectors_retained": true,
  "dim":                  768,
  "training_corpus_size": 100000
}

keep_raw and raw_vectors_retained are different questions and are reported separately on purpose. The first is a query-time policy: may the exact re-rank read the raw vector back. The second is the footprint: are the raw vectors still on disk.

request fields

how it works

The build reads the stored corpus first, because the preset resolves against it. The cell count comes from the corpus size - roughly four times its square root, rounded to a power of two and clamped between 64 and 65,536 - and the quantizer's subvector count comes from the vector dimension: the divisor of dim (at most 64) whose subspace width lands closest to the preset's target, which is 8 for high_recall and 16 for compressed. Codes are 8 bits wide either way.

Then it trains, installs the codebook and the centroids, sets the keep_raw policy, and populates the cell postings in bounded chunks so a large corpus never holds the write lock across the whole build.

Queries reach the index with "index": "ivf_pq" on the ordinary top-k route. On a table built with keep_raw, refine_k_factor controls how many candidates the exact re-rank re-scores; on a codes-only table there is nothing to re-score against and the field is ignored.