Build a compressed vector index
The language tabs use earlier client interfaces. Check SDK versions and connection setup before running them.
IVF-PQ represents vector groups with compact codes. Choose a preset and decide whether original vectors must remain available.
Before you run it: drop_raw_vectors removes original vectors. Restoring them requires ingestion again; it is a destructive choice, not a routine optimization.
First store raw vectors in shop.products; an empty table cannot be trained. IVF groups nearby vectors, and product quantization (PQ) stores compact approximations of them. Keep the original vectors for this first test and measure whether the compressed index still finds the matches your application needs.
- Start with stored raw vectorsUse a populated table with reproducible source embeddings.
- Choose the presetReview compression and exact re-ranking behavior.
- Build the compressed indexConfirm raw_vectors_retained in the response.
- Query with ivf_pqCompare useful matches before adopting the compressed index.
Follow the example
- Start with stored raw vectors.
- Choose high_recall or compressed.
- Review keep_raw and drop_raw_vectors separately.
- Build and query with index: ivf_pq.
Run the request
curl -X POST "https://$OC_HOST/v1/tenants/$OC_TENANT/vector/shop.products/create-ivf-pq-index" \
-H "Authorization: Bearer $OC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"preset": "high_recall",
"seed": 42
}'Check the result
The response reports the installed settings and retained data.
{
"trained": true,
"installed": true,
"preset": "high_recall",
"partitions": 1024,
"pq_m": 64,
"pq_bits": 8,
"keep_raw": true,
"raw_vectors_retained": true,
"dim": 768,
"training_corpus_size": 100000
}keep_raw and raw_vectors_retained are different questions and are reported separately on purpose. The first is a query-time policy: may the exact re-rank read the raw vector back. The second is the footprint: are the raw vectors still on disk.
Request options
| Field | Required | Notes |
|---|---|---|
| preset | yes | "high_recall" or "compressed". high_recall uses a finer quantizer and keeps the raw vector so the exact re-rank can run; compressed uses a coarser one and does not re-rank. |
| seed | no | PRNG seed for the training run. Defaults to 0. Pass one if you need two builds over the same corpus to match. |
| drop_raw_vectors | no | Destructive and opt-in. Discard the raw float embeddings table-wide as part of the build, keeping only the codes. Defaults to false: building an index never destroys data. There is no undo - set it only when the vectors are reproducible from your own source of truth. |
Understand the behavior
The build reads the stored corpus first, because the preset resolves against it. The cell count comes from the corpus size - roughly four times its square root, rounded to a power of two and clamped between 64 and 65,536 - and the quantizer's subvector count comes from the vector dimension: the divisor of dim (at most 64) whose subspace width lands closest to the preset's target, which is 8 for high_recall and 16 for compressed. Codes are 8 bits wide either way.
Then it trains, installs the codebook and the centroids, sets the keep_raw policy, and populates the cell postings in bounded chunks so a large corpus never holds the write lock across the whole build.
Queries reach the index with "index": "ivf_pq" on the ordinary top-k route. On a table built with keep_raw, refine_k_factor controls how many candidates the exact re-rank re-scores; on a codes-only table there is nothing to re-score against and the field is ignored.
Important limits and checks
- Asking for high_recall and drop_raw_vectors. That combination is refused with 400, and it should be: high_recall's accuracy comes from the exact re-rank, which reads the raw vector back. Honouring the drop would leave the policy pointing at deleted data and quietly degrade every query.
- Rebuilding a table whose raw vectors were dropped. An index build needs float vectors to train on, and codes cannot be turned back into them. That is a 400 telling you to re-ingest - the only fix.
- Building on an empty table. 400. Write the vectors first; the preset has nothing to resolve against otherwise.
- Reading
keep_raw: falseas "the vectors are gone". It means the exact re-rank will not run. Unless you passeddrop_raw_vectors,raw_vectors_retainedis still true and the compressed index costs the raw vectors on top of its codes. - Calling it on a quorum-replicated tenant. The route refuses with 501 there. The build re-writes the whole corpus and is not expressible as one replicated frame, so accepting it could leave a promoted node with a half-built index.