A search demonstration can look convincing because the author knows which question produces a good answer. A released application must cope with paraphrases, missing information, obsolete passages and people who are not allowed to read every source.

This guide builds a practical evaluation loop for an OriginChainDB FTS, vector or combined retrieval workflow. It does not require a large benchmark to get started. It requires explicit questions, known evidence and enough recording of the experiment to explain what changed.

1. Define the user task

Use a fictional employee handbook with five passages: travel receipts, equipment requests, office access, public holidays and an obsolete travel rule. The application should help a reader find a current, permitted passage that supports the answer.

Success is not “return five rows.” A result should help the person complete the task without relying on stale or inaccessible evidence. Define whether the workflow returns documents, chunks or generated answers, because each output needs different checks.

2. Create a labeled question file

Start with a small file such as:

[
  {"id":"q1","query":"When do I submit travel receipts?","relevant":["travel-r2"],"allowed":["travel-r2","equipment-r1"]},
  {"id":"q2","query":"How do I request a laptop?","relevant":["equipment-r1"],"allowed":["travel-r2","equipment-r1"]},
  {"id":"q3","query":"Can I bring a pet to the office?","relevant":[],"allowed":["travel-r2","equipment-r1"]}
]

These are synthetic fixtures. The allowed list is an evaluation fixture, not a production authorization policy. In the real application, permission comes from trusted server-side policy evaluation for the authenticated principal.

An empty relevant list means this test set contains no supporting source. It does not mean the system should fabricate a policy to make every question appear answered.

3. Record a reproducible retrieval run

For each question, save the source revision, retrieval configuration, returned IDs and rank order. For vector retrieval, include the model identifier, dimension and preprocessing convention. For FTS, record the indexed field and analysis settings that affect matching.

Do not store secrets in the evaluation artifact. Synthetic queries and record IDs should be sufficient for the first exercise. Keep sensitive real-world evaluation material under its own access and retention rules.

Compare configurations on the same labeled questions and source snapshot. Changing the corpus and retrieval settings simultaneously makes an apparent improvement harder to interpret.

4. Calculate simple measures correctly

The following helper evaluates a bounded returned list against a known relevant set:

def assess(returned_ids, relevant_ids, allowed_ids):
    ranked = list(dict.fromkeys(returned_ids))
    relevant = set(relevant_ids)
    allowed = set(allowed_ids)
    hits = set(ranked) & relevant
    return {
        'precision': len(hits) / len(ranked) if ranked else None,
        'recall': len(hits) / len(relevant) if relevant else None,
        'unauthorized_ids': [x for x in ranked if x not in allowed],
        'missing_relevant_ids': sorted(relevant - set(ranked)),
    }

This example deduplicates IDs while retaining their order. It leaves precision undefined for an empty returned set and recall undefined when the labeled relevant set is empty. Document those conventions instead of silently replacing undefined values with a flattering score.

For q1, returning only travel-r2 gives perfect precision and recall under this tiny fixture. Returning travel-r2 and an irrelevant allowed document lowers precision. Returning an unauthorized document is an access failure regardless of the relevance score.

5. Inspect the failure, not just the average

For every miss, read the expected source. Was it indexed? Did chunking remove the condition? Is the query using the compatible embedding setup? Did an FTS analyzer treat an identifier differently from what you expected?

For every irrelevant result, ask whether the label is correct and whether the passage actually addresses the task. A high similarity score is evidence about a retrieval calculation, not a universal probability that the answer is correct.

Keep ranking quality separate from source freshness and authorization. An obsolete rule can be an excellent wording match. A private document can be the nearest vector. Both may still be unsuitable evidence for the user.

6. Evaluate generation as an additional stage

If a model writes the answer, record which permitted passages it received. Check whether the answer is supported by those passages and whether citations identify the correct source revision. A citation attached to a sentence does not prove that the passage supports it.

Use q3 to test the fallback. A useful assistant can say the handbook does not provide an answer and suggest an appropriate next step. Treat unsupported certainty as a failure, even if the prose is fluent.

7. Make release decisions explicit

Choose acceptance thresholds and critical failure rules before comparing runs. Permission leaks and misleading claims may need different treatment from a lower-ranked useful result. Keep a held-out set of questions so repeated tuning does not simply memorize the original examples.

Save the chosen configuration, fixture revision and known limitations with the release. After a source, model or indexing change, repeat the relevant tests. Evaluation becomes useful when it is a repeatable part of the workflow rather than a one-time screenshot.

Start with the OriginChainDB retrieval documentation, then build a small test set that reflects your actual users’ questions.

Run the helper against deliberately different outcomes

Use the original assess function on the same labeled question while changing only the returned IDs. For q1, the relevant set contains travel-r2 and the allowed set contains travel-r2 plus equipment-r1:

allowed = ['travel-r2', 'equipment-r1']
relevant = ['travel-r2']
assert assess(['travel-r2'], relevant, allowed)['precision'] == 1.0
assert assess(['travel-r2', 'equipment-r1'], relevant, allowed)['precision'] == 0.5
assert assess(['equipment-r1'], relevant, allowed)['recall'] == 0.0
assert assess(['private-payroll'], relevant, allowed)['unauthorized_ids'] == ['private-payroll']
assert assess([], relevant, allowed)['precision'] is None

These checks demonstrate distinct failures. Returning an irrelevant allowed passage harms ranking quality. Returning private-payroll is an authorization failure even if a separate relevance label mistakenly marks it useful. Returning nothing requires a clear convention for undefined precision and an understandable user-facing fallback.

The helper does not measure answer truth, citation support, latency, document freshness or rank position. It also does not prove a production policy merely because a fixture contains an allowed list. Document those limits alongside the score so a dashboard cannot imply broader assurance than the experiment provides.

Add rank-aware inspection without hiding the source

Two configurations can return the same relevant passage but place it at different ranks. If the interface shows only the first result, that order matters. Record the first useful rank or a chosen rank-aware metric in addition to set-level precision and recall. Define the cutoff before comparing runs; evaluating one system at k=3 and another at k=20 changes the question.

Review every critical failure by opening the actual passage. Was the answer condition split into another chunk? Was an obsolete revision labeled as relevant? Did the allowed corpus change between runs? A score should direct investigation, not end it. Keep a short failure note tied to the query ID and the exact source snapshot.

Keep the evaluation set honest as the system improves

Separate a development set from a held-out set. Use the development questions to improve source coverage and retrieval configuration. Run the held-out set at decision points without repeatedly editing the system to memorize its wording. If a held-out question becomes part of everyday tuning, move it into the development set and add new held-out cases.

Include natural variation rather than only paraphrases written by the same author. For a small internal tool, ask a few permitted colleagues how they would search for the task, then remove sensitive information before making reusable fixtures. Record where labels are uncertain and have a subject expert resolve important disagreements. A large number of weak labels is not automatically better than a small well-understood test set.

Make the release gate a written decision

Define which failures block release before seeing the latest score. A possible application policy treats any confirmed permission leak or unsupported high-impact claim as blocking, while a lower-ranked useful result triggers a quality review. The exact thresholds belong to the product’s risk and user requirements; this article does not prescribe one universal acceptable percentage.

Compare the chosen candidate with a baseline on the same corpus and queries. Preserve the configuration, source revision, model contract, returned IDs, measured durations and reviewer notes. When you change source text, a model, an analyzer or an index setting, state which experiment changed and rerun the affected checks.

After release, inspect failure patterns from real usage through an appropriate privacy process. Add representative, sanitized cases to the test set rather than copying raw customer conversations into a public benchmark. The practical result is a repeatable feedback loop: explicit expectations, a reproducible run, separate measures, evidence review and a documented release decision.