Lemmatization (engine, not yet exposed).
Lemmatization exists in the OriginChainDB engine, but it is not yet selectable through the HTTP API. There is no language or analyzer query parameter today. The analyzer the FTS API actually applies is Unicode tokenize + lowercase, and nothing else - no stemming and no lemmatization. Concretely: q=ran will not match a document that says "running" through today's API. This page describes what lemmatization will do once it is exposed; treat it as roadmap, not as something you can call now.
Lemmatization reduces every inflected form to its canonical lemma using a dictionary look-up. Unlike Snowball stemming (a deterministic suffix-strip), lemmatization handles irregular forms — "ran" → "run", "geese" → "goose", "better" → "good". The trade is bigger memory (the dictionary) for higher precision on natural-language fields.
Stem vs lemma — worked example.
running run run ran ran run runs run run better bet good geese gees goose mice mice mouse Notice "better" and "geese" — Snowball can't handle either correctly. Lemma does, because the dictionary maps them directly to the canonical form. Both columns are what the engine computes internally; neither is reachable from the API today.
9 languages (planned).
These are the languages the engine has lemma dictionaries for. Once the analyzer becomes API-selectable, coverage will track Snowball stemming (18 languages) over time. Until then, none of these can be turned on from a query.
What you can rely on today.
Until lemmatization is exposed, the FTS API normalises text in exactly two steps, applied identically at index time and query time:
- Unicode tokenize — split into words by Unicode word boundaries (UAX #29).
- Lowercase — so
Wirelessmatcheswireless. - The one customer-controlled transform that is wired through is synonyms — install a per-(table, field) map via
POST /fts/:t/:f/synonymsand members of a class match each other. If you need "ran" to find "run" today, model it as a synonym class, not as lemmatization.
Where lemma will help (once exposed).
- Natural-language fields (article bodies, support tickets, product descriptions) where false positives from stem-collisions hurt precision.
- You want "ran" and "running" to match "run" — Snowball can't handle the irregular form, and today's API matches neither.
- Field is searchable by humans, not by structured tokens like SKUs or log codes.