Lesson 0003 · Relevance & Internals

Analysis & the term-mismatch trap

How text becomes terms — and the silent bug where a query matches nothing.

~10 minWarm-up + quiz + labCheat sheet: here

The analyzer: three stages, in order

An analyzer transforms a text value into a stream of terms (tokens) through exactly three stages, always in this order: — Analyzer reference

The default standard analyzer is standard tokenizer + lowercase filter. So the field value "The Quick-Brown Fox" becomes three terms:

thequickbrownfox — lowercased, and the hyphen split into a boundary. Those are what land in the index. The original string is gone (kept only in _source, which is never searched).

Two runs of the pipeline: index time vs search time

Analysis happens twice, and this is the crux:

• Index time — when a document is indexed, each text field is analyzed and the resulting terms are written into the inverted index.
• Search time — when a full-text query (match) hits that field, the query string is analyzed the same way, and the resulting terms are looked up.

Matching is term-against-term. By default the same analyzer runs on both sides, so they line up. You can set a different search_analyzer (a real technique — e.g. index with an edge-ngram analyzer for autocomplete, search with a plain one). But if the two pipelines ever produce different tokens for the same word, lookups miss. — Analyzer reference

text vs keyword: analyzed or not

Whether a field is analyzed at all depends on its mapping type — the single most common source of “why didn't my query match” confusion: — Mapping reference

A keyword field stores the whole value as one untokenized term, byte-for-byte. No lowercasing, no splitting. That's why you sort and aggregate on keyword, never on text. A dynamically-mapped string becomes both: a text field plus a .keyword sub-field.

The term-mismatch trap

Now the interview-grade payoff. Two query types treat analysis oppositely:

So on a text field holding the document "Elasticsearch" (indexed, after lowercasing, as the term elasticsearch):

GET /docs/_search
{ "query": { "term": { "title": "Elasticsearch" } } }   // zero hits!

Zero hits — because term searches for the literal Elasticsearch (capital E), but the index only contains elasticsearch. The query never got lowercased, because term skips analysis. Swap to match (which analyzes Elasticsearch → elasticsearch) and it matches.

The mirror-image mistake: running match against a keyword field. There the stored term is the whole verbatim string, so a full-text match on a single word inside it also misses. The fix is always the same diagnostic: ask what terms actually exist.

Check yourself

From memory — effortful recall is the point. Feedback is immediate.

Hands-on: make the trap happen, then read the tokens

You have a live local cluster. Run these and watch analysis with your own eyes.

1 · Watch the pipeline turn text into terms

POST /_analyze
{ "analyzer": "standard", "text": "The Quick-Brown Fox" }

The response lists each token with its start_offset/end_offset. Confirm you get thequickbrownfox — lowercased and split on the hyphen. This is the ground truth of what's in your index.

2 · Reproduce the zero-hit bug

PUT /lab_analysis/_doc/1
{ "title": "Elasticsearch" }                            // text field, dynamically mapped

GET /lab_analysis/_search
{ "query": { "term":  { "title": "Elasticsearch" } } }  // predict: hits?

GET /lab_analysis/_search
{ "query": { "match": { "title": "Elasticsearch" } } }  // predict: hits?

Predict each before you run it. The term query returns nothing; match returns doc 1. Explain the difference out loud in one sentence — that sentence is the interview answer.

3 · Query the auto-created .keyword sub-field

GET /lab_analysis/_search
{ "query": { "term": { "title.keyword": "Elasticsearch" } } }   // now it hits

Here term works: the .keyword sub-field stored the verbatim Elasticsearch, so the exact string matches. Then DELETE /lab_analysis to clean up. Bring any surprise to your teacher.