Lesson 0006 · Query DSL

The bool query

The workhorse compound query — four clause types, one scoring rule, one gotcha.

~11 minWarm-up + quiz + labCheat sheet: here

Warm-up — recall from 0001

From memory: of the four bool keys, which two run in query context (produce a _score) and which two run in filter context (no score, cacheable)? — Scored: must, should. Unscored + cached: filter, must_not. That single split (Lesson 0001) is the backbone of everything below.

Spaced recall (30s) — from 0004

Before the new material, one carry-over you needed two passes on last time: IDF's n equals what, physically? — the length of the term's postings list (its document frequency). Say it, then read on.

The four occurrence types

bool combines clauses under four keys. Each key answers two questions at once: does it filter the result set? and does it affect the score? — Query DSL: bool query

ClauseMeaningScores?Cached?SQL-ish
mustmust matchyesnoAND (scored)
shouldpreferably matchesyesnoOR / boost
filtermust matchnoyesAND (unscored)
must_notmust not matchnoyesAND NOT

Note that must and filter are the same logical constraint — both require the clause to match. The only difference is scoring: must contributes to _score; filter ignores it and becomes cacheable. “The filter and must_not clauses… include or exclude results without impacting the score.” — bool query

The scoring rule: more matches is better

“The bool query takes a more-matches-is-better approach, so the score from each matching must or should clause will be added together to provide the final _score.” — bool query Each scored clause computes BM25 (0002); bool just sums them. So should clauses are how you say “also nice if it matches this” — they lift the score of docs that satisfy them.

The gotcha: what should means depends on its neighbours

This is the single most-missed bool behaviour, driven by the minimum_should_match default:

The rule, exactly

“If the bool query includes at least one should clause and no must or filter clauses, the default value is 1. Otherwise, the default value is 0.” — bool query

“I added a should and got fewer results” (or “more”) is almost always this rule biting.

The canonical shape

Put each concern in the clause that fits its scoring and caching intent — this is the design skill:

{ "query": { "bool": {
  "must":     { "match": { "title": "tuning" } },        // relevance — scored
  "filter":   [ { "term":  { "status": "published" } },  // yes/no — cached
                { "range": { "year": { "gte": 2020 } } } ],
  "must_not":   { "match": { "title": "deprecated" } },  // exclude — cached
  "should":     { "match": { "title": "elasticsearch" } } // boost — optional
} } }

Full-text relevance → must/should. Exact/range/enum constraints → filter (cheap, cached, per 0001). Exclusions → must_not.

Check yourself

From memory — effortful recall is the point. Feedback is immediate.

Which two clause types contribute to _score?

must and should run in query context and their BM25 scores are summed. filter and must_not run in filter context — no score.

A filter clause affects results by:

filter is the same "must match" constraint as must, but its score is ignored — which also makes it cacheable.

With a must clause present, should clauses become:

When a must/filter exists, minimum_should_match defaults to 0, so should only re-ranks — it never filters the result set.

With no must or filter, how many should must match?

With only should clauses, minimum_should_match defaults to 1 — at least one must match, so should acts like OR.

must_not and filter are cheap on repeat because they are:

Filter-context clauses skip scoring and are eligible for the node query cache (bitsets of matching docs) — the 0001 caching story.

Hands-on: place clauses, then watch should behave two ways

On your live cluster. Seed a tiny index:

PUT /lab_bool
{ "mappings": { "properties": {
  "title":  { "type": "text" },
  "status": { "type": "keyword" },
  "year":   { "type": "integer" }
}}}

POST /lab_bool/_bulk
{"index":{"_id":1}}
{"title":"elasticsearch tuning guide","status":"published","year":2021}
{"index":{"_id":2}}
{"title":"elasticsearch scaling guide","status":"published","year":2022}
{"index":{"_id":3}}
{"title":"postgres tuning guide","status":"published","year":2019}
{"index":{"_id":4}}
{"title":"elasticsearch basics","status":"draft","year":2023}

1 · The canonical shape — predict the survivors

GET /lab_bool/_search
{ "query": { "bool": {
  "must":   { "match": { "title": "guide" } },
  "filter": [ { "term": { "status": "published" } },
              { "range": { "year": { "gte": 2020 } } } ],
  "should": { "match": { "title": "tuning" } }
} } }

Predict before running: docs 1 and 2 survive (3 is pre-2020, 4 has no “guide” and is draft), and should pushes doc 1 above doc 2 because it also matches “tuning” — but both still appear. The should re-ranked; it did not filter.

2 · Drop the must/filter — same should, new meaning

GET /lab_bool/_search
{ "query": { "bool": {
  "should": [ { "match": { "title": "tuning" } },
              { "match": { "title": "scaling" } } ]
} } }

Now minimum_should_match is 1, so only docs matching “tuning” or “scaling” return (1, 2, 3 — not 4). Same should clauses, opposite role. Then DELETE /lab_bool and bring any surprise to your teacher.

Primary source — read this next

Query DSL — Boolean query (Elastic Docs, current). The authoritative reference for the four occurrence types, the minimum_should_match default, and the more-matches-is-better scoring rule. Current-version, so the syntax is safe to copy.

Where this goes next

You can now filter and rank. The other half of the DSL is aggregations: instead of “which docs match?”, ask “what are the shapes in my data?” — counts, averages, and buckets over the same matching set. That's Lesson 0007.

← Lesson 0005 Lesson 0001 (context) ↩ bool cheat sheet → Glossary → All lessons
Got a real query where results changed unexpectedly when you added a clause? Bring me the bool and we'll trace which occurrence type — and which minimum_should_match default — is responsible.