Lesson 0007 · Query DSL
From “which documents match?” to “what is the shape of my data?”
From memory: a string mapped as text is analyzed into many tokens; a
keyword is stored as one exact term. Which of the two do you sort and aggregate
on? — keyword. Hold that: it's the number-one aggregation footgun, and it comes
straight from Lesson 0003.
Everything so far answered “which documents match?”. Aggregations answer a different
question — “what are the counts, averages, and groupings across those documents?” — in the
same request. The query selects a document set; the aggs
summarize that set.
— Aggregations reference
Because aggregations run over the query-matched documents, you compose them with everything from
Lesson 0006: filter to a subset with bool, then
summarize it. Set the top-level "size": 0 to skip the hits and return
only the aggregations — the common shape for dashboards.
— Aggregations reference
| Family | Answers | Produces | Examples |
|---|---|---|---|
| metric | “what's the number?” | a value over the docs | avg, sum, min/max, stats, cardinality |
| bucket | “what are the groups?” | buckets, each with a doc_count | terms, range, histogram, date_histogram |
A metric “calculates metrics, such as a sum or average, from field values.” A bucket “groups documents into buckets… based on field values, ranges, or other criteria,” each bucket being “a collection of documents that meet certain criteria.” — Aggregations reference · Definitive Guide: Buckets (A third family, pipeline, feeds on other aggregations' output rather than documents — park it for later.)
Buckets alone give you counts. The real analysis comes from putting an aggregation inside a
bucket's aggs — “buckets can also be nested inside other buckets, giving you a hierarchy,”
and metrics can be computed per bucket. This is drill-down:
— Definitive Guide: Buckets
Read it as: “group by status; within each group, average views.” The same
nesting continues — a bucket inside a bucket inside a bucket — to slice by status, then by year, then
average views, all in one pass.
terms aggregation: two things that trip people upterms is the aggregation you'll reach for most — one bucket per unique value, each with a
doc_count. Two realities to internalize:
keyword, not text
“By default, you cannot run a terms aggregation on a text field. Use a
keyword sub-field instead.” A text field is analyzed into tokens
(0003), so bucketing it would count word fragments,
not values — and it's disabled unless you turn on costly fielddata.
— terms aggregation
Default size is 10 (top ten buckets). Because each shard picks its own
top terms before results merge, “doc_count values for a terms aggregation
may be approximate.” The response tells you how much to worry via
doc_count_error_upper_bound and sum_other_doc_count.
— terms aggregation
(Same per-shard-then-merge shape as BM25's IDF in 0002 — distribution makes exactness expensive.)
From memory — effortful recall is the point. Feedback is immediate.
A metric aggregation produces:
avg, sum, cardinality…) compute a value from field values across the docs. Grouping into buckets is the bucket family's job.A bucket aggregation groups documents by:
doc_count.Aggregations are computed over:
query selects the scope; aggs summarize exactly that set. With no query it's all docs; with a bool it's the filtered subset.To compute a metric per bucket, you:
aggs. Nesting is what turns counts into drill-down analysis.A terms aggregation should run on:
keyword (one exact term). A text field is tokenized, so it's disabled for aggs by default and would bucket word fragments.On your live cluster. Seed a tiny index:
PUT /lab_aggs
{ "mappings": { "properties": {
"title": { "type": "text" },
"status": { "type": "keyword" },
"year": { "type": "integer" },
"views": { "type": "integer" }
}}}
POST /lab_aggs/_bulk
{"index":{"_id":1}}
{"title":"elasticsearch tuning","status":"published","year":2021,"views":900}
{"index":{"_id":2}}
{"title":"elasticsearch scaling","status":"published","year":2022,"views":1500}
{"index":{"_id":3}}
{"title":"elasticsearch basics","status":"draft","year":2023,"views":300}
{"index":{"_id":4}}
{"title":"postgres tuning","status":"published","year":2019,"views":200}size: 0)GET /lab_aggs/_search
{
"size": 0,
"query": { "match": { "title": "elasticsearch" } },
"aggs": { "avg_views": { "avg": { "field": "views" } } }
}Predict first: the query scopes to docs 1–3, so avg_views = (900+1500+300)/3 =
900, and hits is empty because size:0.
GET /lab_aggs/_search
{
"size": 0,
"aggs": {
"by_status": {
"terms": { "field": "status" },
"aggs": { "avg_views": { "avg": { "field": "views" } } }
}
}
}Predict the buckets: published (doc_count 3, avg of 900/1500/200 ≈ 867) and
draft (doc_count 1, avg 300). Note doc_count_error_upper_bound and
sum_other_doc_count in the response — both 0 on this tiny index, but that's where
approximation would show.
GET /lab_aggs/_search
{ "size": 0, "aggs": { "by_title": { "terms": { "field": "title" } } } } // title is text → errorRead the error — “Fielddata is disabled… Text fields are not optimised for operations that require
per-document field data like aggregations and sorting.” That's the 0003 rule enforced by the engine.
Then DELETE /lab_aggs and bring the response to your teacher.
Aggregations reference (Elastic Docs, current)
for the family overview, and the terms aggregation
page for the approximation and size/shard_size mechanics. Current-version — syntax is safe to copy.