Lesson 0005 · Internals & Performance
How an immutable index absorbs a stream of changes — and why it matters for speed.
From memory: the inverted index was described as sorted and fixed. What's the tension the moment new documents arrive? — You can't edit an immutable structure in place. This lesson is the resolution: ES never edits a segment; it writes new ones and merges. First, the key identity: a segment is an inverted index — exactly the structure from 0004. A shard is just a pile of them.
A Lucene segment is a self-contained inverted index, and once written it is never modified. Immutability is a feature, not a limitation: no locking, aggressive filesystem caching, and cheap compression all follow from "it never changes." — Definitive Guide: Dynamically updatable indices
So how do you update something you can't change? You don't. “Instead of rewriting the whole inverted index, add new supplementary indices to reflect more-recent changes.” A shard is a Lucene index = a collection of segments + a commit point (a file listing the live segments). — Dynamically updatable indices
A search doesn't run against "the index" — it runs against every segment in the shard (each a mini inverted index from 0004), and the results are combined. That's the hidden cost lurking below: more segments → more per-segment lookups per query.
New documents don't land in a segment instantly. They sit in an in-memory buffer, not yet searchable. Once per second (default), a refresh writes the buffer to a new segment — and only then are those docs visible to search. — Definitive Guide: Near real-time search
Refresh is cheap because the new segment is written to the filesystem cache first (readable immediately) and only later fsync'd to disk. This is precisely why we say ES is “near real time”: “document changes are not visible to search immediately, but will become visible within 1 second.” — Near real-time search
“I indexed a doc and my search doesn't see it” → almost always the refresh interval,
not a bug. And the flip side: heavy bulk indexing raises refresh_interval (or disables it)
to make fewer, larger segments — less merge pressure, faster indexing.
Segments are immutable, so a delete can't physically remove a document. Instead: “when a document
is ‘deleted,’ it is actually just marked as deleted in the .del file.”
The doc still sits in its segment; searches just filter it out.
— Dynamically updatable indices
An update is therefore a delete + re-index: the old version is marked deleted, the new version is written into a new segment. The old bytes linger until a merge sweeps them away.
A refresh every second means segments “explode” in number. A background merge continuously combines smaller segments into bigger ones. The crucial part: “deleted documents (or old versions of updated documents) are not copied over to the new bigger segment.” Merge is when deleted docs are truly purged and disk is reclaimed. — Definitive Guide: Segment merging
You don't enable merging — “it happens automatically while you are indexing and searching.” But big merges cost I/O and CPU, so “Elasticsearch throttles the merge process so that search still has enough resources.” — Segment merging
_forcemerge down to one segment is great for a read-only / no-longer-written
index (fewer segments → faster search, deleted docs gone). Running it on an actively indexed
index is an anti-pattern: it produces huge segments the normal merge policy won't touch, and can hurt
long-term. Say that and you sound senior.
From memory — effortful recall is the point. Feedback is immediate.
A Lucene segment, once written, is:
A newly indexed document becomes searchable:
Deleting a document at first only:
.del file and filtered from results, but stays on disk. An update is just a delete-mark plus a re-index into a new segment.The background merge reclaims space by:
ES is “near real-time” because search sees changes:
On your live cluster. This makes all three mechanisms visible in one sitting.
PUT /lab_seg
{ "settings": { "index": { "refresh_interval": "-1" } } } // disable auto refresh
PUT /lab_seg/_doc/1?refresh=false
{ "title": "quick brown fox" }
GET /lab_seg/_search
{ "query": { "match_all": {} } } // predict: 0 hits — not refreshed yet
POST /lab_seg/_refresh // now flush buffer → segment
GET /lab_seg/_search { "query": { "match_all": {} } } // now 1 hitPUT /lab_seg/_doc/2?refresh=true
{ "title": "lazy brown dog" }
GET /_cat/segments/lab_seg?v&h=segment,docs.count,docs.deleted // note deleted count
DELETE /lab_seg/_doc/1?refresh=true
GET /_cat/segments/lab_seg?v&h=segment,docs.count,docs.deleted // docs.deleted went UP — still on diskPOST /lab_seg/_forcemerge?max_num_segments=1
GET /_cat/segments/lab_seg?v&h=segment,docs.count,docs.deleted // one segment, docs.deleted back to 0Predict each docs.deleted value before you run it. Then DELETE /lab_seg.
Bring the segment listing to your teacher if anything surprises you.
Elasticsearch: The Definitive Guide — “Inside a Shard” (the short run of chapters: Dynamically Updatable Indices → Near Real-Time Search → Segment Merging). The clearest end-to-end account of this lifecycle. ⚠️ Legacy 2.x — trust the concepts, verify any settings syntax against current docs.