Lesson 0008 · Performance & Ops

Shard sizing & the over-sharding trap

How many shards should an index have? — and the failure mode that quietly wrecks clusters.

~12 minWarm-up + quiz + labCheat sheet: here

Warm-up — recall from 0005

From memory: a shard is made of what, and where does per-segment metadata live? — A shard is a Lucene index built from segments, and ES keeps segment metadata in heap (0005). That single fact — a shard costs heap even when nearly empty — is the whole reason this lesson exists.

A shard is not free

Each shard is a full Lucene index, and it carries fixed overhead regardless of how little data it holds: “Every index and every shard requires some memory and CPU resources… In most cases, a small set of large shards uses fewer resources than many small shards.” — Size your shards

Heap — segment metadata is held in heap, plus “a small amount of heap memory for each mapped field,” per segment.
A search thread — “each shard runs the search on a single CPU thread.” A query touching many shards consumes many threads.
Cluster state — every shard and index is tracked in cluster state, gossiped to every node.
File handles and recovery bookkeeping.

The central trade-off: parallelism vs overhead

Sharding buys parallelism — a search fans out across shards and runs them concurrently. But every shard added is more of the overhead above. The two ways to get it wrong:

too few / too bigSlow recovery after failure, less parallelism, “hot” shards that can't spread load.

the sweet spot10–50 GB per shard, under 200M docs — big enough to amortize overhead, small enough to move and recover.

too many / too smallOver-sharding: heap pressure, thread-pool exhaustion, cluster-state bloat, slower searches.

The docs put the cost starkly: “Searching a thousand 50MB shards will be substantially more expensive than searching a single 50GB shard containing the same data.” — Size your shards

The numbers worth memorizing

Guideline	Value	Why
Shard size	10–50 GB	50 GB is the common ILM rollover trigger; 10 GB a sensible floor.
Docs per shard	< 200 million	Lucene addressing & performance headroom.
Shards per data node	≤ ~1,000	“Each non-frozen data node supports up to 1,000 shards.”
Master heap	~1 GB / 3,000 indices	Cluster-state bookkeeping lives on master-eligible nodes.

Numbers: Size your shards · Node & shard size best practices

Interview edge — retire the old rule

The classic “20 shards per GB of heap” guideline was deprecated in Elasticsearch 8.3. Per-shard overhead dropped sharply in 7.x/8.x (compact metadata, off-heap structures, compressed cluster state), so quoting it now signals stale knowledge. Reason from shard size and per-node shard count instead. — Node & shard size best practices

Over-sharding: the quiet killer

“Too many shards can degrade search performance and make the cluster unstable. This is referred to as oversharding.” The thread-pool mechanism is the sharp edge: “Most searches hit multiple shards. Each shard runs the search on a single CPU thread… searches across a large number of shards can deplete a node's search thread pool.” — Size your shards

It usually creeps in via many small indices (per-tenant, per-day time series with low volume) each with default primaries. Fixes: consolidate with _reindex, reduce primaries with _shrink, or use ILM rollover so a new shard starts only at ~50 GB — not every day regardless of size.

Sizing in one move

Estimate total data, divide by a target shard size (~40 GB is a safe mid-band), round up:
primaries ≈ ceil(total_GB / 40). 300 GB → 8 primaries. Then check it against the per-node shard cap and your docs-per-shard headroom. Test with realistic data before committing — the docs' one universal rule is benchmark.

Check yourself

From memory — effortful recall is the point. Feedback is immediate.

Compared to many small shards, a few large shards use:

Per-shard overhead (heap for segment metadata, threads, cluster state) is fixed, so consolidating the same data into fewer, larger shards costs less.

Each shard runs a given search on:

One thread per shard per search. Fan a query across thousands of shards and you deplete the node's search thread pool — the core over-sharding failure.

The recommended shard size range is about:

10–50 GB: large enough to amortize per-shard overhead, small enough to relocate and recover quickly. Also keep under 200M docs per shard.

Too many tiny shards primarily causes:

Over-sharding burns heap (segment metadata + per-field), floods the search thread pool, and bloats cluster state. It doesn't lose data or change scoring.

The old “20 shards per GB heap” rule is now:

Retired in 8.3 after per-shard overhead dropped. Reason from shard size (10–50 GB) and per-node shard count instead.

Hands-on: read your cluster, then over-shard on purpose

On your live cluster. This makes the overhead visible in real numbers.

1 · Survey what you have

GET /_cluster/health           // active_shards, active_primary_shards
GET /_cat/nodes?v&h=name,heap.percent,ram.percent,node.role
GET /_cat/shards?v&s=index    // every shard, its size and node

Note your total shard count and per-node heap. On a single-node dev box you'll see replicas sit UNASSIGNED — that's expected (a replica never shares a node with its primary).

2 · Create a deliberately over-sharded index

PUT /lab_over
{ "settings": { "number_of_shards": 12, "number_of_replicas": 0 } }

PUT /lab_over/_doc/1?refresh=true
{ "title": "one tiny document" }

GET /_cat/shards/lab_over?v&h=shard,prirep,docs,store   // 12 shards for 1 doc

Twelve shards, eleven of them holding zero documents — each still a Lucene index with its own heap and file-handle cost. This is over-sharding in miniature.

3 · Size it properly, then clean up

Compute the right count for a real dataset: for an expected 300 GB, ceil(300 / 40) ≈ 8 primaries. Confirm the arithmetic, then DELETE /lab_over. Bring your _cat/shards survey to your teacher and we'll sanity-check your real indices.

Primary source — read this next

Size your shards (Elastic Docs, current) — the authoritative guidance, with the reasoning behind every number. Pair it with the Labs post Node & shard size best practices for the modern (post-8.3) rules.

Where this goes next

That completes the core arc: relevance (0001–0002), internals (0003–0005), and Query DSL + ops (0006–0008). The highest-value next session isn't a new topic — it's a mixed, interleaved review that pulls questions across all eight lessons to convert fluency into durable storage strength. Optional deeper dives after that: mappings design, cluster coordination, or nested/parent-child modeling.

← Lesson 0007 Lesson 0005 (segments) ↩ Shard sizing cheat sheet → Glossary → All lessons

Suspect a cluster is over-sharded? Bring me _cat/shards and _cat/nodes and we'll count shards per node, spot the tiny indices, and pick between _shrink, _reindex, and rollover.