Lesson 0007 · High availability & scaling

Automatic failover: Sentinel & Cluster

Who notices the primary died and promotes a replica — and how to scale writes past one node.

~12 minRetrieval quiz at the endCheat sheet: here

In Lesson 0006 you could promote a replica by hand (REPLICAOF NO ONE). But nobody wants to do that at 3 a.m., and a single primary still caps your write throughput and memory at one machine. Redis answers with two systems that automate the failover you already reasoned about: Sentinel (high availability for a single primary) and Redis Cluster (high availability plus sharding across many primaries).

Part 1 — Sentinel: automatic failover for one primary

Sentinel is a companion process with four jobs: — Redis Docs — Sentinel

Crucially, "Redis Sentinel is a distributed system": you run several Sentinels that cooperate — "There is no fun in having a failover system which is itself a single point of failure." That's what makes the next idea work.

The two thresholds — this answers your split-brain worry

SDOWN (subjectively down): one Sentinel stops getting valid PINGs and privately suspects the primary. ODOWN (objectively down): reached when "enough Sentinels (at least the number configured as the quorum)" agree. But detection and action are separate: the quorum is only used to detect the failure; to actually fail over, a Sentinel must be "elected leader… with the vote of the majority of the Sentinel processes."

This two-gate design is why a minority partition can't spuriously fail over: a lone (or minority) group of Sentinels can reach quorum to suspect the primary, but can never reach a majority to promote one — so a split brain can't elect two primaries. It's also why you run an odd number ≥ 3 Sentinels. The failover result: "a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications… are informed about the new address." — Redis Docs — Sentinel

Part 2 — Redis Cluster: shard writes across many primaries

Sentinel keeps one primary alive; it doesn't grow past one machine. Redis Cluster splits the keyspace across many primaries using 16384 hash slots: "to compute the hash slot for a given key, we simply take the CRC16 of the key modulo 16384" — i.e. HASH_SLOT = CRC16(key) mod 16384. — Redis Docs — Scaling · Cluster spec

Each primary owns a subset of the 16384 slots. Adding or removing a node just moves slots between nodes (resharding), with no downtime. Every shard is itself a mini primary+replicas group with its own automatic failover — so Cluster gives you HA and horizontal scale.

Client routing: MOVED vs ASK

A client keeps a map of slot → node. If it guesses wrong, the node redirects it — and the two redirect kinds mean different things: — Cluster spec

ReplyMeaningClient does
-MOVED slot host:portThis slot permanently lives on another node.Update its slot map, retry there (and for future keys).
-ASK host:portThis slot is currently migrating; just this key may already be on the target.Send ASKING + retry only this one query there. Map unchanged.

The multi-key constraint (hash tags)

Because keys live on different shards, a multi-key command whose keys span slots fails with a CROSSSLOT error — Redis "does not support multi-key operations… unless all of the keys… belong to the same hash slot." The escape hatch is a hash tag: "only the substring between { and } is hashed." So {user1000}.following and {user1000}.followers land in the same slot and can be used together. — Cluster spec

Cluster is still not strongly consistent (recall Lesson 0006)

"Redis Cluster does not guarantee strong consistency… it is possible that Redis Cluster will lose writes that were acknowledged" — "because it uses asynchronous replication." The same async trade-off you learned last lesson, now per-shard. On a partition, Cluster stays up only where a majority of primaries is reachable and each unreachable primary has a reachable replica; a primary that can't reach the majority stops accepting writes after node-timeout (so the minority side can't diverge). — Redis Docs — Scaling

Which one?

SentinelCluster
Gives youAutomatic failover (HA)HA + horizontal sharding
PrimariesOne (with replicas)Many (each with replicas)
Use whenData fits one node; you just need failoverData/writes exceed one node
CostSimple; no multi-key limitsMulti-key needs same slot (hash tags)

Check yourself

Answer from memory — effortful recall builds retention. Two questions revisit earlier lessons (spacing).

A Redis Sentinel deployment avoids a single point of failure by:

Sentinel is a distributed system: multiple Sentinels cooperate, so it keeps working even if some Sentinels are down. A failover system that's itself a SPOF is no use.

The quorum number of Sentinels is what's needed to:

Quorum only DETECTS failure (reaching ODOWN). Actually performing the failover needs a Sentinel elected leader by a MAJORITY of all Sentinels — that separation stops a minority partition failing over.

A key's hash slot in Redis Cluster is:

There are 16384 slots; HASH_SLOT = CRC16(key) mod 16384. Each primary owns a subset of those slots, and resharding moves slots between nodes with no downtime.

A client gets -MOVED. Compared to -ASK, it should:

MOVED = the slot permanently lives elsewhere, so update the map and route future keys there. ASK = slot is mid-migration, a one-off redirect (ASKING + retry just this query), map unchanged.

MGET a b across two shards fails. The fix that forces one slot is:

Multi-key ops need one slot or you get CROSSSLOT. A hash tag ({...}) hashes only the braced substring, so {u1}:a and {u1}:b share a slot and can be used together.

Interleave — Redis Cluster can lose acknowledged writes for the same reason as Lesson 0006:

Each shard replicates asynchronously, so a primary can ack a write and fail before a replica gets it — Cluster "does not guarantee strong consistency," just like a plain primary/replica setup.

Hands-on: hash slots and redirection

A full cluster needs 6 nodes; you can explore slot math on a single instance, and spin up a real cluster if you have Docker.

1 · See the slot math (works on any Redis)

redis-cli CLUSTER KEYSLOT user1000               # the CRC16 mod 16384 slot
redis-cli CLUSTER KEYSLOT "{user1000}.following"
redis-cli CLUSTER KEYSLOT "{user1000}.followers" # SAME slot as above — hash tag
redis-cli CLUSTER KEYSLOT "user1000.followers"   # DIFFERENT — no tag

2 · Spin up a real cluster (optional, needs Docker)

docker run --rm -p 7000-7005:7000-7005 -e "IP=0.0.0.0" grokzen/redis-cluster:latest
redis-cli -c -p 7000 SET foo bar     # -c follows MOVED/ASK automatically
redis-cli -c -p 7000 CLUSTER SHARDS  # which node owns which slots
redis-cli -p 7000 MSET a 1 b 2       # likely (error) CROSSSLOT ... 
redis-cli -p 7000 MSET "{t}a" 1 "{t}b" 2   # OK — same slot via hash tag
Predict first

Before step 1: will the two {user1000}... keys share a slot? Why does the un-tagged user1000.followers differ? And in step 2, why does plain MSET a b risk CROSSSLOT but the tagged one doesn't? Commit, then run it.

Bring your CLUSTER KEYSLOT numbers (and any CROSSSLOT error) to your teacher.

Primary source — read this next

Redis Docs — Scale with Redis Cluster for the hands-on model, and Sentinel for HA on a single primary. For internals-grade depth, the Cluster specification covers slots, redirection, and the failure model in full.

Where this goes next

That completes the arc: from the single thread (0001) all the way to a self-healing, sharded cluster. Lesson 0008 is a mixed interleaved review — a cross-lesson quiz that jumps between structures, caching, persistence, and HA to convert your fluency into durable long-term memory (the real goal). It's the desirable difficulty that makes all of this stick.

← Lesson 0006 Cheat sheet → Glossary → All lessons
I'm your teacher for this — bring me a scaling requirement and we'll decide Sentinel vs Cluster, or reason through why an even number of Sentinels is a bad idea, or how a hash tag saves a multi-key transaction.