Redis Sentinel Mode
Redis Sentinel Core Implementation
Sentinel must be able to discover nodes — the master, replicas, and other sentinels — i.e. it needs to know the current Redis deployment network view.
Sentinel must evaluate node health and determine whether the master is working normally.
Sentinel must reach consensus that the master is down and perform a failover, electing a leader node to carry out the failover.
Sentinel Initialization
Sentinel initialization mainly establishes connections to the master and any newly discovered nodes, subscribes to the master/replica hello channel, and starts the periodic timer tasks. The periodic tasks repeatedly check node validity, decide whether a node is subjectively or objectively down, and determine whether a failover should be triggered, entering the failover state machine.
The sentinel initialization flow is as follows:
main(int argc, char **argv)
{
// Check whether this is sentinel mode
server.sentinel_mode = checkForSentinelMode(argc,argv, exec_name);
ACLInit(); // Initialize ACL
if (server.sentinel_mode) {
initSentinelConfig();
initSentinel();
}
if (argc >= 2) {
loadServerConfig(server.configfile, config_from_stdin, options);
if (server.sentinel_mode) loadSentinelConfigFromQueue();
}
// A sentinel node must have a writable config file; it persists
// discovered replicas and other sentinels into the config file.
if (server.sentinel_mode) sentinelCheckConfigFile();
server.supervised = redisIsSupervised(server.supervised_mode);
initServer();
if (server.sentinel_mode) {
ACLLoadUsersAtStartup();
InitServerLast();
sentinelIsRunning(){ // Generate the sentinel ID and write it to the config
sentinelGenerateInitialMonitorEvents(){
sentinelEvent(LL_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
}
}
}
aeMain(server.el); // Start the event loop
}In sentinelIsRunning() the sentinel ID is generated and written to the config file, and a +monitor event is produced; at startup the sentinel broadcasts the list of masters it is monitoring.
205502:X 27 Jan 2026 19:43:07.403 * Sentinel new configuration saved on disk
205502:X 27 Jan 2026 19:43:09.212 # Sentinel ID is b3461fcadb4a23ebc9534c08228612c72e13d3a8
The current Sentinel node begins monitoring the master named mymaster, and at least 2 Sentinel nodes must agree to declare it failed
205502:X 27 Jan 2026 19:47:39.470 # +monitor master mymaster 192.168.232.128 6379 quorum 2Sentinel must be configured with a writable config file; when it discovers replicas and other sentinels, it writes that information into the config file for persistent storage.
The initialization flow also calls the serverCron function:
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
/* Run the Sentinel timer if we are in sentinel mode. */
if (server.sentinel_mode) sentinelTimer();
}
void sentinelTimer(void) {
sentinelCheckTiltCondition();
sentinelHandleDictOfRedisInstances(sentinel.masters);
sentinelRunPendingScripts();
sentinelCollectTerminatedScripts();
sentinelKillTimedoutScripts();
/* We continuously change the frequency of the Redis "timer interrupt"
* in order to desynchronize every Sentinel from every other.
* This non-determinism avoids that Sentinels started at the same time
* exactly continue to stay synchronized asking to be voted at the
* same time again and again (resulting in nobody likely winning the
* election because of split brain voting). */
server.hz = CONFIG_DEFAULT_HZ + rand() % CONFIG_DEFAULT_HZ;
}Node Discovery and Failure Detection
Node Discovery
One of Sentinel’s core responsibilities is to determine which node has failed and perform a failover. To judge node failure, the first requirement is to know the information of all nodes in the cluster.
Node discovery naturally brings the gossip protocol to mind. In fact, the idea behind Redis node discovery is exactly the gossip protocol: the sentinel config file is configured with the master node information. That is, the sentinel discovers other node information through the master, and persists it in the config file. Every sentinel is configured with the master it monitors, so it can obtain all node information via the master.
sentinel monitor mymaster 127.0.0.1 6379 2 # sentinel config file, monitoring the masterIn its initial state the sentinel only knows the master information and must discover the rest automatically:
- Replica information (obtained periodically by running the INFO command against the master to get the replica list)
- Sentinel node information (via the Pub/Sub subscription mechanism: a sentinel periodically publishes its own hello message on that channel, equivalent to the push message in gossip, and other nodes update their view upon receiving it)
Sentinel Node Discovery
Sentinel uses the Pub/Sub (publish/subscribe) mechanism to auto-discover other sentinels. All sentinels monitoring the same master subscribe to a special channel __sentinel__:hello and periodically publish their own information there, achieving mutual discovery and information synchronization.
#define SENTINEL_HELLO_CHANNEL "__sentinel__:hello"The sentinel node discovery flow is as follows:
sequenceDiagram
participant S1 as Sentinel 1
participant M as Master
participant S2 as Sentinel 2
participant S3 as Sentinel 3
Note over S1,S3: All Sentinels subscribe to __sentinel__:hello
S1->>M: SUBSCRIBE __sentinel__:hello
S2->>M: SUBSCRIBE __sentinel__:hello
S3->>M: SUBSCRIBE __sentinel__:hello
Note over S1,S3: Publish a Hello message every 2 seconds
S1->>M: PUBLISH __sentinel__:hello<br/>"IP,Port,RunID,Epoch,..."
M->>S2: Forward S1's message
M->>S3: Forward S1's message
Note over S2: Discovers Sentinel 1
Note over S3: Discovers Sentinel 1
S2->>M: PUBLISH __sentinel__:hello<br/>"IP,Port,RunID,Epoch,..."
M->>S1: Forward S2's message
M->>S3: Forward S2's message
Note over S1: Discovers Sentinel 2
Note over S3: Discovers Sentinel 2
Note over S1,S3: All Sentinels have discovered each otherReplica Information Discovery
The sentinel periodically sends the INFO command to the master to obtain replica information. The master knows all of its replicas; running the INFO command:
127.0.0.1:6379> info
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.232.137,port=6379,state=online,offset=524642,lag=0By parsing the result of the INFO command, replica information can be obtained and the config file updated.
// Sentinel node discovers replica information
58221:X 29 Jan 2026 11:37:12.703 * +slave slave 192.168.232.137:6379 192.168.232.137 6379 @ mymaster 192.168.232.128 6379Periodically send the INFO, PING commands and the hello message.
void sentinelSendPeriodicCommands(sentinelRedisInstance *ri) {
// Send the INFO command to masters and replicas
if ((ri->flags & SRI_SENTINEL) == 0 &&
(ri->info_refresh == 0 ||
(now - ri->info_refresh) > info_period))
{
// Callback for handling the INFO command reply
retval = redisAsyncCommand(ri->link->cc,
sentinelInfoReplyCallback, ri, "%s",
sentinelInstanceMapCommand(ri,"INFO"));
if (retval == C_OK) ri->link->pending_commands++;
}
// Send the PING command to masters, replicas, and sentinels
if ((now - ri->link->last_pong_time) > ping_period &&
(now - ri->link->last_ping_time) > ping_period/2) {
sentinelSendPing(ri);
}
// Publish the hello message
if ((now - ri->last_pub_time) > sentinel_publish_period) {
sentinelSendHello(ri);
}
}Handling the INFO command result:
void sentinelInfoReplyCallback(redisAsyncContext *c, void *reply, void *privdata) {
// Get the INFO command result
if (r->type == REDIS_REPLY_STRING)
sentinelRefreshInstanceInfo(ri,r->str);
}Establishing Connections
When a Sentinel establishes a connection with a Master or a Slave, it creates two independent connections:
- Command connection: used to send PING, INFO, etc. The sentinel establishes a command connection with every other node (master, replica, sentinel).
- Message channel: subscribes to the
__sentinel__:hellochannel. The sentinel only establishes a message channel with masters and replicas, subscribing to__sentinel__:hello.
// Called by serverCron
void sentinelTimer(void) {
// Recursively run periodic tasks for every Redis instance the sentinel monitors (master, replica, other sentinels)
sentinelHandleDictOfRedisInstances(sentinel.masters);
}
void sentinelHandleDictOfRedisInstances(dict *instances) {
dictIterator *di;
dictEntry *de;
sentinelRedisInstance *switch_to_promoted = NULL;
/* There are a number of things we need to perform against every master. */
di = dictGetIterator(instances);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *ri = dictGetVal(de);
// Run periodic tasks for the current instance
sentinelHandleRedisInstance(ri);
if (ri->flags & SRI_MASTER) { // current instance is a master
sentinelHandleDictOfRedisInstances(ri->slaves); // recurse into replicas
sentinelHandleDictOfRedisInstances(ri->sentinels); // recurse into sentinels
if (ri->failover_state == SENTINEL_FAILOVER_STATE_UPDATE_CONFIG) {
switch_to_promoted = ri;
}
}
}
if (switch_to_promoted)
sentinelFailoverSwitchToPromotedSlave(switch_to_promoted);
dictReleaseIterator(di);
}
// The sentinel's core periodic task handler
void sentinelHandleRedisInstance(sentinelRedisInstance *ri) {
/* ========== MONITORING HALF ============ */
/* Every kind of instance */
sentinelReconnectInstance(ri); // (Re)connect if the connection to the instance dropped
sentinelSendPeriodicCommands(ri); // Periodically send commands to probe the instance
/* ============== ACTING HALF ============= */
/* We don't proceed with the acting half if we are in TILT mode.
* TILT happens when we find something odd with the time, like a
* sudden change in the clock. */
if (sentinel.tilt) {
if (mstime()-sentinel.tilt_start_time < sentinel_tilt_period) return;
sentinel.tilt = 0;
sentinelEvent(LL_WARNING,"-tilt",NULL,"#tilt mode exited");
}
// Check whether the instance is subjectively down
sentinelCheckSubjectivelyDown(ri);
/* Masters and slaves */
if (ri->flags & (SRI_MASTER|SRI_SLAVE)) {
/* Nothing so far. */
}
// Periodically check whether the master is objectively down
if (ri->flags & SRI_MASTER) {
sentinelCheckObjectivelyDown(ri);
if (sentinelStartFailoverIfNeeded(ri)) // Do we need to start a failover?
sentinelAskMasterStateToOtherSentinels(ri,SENTINEL_ASK_FORCED);
sentinelFailoverStateMachine(ri);
// Ask other sentinels about the master state
sentinelAskMasterStateToOtherSentinels(ri,SENTINEL_NO_FLAGS);
}
}The two connections are established as follows:
void sentinelReconnectInstance(sentinelRedisInstance *ri) {
// If the command connection dropped, re-establish it
if (link->cc == NULL) {
link->cc = redisAsyncConnectBind(ri->addr->ip,ri->addr->port,server.bind_source_addr);
redisAeAttach(server.el,link->cc);
// Send the AUTH command; the sentinel acts as a client and must authenticate
sentinelSendAuthIfNeeded(ri,link->cc);
// Send PING
sentinelSendPing(ri);
}
/* Pub / Sub */
if ((ri->flags & (SRI_MASTER|SRI_SLAVE)) && link->pc == NULL) {
link->pc = redisAsyncConnectBind(ri->addr->ip,ri->addr->port,server.bind_source_addr);
if (link->pc && !link->pc->err) anetCloexec(link->pc->c.fd);
if (!link->pc) {
sentinelEvent(LL_DEBUG,"-pubsub-link-reconnection",ri,"%@ #Failed to establish connection");
} else if (!link->pc->err && server.tls_replication &&
(instanceLinkNegotiateTLS(link->pc) == C_ERR)) {
sentinelEvent(LL_DEBUG,"-pubsub-link-reconnection",ri,"%@ #Failed to initialize TLS");
} else if (link->pc->err) {
sentinelEvent(LL_DEBUG,"-pubsub-link-reconnection",ri,"%@ #%s",
link->pc->errstr);
instanceLinkCloseConnection(link,link->pc);
} else {
int retval;
link->pc_conn_time = mstime();
link->pc->data = link;
redisAeAttach(server.el,link->pc);
redisAsyncSetConnectCallback(link->pc,
sentinelLinkEstablishedCallback);
redisAsyncSetDisconnectCallback(link->pc,
sentinelDisconnectCallback);
sentinelSendAuthIfNeeded(ri,link->pc);
sentinelSetClientName(ri,link->pc,"pubsub");
// Subscribe to the __sentinel__:hello channel
// Set the callback that handles hello messages on the subscribed channel
retval = redisAsyncCommand(link->pc,
sentinelReceiveHelloMessages, ri, "%s %s",
sentinelInstanceMapCommand(ri,"SUBSCRIBE"),
SENTINEL_HELLO_CHANNEL);
if (retval != C_OK) {
/* If we can't subscribe, the Pub/Sub connection is useless
* and we can simply disconnect it and try again. */
instanceLinkCloseConnection(link,link->pc);
return;
}
}
}
}Sending Hello
After establishing the command connection and subscribing to the __sentinel__:hello channel, the sentinel periodically (default 2s) sends a hello message to __sentinel__:hello. The hello message contains its own information and the monitored master configuration.
The sentinel receives other sentinels’ hello messages from the subscribed __sentinel__:hello channel, parses them, and updates its local cluster view.
Hello message format:
sentinel_ip,sentinel_port,sentinel_runid,current_epoch, master_name,master_ip,master_port,master_config_epoch
| Field | Description | Purpose |
|---|---|---|
| 0 | Sentinel IP | Node network address |
| 1 | Sentinel port | Node communication port |
| 2 | Sentinel runid | Unique node identifier |
| 3 | current_epoch | Logical clock (used for failover) |
| 4 | master_name | Monitored master name |
| 5 | master_ip | Master’s current IP |
| 6 | master_port | Master’s current port |
| 7 | master_config_epoch | Master config epoch |
The epoch is used to prevent split brain, ensuring the newest configuration wins.
- Each failover increments the epoch
- A sentinel only updates its local config when it receives a higher config_epoch
A configuration with a higher config_epoch is the new master configuration, preventing an old configuration from overwriting a new one.
Failure Detection
Sentinel must be able to detect failed nodes. How? By periodically sending PING to all (master, replica, sentinel) nodes, checking the PONG reply, and recording the time the last PING was sent and the time the last PONG was received.
sentinelTimer
--> sentinelHandleDictOfRedisInstances
--> sentinelHandleRedisInstance
--> sentinelSendPeriodicCommands
--> sentinelSendPingBy default PING is sent every 1s:
/* Send PING to all the three kinds of instances. */
if ((now - ri->link->last_pong_time) > ping_period &&
(now - ri->link->last_ping_time) > ping_period/2) {
sentinelSendPing(ri);
}Every 1s:
#define SENTINEL_PING_PERIOD 1000
static mstime_t sentinel_info_period = 10000;
static mstime_t sentinel_ping_period = SENTINEL_PING_PERIOD;A single node’s judgment is a subjective down; when more than quorum nodes consider an instance failed, it is objectively down.
The logic for judging subjective down:
- Time without a PONG reply > the
down-after-millisecondssetting - For a master, it also checks the INFO reply (to rule out transient network jitter)
void sentinelCheckSubjectivelyDown(sentinelRedisInstance *ri) {
mstime_t elapsed = 0;
if (ri->link->act_ping_time) // How long since the last PING
elapsed = mstime() - ri->link->act_ping_time;
else if (ri->link->disconnected)
elapsed = mstime() - ri->link->last_avail_time;
/* Update the SDOWN flag. We believe the instance is SDOWN if:
*
* 1) It is not replying.
* 2) We believe it is a master, it reports to be a slave for enough time
* to meet the down_after_period, plus enough time to get two times
* INFO report from the instance. */
if (elapsed > ri->down_after_period ||
(ri->flags & SRI_MASTER &&
ri->role_reported == SRI_SLAVE &&
mstime() - ri->role_reported_time >
(ri->down_after_period+sentinel_info_period*2)) ||
(ri->flags & SRI_MASTER_REBOOT &&
mstime()-ri->master_reboot_since_time > ri->master_reboot_down_after_period))
{
/* Is subjectively down */
if ((ri->flags & SRI_S_DOWN) == 0) {
sentinelEvent(LL_WARNING,"+sdown",ri,"%@");
ri->s_down_since_time = mstime();
ri->flags |= SRI_S_DOWN;
}
} else {
/* Is subjectively up */
if (ri->flags & SRI_S_DOWN) {
sentinelEvent(LL_WARNING,"-sdown",ri,"%@");
ri->flags &= ~(SRI_S_DOWN|SRI_SCRIPT_KILL_SENT);
}
}
}Configure the subjective-down timeout:
sentinel down-after-milliseconds mymaster 30000 # sentinel config file, set mymaster failure timeThe sentinel periodically checks whether the master is objectively down. When this sentinel considers the master subjectively down, it sends a SENTINEL IS-MASTER-DOWN-BY-ADDR request to other sentinels asking for their view of the master.
sentinelHandleRedisInstance()
--> sentinelAskMasterStateToOtherSentinels(ri,SENTINEL_NO_FLAGS) {
retval = redisAsyncCommand(ri->link->cc,
sentinelReceiveIsMasterDownReply, ri,
"%s is-master-down-by-addr %s %s %llu %s",
sentinelInstanceMapCommand(ri,"SENTINEL"),
announceSentinelAddr(master->addr), port,
sentinel.current_epoch,
(master->failover_state > SENTINEL_FAILOVER_STATE_NONE) ?
sentinel.myid : "*");
}On receiving the SENTINEL IS-MASTER-DOWN-BY-ADDR request, the other sentinel replies whether it considers the master subjectively down:
addReply(c, isdown ? shared.cone : shared.czero);The sentinel periodically calls sentinelCheckObjectivelyDown to check whether the master is objectively down.
void sentinelCheckObjectivelyDown(sentinelRedisInstance *master) {
dictIterator *di;
dictEntry *de;
unsigned int quorum = 0, odown = 0;
if (master->flags & SRI_S_DOWN) {
/* Is down for enough sentinels? */
quorum = 1; /* the current sentinel. */
/* Count all the other sentinels. */
di = dictGetIterator(master->sentinels);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *ri = dictGetVal(de);
// Collect the sentinels' view of the master
if (ri->flags & SRI_MASTER_DOWN) quorum++;
}
dictReleaseIterator(di);
if (quorum >= master->quorum) odown = 1;
}
/* Set the flag accordingly to the outcome. */
if (odown) {
if ((master->flags & SRI_O_DOWN) == 0) {
sentinelEvent(LL_WARNING,"+odown",master,"%@ #quorum %d/%d",
quorum, master->quorum);
master->flags |= SRI_O_DOWN; // Mark master objectively down
master->o_down_since_time = mstime();
}
} else {
if (master->flags & SRI_O_DOWN) {
sentinelEvent(LL_WARNING,"-odown",master,"%@");
master->flags &= ~SRI_O_DOWN;
}
}
}After determining objective down, it decides whether a failover is needed. The conditions:
- The master is in the objective-down state
- No failover is currently in progress
- It has not recently already attempted a failover (to avoid too-frequent failover attempts)
void sentinelHandleRedisInstance(sentinelRedisInstance *ri) {
/* Only masters */
if (ri->flags & SRI_MASTER) {
// Check whether the master is objectively down
sentinelCheckObjectivelyDown(ri);
// Decide whether a failover is needed
if (sentinelStartFailoverIfNeeded(ri))
// With SENTINEL_ASK_FORCED, ask sentinels to vote and elect a leader
sentinelAskMasterStateToOtherSentinels(ri,SENTINEL_ASK_FORCED);
// Failover state machine
sentinelFailoverStateMachine(ri);
sentinelAskMasterStateToOtherSentinels(ri,SENTINEL_NO_FLAGS);
}
}If the conditions pass, sentinelStartFailover is called to begin the failover:
/* Setup the master state to start a failover. */
void sentinelStartFailover(sentinelRedisInstance *master) {
serverAssert(master->flags & SRI_MASTER);
master->failover_state = SENTINEL_FAILOVER_STATE_WAIT_START;
master->flags |= SRI_FAILOVER_IN_PROGRESS;
master->failover_epoch = ++sentinel.current_epoch; // increment epoch
sentinelEvent(LL_WARNING,"+new-epoch",master,"%llu",
(unsigned long long) sentinel.current_epoch);
sentinelEvent(LL_WARNING,"+try-failover",master,"%@");
master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
master->failover_state_change_time = mstime();
}Failover
Failover has two main phases: the leader election phase and the actual failover task execution.
Node Election
After the sentinel enters the SENTINEL_FAILOVER_STATE_WAIT_START state — i.e. once the master is confirmed objectively down and a failover is needed — it enters the leader election phase. The first sentinel to reach this state starts requesting votes from the others.
void sentinelAskMasterStateToOtherSentinels(sentinelRedisInstance *master, int flags) {
// * means "request a vote for myself"
retval = redisAsyncCommand(ri->link->cc,
sentinelReceiveIsMasterDownReply, ri,
"%s is-master-down-by-addr %s %s %llu %s",
sentinelInstanceMapCommand(ri,"SENTINEL"),
announceSentinelAddr(master->addr), port,
sentinel.current_epoch,
(master->failover_state > SENTINEL_FAILOVER_STATE_NONE) ?
sentinel.myid : "*");
}Vote format:
SENTINEL is-master-down-by-addr <ip> <port> <current-epoch> <runid>- runid of
*means “I request a vote” - a concrete runid means “I vote for that ID”
On receiving the request, the other node decides whether to vote:
void sentinelCommand(client *c) {
/* Vote for the master (or fetch the previous vote) if the request
* includes a runid, otherwise the sender is not seeking for a vote. */
if (ri && ri->flags & SRI_MASTER && strcasecmp(c->argv[5]->ptr,"*")) {
leader = sentinelVoteLeader(ri,(uint64_t)req_epoch,
c->argv[5]->ptr,
&leader_epoch);
}
}The sentinels vote among themselves to elect a leader; the others act as observers. Why elect a leader? To prevent split brain. A vote is only granted if the epoch is greater than the local one, so for a given logical epoch (i.e. this failover round) a sentinel can vote for only one candidate.
char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch) {
// Sync the global epoch locally and update current_epoch in sentinel.conf
if (req_epoch > sentinel.current_epoch) {
sentinel.current_epoch = req_epoch;
sentinelFlushConfig();
sentinelEvent(LL_WARNING,"+new-epoch",master,"%llu",
(unsigned long long) sentinel.current_epoch);
}
// Decide whether to vote
// 1. This sentinel has not yet voted in this epoch
// 2. The vote epoch must be >= the current epoch
if (master->leader_epoch < req_epoch && sentinel.current_epoch <= req_epoch)
{
sdsfree(master->leader);
master->leader = sdsnew(req_runid);
master->leader_epoch = sentinel.current_epoch;
sentinelFlushConfig();
sentinelEvent(LL_WARNING,"+vote-for-leader",master,"%s %llu",
master->leader, (unsigned long long) master->leader_epoch);
// If not voting for self, set a random delay to reduce competing failovers.
// Prevents multiple sentinels all thinking they should be leader, like leader election in raft.
if (strcasecmp(master->leader,sentinel.myid))
master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
}
*leader_epoch = master->leader_epoch;
return master->leader ? sdsnew(master->leader) : NULL;
}Flow diagram:
sequenceDiagram
participant S1 as Sentinel A (detects O_DOWN)
participant S2 as Sentinel B (asked to vote)
participant M as Redis Master
S1->>S2: SENTINEL is-master-down-by-addr<br/>+ req_epoch + req_runid
S2->>S2: sentinelVoteLeader(master, req_epoch, req_runid, &epoch)
alt Vote allowed
S2-->>S1: +vote-for-leader (returns new leader)
S2->>S2: Set failover_start_time (if not self-vote)
else Already voted / epoch expired
S2-->>S1: Return current leader (NULL or old value)
end
S1->>S1: Count votes >= quorum?
alt Yes
S1->>M: Start failover (sentinelStartFailover)
endThere is also the question of which node this sentinel itself votes for:
char *sentinelGetLeader(sentinelRedisInstance *master, uint64_t epoch) {
dict *counters;
dictIterator *di;
dictEntry *de;
unsigned int voters = 0, voters_quorum;
char *myvote;
char *winner = NULL;
uint64_t leader_epoch;
uint64_t max_votes = 0;
serverAssert(master->flags & (SRI_O_DOWN|SRI_FAILOVER_IN_PROGRESS));
counters = dictCreate(&leaderVotesDictType);
voters = dictSize(master->sentinels)+1; /* All the other sentinels and me.*/
/* Count other sentinels votes */
// Tally the votes from other sentinels
di = dictGetIterator(master->sentinels);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *ri = dictGetVal(de);
if (ri->leader != NULL && ri->leader_epoch == sentinel.current_epoch)
sentinelLeaderIncr(counters,ri->leader);
}
dictReleaseIterator(di);
/* Check what's the winner. For the winner to win, it needs two conditions:
* 1) Absolute majority between voters (50% + 1).
* 2) And anyway at least master->quorum votes. */
// Find the candidate currently leading in votes
di = dictGetIterator(counters);
while((de = dictNext(di)) != NULL) {
uint64_t votes = dictGetUnsignedIntegerVal(de);
if (votes > max_votes) {
max_votes = votes;
winner = dictGetKey(de);
}
}
dictReleaseIterator(di);
/* Count this Sentinel vote:
* if this Sentinel did not voted yet, either vote for the most
* common voted sentinel, or for itself if no vote exists at all. */
// If there is a leading candidate, vote for it first
// Otherwise vote for self
if (winner)
myvote = sentinelVoteLeader(master,epoch,winner,&leader_epoch);
else
myvote = sentinelVoteLeader(master,epoch,sentinel.myid,&leader_epoch);
if (myvote && leader_epoch == epoch) {
uint64_t votes = sentinelLeaderIncr(counters,myvote);
if (votes > max_votes) {
max_votes = votes;
winner = myvote;
}
}
voters_quorum = voters/2+1;
if (winner && (max_votes < voters_quorum || max_votes < master->quorum))
winner = NULL;
winner = winner ? sdsnew(winner) : NULL;
sdsfree(myvote);
dictRelease(counters);
return winner;
}After the voting phase, the failover state machine determines the current leader and begins the failover.
void sentinelFailoverWaitStart(sentinelRedisInstance *ri) {
char *leader;
int isleader;
// Get the current leader
leader = sentinelGetLeader(ri, ri->failover_epoch);
// Determine whether we are the leader
isleader = leader && strcasecmp(leader,sentinel.myid) == 0;
sdsfree(leader);
/* If I'm not the leader, and it is not a forced failover via
* SENTINEL FAILOVER, then I can't continue with the failover. */
if (!isleader && !(ri->flags & SRI_FORCE_FAILOVER)) {
mstime_t election_timeout = sentinel_election_timeout;
/* The election timeout is the MIN between SENTINEL_ELECTION_TIMEOUT
* and the configured failover timeout. */
if (election_timeout > ri->failover_timeout)
election_timeout = ri->failover_timeout;
/* Abort the failover if I'm not the leader after some time. */
if (mstime() - ri->failover_start_time > election_timeout) {
sentinelEvent(LL_WARNING,"-failover-abort-not-elected",ri,"%@");
sentinelAbortFailover(ri);
}
return;
}
sentinelEvent(LL_WARNING,"+elected-leader",ri,"%@");
if (sentinel.simfailure_flags & SENTINEL_SIMFAILURE_CRASH_AFTER_ELECTION)
sentinelSimFailureCrash();
// After the leader is elected, move to the next state
ri->failover_state = SENTINEL_FAILOVER_STATE_SELECT_SLAVE;
ri->failover_state_change_time = mstime();
sentinelEvent(LL_WARNING,"+failover-state-select-slave",ri,"%@");
}Failover Task Execution
After a leader sentinel is elected, the failover state machine begins:
- SENTINEL_FAILOVER_STATE_WAIT_START: wait to start the failover
- SENTINEL_FAILOVER_STATE_SELECT_SLAVE: choose a replica as the new master
- SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE: send
slaveof no oneto the promoted replica - SENTINEL_FAILOVER_STATE_WAIT_PROMOTION: wait for the new master’s promotion
- SENTINEL_FAILOVER_STATE_RECONF_SLAVES: reconfigure the master’s replicas; the old replicas must be updated to the new master
void sentinelFailoverStateMachine(sentinelRedisInstance *ri) {
serverAssert(ri->flags & SRI_MASTER);
if (!(ri->flags & SRI_FAILOVER_IN_PROGRESS)) return;
switch(ri->failover_state) {
case SENTINEL_FAILOVER_STATE_WAIT_START:
sentinelFailoverWaitStart(ri);
break;
case SENTINEL_FAILOVER_STATE_SELECT_SLAVE:
sentinelFailoverSelectSlave(ri);
break;
case SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE:
sentinelFailoverSendSlaveOfNoOne(ri);
break;
case SENTINEL_FAILOVER_STATE_WAIT_PROMOTION:
sentinelFailoverWaitPromotion(ri);
break;
case SENTINEL_FAILOVER_STATE_RECONF_SLAVES:
sentinelFailoverReconfNextSlave(ri);
break;
}
}Choosing a Replica as the New Master
After the leader is elected, the first task is to pick one of the existing replicas as the new master.
void sentinelFailoverSelectSlave(sentinelRedisInstance *ri) {
sentinelRedisInstance *slave = sentinelSelectSlave(ri);
// ...
}What are the selection principles?
- First get all replicas
- Filter by health: exclude replicas that are subjectively or objectively down
- Exclude nodes with abnormal connections (e.g. command connection dropped)
- Exclude nodes with
replica-priority=0 - Exclude nodes that have been disconnected from the old master longer than a threshold
- Exclude nodes whose INFO info is too stale
- The remaining nodes are candidates, sorted as follows
- Three-level sort:
- Primary: replica priority (
replica-priority) - Secondary: replication offset, prefer the replica least behind the master
- Finally: sort by runid as a tie-breaker
- Primary: replica priority (
Promoting the Replica to Master
After choosing the replica to promote, sentinelFailoverSendSlaveOfNoOne is executed to promote it to master.
void sentinelFailoverSendSlaveOfNoOne(sentinelRedisInstance *ri) {
int retval;
/* We can't send the command to the promoted slave if it is now
* disconnected. Retry again and again with this state until the timeout
* is reached, then abort the failover. */
if (ri->promoted_slave->link->disconnected) {
if (mstime() - ri->failover_state_change_time > ri->failover_timeout) {
sentinelEvent(LL_WARNING,"-failover-abort-slave-timeout",ri,"%@");
sentinelAbortFailover(ri);
}
return;
}
/* Send SLAVEOF NO ONE command to turn the slave into a master.
* We actually register a generic callback for this command as we don't
* really care about the reply. We check if it worked indirectly observing
* if INFO returns a different role (master instead of slave). */
retval = sentinelSendSlaveOf(ri->promoted_slave,NULL);
if (retval != C_OK) return;
sentinelEvent(LL_NOTICE, "+failover-state-wait-promotion",
ri->promoted_slave,"%@");
ri->failover_state = SENTINEL_FAILOVER_STATE_WAIT_PROMOTION;
ri->failover_state_change_time = mstime();
}The promotion process:
Through a transaction:
- MULTI start the transaction
- SLAVEOF NO ONE promote the replica to master
- CONFIG REWRITE rewrite the new config into the config file
- CLIENT KILL TYPE normal disconnect normal clients
- CLIENT KILL TYPE pubsub disconnect pub/sub clients
- EXEC execute the transaction
After sending the promotion command, wait for the replica to finish being promoted. This is done indirectly via the INFO command.
// Check whether the failover timed out
void sentinelFailoverWaitPromotion(sentinelRedisInstance *ri) {
/* Just handle the timeout. Switching to the next state is handled
* by the function parsing the INFO command of the promoted slave. */
if (mstime() - ri->failover_state_change_time > ri->failover_timeout) {
sentinelEvent(LL_WARNING,"-failover-abort-slave-timeout",ri,"%@");
sentinelAbortFailover(ri);
}
}The sentinel periodically sends INFO to the master and replicas.
void sentinelSendPeriodicCommands(sentinelRedisInstance *ri) {
// Send INFO to master and replica; handled by sentinelInfoReplyCallback
if ((ri->flags & SRI_SENTINEL) == 0 &&
(ri->info_refresh == 0 ||
(now - ri->info_refresh) > info_period))
{
retval = redisAsyncCommand(ri->link->cc,
sentinelInfoReplyCallback, ri, "%s",
sentinelInstanceMapCommand(ri,"INFO"));
if (retval == C_OK) ri->link->pending_commands++;
}
}The sentinel handles the master/replica INFO replies. When it detects a replica’s role field changed to master, it triggers the promotion-confirmation logic.
void sentinelInfoReplyCallback(redisAsyncContext *c, void *reply, void *privdata) {
if (r->type == REDIS_REPLY_STRING)
sentinelRefreshInstanceInfo(ri,r->str);
}
void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
/* Handle slave -> master role switch. */
if ((ri->flags & SRI_SLAVE) && role == SRI_MASTER) {
/* If this is a promoted slave we can change state to the
* failover state machine. */
if ((ri->flags & SRI_PROMOTED) &&
(ri->master->flags & SRI_FAILOVER_IN_PROGRESS) &&
(ri->master->failover_state ==
SENTINEL_FAILOVER_STATE_WAIT_PROMOTION))
{
/* Now that we are sure the slave was reconfigured as a master
* set the master configuration epoch to the epoch we won the
* election to perform this failover. This will force the other
* Sentinels to update their config (assuming there is not
* a newer one already available). */
// Confirm the replica has been promoted to master, rewrite config
// Trigger the leader sentinel to run the client reconfig script
ri->master->config_epoch = ri->master->failover_epoch;
ri->master->failover_state = SENTINEL_FAILOVER_STATE_RECONF_SLAVES; // next state
ri->master->failover_state_change_time = mstime();
sentinelFlushConfig();
sentinelEvent(LL_WARNING,"+promoted-slave",ri,"%@");
if (sentinel.simfailure_flags &
SENTINEL_SIMFAILURE_CRASH_AFTER_PROMOTION)
sentinelSimFailureCrash();
sentinelEvent(LL_WARNING,"+failover-state-reconf-slaves",
ri->master,"%@");
sentinelCallClientReconfScript(ri->master,SENTINEL_LEADER,
"start",ri->master->addr,ri->addr);
sentinelForceHelloUpdateForMaster(ri->master);
} else {
/* A slave turned into a master. We want to force our view and
* reconfigure as slave. Wait some time after the change before
* going forward, to receive new configs if any. */
mstime_t wait_time = sentinel_publish_period*4;
// A replica turned master unexpectedly; force it back to replica
if (!(ri->flags & SRI_PROMOTED) &&
sentinelMasterLooksSane(ri->master) &&
sentinelRedisInstanceNoDownFor(ri,wait_time) &&
mstime() - ri->role_reported_time > wait_time)
{
int retval = sentinelSendSlaveOf(ri,ri->master->addr);
if (retval == C_OK)
sentinelEvent(LL_NOTICE,"+convert-to-slave",ri,"%@");
}
}
}
}Reconfiguring the Other Replicas
After the chosen replica is promoted to master, the other replicas must be reconfigured, mainly changing their slaveof target from the old master to the new master: SLAVE OF <new master address>
void sentinelFailoverReconfNextSlave(sentinelRedisInstance *master) {
// Send slaveof newmaster to the other replicas
/* Send SLAVEOF <new master>. */
retval = sentinelSendSlaveOf(slave,master->promoted_slave->addr);
if (retval == C_OK) {
slave->flags |= SRI_RECONF_SENT;
slave->slave_reconf_sent_time = mstime();
sentinelEvent(LL_NOTICE,"+slave-reconf-sent",slave,"%@");
in_progress++;
}
// Detect whether the replica reconfiguration is done
sentinelFailoverDetectEnd(master);
}58221:X 29 Jan 2026 13:56:24.668 # +sdown master mymaster 192.168.232.128 6379
58221:X 29 Jan 2026 13:56:24.682 * Sentinel new configuration saved on disk
58221:X 29 Jan 2026 13:56:24.682 # +new-epoch 1
58221:X 29 Jan 2026 13:56:24.686 * Sentinel new configuration saved on disk
58221:X 29 Jan 2026 13:56:24.686 # +vote-for-leader fe6e146804ee6df95a04afa50d27a7d651ab33d2 1
58221:X 29 Jan 2026 13:56:24.745 # +odown master mymaster 192.168.232.128 6379 #quorum 3/2
58221:X 29 Jan 2026 13:56:24.745 # Next failover delay: I will not start a failover before Thu Jan 29 13:56:45 2026
58221:X 29 Jan 2026 13:56:45.133 * Sentinel new configuration saved on disk
58221:X 29 Jan 2026 13:56:45.133 # +new-epoch 2
58221:X 29 Jan 2026 13:56:45.135 * Sentinel new configuration saved on disk
58221:X 29 Jan 2026 13:56:45.135 # +vote-for-leader 556372190acd0c77c0bc6760178830ed66ddac08 2Sentinel Node Roles
A sentinel node has two roles:
- leader: elected
- observer: if not the leader, it can only observe
#define SENTINEL_LEADER (1<<17)
#define SENTINEL_OBSERVER (1<<18)Commands Sentinel Needs
Commands the sentinel executes:
- INFO get replica info and replication offset
- PING node probing
- MULTI, SLAVEOF, CONFIG REWRITE, EXEC, CLIENT promote a replica to master and rewrite the config into the config file