Redis Cluster
Redis Cluster is the official Redis clustering solution. Redis Cluster is decentralized.
Design Goals
Redis Cluster is Redis’s distributed implementation. Its design goals: high performance and linear scalability, up to 1000 nodes. Achieving very high performance and scalability while keeping weaker but reasonable write safety and availability is the main goal of Redis Cluster.
Performance is nearly linearly scalable. Under normal circumstances the client caches the cluster’s slot information and talks directly to the correct node; only a few cases trigger a redirection.
Basic Implementation Principles
In Redis Cluster, each node is responsible for storing data and maintaining cluster state, including mapping keys to the correct node. Cluster nodes can automatically discover other nodes, detect unavailable nodes, and promote a replica to master when a failure happens so the cluster keeps running.
Redis Cluster splits all data into 16384 slots, and each node is responsible for a subset of slots. The slot information is stored on every node, unlike codis which needs a separate distributed store for slot information. When a Redis Cluster client connects to the cluster it also gets a copy of the cluster’s slot configuration. So when the client wants to look up a key, it can locate the target node directly. To locate the node for a specific key, the client must cache the slot information so it can find the node accurately and quickly. And because the client’s and server’s slot information can be inconsistent, a correction mechanism is needed to validate and adjust the slot mapping.
#define CLUSTER_SLOTS 16384 Also, each Redis Cluster node persists its cluster config to a config file, so the config file must be writable, and you should not edit it manually.
When the cluster scales up or down, the number of slots stays the same — slots migrate between nodes and the slot computation logic is unchanged. The design idea is essentially consistent hashing.
Nodes use the Gossip protocol to synchronize cluster information, discover new nodes, send PING packets to ensure all other nodes are working, and send the required cluster messages to signal specific situations.
Slot Location Algorithm
By default Redis Cluster hashes the key with the crc16 algorithm to get an integer, then takes that integer modulo 16384 to get the specific slot.
Hash tags: Redis Cluster also lets a user force a key onto a specific slot by embedding a tag marker in the key string, so the slot the key maps to equals the slot of the tag. By marking part of the key with {}, you force the key to map to the same slot (e.g. user:{123}:name and user:{123}:email land in the same slot). For a key with a tag, only the part inside {} is hashed, so it maps to the same slot.
/* We have 16384 hash slots. Given a key, its hash slot is obtained by
* computing the CRC16 of the key and taking the least significant 14 bits.
* However, if the key contains the {...} pattern, only the part between
* { and } is hashed. This mechanism may be used in the future to force
* certain keys into the same node (as long as no resharding is in progress). */
unsigned int keyHashSlot(char *key, int keylen) {
int s, e; /* start-end indexes of { and } */
for (s = 0; s < keylen; s++)
if (key[s] == '{') break;
/* No '{' ? Hash the whole key. This is the base case. */
if (s == keylen) return crc16(key,keylen) & 0x3FFF;
/* '{' found? Check if we have the corresponding '}'. */
for (e = s+1; e < keylen; e++)
if (key[e] == '}') break;
/* No '}' or nothing between {} ? Hash the whole key. */
if (e == keylen || e == s+1) return crc16(key,keylen) & 0x3FFF;
/* If we are here there is both a { and a } on its right. Hash
* what is in the middle between { and }. */
return crc16(key+s+1,e-s-1) & 0x3FFF;
}Why crc16? Because it is fast, and its hash value satisfies the modulo-by-16384 requirement. Hash collisions are not a big concern — as long as the distribution is roughly even, fast speed is what matters most.
Redirection
As mentioned, the Redis client caches the cluster’s slot configuration, but the client’s and server’s slot information can be inconsistent. That is when the redirection mechanism is needed. When the client sends a command to the wrong node, that node notices the key’s slot is not managed by itself, so it sends the client a special redirect command MOVED carrying the target node’s address, telling the client to connect to that node to get the data. After receiving MOVED, the client must immediately correct its local slot mapping table, and all subsequent keys will use the new slot mapping.
When a client sends a command to the Redis server, it goes through the processCommand function.
int processCommand(client *c) {
// In cluster mode, the cluster redirect happens here
// Two cases where no redirect happens:
// 1. The sender is this node's master
// 2. The command contains no key
if (server.cluster_enabled &&
!mustObeyClient(c) &&
!(!(c->cmd->flags&CMD_MOVABLE_KEYS) && c->cmd->key_specs_num == 0 &&
c->cmd->proc != execCommand))
{
int error_code;
// Try to resolve which node this command should route to
clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,
&c->slot,&error_code);
// Target node is not this node, need to redirect
if (n == NULL || n != server.cluster->myself) {
if (c->cmd->proc == execCommand) {
discardTransaction(c);
} else {
flagTransaction(c);
}
// Return MOVED/ASK to the client
clusterRedirectClient(c,n,c->slot,error_code);
c->cmd->rejected_calls++;
return C_OK;
}
}
}Cluster Change Awareness
When a server node changes, the client should be notified immediately to refresh its node-relationship table in real time. How does the client get notified? Two cases:
- The target node is down: the client throws a ConnectionError, then picks a random node to retry. The retried node tells it the new node address for the target slot via a MOVED command.
- An operator manually changed the cluster info, switching the master to another node and removing the old master from the cluster. Then commands sent to the old master receive a ClusterDown error, indicating the current node’s cluster is unavailable (the node no longer belongs to the previous cluster). The client then closes all connections, clears the slot mapping table, and throws an error to the upper layer. When the next command comes, it re-initializes the node info.
Migration
Redis Cluster provides the migration tool redis-trib, letting operators manually adjust slot allocation. It is written in Ruby and composes various native Redis Cluster commands. Redis migrates one slot at a time; while a slot is migrating it is in an intermediate transitional state.
Migration process: the source node runs the dump command on the current key to get the serialized content, then sends the restore command to the target node with the serialized content as a parameter; the target node deserializes it to restore the content into its memory. The full key list for each slot can be obtained via the keysinslot command. In practice the migrate command does the migration; its internal implementation relies on dump and restore.
Fault Tolerance
Redis Cluster can have several replicas for each master. When a master fails, the cluster automatically promotes one of its replicas to master. If a master has no replica, then when it fails the cluster becomes completely unavailable. However, Redis also provides the cluster-require-full-coverage parameter, which allows partial node failures while the other nodes keep serving.
For network jitter, Redis Cluster provides the cluster-node-timeout parameter. Only when a node is unreachable for the timeout duration is it considered failed and a master-replica switch is triggered. Without this option, network jitter would cause frequent master-replica switches.
Possibly Fail (PFail) and Fail: Redis Cluster is decentralized; one node thinking a node is gone does not mean all nodes think so. So the cluster needs a negotiation step — only when the majority of nodes agree a node is gone does the cluster consider it needs a master-replica switch for fault tolerance.
Redis Cluster nodes use the gossip protocol to broadcast their own state and changes to their view of the whole cluster. When a node finds another node gone (PFail), it broadcasts this message to the whole cluster; other nodes receive the failure info. If the number of nodes reporting a node as gone reaches a cluster majority, that node can be marked as definitely down (Fail), and then broadcast to the whole cluster, forcing other nodes to accept that it is down and immediately performing the master-replica switch on it.
Notes
- Hash collisions: different keys may map to the same slot, but it is fine as long as the data is evenly distributed.
- Cluster does not support transactions: Redis Cluster is distributed; current Redis only supports single-node transactions, not distributed transactions.
- Cluster’s mget is much slower than on a single Redis, because it is split into multiple get commands.
- Cluster’s rename command is no longer atomic; it needs to move data from the source node to the target node. After changing the name, the key’s hash changes, and so does the slot it maps to.
- Redis Cluster does not support multiple databases; only database 0 is allowed, and the
SELECTcommand is disallowed. - Write safety: Redis Cluster does not guarantee no data loss; even acknowledged writes can be lost. Here is an example scenario that loses acknowledged writes in the majority partition during a failure: a write may reach the master, but while the master replies to the client, the write may not have propagated to the replica through the asynchronous replication used between master and replica. If the master crashes before the write reaches the replica, and the master is unreachable long enough for its replica to be promoted, the write is lost forever. In a fully sudden master failure this is usually hard to observe, because the master tries to reply to the client (ack the write) and to the replica (propagate the write) almost simultaneously. Still, it is a real-world failure mode.
The root cause of data loss is still the master-replica asynchronous replication — the replica is never synced, and is always behind the master.
Source Code Analysis
Node Startup
After the node starts, the verifyClusterConfigWithData function verifies that the data loaded from disk is consistent with the cluster config.
- If it finds keys that belong to hash slots this node should not be responsible for, it handles them as follows:
- If, per the current cluster config, no other node is responsible for those slots, add those slots to this node’s responsibility.
- If, per the current config, another node is already responsible for those slots, mark those slots as
IMPORTINGfrom this node’s perspective. This explains why this node holds data for those slots, and also lets redis-cli notice the problem and try to fix it.
- If data is found in a database other than db0, it returns
C_ERR, telling the caller to exit the server with an error or take other action.
int main(int argc, char **argv) {
initServer() {
if (server.cluster_enabled) clusterInit(); // initialize this cluster node
}
if (!server.sentinel_mode) {
loadDataFromDisk();
if (server.cluster_enabled) {
if (verifyClusterConfigWithData() == C_ERR) {
serverLog(LL_WARNING,
"You can't have keys in a DB different than DB 0 when in "
"Cluster mode. Exiting.");
exit(1);
}
}
}
aeMain(server.el);
}
void clusterInit(void) {
server.cluster->nodes = dictCreate(&clusterNodesDictType);
// Load this node's config
if (clusterLoadConfig(server.cluster_configfile) == C_ERR) {
/* No configuration found. We will just use the random name provided
* by the createClusterNode() function. */
myself = server.cluster->myself =
createClusterNode(NULL,CLUSTER_NODE_MYSELF|CLUSTER_NODE_MASTER);
serverLog(LL_NOTICE,"No cluster configuration found, I'm %.40s",
myself->name);
clusterAddNode(myself);
saveconf = 1;
}
if (saveconf) clusterSaveConfigOrDie(1);
}After startup, the server periodically runs serverCron, which calls clusterCron.
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
/* Run the Redis Cluster cron. */
run_with_period(100) {
if (server.cluster_enabled) clusterCron();
}
// ...
}clusterCron is the core function of Redis Cluster; it runs every 100ms and is responsible for maintaining cluster health, node communication, and failover. Core functions include:
- Node connection maintenance: manage cluster bus connections
- Failure detection: identify possibly-failed nodes
- Topology optimization: handle master-replica relationships and node migration
- State synchronization: update the global cluster view
- Failover coordination: handle automatic/manual failover
void clusterCron(void) {
di = dictGetSafeIterator(server.cluster->nodes);
while((de = dictNext(di)) != NULL) {
clusterNode *node = dictGetVal(de);
/* The sequence goes:
* 1. We try to shrink link buffers if possible.
* 2. We free the links whose buffers are still oversized after possible shrinking.
* 3. We update the latest memory usage of cluster links.
* 4. We immediately attempt reconnecting after freeing links.
*/
clusterNodeCronResizeBuffers(node);
clusterNodeCronFreeLinkOnBufferLimitReached(node);
clusterNodeCronUpdateClusterLinksMemUsage(node);
// Handle connection / connection recovery
if(clusterNodeCronHandleReconnect(node, handshake_timeout, now)) continue;
}
// Failure detection: every 100ms pick a random node and send a PING
if (!(iteration % 10)) {
int j;
/* Check a few random nodes and ping the one with the oldest
* pong_received time. */
for (j = 0; j < 5; j++) {
de = dictGetRandomKey(server.cluster->nodes);
clusterNode *this = dictGetVal(de);
/* Don't ping nodes disconnected or with a ping currently active. */
if (this->link == NULL || this->ping_sent != 0) continue;
if (this->flags & (CLUSTER_NODE_MYSELF|CLUSTER_NODE_HANDSHAKE))
continue;
if (min_pong_node == NULL || min_pong > this->pong_received) {
min_pong_node = this;
min_pong = this->pong_received;
}
}
if (min_pong_node) {
serverLog(LL_DEBUG,"Pinging node %.40s", min_pong_node->name);
clusterSendPing(min_pong_node->link, CLUSTERMSG_TYPE_PING);
}
}
// Check for orphaned masters; if one exists, migrate a replica from the
// master with the most replicas to the orphaned master, requiring the
// source master to have at least 2 healthy replicas.
orphaned_masters = 0;
max_slaves = 0;
this_slaves = 0;
di = dictGetSafeIterator(server.cluster->nodes);
while((de = dictNext(di)) != NULL) {
clusterNode *node = dictGetVal(de);
now = mstime(); /* Use an updated time at every iteration. */
if (node->flags &
(CLUSTER_NODE_MYSELF|CLUSTER_NODE_NOADDR|CLUSTER_NODE_HANDSHAKE))
continue;
/* Orphaned master check, useful only if the current instance
* is a slave that may migrate to another master. */
if (nodeIsSlave(myself) && nodeIsMaster(node) && !nodeFailed(node)) {
int okslaves = clusterCountNonFailingSlaves(node);
/* A master is orphaned if it is serving a non-zero number of
* slots, have no working slaves, but used to have at least one
* slave, or failed over a master that used to have slaves. */
if (okslaves == 0 && node->numslots > 0 &&
node->flags & CLUSTER_NODE_MIGRATE_TO)
{
orphaned_masters++;
}
if (okslaves > max_slaves) max_slaves = okslaves;
if (myself->slaveof == node)
this_slaves = okslaves;
}
}
}Key definition:
typedef struct clusterNode {
mstime_t ctime; /* Node object creation time, for debugging and state tracking */
char name[CLUSTER_NAMELEN]; /* Unique node id, hex string, sha1-size */
int flags; /* Node state flags: CLUSTER_NODE_MASTER master, CLUSTER_NODE_SLAVE replica, CLUSTER_NODE_FAIL marked failed, CLUSTER_NODE_MYSELF this node */
uint64_t configEpoch; /* Config logical clock */
unsigned char slots[CLUSTER_SLOTS/8]; /* Bitmap (16384 slots); each bit says whether this node owns that slot */
uint16_t *slot_info_pairs; /* Slot range pairs (e.g. [0, 5000], [5001, 10000]) for memory/traversal efficiency */
int slot_info_pairs_count; /* Number of valid range pairs */
int numslots; /* Total number of slots this node owns */
int numslaves; /* Number of replicas */
struct clusterNode **slaves; /* Pointer array to replicas */
struct clusterNode *slaveof; /* Pointer to master, replica-only, may be NULL if master not discovered */
unsigned long long last_in_ping_gossip; /* Number of the last carried in the ping gossip section */
mstime_t ping_sent; /* Time the latest ping was sent */
mstime_t pong_received; /* Time the latest pong was received */
mstime_t data_received; /* Time the latest data was received */
mstime_t fail_time; /* Time marked as FAIL */
mstime_t voted_time; /* Last time we voted for a slave of this master */
mstime_t repl_offset_time; /* Replica offset update timestamp */
mstime_t orphaned_time; /* Starting time of orphaned master condition */
long long repl_offset; /* Latest known replication offset */
char ip[NET_IP_STR_LEN]; /* Latest known IP address of this node */
sds hostname; /* The known hostname for this node */
int port; /* Latest known clients port (TLS or plain). */
int pport; /* Plain-text client port, used when port is a TLS port */
int cport; /* Cluster internal communication port */
clusterLink *link; /* TCP/IP link established toward this node */
clusterLink *inbound_link; /* TCP/IP link accepted from this node */
list *fail_reports; /* List of nodes signaling this as failing */
} clusterNode;Cluster Initialization
The redis-cli --cluster create command creates the initial cluster. It mainly does:
- Node validation and connection
- Hash slot allocation
- Master-replica topology construction
- Cluster config synchronization
Overall flow:
graph LR
A[redis-cli --cluster create node1 node2 ...] --> B[Client parses node list]
B --> C[Connect to all nodes, validate state]
C --> D[Allocate hash slots (16384)]
D --> E[Send CLUSTER ADDSLOTS to each master]
E --> F[Send CLUSTER REPLICATE to each replica]
F --> G[Nodes handshake with each other (MEET)]
G --> H[Cluster config takes effect]Slot allocation happens in the redis-cli when executing --cluster create; after allocation the client sends the CLUSTER ADDSLOTS command to the server. On receipt, the server checks whether those slots are already owned, assigns the slots in the command to this node, and updates the internal clusterNode.slots mapping.
Configuring the master-replica relationship is also done by the redis-cli client issuing CLUSTER REPLICATE <master NodeID> for each replica; on receipt, the replica server switches to replica and starts replicating the specified master.
The Redis server only executes commands and maintains state; it does not participate in slot allocation logic.
Core source:
static int clusterManagerCommandCreate(int argc, char **argv) {
cluster_manager.nodes = listCreate();
for (i = 0; i < argc; i++) {
char *addr = argv[i];
char *ip = NULL;
int port = 0;
parseClusterNodeAddress(addr, &ip, &port, NULL)
clusterManagerNode *node = clusterManagerNewNode(ip, port, 0);
clusterManagerNodeConnect(node) // create connection
clusterManagerNodeIsCluster(node, &err) // verify node is in cluster mode
clusterManagerNodeLoadInfo(node, 0, &err)
if (!clusterManagerNodeIsEmpty(node, &err)) { // non-empty node check
clusterManagerPrintNotEmptyNodeError(node, err);
if (err) zfree(err);
freeClusterManagerNode(node);
return 0;
}
listAddNodeTail(cluster_manager.nodes, node);
}
// Verify cluster node count; minimum scale is 3
int node_len = cluster_manager.nodes->len;
int replicas = config.cluster_manager_command.replicas;
int masters_count = CLUSTER_MANAGER_MASTERS_COUNT(node_len, replicas);
if (masters_count < 3) {
clusterManagerLogErr(
"*** ERROR: Invalid configuration for cluster creation.\n"
"*** Redis Cluster requires at least 3 master nodes.\n"
"*** This is not possible with %d nodes and %d replicas per node.",
node_len, replicas);
clusterManagerLogErr("\n*** At least %d nodes are required.\n",
3 * (replicas + 1));
return 0;
}
clusterManagerLogInfo(">>> Performing hash slots allocation "
"on %d nodes...\n", node_len);
// Hash slot allocation, sequential, 16384 / number_of_masters, evenly
for (i = 0; i < masters_count; i++) {
clusterManagerNode *master = masters[i];
long last = lround(cursor + slots_per_node - 1);
if (last > CLUSTER_MANAGER_SLOTS || i == (masters_count - 1))
last = CLUSTER_MANAGER_SLOTS - 1;
if (last < first) last = first;
printf("Master[%d] -> Slots %ld - %ld\n", i, first, last);
master->slots_count = 0;
for (j = first; j <= last; j++) {
master->slots[j] = 1;
master->slots_count++;
}
master->dirty = 1;
first = last + 1;
cursor += slots_per_node;
}
// Anti-affinity node placement: prefer spreading masters across
// different physical machines (group by IP)
// Round-robin: the interleaved array ensures nodes with the same IP
// are not placed consecutively
clusterManagerNode *first_node = interleaved[0];
for (i = 0; i < (interleaved_len - 1); i++)
interleaved[i] = interleaved[i + 1];
interleaved[interleaved_len - 1] = first_node;
int assign_unused = 0, available_count = interleaved_len;
assign_replicas:
for (i = 0; i < masters_count; i++) {
clusterManagerNode *master = masters[i];
int assigned_replicas = 0;
// Build master-replica relationship, prefer a replica on a
// different IP machine
while (assigned_replicas < replicas) {
if (available_count == 0) break;
clusterManagerNode *found = NULL, *slave = NULL;
int firstNodeIdx = -1;
for (j = 0; j < interleaved_len; j++) {
clusterManagerNode *n = interleaved[j];
if (n == NULL) continue;
if (strcmp(n->ip, master->ip)) {
found = n;
interleaved[j] = NULL;
break;
}
if (firstNodeIdx < 0) firstNodeIdx = j;
}
if (found) slave = found;
else if (firstNodeIdx >= 0) {
slave = interleaved[firstNodeIdx];
interleaved_len -= (firstNodeIdx + 1);
interleaved += (firstNodeIdx + 1);
}
if (slave != NULL) {
assigned_replicas++;
available_count--;
if (slave->replicate) sdsfree(slave->replicate);
slave->replicate = sdsnew(master->name);
slave->dirty = 1;
} else break;
printf("Adding replica %s:%d to %s:%d\n", slave->ip, slave->port,
master->ip, master->port);
if (assign_unused) break;
}
}
if (!assign_unused && available_count > 0) {
assign_unused = 1;
printf("Adding extra replicas...\n");
goto assign_replicas;
}
for (i = 0; i < ip_count; i++) {
clusterManagerNodeArray *node_array = ip_nodes + i;
clusterManagerNodeArrayReset(node_array);
}
clusterManagerOptimizeAntiAffinity(ip_nodes, ip_count);
clusterManagerShowNodes();
int ignore_force = 0;
if (confirmWithYes("Can I set the above configuration?", ignore_force)) {
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *node = ln->value;
char *err = NULL;
int flushed = clusterManagerFlushNodeConfig(node, &err); // calls cluster addslots internally
if (!flushed && node->dirty && !node->replicate) {
if (err != NULL) {
CLUSTER_MANAGER_PRINT_REPLY_ERROR(node, err);
zfree(err);
}
success = 0;
goto cleanup;
} else if (err != NULL) zfree(err);
}
clusterManagerLogInfo(">>> Nodes configuration updated\n");
clusterManagerLogInfo(">>> Assign a different config epoch to "
"each node\n");
int config_epoch = 1;
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *node = ln->value;
redisReply *reply = NULL;
reply = CLUSTER_MANAGER_COMMAND(node,
"cluster set-config-epoch %d",
config_epoch++);
if (reply != NULL) freeReplyObject(reply);
}
clusterManagerLogInfo(">>> Sending CLUSTER MEET messages to join "
"the cluster\n");
clusterManagerNode *first = NULL;
char first_ip[NET_IP_STR_LEN]; /* first->ip may be a hostname */
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *node = ln->value;
if (first == NULL) {
first = node;
/* Although hiredis supports connecting to a hostname, CLUSTER
* MEET requires an IP address, so we do a DNS lookup here. */
if (anetResolve(NULL, first->ip, first_ip, sizeof(first_ip), ANET_NONE)
== ANET_ERR)
{
fprintf(stderr, "Invalid IP address or hostname specified: %s\n", first->ip);
success = 0;
goto cleanup;
}
continue;
}
redisReply *reply = NULL;
if (first->bus_port == 0 || (first->bus_port == first->port + CLUSTER_MANAGER_PORT_INCR)) {
/* CLUSTER MEET bus-port parameter was added in 4.0.
* So if (bus_port == 0) or (bus_port == port + CLUSTER_MANAGER_PORT_INCR),
* we just call CLUSTER MEET with 2 arguments, using the old form. */
reply = CLUSTER_MANAGER_COMMAND(node, "cluster meet %s %d",
first_ip, first->port);
} else {
reply = CLUSTER_MANAGER_COMMAND(node, "cluster meet %s %d %d",
first_ip, first->port, first->bus_port);
}
int is_err = 0;
if (reply != NULL) {
if ((is_err = reply->type == REDIS_REPLY_ERROR))
CLUSTER_MANAGER_PRINT_REPLY_ERROR(node, reply->str);
freeReplyObject(reply);
} else {
is_err = 1;
fprintf(stderr, "Failed to send CLUSTER MEET command.\n");
}
if (is_err) {
success = 0;
goto cleanup;
}
}
/* Give one second for the join to start, in order to avoid that
* waiting for cluster join will find all the nodes agree about
* the config as they are still empty with unassigned slots. */
sleep(1);
// Block until the cluster forms
clusterManagerWaitForClusterJoin();
/* Useful for the replicas */
listRewind(cluster_manager.nodes, &li);
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *node = ln->value;
if (!node->dirty) continue;
char *err = NULL;
int flushed = clusterManagerFlushNodeConfig(node, &err);
if (!flushed && !node->replicate) {
if (err != NULL) {
CLUSTER_MANAGER_PRINT_REPLY_ERROR(node, err);
zfree(err);
}
success = 0;
goto cleanup;
} else if (err != NULL) {
zfree(err);
}
}
// Reset Nodes
listRewind(cluster_manager.nodes, &li);
clusterManagerNode *first_node = NULL;
while ((ln = listNext(&li)) != NULL) {
clusterManagerNode *node = ln->value;
if (!first_node) first_node = node;
else freeClusterManagerNode(node);
}
listEmpty(cluster_manager.nodes);
if (!clusterManagerLoadInfoFromNode(first_node)) {
success = 0;
goto cleanup;
}
clusterManagerCheckCluster(0);
}
}Cluster High Availability
Slot allocation strategy:
- Even allocation: ideally, N masters, each owns 16384/N slots
- Uneven allowed: via
redis-cli --cluster reshard - Allocation timing:
- At cluster creation (
--cluster create) - At scaling (dynamic migration)
- At cluster creation (
Slots are only assigned to masters; replicas only replicate the master’s data and can be promoted to master on failure, taking over the original master’s slots.
Replica Placement
Specify at cluster creation:
redis-cli --cluster create \
192.168.1.10:7000 192.168.1.10:7001 192.168.1.10:7002 \ # masters
192.168.1.10:7003 192.168.1.10:7004 192.168.1.10:7005 \ # replicas
--cluster-replicas 1--cluster-replicas 1means one replica per master- redis-cli auto-pairs them, trying to keep master and replica on different physical machines
Add a replica manually
# 1. Start a new node (empty instance)
# 2. Make it a replica of a master
redis-cli -p 7003 CLUSTER REPLICATE <master Node ID>Failover: how a replica is promoted to master
When the master goes down, the replica is promoted via the following flow:
- Failure detection: all nodes periodically PING the master; if no response within
cluster-node-timeout, it is marked subjectively downPFAIL. After a majority of masters agree, it is marked objectively downFAIL. - The replica initiates an election: an eligible replica (fresh data, high priority) sends a
FAILOVER AUTHORIZATIONrequest; the replica that gets the majority of master votes becomes the new master. - The winning replica runs
CLUSTER FAILOVER TAKEOVERto become master. It takes over all the original master’s hash slots and broadcasts the topology change. - Client redirection: the client receives
MOVED <slot>
sequenceDiagram
participant Master
participant Replica1
participant Replica2
participant OtherMasters
Note over Master: Down (e.g. kill -9)
OtherMasters->>OtherMasters: Gossip spreads PFAIL
OtherMasters->>OtherMasters: Majority vote -> marked FAIL
Replica1->>Replica1: Detects master=FAIL
Replica1->>OtherMasters: Initiate failover request
OtherMasters-->>Replica1: Vote authorization
Replica1->>Replica1: Promote to new master
Replica1->>Replica2: Notify topology change
Replica2->>Replica2: Switch replication source (to new master)Building a redis cluster
Each node in the cluster has a unique name. The node name is the hex representation of a 160-bit random number, obtained when the node first starts (usually from /dev/urandom). The node saves its ID in the node config file and uses the same ID forever, unless an admin deletes the node config file or requests a hard reset via the CLUSTER RESET command.
Each Redis Cluster node has an extra TCP port for incoming connections from other Redis Cluster nodes. That port is the data port plus 10000, or can be set via the cluster-port config.
We will build a three-node cluster (no replicas) as an example:
- Important config parameters:
port 7000 # node service port
cluster-enabled yes # must be yes to enable cluster mode
cluster-config-file nodes-7000.conf # cluster state config file, auto-maintained by redis
cluster-node-timeout 15000 # node timeout in ms, decides if a node is unreachable- Start three redis node instances
- On any node, use the
redis-cli --cluster createcommand to create the initial cluster. It will prompt you to confirm the node allocation; type yes to finish.
postgres@slpc:~/redis/cluster$ redis-cli -p 7000 --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460 # slot allocation
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 799ae73e4300b495fc475c3d908c32e4d1018a44 127.0.0.1:7000
slots:[0-5460] (5461 slots) master
M: 06b3b0b4a19e9f1efb94055614d782dea6eee4b0 127.0.0.1:7001
slots:[5461-10922] (5462 slots) master
M: a2fcec028ddf56c3ada5850dcc73f2cf1bb56e03 127.0.0.1:7002
slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 799ae73e4300b495fc475c3d908c32e4d1018a44 127.0.0.1:7000
slots:[0-5460] (5461 slots) master
M: a2fcec028ddf56c3ada5850dcc73f2cf1bb56e03 127.0.0.1:7002
slots:[10923-16383] (5461 slots) master
M: 06b3b0b4a19e9f1efb94055614d782dea6eee4b0 127.0.0.1:7001
slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.- After success, run
cluster infoto check cluster state:
postgres@slpc:~/redis/cluster$ redis-cli -p 7000
127.0.0.1:7000> cluster info
cluster_state:ok # cluster state ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:213
cluster_stats_messages_pong_sent:219
cluster_stats_messages_sent:432
cluster_stats_messages_ping_received:217
cluster_stats_messages_pong_received:213
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:432Check node info:
127.0.0.1:7000> cluster nodes
799ae73e4300b495fc475c3d908c32e4d1018a44 127.0.0.1:7000@17000 myself,master - 0 1761898525000 1 connected 0-5460
a2fcec028ddf56c3ada5850dcc73f2cf1bb56e03 127.0.0.1:7002@17002 master - 0 1761898526417 3 connected 10923-16383
06b3b0b4a19e9f1efb94055614d782dea6eee4b0 127.0.0.1:7001@17001 master - 0 1761898527427 2 connected 5461-10922In cluster mode redis-cli needs the -c flag. Also, if ACL is configured, you need --user and --pass; redirection requires user/password authentication. The AUTH command only applies to the current connection. Redis Cluster requires the client to manage multi-node connections itself; redis-cli is a simple client and does not record auth state across connections unless you provide --user and --pass at startup. Advanced client libraries like redis-py authenticate each node connection automatically and only need to be initialized once.
postgres@slpc:~/works$ redis-cli -c --user admin --pass admin_password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> get k1
-> Redirected to slot [12706] located at 192.168.232.138:6379
(nil)
192.168.232.138:6379> get k2
-> Redirected to slot [449] located at 192.168.232.128:6379
"v2"
192.168.232.128:6379> set k1
(error) ERR wrong number of arguments for 'set' command
192.168.232.128:6379> set k1 v1
-> Redirected to slot [12706] located at 192.168.232.138:6379
OK
Redis Cluster has built-in high availability: it already integrates automatic failover and master-replica switching, and does not need Sentinel.
Building a highly available redis cluster
To build a master-replica HA cluster you add replicas, specifying --cluster-replicas N, giving each master N replicas.
redis-cli --cluster create host1:port1 host2:port2 ... --cluster-replicas NAll nodes must be empty instances (not yet in a cluster, no data).
CLUSTER REPLICATE is one of the core commands for manual Redis Cluster operations, used to flexibly adjust the master-replica topology. In production, prefer redis-cli --cluster add-node --slave (which calls this command internally).
redis cluster scaling
Scaling up (add a master):
Add master add-node -> migrate slots reshard -> rebalance cluster
- Add
root@slpc:/var/log/redis# /usr/local/redis/bin/redis-cli --cluster add-node 192.168.232.128:6381 192.168.232.128:6379 --cluster-master-id 831e4c891d2025233320461e4b6aa3ea3e9c3c97 --user admin --pass admin_password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 192.168.232.128:6381 to cluster 192.168.232.128:6379
>>> Performing Cluster Check (using node 192.168.232.128:6379)
M: 831e4c891d2025233320461e4b6aa3ea3e9c3c97 192.168.232.128:6379
slots:[0-5460] (5461 slots) master
M: ca62daac7e5028e18f65def8a3ffdfb519c7be63 192.168.232.138:6379
slots:[10923-16383] (5461 slots) master
M: e0856d63d6a1170c0755f74782ba828be23e8fc2 192.168.232.137:6379
slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to 192.168.232.128:6381 to verify there is no functions in it
>>> Send FUNCTION RESTORE to 192.168.232.128:6381
>>> Send CLUSTER MEET to node 192.168.232.128:6381 to make it join the cluster.
[OK] New node added correctly.
- Migrate slots: you can manually migrate a slot with reshard, or auto-migrate with rebalance; you can simulate first with
--cluster-simulate.
root@slpc:/var/log/redis# /usr/local/redis/bin/redis-cli --cluster rebalance 192.168.232.128:6379 --cluster-use-empty-masters --cluster-simulate --user admin --pass admin_password- Cluster rebalance: auto-computed; prefer the automatic computation. Manual reshard is only for when the automatic computation cannot meet requirements or for special operations.
root@slpc:/var/log/redis# /usr/local/redis/bin/redis-cli --cluster rebalance 192.168.232.128:6379 --cluster-use-empty-masters --user admin --pass admin_password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 192.168.232.128:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4.00
Moving 1366 slots from 192.168.232.137:6379 to 192.168.232.128:6381
######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Moving 1365 slots from 192.168.232.138:6379 to 192.168.232.128:6381
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Moving 1365 slots from 192.168.232.128:6379 to 192.168.232.128:6381
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Scaling down (remove a master):
Migrate all its slots away reshard -> delete the master del-node
Add a replica:
Add a replica without allocating slots: add-node --cluster-slave
Reset the cluster: reset a node’s cluster state so it no longer belongs to any cluster, clear slot allocation, and remove its node ID: cluster reset hard
| Mode | Command | Behavior |
|---|---|---|
| SOFT (default) | CLUSTER RESET SOFT | - Clear cluster config (node list, slot allocation) - Keep current database data - Generate a new node ID - Node becomes a standalone master (no longer part of a cluster) |
| HARD | CLUSTER RESET HARD | - Everything SOFT does - Plus FLUSHALL (clear all data)- Fully clean reset |
redis cluster backup and restore
Backup flow
Backing up and restoring a Redis Cluster is more complex than a single Redis, because data is spread across multiple nodes (shards) and includes cluster topology metadata.
After a backup, the cluster topology and slot info may change — for example the shard count may grow, or slots may be reallocated. To deal with this.
What to back up:
- RDB/AOF data files: each node’s actual data
- nodes.conf file: cluster topology metadata
If you only back up the rdb file, the cluster state is lost and the node won’t know which slots it owns.
Backup plan 1: per-node RDB + nodes.conf metadata
Steps:
a. Run BGSAVE on every node b. Copy the data rdb and nodes.conf metadata files
Backup plan 2: back up with redis-cli --cluster backup
Steps:
a. Run redis-cli --cluster backup
Neither backup adapts to cluster topology changes and slot reallocation. To solve this, you can borrow the idea of logical backup.
Backup approach:
a. Run bgsave on each shard to produce an rdb snapshot, upload it to object storage.
b. On restore, use the redisshake tool to read the rdb downloaded from object storage, replay the rdb, and load it into the latest cluster.
c. Another way: start a redis instance for each shard and import data from each shard’s redis instance via redis-cli --cluster import.
Either way, a Redis Cluster backup can hardly satisfy data consistency.
Restore flow
Steps:
- Prepare a new cluster (empty instances), configure
cluster-enabled yes, but do not run--cluster create - Restore data files: put the backed-up dump.rdb and nodes.conf into the corresponding node directories
- Start redis
After redis starts, it loads the rdb data, reads nodes.conf to restore the cluster topology, and automatically handshakes (gossip) with other nodes.
redis cluster commands
postgres@slpc:~$ redis-cli --cluster help
Cluster Manager Commands:
create host1:port1 ... hostN:portN # create a new cluster
--cluster-replicas <arg> # number of replicas
check host:port # check cluster health
--cluster-search-multiple-owners # force detect slot conflicts
info host:port # view cluster info
fix host:port # fix cluster: auto-fix unassigned slots, replicas not replicating correctly, etc.
--cluster-search-multiple-owners
--cluster-fix-with-unreachable-masters # try to fix even if master is down (use with care)
reshard host:port # manually migrate slots (core of scaling)
--cluster-from <arg>
--cluster-to <arg> # target master ID
--cluster-slots <arg> # how many slots to migrate
--cluster-yes # skip confirmation (for automation scripts)
--cluster-timeout <arg>
--cluster-pipeline <arg>
--cluster-replace # delete source node after migration
rebalance host:port # cluster rebalance: auto-balance slot distribution
--cluster-weight <node1=w1...nodeN=wN> # node weights, allocate slots by weight
--cluster-use-empty-masters # allow empty masters to participate
--cluster-timeout <arg>
--cluster-simulate # simulate only, do not execute
--cluster-pipeline <arg>
--cluster-threshold <arg>
--cluster-replace
add-node new_host:new_port existing_host:existing_port # add a node (as master); must start empty redis first; after adding, manually reshard (master) or auto-replicate (replica)
--cluster-slave # add as replica
--cluster-master-id <arg> # specify master ID
del-node host:port node_id # delete a node; if master, migrate all slots away first (reshard); if replica, delete directly. After deletion the node can rejoin other clusters
call host:port command arg arg .. arg # run a command in batch
--cluster-only-masters # run on all masters
--cluster-only-replicas # run on all replicas
set-timeout host:port milliseconds
import host:port # import data from an external redis
--cluster-from <arg>
--cluster-from-user <arg>
--cluster-from-pass <arg>
--cluster-from-askpass
--cluster-copy # keep source data
--cluster-replace # overwrite existing keys in cluster
backup host:port backup_directory # back up the cluster
help
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
Cluster Manager Options:
--cluster-yes Automatic yes to cluster commands promptsReference docs: