Redis Study Notes

These are some notes taken while learning Redis, focusing on its features and the implementation principles behind some of them. Redis is open source, so you can analyze its code, and there are many books—such as Redis Design and Implementation—to learn from.

redisdatabase.svg

The Redis Protocol

To implement Redis, you must first implement the Redis protocol—the communication protocol between Redis clients and the Redis server, and the foundation for implementing Redis clients in various languages.

See Redis Serialization Protocol Specification

Redis Data Structures

Redis supports many data structures, such as strings, hashes, lists, sets, sorted sets, HyperLogLog, streams, geospatial, and more. The five most commonly used are the basic data types:

String Type

The string type is supported; a Redis string stores a sequence of bytes, including text, serialized objects, and binary arrays. A value can be any kind of string (including binary data), and a single value cannot exceed 512 MB.

String commands:

  • SET key value Store a string value
  • GET key Retrieve a string value
  • GETRANGE key start end Return a substring of the string stored at the key
  • GETSET key value Delete the key and return its string value
  • GETEX Set the key’s expiration and return its string value
  • MGET key1 [key2..] Retrieve multiple string values in a single operation
  • SETEX key seconds value Set the string value and expiration; create if missing
  • SETNX key value Store a string value only if the key does not exist
  • SETRANGE key offset value Overwrite part of the string via an offset; create if missing
  • STRLEN key Return the length of the string value
  • MSET key value [key value …] Atomically create or modify one or more string values
  • MSETNX key value [key value …] Atomically modify one or more string values only if all keys are absent
  • PSETEX key milliseconds value Set the string value and expiration in milliseconds; create if missing
  • INCR key Parse the string as an integer, increment by one, and set it; atomic—no race even with concurrent clients
  • INCRBY key increment Atomically increment (or decrement with a negative number) the stored counter
  • INCRBYFLOAT key increment Floating-point counter
  • DECR key Decrement the integer value by one; use 0 as the initial value if missing
  • DECRBY key decrement Subtract a number from the integer value; use 0 as the initial value if missing
  • APPEND key value Append to the key’s value; create if missing

The bitmap is a special case—not a standalone data type, but a set of bit operations defined on the string type. Related commands:

  • GETBIT key offset Return the bit value at the offset
  • SETBIT key offset value Set or clear the bit at the offset; create if missing
  • BITCOUNT Count the number of set bits in the string
  • BITOP Perform bitwise operations on multiple strings and store the result
  • BITFIELD Perform arbitrary bit-field integer operations on the string
  • BITFIELD_RO Perform arbitrary read-only bit-field integer operations on the string
  • BITPOS Find the first set or cleared bit in the string

String command reference

Hash Type

A Redis hash is a record type modeled as a collection of field-value pairs. The HSET command does not distinguish insert from update: if the field does not exist it inserts, otherwise it updates. A hash key can contain up to 2^32-1 fields.

Hash commands:

  • HDEL key field1 [field2] Delete one or more fields and their values; delete the hash if empty
  • HEXISTS key field Check whether a field exists in the hash
  • HGET key field Get the value of a field
  • HGETALL key Get all fields and values in the hash
  • HINCRBY key field increment Increment the integer value of a field; use 0 if missing
  • HINCRBYFLOAT key field increment Increment the float value of a field; use 0 if missing
  • HKEYS key Return all fields in the hash
  • HLEN key Return the number of fields in the hash
  • HMGET key field1 [field2] Return the values of multiple fields
  • HMSET key field1 value1 [field2 value2 ] Set multiple field values (deprecated since v4.0.0)
  • HSET key field value Set multiple field values
  • HSETNX key field value Set a field’s value only if it does not exist
  • HVALS key Return all values in the hash
  • HSCAN key cursor [MATCH pattern] [COUNT count] Iterate fields and values
  • HRANGEFIELD Return one or more random fields from the hash
  • HSTRLEN Return the string length of the value associated with a field; 0 if key or field is missing
  • HEXPIRE Set a field’s expiration with a relative TTL v7.4.0
  • HEXPIREAT Set a field’s expiration with an absolute timestamp v7.4.0
  • HEXPIRETIME Return a field’s expiration as a Unix timestamp in seconds v7.4.0
  • HGETDEL Return a field’s value and delete it v8.0.0
  • HGETEX Get one or more fields’ values and optionally set their expiration v8.0.0
  • HPERSIST Remove the expiration of each specified field v7.4.0
  • HPEXPIRE Set a field’s expiration with a relative TTL in milliseconds v7.4.0
  • HPEXPIREAT Set a field’s expiration with an absolute timestamp in milliseconds v7.4.0
  • HPEXPIRETIME Return a field’s expiration as a Unix timestamp in milliseconds v7.4.0
  • HPTTL Return a field’s remaining TTL v7.4.0

Command reference

List Type

A Redis list is a string list ordered by insertion. Internally a list is implemented with a doubly linked list, so adding elements to either end is O(1).

List commands:

  • BLPOP key1 [key2 ] timeout Remove and return the first element of the list
  • BRPOP key1 [key2 ] timeout Remove and return the last element of the list
  • BRPOPLPUSH source destination timeout Pop a value and push it to another list, then return it (deprecated since v6.2.0)
  • LINDEX key index Return the element at the index
  • LINSERT key BEFORE|AFTER pivot value Insert before or after another element
  • LLEN key Return the length of the list
  • LPOP key Remove and return the first element
  • LPUSH key value1 [value2] Prepend one or more elements to the head
  • LPUSHX key value Prepend one or more elements only if the list exists
  • LRANGE key start stop Return elements in the specified range
  • LREM key count value Remove elements from the list
  • LSET key index value Set the element at the index
  • LTRIM key start stop Trim the list to the specified range
  • RPOP key Remove and return the last element
  • RPOPLPUSH source destination Remove the last element and push to another list, then return it (deprecated since v6.2.0)
  • RPUSH key value1 [value2] Append one or more elements
  • RPUSHX key value Append one or more elements only if the list exists
  • BLMOVE Pop from one list, push to another, and return it; block if empty; delete the list if the last element moved
  • BLPOP Remove and return the first element; block if empty; delete the list if the last element popped
  • BRPOP Remove and return the last element; block if empty; delete the list if the last element popped
  • LMOVE Pop from one list, push to another, and return it; delete the list if the last element moved
  • LMPOP Remove and return multiple elements from a list; delete the list if the last element moved v7.0.0
  • BLMPOP Pop the first element from one of multiple lists; block if empty; delete the list if the last element popped v7.0.0

List command reference

A Redis list holds at most 2^32-1 elements.

Set Type

A Redis set is a unique, unordered collection of strings, implemented internally as a hash table with empty values. Set commands:

  • SADD key member1 [member2] Add one or more members
  • SCARD key Get the number of members
  • SDIFF key1 [key2] Get the difference of multiple sets
  • SDIFFSTORE destination key1 [key2] Store the difference of multiple sets
  • SINTER key1 [key2] Get the intersection of multiple sets
  • SINTERSTORE destination key1 [key2] Store the intersection of multiple sets
  • SISMEMBER key member Check whether a member is in the set
  • SMEMBERS key Get all members
  • SMOVE source destination member Move a member from source to destination
  • SPOP key Randomly return and remove one or more members
  • SRANDMEMBER key [count] Randomly return one or more members
  • SREM key member1 [member2] Remove one or more members
  • SUNION key1 [key2] Get the union of multiple sets
  • SUNIONSTORE destination key1 [key2] Store the union of multiple sets
  • SSCAN key cursor [MATCH pattern] [COUNT count] Iterate over set elements

Sorted Set Type

A Redis sorted set is a unique collection of strings ordered by a score associated with each string. Internally it is implemented with a hash table and a skip list. Commands:

  • ZADD key score1 member1 [score2 member2] Add elements
  • ZCARD key Get the number of members
  • ZCOUNT key min max Count members in a score range
  • ZINCRBY key increment member Increment a member’s score
  • ZINTERSTORE destination numkeys key [key …] Store the intersection of multiple sorted sets
  • ZLEXCOUNT key min max Count members in a lexicographic range
  • ZRANGE key start stop [WITHSCORES] Get members in a range
  • ZRANGEBYLEX key min max [LIMIT offset count] Get members in a lexicographic range
  • ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT] Get members in a score range
  • ZRANK key member Get the ascending index of a member
  • ZREM key member [member …] Remove one or more members; delete the sorted set if all removed
  • ZREMRANGEBYLEX key min max Remove all members in a lexicographic range
  • ZREMRANGEBYRANK key start stop Remove all members in a rank range
  • ZREMRANGEBYSCORE key min max Remove all members in a score range
  • ZREVRANGE key start stop [WITHSCORES] Get members in a range, in reverse order
  • ZREVRANGEBYSCORE key max min [WITHSCORES] Get members in a score range, in reverse order
  • ZREVRANK key member Get the descending index of a member
  • ZSCORE key member Get a member’s score
  • ZUNIONSTORE destination numkeys key [key …] Store the union of multiple sorted sets
  • ZSCAN key cursor [MATCH pattern] [COUNT count] Iterate over sorted set elements
  • BZPOPMAX Remove and return the highest-scoring member from one or more sorted sets
  • BZPOPMIN Remove and return the lowest-scoring member from one or more sorted sets
  • BZMPOP Remove and return members by score from one or more sorted sets; block if none v7.0.0

Other Data Types

Beyond the five basic types, Redis has the following:

  • Vector Sets: a dedicated type for managing high-dimensional vector data, enabling fast and efficient vector similarity search v8.0.0
  • Streams: a data structure that behaves like an append-only log, useful for recording events in order and processing them jointly
  • Bitmap: not a standalone type; bitmaps let you perform bit operations on strings
  • Bitfield: efficiently encodes multiple counters in a string value; provides atomic get/set/increment and supports different overflow policies
  • Geospatial: geospatial indexes let you store coordinates and search them
  • JSON: provides JavaScript Object Notation (JSON) support; store, update, and retrieve JSON values like any other Redis type
  • Probabilistic types: allow you to collect and compute statistics in an approximate but efficient way
    • HyperLogLog: probabilistic estimate of the cardinality (number of elements) of large sets
    • Bloom filter: check whether an element exists in a set
    • Cuckoo filter: check whether an element exists in a set
    • t-digest: estimate percentiles of a stream of values
    • Top-K: estimate the rank of data points in a stream
    • Count-min sketch: estimate the frequency of data points in a stream
  • Time series
  • LCS Find the longest common substring redis-7.0.0

Keys

In Redis, a key is a string that may consist of letters, digits, and special characters, with a maximum length of 512 MB. A key can be arbitrary binary data.

Keys are globally unique—meaning unique within a single database; the same key can exist in multiple databases.

Key Commands
  • DEL key Delete a key and its associated value
  • DUMP key Serialize the given key and return the serialized value
  • EXISTS key Indicate whether the key exists in the database
  • EXPIRE key seconds Set an expiration
  • EXPIREAT key timestamp Set an expiration by timestamp
  • PEXPIREAT key milliseconds-timestamp Set an expiration in milliseconds
  • KEYS pattern Block the server until all matching keys are returned
  • MOVE key db Move the key from the current database to db
  • PERSIST key Remove the expiration; the key persists
  • PTTL key Return the remaining expiration in milliseconds
  • TTL key Return the remaining TTL in seconds
  • RANDOMKEY Randomly return a key from the current database
  • RENAME key newkey Rename the key
  • RENAMENX key newkey Rename only if newkey does not exist
  • SCAN cursor [MATCH pattern] [COUNT count] Incrementally iterate a collection
  • TYPE key Return the type of the value stored at the key
Expired Key Handling

Key expiration sets a timeout (TTL) on a key; when it elapses, the key is automatically destroyed. Redis stores expiration times in an expires dictionary (the key points to the actual key, the value is the expiration time). How are expired keys deleted? Redis provides two approaches:

Lazy Deletion (passive)

Triggered: when a client tries to access a key.

Flow:

  1. Check whether the key exists in the expires dictionary.
  2. If it exists and is expired, delete it directly.
  3. Return nil or proceed with the operation.
Active Deletion (periodic)

Triggered: periodically by Redis, configurable via the hz setting.

Flow:

  1. Randomly sample N keys and check for expiration.
  2. Delete all expired keys.
  3. If the expired ratio exceeds 25%, repeat until it drops below 25% or the time limit is reached.

Async deletion optimization:

The UNLINK command lets the main thread mark a key for deletion while a background thread actually frees the memory, avoiding blocking the main thread when deleting large keys.

Automatic Key Creation and Removal
  1. When adding elements to an aggregate data type, if the target key does not exist, an empty aggregate is created before adding.
  2. When removing elements from an aggregate, if the value becomes empty, the key is automatically destroyed. Streams are the only exception.
  3. A read-only command on an empty key (e.g., LLEN returning list length) or a write that removes elements behaves as if the key held the expected empty aggregate.
Key Eviction

Redis is commonly used as a cache to speed up reads against slower servers or databases, holding a copy of data persisted elsewhere (e.g., client—redis—postgres, with data in postgres and redis as cache). Evicting them when memory is low is therefore safe (the data can be fetched from postgres). Redis lets you specify an eviction policy so keys are evicted automatically when the cache exceeds the configured limit. Whenever a client adds to the cache, Redis checks memory usage and, if over the limit, evicts keys per the chosen policy until total memory falls back below the limit.

Set maxmemory to cap memory; when exceeded, Redis evicts per maxmemory-policy. Available policies:

  • noeviction: keys are not evicted, but commands that store new data return an error (reads still work; with replication this applies to masters only)
  • allkeys-lru: evict least-recently-used keys
  • allkeys-lfu: evict least-frequently-used keys
  • allkeys-random: evict random keys
  • volatile-lru: evict LRU keys among those with an expire set
  • volatile-lfu: evict LFU keys among those with an expire set
  • volatile-random: evict random keys among those with an expire set
  • volatile-ttl: evict among those with an expire set the keys with the shortest remaining TTL

Key eviction

For the LRU algorithm, Redis uses an approximate LRU: instead of exact computation, it randomly samples a few keys and evicts the one least recently accessed. The reason for approximate rather than true LRU is that true LRU consumes more memory.

Multiple Database Support

Redis supports 16 databases by default; a client automatically selects database 0 after connecting and can switch with the select command. Redis does not support custom database names—each is numbered. Databases are not fully isolated; for example, flushall can clear all databases in an instance, so databases are more like namespaces.

Because flushdb and flushall are time-consuming, Redis can run them asynchronously: flushall async, delegated to a background thread.

Persistence

Redis supports two persistence methods: RDB and AOF. The former periodically stores in-memory data to disk per configured rules; the latter records the commands themselves after each execution.

RDB

RDB persistence is done via snapshots. When certain conditions are met, Redis automatically generates a copy of all in-memory data and stores it on disk. Snapshots occur in these cases:

  • Executing save or bgsave (save is synchronous and blocks all client requests; bgsave snapshots asynchronously in the background and keeps serving clients)
  • Automatic snapshot per configuration rules
  • Executing flushall
  • During replication

Snapshot process:

  1. Redis uses fork to duplicate the current process.
  2. The parent process keeps receiving and processing client commands, while the child process writes in-memory data to a temporary file on disk.
  3. Once the child finishes writing, it replaces the old RDB file with the temporary one, completing the snapshot.

Snapshot principle:

At fork, the OS uses copy-on-write: at the instant fork happens, parent and child share the same memory. When the parent modifies a region (e.g., a write command), the OS copies that region so the child’s data is unaffected. Thus the new RDB file stores memory as of the fork instant.

After startup, Redis reads the RDB snapshot to load data from disk into memory. With RDB, an abnormal exit loses all changes made after the last snapshot.

AOF

When storing non-temporary data, AOF persistence is generally enabled to reduce data loss from process termination. AOF appends every write command Redis executes to a disk file, which obviously reduces performance—whether to enable it depends on the business. AOF is off by default.

Enable it with appendonly yes. Once enabled, every write command is written to the AOF file on disk.

The AOF file records write commands as plain text—its content is exactly the raw communication protocol the client sends to Redis.

At startup, Redis executes the AOF commands one by one to load data from disk into memory; loading is slower than RDB.

Transactions

A client often issues a series of commands to make a set of related changes to a data object. Another client might modify the same object with similar commands in between, causing corruption or inconsistency. A transaction groups a client’s multiple commands as a unit; the commands are guaranteed to execute in order, undisturbed by other clients’ commands.

Why is Redis’s transaction implementation much simpler than PostgreSQL’s? Because Redis’s critical processing is single-threaded—only one client’s commands execute at a time, and the next client’s commands run only after the current one finishes. PostgreSQL is multi-process with concurrent execution, so its transaction implementation is far more complex and its guarantees much stronger.

Also, Redis transactions do not support rollback: if a command fails, subsequent commands still execute and executed operations cannot be undone. The client must handle transaction failures accordingly.

Query Engine

The Redis query engine lets you retrieve data by content rather than by key. The data types introduced above are all looked up by key.

Pub/Sub

Redis has pub/sub: a client can subscribe to one or more channels and receive messages published by other clients.

Redis Pipelining

Redis Pipelining is a technique to improve performance by sending multiple commands at once without waiting for each individual response.

Redis is a TCP server using the C/S model and a request/response protocol. Normally a request goes:

  • The client sends a request to Redis and reads the response from the socket.
  • Redis processes the command and sends the response back.

With pipelining, the server can process a new request even before the client has read the old response, so multiple commands can be sent without waiting for any reply, and all replies read at once.

Note that pipelining does not guarantee atomicity; if you need transactional guarantees, start a transaction.

The essence of the pipeline is a performance gain from changing the read/write order on the client: from write-read-write-read to write-write-read-read, reducing round trips between client and server.

Client-Side Caching

Client-side caching reduces network traffic between Redis clients and the server, usually improving performance. When enabled, the Redis server remembers or tracks the set of keys each client connection has read before—whether the client reads data directly (e.g., GET) or the server computes a value from stored data (e.g., STRLEN). When any client writes new data to a tracked key, the server sends an invalidation message to all clients that previously accessed that key. This warns the client that its cached copy is no longer valid, and the client evicts the stale data. The next time it reads the same key, it goes to the database and refreshes its cache with the updated data.

With client-side caching, the client library maintains a local cache of data items as it retrieves them from the database. When the same item is needed again, the client satisfies the read from the cache instead of hitting the database.

image

Recommended scenario: only a small subset of data is accessed far more frequently than the rest.

A problem caches face: data consistency—how to update the cache when data changes?

All caching systems must implement a scheme to update cached data when the corresponding data changes in the primary database. Redis uses a method called tracking.

Client-side caching implementation:

Redis client-side caching is called tracking, and it has two modes:

  • Default mode: the server remembers which keys a given client accessed and sends invalidation messages when those keys are modified. This uses server-side memory, but invalidations are sent only for keys the client might cache.
  • Broadcast mode: the server does not try to remember which keys a client accessed, so it is memory-safe on the server. Instead, clients subscribe to key prefixes, e.g. object: or user:, and are notified whenever a matching key is touched.

Default mode:

  1. A client can start tracking if it wants; tracking is off by default at connection start.
  2. Once tracking is on, the server remembers which keys each client requested during the connection lifetime.
  3. When a client modifies a key, or it is evicted due to expiration or the maxmemory policy, every client with tracking on that may have cached the key receives an invalidation notification.
  4. On receiving an invalidation, the client must remove the corresponding key to avoid serving stale data.

This sounds great, but imagine 10,000 connected clients requesting millions of keys over long-lived connections—the server would end up storing too much information. So Redis uses two key ideas to bound server-side memory and the CPU overhead of the data structures:

  • The server records, in a global table, the list of clients that might cache a given key. This table is called the invalidation table. It holds a maximum number of entries. If a new key is inserted, the server may evict an old entry by pretending the key was modified (even if it wasn’t) and sending an invalidation—reclaiming memory for that key, even though it forces clients holding a local copy to evict it.
  • In the invalidation table we don’t actually need to store pointers to client structures (which would force garbage collection on disconnect); instead we store only the client ID (each Redis client has a unique numeric ID). If a client disconnects, the information is incrementally garbage-collected as cache slots become invalid.
  • The key namespace is single, not partitioned by database number. So if a client cached key foo in database 2 while another client changed foo in database 3, the invalidation is still sent—letting us ignore the database number and reduce memory use and implementation complexity.

Client-side caching is off by default. How to enable it? In RESP3, client-side caching has two modes:

  • Default mode: CLIENT TRACKING ON to enable, CLIENT TRACKING OFF to disable
  • Broadcast mode: CLIENT TRACKING ON BCAST to enable; CLIENT TRACKING ON BCAST PREFIX prefix1 prefix2 enables broadcast for specific prefixes, so you receive invalidations only for those prefixes rather than all keys.

In RESP2, a redirect mode forwards invalidations to another client via pub/sub.

# telnet B
client id
:368
subscribe _redis_:invalidate  # subscribe to the __redis__:invalidate channel

# telnet A, enable tracking and redirect to B
client tracking on bcast redirect 368

# telnet B, when a key is modified, receives the __redis__:invalidate channel message
message
$20
__redis__:invalidate
*1
$1
a

Opt-in and Opt-out Caching

Opt-in

A client implementation may want to cache only selected keys and explicitly tell the server what it will and won’t cache. This costs more bandwidth when caching new objects but reduces the data the server must remember and the invalidations the client receives.

To do this, enable tracking with the OPTIN option:

CLIENT TRACKING ON REDIRECT 1234 OPTIN

In this mode, keys mentioned in read queries should not be cached by default; instead, when the client wants to cache something, it must send a special command immediately before the command that actually retrieves the data:

CLIENT CACHING YES
+OK
GET foo
"bar"
Opt-out

Opt-out caching lets the client automatically cache keys locally without explicitly opting in for each one. This ensures all keys are cached by default unless specified otherwise, simplifying client-side caching by removing the need for an explicit command per key.

Enable tracking with the OPTOUT option:

CLIENT TRACKING ON OPTOUT

To exclude a specific key from tracking and caching, use the CLIENT UNTRACKING command:

CLIENT UNTRACKING key

Broadcast Mode

So far we described the first client-side caching model. There is another, broadcast mode, which views the trade-off differently: it uses no server-side memory but sends more invalidation messages to clients. Its main behaviors:

  • A client enables client-side caching with the BCAST option and specifies one or more prefixes with PREFIX. For example: CLIENT TRACKING on REDIRECT 10 BCAST PREFIX object: PREFIX user:. If no prefix is specified at all, the prefix is treated as the empty string, so the client receives invalidations for all modified keys. With one or more prefixes, only keys matching one of them are sent.
  • The server stores nothing in the invalidation table. Instead it uses a separate prefix table, each prefix associated with a list of clients.
  • Any two prefixes must not overlap in the keyspace. For example, prefixes “foo” and “foob” are not allowed because both would trigger invalidation for key “foobar”. Using just “foo” is sufficient.
  • Every time a key matching any prefix is modified, all clients subscribed to that prefix receive an invalidation.
  • The CPU the server consumes is proportional to the number of registered prefixes. With few prefixes the difference is hard to notice; with many, the CPU overhead can be substantial.
  • In this mode the server can optimize by creating a single reply for all clients subscribed to a given prefix and sending the same reply to all—helping reduce CPU usage.

Client-side caching introduction Client-side caching reference

Replication

Redis provides replication: when data is updated on one database, it can automatically sync the updates to others. Just configure the replica with --slaveof masterIP masterPort.

When a replica starts, it sends a sync command to the master. On receiving sync, the master begins saving a snapshot in the background and caches the commands received during the save. Once the snapshot completes, Redis sends the snapshot file and all cached commands to the replica. The replica loads the snapshot and executes the cached commands. This is replication initialization; afterward, whenever the master receives a write command it syncs it to the replica, keeping the two consistent.

References:

Redis Command Reference Reading and Writing the Redis RESP3 Protocol and Redis 6.0 Client-Side Caching