Redis stalls caused by bgsave

When a Redis instance holds a large amount of data—tens of GB, for example—running bgsave may cause Redis to stall (e.g. for about 1 second). The core reason is that Redis is single-process: when bgsave runs, it calls fork() to create a child process, and because the instance’s memory is large, this fork() call takes too long, stalling Redis.

Indeed, running bgsave on a large dataset very likely causes noticeable Redis stalls, sometimes lasting several seconds. The stall does not come from disk I/O; its root cause lies in the fork system call and the operating system’s memory management.

Under Linux, fork() is implemented using copy‐on‐write pages, so the 
only penalty that it incurs is the time and memory required to duplicate the 
parent's page tables, and to create a unique task structure for the child.

In Chinese: On Linux, fork is implemented using copy-on-write pages, so its only overhead is the time and memory required to duplicate the parent’s page tables and to create a unique task structure for the child.

The larger the dataset, the longer it takes to copy the parent’s page tables, and the worse the Redis stall becomes.

The parameter latest_fork_usec indicates the time spent by the most recent fork() system call, in microseconds (1 second = 1,000,000 microseconds).

We can break the stall caused by bgsave into two key phases:

💡 The two core phases of bgsave stalls

  1. Phase 1: fork() the child process — the culprit of the stall

    • When you run bgsave, the Redis main process calls fork() to create a child process for persistence.
    • One of fork()’s core jobs is to copy the parent’s page tables. You can think of the page table as the “index directory” of memory—it records where memory data lives.
    • The larger the dataset, the bigger this “index directory”, and the longer it takes to copy. While fork() is executing, the Redis main process is completely blocked and cannot serve any client requests. This is the direct source of the stall.
    • A real-world case: a Redis instance with a resident set size (RSS) of 16 GB had a page table of 33 MB, and its fork() took as long as 1.01 seconds, directly causing periodic application-layer stalls.

  2. Phase 2: Copy-on-Write — a potential secondary impact

    • After fork() completes, the child process begins writing the in-memory data to disk. At this point, the parent and child share memory.
    • If the parent process (the main Redis) needs to modify a memory page (e.g. to handle a write request), the kernel copies that page before applying the modification. This is the Copy-on-Write (COW) mechanism.
    • Under extreme high-concurrency write scenarios, a large number of memory pages may be copied in a short time. This copying consumes CPU and may introduce extra memory overhead and operation latency. According to testing, under very high write pressure the latency spike caused by COW can exceed 300 milliseconds.

🔍 How to monitor and diagnose the problem

The most direct monitoring metric is latest_fork_usec returned by the INFO persistence command.

  • This value records the number of microseconds consumed by the most recent fork() operation. By periodically collecting this metric, you can tell whether fork() latency is growing and whether it has reached a level that could cause business stalls (e.g. persistently exceeding several hundred milliseconds or even 1 second).

🛠️ How to optimize and avoid the stall

Knowing the cause, we can optimize from several angles:

  • OS level:

    • Set vm.overcommit_memory = 1: ensures fork() succeeds even when memory is tight, avoiding more severe problems caused by memory allocation failures.
    • Disable Transparent Huge Pages (THP): although THP aims to improve performance, it actually increases the memory-copy overhead and latency in fork-heavy scenarios like Redis. It is recommended to run echo never > /sys/kernel/mm/transparent_hugepage/enabled in the system startup script.

  • Redis configuration level:

    • Control the single-instance memory cap: this is the most fundamental fix. Experience suggests keeping a single Redis instance under 10 GB to 20 GB of memory. The larger the memory, the longer fork() takes.
    • Enable active defragmentation: if memory growth is unavoidable, enable activedefrag yes. This effectively reduces memory fragmentation and lowers RSS, indirectly reducing fork() pressure.
    • Optimize the persistence strategy:
      • If the business allows, consider lowering the trigger frequency of the save configuration.
      • If data safety is critical, weigh whether to use AOF + mixed persistence (aof-use-rdb-preamble yes). Mixed persistence also triggers fork() when rewriting the AOF, but its temp files and downstream impact differ slightly from RDB, so it can serve as an alternative.

  • Architecture level:

    • Use Redis Cluster: once a single instance exceeds 20 GB, the most thorough solution is to adopt Redis Cluster. By spreading data across multiple nodes, each node (shard) holds much less data, fundamentally solving the problem of a single instance’s excessively high fork() latency. This also echoes the horizontal-scaling capability of Cluster you learned about earlier.

In short, monitoring latest_fork_usec is how you discover the problem, while controlling single-instance memory size and tuning kernel parameters is the core method to solve it. As data keeps growing, moving to Redis Cluster is the better long-term architecture.

Reference: Failure Analysis | A Case of Periodic Redis Stalls Caused by bgsave