High-Throughput Redis Cluster Architecture: Memory Eviction, Persistence & Invalidation

1. Hash Slot Topologies & CRC16 Distribution Mechanics

A distributed Redis Cluster partitions its keyspace deterministically across 16,384 discrete logical hash slots. Unlike consistent hashing rings used in distributed DHTs, Redis uses a fixed-size slot array where each primary node owns a subset of contiguous or fragmented slots.

When a client issues a command for a key, the target hash slot is computed using the CRC16 checksum modulo 16384:

$$\text{Hash Slot} = \text{CRC16}(\text{key}) \pmod{16384}$$

If a client sends a query to Node A, but the key hashes to a slot owned by Node B, Node A does not proxy the request. Instead, it responds with a redirection error: -MOVED 3999 10.0.1.15:6379. High-throughput client SDKs (such as Jedis, Lettuce, or redis-py) cache this slot-to-node topology locally to achieve zero-hop routing.

2. Memory Eviction Algorithms: LRU vs. LFU Internal Approximations

When dataset growth exceeds the maxmemory ceiling, Redis does not maintain an exact doubly-linked list of millions of keys due to extreme memory overhead. Instead, it employs probabilistic sampled approximations.

1. Approximated Least Recently Used (LRU)

Every Redis object header contains a 24-bit lru field recording the server clock timestamp (1-second resolution). During eviction cycles, Redis randomly samples maxmemory-samples keys (default: 5) and evicts the key with the oldest idle time in the sample pool. Setting maxmemory-samples 10 achieves a mathematical distribution almost indistinguishable from true LRU while conserving metadata space.

2. Approximated Least Frequently Used (LFU)

Redis divides the 24-bit field into two components: an 8-bit Logarithmic Access Counter (0–255) and a 16-bit Last Decay Time (minute resolution). The counter increments with diminishing probability governed by lfu-log-factor and decays over time based on lfu-decay-time:

# Probability of counter increment on key access:
# Probability = 1 / (current_counter * lfu-log-factor + 1)

3. Persistence Latency: Fork Overhead, CoW Penalties & Multi-Part AOF

Redis persistence operates through snapshotting (RDB) and write-ahead logging (Append Only File - AOF). Both mechanisms utilize the Linux fork() system call to perform disk I/O in a background child process.

# Disable Transparent Huge Pages permanently on database nodes
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Ensure Linux allows memory overcommit for fork safety
sysctl vm.overcommit_memory=1

4. RESP3 Protocol & Server-Assisted Client-Side Caching

Redis 7+ utilizes the RESP3 protocol to support server-assisted client-side caching. Instead of issuing network round-trips for hot keys, client runtimes maintain an in-process memory cache. The Redis cluster tracks active keys and dispatches asynchronous invalidation messages via dedicated tracking tables.

Client runtimes leverage two primary invalidation modes:

• Default Tracking Mode: The server remembers which keys each client connection has read. When another client modifies a tracked key, the server sends an invalidation payload to the reading client. Requires server memory to maintain key-to-client tracking tables.
• Broadcasting Mode (BCAST): The server does not track per-connection key lists. Instead, clients subscribe to key prefixes (e.g., users:). Any write under that prefix broadcasts an invalidation notice, conserving server memory at the cost of slight network overhead.

5. Production Incident Postmortem: Large Key Deletions & Event Loop Freezes

Incident Summary

An e-commerce API platform experienced catastrophic p99 latency spikes reaching 14,000ms on their primary Redis master. Upstream connection pools saturated, leading to thousands of HTTP 504 gateway timeouts.

Root-Cause Sequence

1. The Trigger: An automated batch job executed synchronous DEL analytics:daily_events on a Set containing 4.8 million member elements.
2. Event Loop Block: Because Redis processes commands via a single-threaded event loop, freeing memory for 4.8 million elements required the CPU to walk the entire hash table and reclaim memory allocator arenas synchronously, blocking the main thread for 1.42 seconds.
3. Cascading Cluster Failover: Neighboring cluster nodes missed heartbeat pings (cluster-node-timeout 1000ms) and initiated an uncoordinated master failover during the freeze.
4. Remediation: Enabled lazyfree-lazy-server-del and transitioned all deletion workflows to asynchronous UNLINK commands.

6. Production-Hardened redis.conf Blueprint with Inline Architectural Annotations

# ==============================================================================
# Production High-Throughput Redis Cluster Configuration
# Target: 100,000+ Operations/Sec per Node | Low Tail Latency
# ==============================================================================

# Network & Connection Boundaries
bind 0.0.0.0
port 6379
tcp-backlog 65535
timeout 0
tcp-keepalive 300
protected-mode yes

# Memory Management & Eviction
maxmemory 32gb
maxmemory-policy allkeys-lru
maxmemory-samples 10
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10
active-defrag-threshold-upper 30

# Multi-Part AOF Persistence Hardening
appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes

# Snapshotting Controls (Relaxed to avoid fork storms)
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes

# Non-Blocking Background Memory Reclamation (Lazyfree)
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes
lazyfree-lazy-user-del yes

# Cluster High Availability & Health Probing
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-migration-barrier 1
cluster-require-full-coverage no
cluster-allow-reads-when-down no

# Client Output Buffer Protections
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 2gb 512mb 60
client-output-buffer-limit pubsub 64mb 16mb 60

7. Failure Mode and Effects Analysis (FMEA Matrix) for In-Memory Clusters

8. Engineering FAQ & Redis Protocol Specifications

Q1: Why does Redis Cluster use 16,384 hash slots instead of 65,536?

Technical Explanation: Redis cluster nodes exchange heartbeat ping/pong packets every second. The heartbeat payload contains the node's hash slot configuration bitmap. A 16,384 slot allocation requires a compact 2KB bitmap ($16384 / 8 = 2048\text{ Bytes}$). Expanding to 65,536 slots would quadruple the bitmap size to 8KB per heartbeat. Across clusters with hundreds of nodes, this creates excessive network gossip traffic while 16,384 slots provide more than enough granularity for scaling up to 1,000 physical master instances.

Q2: What is the difference between DEL and UNLINK?

Technical Explanation: DEL synchronously removes the key and immediately frees all underlying memory allocations on the main event loop thread. For large complex data structures, this causes CPU stalls. UNLINK performs an $O(1)$ removal of the key pointer from the keyspace namespace instantly on the main thread, and then delegates the asynchronous memory deallocation of the actual data elements to a background I/O worker thread.