Linux OOM-Killer Architecture & Virtual Memory Subsystem: The Production Playbook

1. Linux Virtual Memory Layout: Page Cache, Anonymous Memory & Slab Allocators

The Linux memory management subsystem abstracts physical RAM into 4KB virtual pages mapped via multi-level page tables. At the physical layer, RAM is categorized into distinct memory zones (ZONE_DMA32, ZONE_NORMAL, ZONE_MOVABLE) governed by the Buddy Allocator, which manages contiguous power-of-two page orders (Order 0 through Order 10).

From an operating system perspective, host memory is partitioned into three major consumers:

• Anonymous Memory: Heap, stack, and mmap memory allocations owned by running processes. Because anonymous memory has no persistent backing file on disk, it cannot be discarded during memory pressure without moving to Swap space.
• Page Cache: File-backed disk pages cached in RAM to accelerate disk read and write operations. Under memory pressure, clean page cache pages can be reclaimed instantly without I/O overhead.
• Kernel Slab Allocators (SLUB/SLAB): Kernel data structures, dentry (directory entry) caches, inode tables, and socket control buffers. Reclaimable slab is tracked under SReclaimable inside /proc/meminfo.

2. Page Allocation Lifecycles, Watermarks (kswapd) & Direct Reclamation Stalls

Every memory zone defines three deterministic watermark thresholds: WMARK_MIN, WMARK_LOW, and WMARK_HIGH, calculated dynamically based on the kernel's vm.min_free_kbytes parameter:

# Check active memory watermarks across NUMA nodes
cat /proc/zoneinfo | grep -E "(Node|zone|min|low|high|nr_free)"

The Three Watermark Operational Phases

1. Zone Free > WMARK_HIGH: The zone is healthy. Allocations execute synchronously at near-zero latency from free buddy lists.
2. Zone Free < WMARK_LOW: The kernel wakes the asynchronous page reclamation daemon (kswapd) in background mode to evict clean page cache and compress memory up to WMARK_HIGH. User processes continue executing uninterrupted.
3. Zone Free < WMARK_MIN: kswapd is overwhelmed. The kernel forces calling user processes into Direct Reclamation. Application execution threads stall, writing dirty pages to disk synchronously. If memory remains below WMARK_MIN, out_of_memory() is invoked in kernel space.

3. The oom_badness() Heuristic Algorithm & Mathematical Scoring

When the kernel determines that physical page allocation cannot proceed, it invokes out_of_memory() in mm/oom_kill.c. The kernel iterates across all processes in the task list to compute an integer score from 0 to 1000 via the internal oom_badness() function:

$$\text{Points} = \frac{\text{Anonymous RSS Pages} + \text{Page Table Pages} + \text{Swap Pages}}{\text{Total Usable RAM}} \times 1000$$

The score is then adjusted based on process priority and configuration knobs:

• Root Process Bias: Processes executing with CAP_SYS_ADMIN or running under UID 0 receive a slight reduction (-30 points) to protect system daemons.
• User-Configured Adjustment Knob (oom_score_adj): Values range from -1000 (completely exempt from OOM-killing) to +1000 (guaranteed first target).

# Protect critical daemon (e.g., PgBouncer or SSHD) from OOM termination
echo -1000 > /proc/$(pgrep -f pgbouncer)/oom_score_adj

# Check calculated OOM badness for an application process
cat /proc/<PID>/oom_score
cat /proc/<PID>/oom_score_adj

4. Dirty Page Flushing Architecture: vm.dirty_ratio vs. Writeback Stalls

When processes write to disk files, data is written to the Page Cache as "dirty" pages. Background kernel threads (flushers) flush dirty pages to physical NVMe/SSD storage. If applications generate dirty data faster than disk write throughput, severe I/O stalls occur.

5. Cgroups v2 Memory Isolation: memory.high Throttling vs. memory.max Hard Kills

In modern containerized infrastructures (Kubernetes 1.25+ with Linux Cgroups v2), memory boundaries operate under hierarchical control controllers rather than global host limits.

The Cgroups v2 Memory Interface

• memory.min: Hard memory guarantee. Pages below this boundary are completely protected from host memory reclamation.
• memory.low: Best-effort memory protection. Pages are reclaimed only if no other unprotected cgroups can yield memory.
• memory.high: The proactive throttling barrier. If a container breaches memory.high, the kernel slows down the container processes with intentional CPU allocation delays while reclaiming memory in the background, avoiding an OOM kill.
• memory.max: The hard ceiling. Exceeding this boundary immediately triggers an internal Cgroup-scoped OOM-killer termination (Kubernetes OOMKilled status).

6. Production Incident Postmortem: Silent Database Node Evictions via Page Cache Bloat

Incident Summary

A primary Redis/PostgreSQL database host with 128GB RAM crashed unexpectedly with zero application-level core dumps. Monitoring dashboards showed host memory hovering at 94% utilization for days prior to failure.

Root-Cause Investigation Sequence

1. The Alert: Prometheus alerted on HostOOMKillDetected. Database processes were terminated abruptly.
2. Memory Allocation Analysis: Inspecting historical /proc/meminfo revealed that unconstrained streaming log uploads consumed 72GB of RAM as Active(file) Page Cache.
3. Watermark Breach: A burst of write transactions pushed allocation below WMARK_MIN. The kernel entered Direct Reclamation, but because disk I/O was saturated writing dirty pages, reclamation stalled.
4. OOM Execution: oom_badness() selected the primary database process (holding 48GB Anonymous RSS) as the single largest memory consumer on the task list and terminated it with SIGKILL.
5. Remediation: SREs configured vm.dirty_background_ratio = 3, elevated vm.min_free_kbytes = 1048576, and isolated database processes within dedicated Cgroups v2 slices with oom_score_adj = -800.

7. Hardened sysctl.conf & Cgroups Production Blueprint

/etc/sysctl.d/99-memory-subsystem.conf

# ==============================================================================
# Production Linux Memory Management & OOM Hardening Blueprint
# Target: High-Concurrency Database, Gateway & Container Host Nodes
# ==============================================================================

# Memory Overcommit Controls
vm.overcommit_memory = 1
vm.overcommit_ratio = 50

# Ensure adequate atomic allocation headroom for kernel networking
vm.min_free_kbytes = 1048576
vm.vfs_cache_pressure = 50

# Dirty Page Background Writeback Tuning (Prevent I/O Spikes)
vm.dirty_background_ratio = 3
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 1500
vm.dirty_writeback_centisecs = 500

# Swap Aggressiveness (Keep minimal for server workloads)
vm.swappiness = 10

# Disable Transparent Huge Pages Defragmentation Delays
vm.zone_reclaim_mode = 0

8. Failure Mode and Effects Analysis (FMEA Matrix) for Host Memory

9. Engineering FAQ & Linux mm/oom_kill.c Kernel Source Reference

Q1: Why does the Linux kernel kill the largest process rather than the process that triggered the OOM?

Specification: Linux Kernel Source mm/oom_kill.c (out_of_memory).
Technical Explanation: The process that triggers the final page allocation failure is often an innocent lightweight daemon (such as an NTP client or a cron job) that happened to request a single page when RAM was exhausted. Killing that process would only reclaim 4KB of memory, causing the very next memory request to fail milliseconds later. The kernel calculates oom_badness() to identify the single process holding the largest volume of physical anonymous memory, terminating it to immediately restore stability across the entire operating system.

Q2: What is the exact difference between exit code 137 and exit code 139 in container runtimes?

Technical Explanation: Exit code 137 indicates the container was terminated via Signal 9 (SIGKILL) ($128 + 9 = 137$), which is the standard signal dispatched by the Linux OOM-Killer or Kubelet when memory limits are exceeded. Exit code 139 indicates Signal 11 (SIGSEGV) ($128 + 11 = 139$), signifying a Segmentation Fault where an application attempted to read or write to an invalid virtual memory address or unmapped memory page.