Container Runtime Storage Exhaustion & OverlayFS Inode Depletion: The Production Playbook

1. OverlayFS Internals: Layer Stacking, Copy-Up Overhead, and Whiteouts

Modern container engines—predominantly containerd and Docker using the overlay2 driver—rely on Linux OverlayFS (a modern union mount filesystem) to assemble container filesystems efficiently. Understanding its internal mechanisms is critical for troubleshooting catastrophic node disk exhaustion.

OverlayFS constructs a single unified view by layering four discrete directory abstractions:

• Lower Directories (lowerdir): Read-only immutable container image layers stacked in order. Multiple running containers share the same lowerdir layers without duplicating physical disk space.
• Upper Directory (upperdir): The read-write layer unique to each active container instance. All file additions, modifications, and runtime writes land in this directory.
• Work Directory (workdir): An internal kernel-exclusive staging directory on the same backing filesystem used to prepare atomic file operations before exposing them in the upper layer.
• Merged Directory (merged): The composite mount point presented inside the container mount namespace as the root filesystem (/).

When a container deletes an inherited file from an image layer, OverlayFS cannot alter the underlying read-only lowerdir. Instead, it writes a character device with major/minor numbers 0/0 into the upperdir, known as a whiteout device. While the file becomes invisible inside the container, the physical blocks in the lower layer remain occupied on disk.

2. Block Storage Saturation vs. Inode Table Starvation

Filesystem failure under high-density container workloads occurs via two distinct, often independent depletion vectors:

1. Block Allocation Saturation (Zero Bytes Free)

Occurs when total data volume exceeds the block capacity of the underlying filesystem (e.g., ext4 or XFS). Common culprits include unrotated container stdout/stderr logs written to /var/log/pods/, application core dumps, or unmounted local volumes accumulating state in ephemeral container roots.

2. Index Node (Inode) Starvation (Zero File Descriptors Free)

Every file, directory, symlink, socket, and whiteout device requires exactly one inode entry in the filesystem allocation table. Inodes store metadata (permissions, ownership, physical block pointers). When micro-file intensive workloads (e.g., large Node.js node_modules trees, temporary session caches, or millions of tiny extracted build artifacts) generate millions of files, every available inode is consumed while hundreds of gigabytes of raw block capacity remain completely free.

3. Production Incident Postmortem: Kubelet Eviction Storm & Snapshotter Leaks

Incident Summary

At 03:14 UTC, a 40-node production Kubernetes cluster suffered a cascading eviction storm. Over 180 critical microservice Pods were evicted simultaneously with the status Evicted: The node had condition: [DiskPressure]. Attempts to schedule replacements triggered immediate CreateContainerError and SystemOOM across nodes.

Timeline and Root-Cause Sequence

1. Initial Alert: Prometheus fired KubeNodeInodesFree < 3% across worker nodes. Physical disk usage (df -h) reported 68% free space, masking the critical failure.
2. Diagnostic Discovery: Running df -ih /var/lib/containerd revealed 100% Inode utilization (0 inodes available).
3. Corrupted Snapshotter Root Cause: A CI/CD build runner running unprivileged Docker-in-Docker containers terminated uncleanly due to OOM kills. The containerd overlayfs snapshotter failed to clean up unmounted lower/upper mount points, leaving over 14 million orphaned temp directories and stale whiteouts pinned in the VFS dentry cache.
4. Resolution: SREs executed containerd metadata garbage collection via ctr images prune and cleared unlinked open file handles held by dangling containerd-shim processes.

4. Step-by-Step Interactive Diagnostics & Identification Runbook

Execute the following commands in sequence during an active node disk pressure event:

# 1. Inspect both block capacity and inode consumption across all mounts
df -h
df -ih /var/lib/containerd /var/lib/docker /var/log

# 2. Locate the top 10 directories consuming the highest inode counts
find /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -nr | head -n 10

# 3. Identify running container IDs mapping to a specific high-usage OverlayFS layer
# Replace  with the identifier found in step 2
crictl ps -a -q | xargs -I {} sh -c 'echo -n "{}: "; crictl inspect {} | grep "" || true'

# 4. Detect deleted but open file handles consuming active disk blocks
lsof +L1 /var/lib/containerd /var/log/pods

# 5. Measure raw stdout/stderr JSON log accumulation across all Pods
du -sh /var/log/pods/* | sort -hr | head -n 10

5. Automated Layer Reclamation & Daemon Hardening Configuration

Hardening the container runtime configuration prevents log accumulation and enables aggressive garbage collection.

containerd Production Configuration (/etc/containerd/config.toml)

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  # Dedicated image filesystem configuration
  sandbox_image = "registry.k8s.io/pause:3.9"
  max_container_log_line_size = 262144

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"
    discard_unpacked_layers = true

  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.gc.v1.scheduler"]
  # Enforce proactive asynchronous garbage collection
  pause_threshold = 0.02
  deletion_threshold = 20
  mutation_threshold = 100
  schedule_delay = "500ms"
  startup_delay = "100ms"

Automated Emergency Reclamation Script (/usr/local/bin/container-storage-reclaim.sh)

#!/usr/bin/env bash
# ==============================================================================
# Container Storage Emergency Reclamation Script
# Cleans dead containers, dangling images, and truncates bloated logs safely
# ==============================================================================
set -euo pipefail

echo "==> [Phase 1] Purging exited container sandboxes..."
crictl rm $(crictl ps -a -q --state Exited) 2>/dev/null || true

echo "==> [Phase 2] Pruning dangling unreferenced container images..."
crictl rmi --prune || true

echo "==> [Phase 3] Truncating pod stderr/stdout logs exceeding 100MB..."
find /var/log/pods/ -name "*.log" -type f -size +100M -exec truncate -s 0 {} \;

echo "==> [Phase 4] Reclaiming containerd builder & snapshot cache..."
if command -v ctr &> /dev/null; then
    ctr --namespace k8s.io images prune --all || true
fi

echo "==> [Done] Current Storage & Inode Status:"
df -h /var/lib/containerd
df -ih /var/lib/containerd

6. Failure Mode and Effects Analysis (FMEA Matrix) for Container Filesystems

7. Engineering FAQ & Linux VFS/POSIX Standards Reference

Q1: Why does `rm -rf` inside a running container not reduce image layer disk usage?

Technical Explanation: Docker and OCI images consist of immutable, read-only tar archive layers stacked via OverlayFS. When you execute rm -rf /some/file inside a container, the file residing in the read-only lowerdir cannot be modified. OverlayFS writes a whiteout character device (inode with major/minor number 0/0) into the mutable upperdir to mask the lower file from VFS readdir operations. The physical disk blocks occupied by the file in the lower layer remain unchanged on disk.

Q2: How does XFS Project Quota enforcement prevent one container from crashing a shared node?

Technical Explanation: Standard POSIX user and group quotas are insufficient in containerized environments where multiple untrusted containers run under arbitrary UIDs. XFS Project Quotas assign a discrete 32-bit project identifier to directory subtrees (e.g., /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/<id>). The Linux kernel enforces hard limits on total block allocation and inode count per project directory. If a single container attempts to flood disk space, writes to that specific container fail with EDQUOT without impacting neighboring containers or host node stability.