Why Processes Die of OOM with Memory to Spare: Overcommit, cgroups, the OOM Killer

Infrastructure Linux
5 min read

The monitoring graph shows memory to spare, yet a process is dead, and dmesg carries the OOM killer’s fingerprints. “There’s memory left — why OOM?” is one of the most common paradoxes in Linux operations, and the answer has three layers. Linux only promises memory and delivers it later (overcommit); the threshold for death may be a cgroup limit rather than the whole system; and the graph’s “remaining memory” may have been the wrong number to read in the first place. Let’s unpack them one at a time.

The premise: malloc does not give you memory #

The first key is overcommit. When a process requests 1GB via malloc, the kernel does not reserve physical memory — it reserves virtual address space. Actual physical pages are allocated the moment the process first writes to those addresses. That is how Linux can calmly promise more than physical memory exists, and most of the time it works out, because processes rarely use everything they asked for. It is the same philosophy as fork working via copy-on-write.

The trouble comes when the promises get called in at once. When processes actually start writing to the memory they were promised and physical memory truly runs out, the kernel has no options left: it cannot renege on committed memory, and it cannot halt the whole system. Enter the OOM killer — the kernel’s last resort of killing someone to reclaim memory.

The behavior can be changed with vm.overcommit_memory (2 enables strict mode), but the default on desktops and general servers is heuristic overcommit, and most environments run on that premise.

What “free memory” in free actually means #

The second key is reading the metrics. In free output, the column to watch is not free but available.

  • free: memory used for nothing at all. Linux uses spare memory as page cache (a cache of disk contents), so the healthier the system, the smaller this number.
  • available: an estimate of what a new process could actually use, including the reclaimable part of the page cache.

A small free is normal; a bottomed-out available is the danger signal. But processes can die even when available looks plentiful — that is the third key.

Container-era OOM: the boundary is the cgroup, not the node #

Most OOMs in modern environments are not whole-system memory exhaustion but cgroup limit hits. Containers (Docker, Kubernetes) cap memory with cgroups, and when the processes inside reach that cap, a cgroup-scoped OOM kill fires no matter how much memory the node has left.

That is the true identity of “the node graph shows headroom but the pod got OOMKilled.” The places to check differ too.

Check 
# whole-system OOM: kernel log
dmesg -T | grep -i "out of memory"
journalctl -k | grep -i oom

# Kubernetes: exit code 137, reason OOMKilled
kubectl describe pod mypod   # Last State: Terminated, Reason: OOMKilled

Exit code 137 (128 + SIGKILL’s 9) is OOM’s signature. The kernel log records which cgroup, which process, and the memory state at the time, so the first command of any OOM investigation is always dmesg.

The classic JVM-in-a-container incident lives here too. Set the max heap equal to the container limit, and the moment off-heap memory (metaspace, thread stacks, native) piles on top, the limit is breached and the container dies. The container limit applies to the process’s total memory use, and must sit comfortably above the application’s configured heap cap.

How the victim is chosen: oom_score #

The OOM killer does not kill at random. Each process gets an oom_score (roughly based on its share of memory use), and the highest scorer dies. You can weight this with oom_score_adj (-1000〜1000).

  • -1000 is exemption — why essential daemons like sshd carry low values — and conversely, giving expendable cache-like processes a positive value volunteers them first.
  • Kubernetes wires this into QoS classes. Pods with requests equal to limits (Guaranteed) are protected; pods with no requests (BestEffort) die first. If “our important service keeps dying first,” the first thing to check is requests and limits — the QoS class.

Response: there is an order #

  1. Confirm the cause of death: dmesg and kubectl tell you whether it was a system OOM or a cgroup OOM, who died, and who held the memory at the time. Skip this and jump to “add more memory,” and the incident repeats.
  2. Limit problem or usage problem?: if normal load genuinely needs more than the limit, raising limits is correct; if usage climbs without end, it is a memory-leak investigation. For Python, profiling with py-spy is the tool.
  3. Swap and pressure signals: nodes without swap go straight from memory pressure to OOM. To see the signal before the death, put PSI (pressure stall information) or working-set trends into monitoring. Other reasons systems slow down were covered in The Truth About Load Average.

Summary #

  • Linux overcommits: memory is promised, and physical pages are allocated on first write. When promises get called in at once, the OOM killer steps in.
  • Watch available, not free. A small free is normal because of the page cache.
  • Most modern OOMs are cgroup limit hits, not node exhaustion — containers die with memory to spare on the node. Exit code 137 and dmesg are the confirmation tools.
  • Victims are chosen by oom_score, and in Kubernetes the QoS class created by requests and limits sets that ranking.
  • The response order: confirm the cause → separate limit problems from leak problems → add pressure metrics to monitoring. “Add memory” without confirmation is a scheduled recurrence.
X