The Linux OOM killer terminates processes when the system runs out of memory. This guide explains how to identify OOM kills, interpret kernel logs, adjust oom_score to protect critical processes, and configure systemd-oomd on Ubuntu 24.04.

What the OOM killer does

When Linux exhausts available memory, the kernel’s out-of-memory (OOM) killer selects a process and terminates it to prevent a complete system hang. The kernel calculates a badness score for each running process and kills the one with the highest score. The killed process receives a SIGKILL signal and exits immediately, usually with exit code 137.

The OOM killer is a last-resort mechanism. It does not ask for permission, and it does not choose wisely by default. A critical database process can be killed while a less important background job survives.

How to identify an OOM kill

If a process disappears without a clear error, check kernel logs first.

“`bash
dmesg | grep -i killed
“`

Example output:

“`
[ 123.456789] Out of memory: Kill process 1242 (mysqld) score 850 or sacrifice child
[ 123.456789] Killed process 1242 (mysqld) vsz: 1048576kB, anon-rss: 512000kB, file-rss: 0kB
“`

You can also check systemd journal:

“`bash
journalctl -k | grep -i “out of memory”
“`

Or look for exit code 137 in application logs:

“`bash
echo $?
“`

If the last command returned 137, the process was killed by SIGKILL, which is the OOM killer’s signal.

Step 1: Check oom_score for running processes

Each process has an oom_score in /proc that reflects how likely the kernel is to kill it. Higher scores mean higher risk.

“`bash
ps -eo pid,comm,args –sort=-%mem | head -20
“`

Then check a specific process’s score:

“`bash
cat /proc/1242/oom_score
“`

Example output:

“`
850
“`

A score of 850 means the process is using approximately 85% of available memory and is at high risk of being killed.

Step 2: Adjust oom_score_adj to protect critical processes

You can raise or lower a process’s oom_score_adj value to influence the OOM killer’s decision. The range is -1000 to 1000. A value of -1000 makes the process unkillable by the OOM killer.

To protect a critical process (for example, a database with PID 1242):

“`bash
echo -500 | sudo tee /proc/1242/oom_score_adj
“`

To make a process more likely to be killed (for example, a background batch job):

“`bash
echo 500 | sudo tee /proc/5678/oom_score_adj
“`

These changes take effect immediately but are lost on reboot. To make them persistent, set them in the systemd unit file:

“`ini
[Service]
OOMScoreAdjust=-500
“`

Then reload and restart:

“`bash
sudo systemctl daemon-reload
sudo systemctl restart mysqld
“`

Step 3: Configure systemd-oomd on Ubuntu 24.04

Ubuntu 24.04 uses systemd-oomd as a userspace OOM killer that acts before the kernel OOM killer. It monitors memory pressure using cgroups-v2 and pressure stall information (PSI).

Check if systemd-oomd is active:

“`bash
systemctl status systemd-oomd.service
“`

To configure memory pressure thresholds for a specific service, edit its unit file:

“`ini
[Service]
ManagedOOMMemoryPressure=75%
“`

This tells systemd-oomd to take action when memory pressure exceeds 75%. The default is usually 90%. Lower values make systemd-oomd more aggressive.

Common OOM scenarios

Java applications

Java processes often die with exit code 137 because the JVM heap is too large for the container or system memory. Fix by reducing -Xmx or increasing memory limits.

Docker containers

Containers have memory limits. When a container exceeds its limit, Docker reports OOMKilled. Check with:

“`bash
docker inspect –format='{{.State.OOMKilled}}’
“`

Web servers under traffic spikes

Apache or Nginx can spawn too many worker processes during traffic spikes, exhausting memory. Configure MaxRequestWorkers or worker_processes to match available memory.

What not to do

  • Do not disable the OOM killer entirely. Without it, the entire system can hang when memory is exhausted.
  • Do not set oom_score_adj to -1000 on every process. If nothing can be killed, the kernel panics instead.
  • Do not add swap as a permanent fix. Swap is slow and masks memory problems rather than solving them.

For memory monitoring commands that help prevent OOM kills before they happen, see Linux Process & Memory Commands Cheat Sheet. For disk-full issues that can accompany memory pressure, see Linux Disk Full: A Troubleshooting Runbook.