Linux Disk Full: A Troubleshooting Runbook

“My disk is full” and “du says there is plenty of space” is the most confusing state a Linux machine can be in, and it is not a paradox. df reports filesystem allocation; du reports the size of files it can see. When those disagree, the difference is always information, and the usual cause is a file that has been deleted but is still held open by a running process.

Three questions resolve almost every full-disk incident, and they are cheap. How many inodes are free, which single directory is largest at depth one, and is anything deleted-but-open — checkable with lsof +L1. Each of those three answers eliminates one of the common causes: too many small files, one runaway directory, and space reclaimed but never actually released. The rest of this reference is the details and the tools that make each step faster.

Key takeaways:

df measures filesystem allocation. du measures what it can see. A large gap between them means deleted-but-open files.
A filesystem can be at 0% free with plenty of inodes left, and the reverse. Always check both.
Deleting a file that a process holds open frees nothing until that process closes it or is restarted. This is not a bug; it is the reference-counting model working correctly.
du -x stops at filesystem boundaries. Without it, one du run can walk into a mount point and report nonsense.
Ext4 reserves blocks for root by default, which is part of the gap between df and du on a nearly full filesystem.
Journal and log files are usually the culprit on a service host. journalctl --vacuum-size= and proper logrotate are the fix.

Table of Contents

Question one: is it blocks or inodes?

df -h                       # human readable, all filesystems
df -hT                      # add the filesystem type
df -i                       # inode usage instead of space
df -h --output=source,fstype,size,used,avail,pcent,target

Inodes are the fixed-size structures that represent files and directories, and they are finite. A filesystem with gigabytes free and 0% inodes free cannot create a new file no matter how much space is left. This happens with caches, mail spools, session directories and anything that creates many small temporary files.

Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1      6553600 6553599      1  100% /

# Which directories are holding inodes
for d in /var/* /tmp; do
  printf '%s %sn' "$(find "$d" -xdev -type f 2>/dev/null | wc -l)" "$d"
done | sort -rn | head

The inode-exhaustion signature is a specific failure: creating a new file fails with No space left on device even though df -h shows space free, and existing files can still be appended to. When you see that combination, stop looking at sizes.

Question two: where is the space?

Go top-down rather than trying to list everything:

du -x -h --max-depth=1 / 2>/dev/null | sort -h | tail -20

Three details in that command matter.

-x stops at filesystem boundaries. Without it, du / descends into mounted network and container filesystems and can report enormous or infinite-looking totals. On a host with NFS or overlay mounts this is the difference between a usable answer and a useless one.
--max-depth=1 summarises instead of listing. Then repeat on whichever directory wins.
2>/dev/null suppresses permission-denied noise so the sort is not polluted. When running as root you can drop it.

Then descend, and switch to ncdu once you have a suspect directory. It is an interactive du that lets you walk down the tree and see the sorted breakdown as you go, which is dramatically faster than repeated shell commands:

ncdu /var            # interactive, sorted, navigable
du -x -h --max-depth=2 /var/lib | sort -h | tail -30

To find large individual files rather than directories:

find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

# Files not touched in a year, a common archive candidate list
find /var -xdev -type f -atime +365 -exec ls -lh {} + 2>/dev/null | head -50

Question three: is anything deleted but still open?

This is the case that produces the confusing symptom, and it is worth understanding rather than memorising a fix.

A file on disk is a name (a directory entry) plus the data blocks. Opening a file creates a reference from the process to those blocks. Deleting the file removes only the name. The blocks are reclaimed when the last reference goes away — which happens when the process closes the file descriptor or exits.

So if a process has been writing to a log for a year, and you delete that log, the space is still consumed. The process is still writing, into an unnamed file, and the filesystem cannot free blocks that are still allocated. Space returns only when the process lets go.

# Files with link count 0: deleted, but still held open
lsof +L1

# Same thing, sorted by size, showing what to signal
lsof +L1 2>/dev/null | sort -k7 -n -r | head -20

+L1 means “link count less than 1”, which in practice means deleted-but-open. The fix depends on the process:

If it is a log you can live without losing: truncate rather than delete. : > /var/log/big.log or truncate -s 0 /var/log/big.log frees the blocks immediately without restarting anything.
If you can restart the process: restart it. That is cleaner and also fixes the underlying leak.
If it is a database: do not truncate its files. Use the database’s own tooling. This is how databases get corrupted.

For a system-level diagnosis, /proc holds the same information:

ls -l /proc/*/fd/ 2>/dev/null 
  | grep '(deleted)' | head -20

The lasting fix for log-shaped cases is logrotate with copytruncate, which copies then truncates rather than renaming. It gives up atomicity in exchange for the running process keeping its descriptor, and for a log you do not mind losing a few lines of, that is the right trade.

Other reasons df and du disagree

Cause	What to check
Deleted but open files	`lsof +L1`. The usual cause. Looked at first because it is easiest to rule out
Reserved blocks	Ext4 reserves a percentage for root by default, so `df` shows less available to a normal user than the total
Sparse files	`du --apparent-size` versus plain `du`. A file with holes reports its logical size but occupies little
Hard links	`du` counts a hard-linked file once per invocation, so counting subdirectories separately double-counts
Filesystem overhead and metadata	Ext4 and XFS reserve space for journal and metadata. Small on a large filesystem, visible on a small one
Different mount points	You ran `du` without `-x` and it walked into another filesystem
Snapshots	On btrfs or ZFS, `df` reflects snapshot retention and can report space as used with no visible file owning it

The snapshot case is worth remembering on modern hosts. On a copy-on-write filesystem, a snapshot pins blocks that no current file references, and the resulting df versus du gap has nothing to do with open file descriptors.

The usual suspects on a server

# systemd journal
journalctl --disk-usage
journalctl --vacuum-size=500M
du -sh /var/log/journal /run/log/journal 2>/dev/null

# Rotated logs nobody has cleaned up
du -ah /var/log | sort -h | tail -20
ls -lh /var/log/*.gz /var/log/*.1 2>/dev/null

# Caches
du -sh /var/cache/* 2>/dev/null | sort -h | tail
du -sh /root/.cache /home/*/.cache 2>/dev/null

# Container layers, which is a different filesystem and needs -x discipline
docker system df 2>/dev/null

# Old kernels if this machine has ever had a kernel upgrade
du -sh /boot
dpkg -l 'linux-image-*' 2>/dev/null | tail

A runbook

# 1. Confirm the filesystem is actually full
df -h /var

# 2. Confirm it is blocks, not inodes
df -i /var

# 3. Rule out deleted-but-open first — fastest, most common
lsof +L1 2>/dev/null | head

# 4. Top-level breakdown, staying on one filesystem
du -x -h --max-depth=1 /var 2>/dev/null | sort -h | tail

# 5. Descend into the winner
du -x -h --max-depth=2 /var/lib 2>/dev/null | sort -h | tail -30

# 6. Largest individual files
find /var -xdev -type f -size +200M -exec ls -lh {} + 2>/dev/null

# 7. Reclaim the safe things
journalctl --vacuum-size=500M

# 8. Verify
df -h /var

Order matters here. Steps 1 to 3 take seconds and eliminate the majority of incidents. Descending the tree with du before checking for deleted-but-open files wastes time, because no matter how large the tree, it will never account for space that no longer has a name.

Two things not to do

Do not delete files under /var/lib to free space. Databases, package state and container layers all live there and do not tolerate manual removal. Find out what owns the space first.
Do not use rm -rf as a first response to “disk is full”. The disk being full means a running process has been failing to write, which means partially written files and possibly half-completed operations. Clean up deliberately, one known-good target at a time.

The same reclaiming-a-full-disk problem exists in cloud environments, with different tooling and the same shape of reasoning. finding and deleting unused AWS EBS volumes automatically covers that side, and an AI-powered Linux security monitor covers why log growth is worth watching before it becomes an incident.