df du differ: find deleted files Linux

Why df and du Disagree: Finding Hidden Gigabytes on Linux

The df -h command shows 95% disk usage, while du -sh /* sums up only 20%. The remaining 75% are invisible to du but occupy disk blocks. The reason is open file descriptors from processes pointing to deleted files. df reads the file system superblock directly, du traverses the directory tree and ignores files without entries in it.

An inode stores file metadata, including the list of data blocks. rm only removes the directory entry (hard link), the inode link count drops, but if a process holds a descriptor, the blocks remain allocated.

Diagram:

Before rm:
directory        inode          blocks
/var/log/  --> app.log  -->  [block1][block2]
               ^
               nginx descriptor (pid 1234, fd 7)

After rm:
directory        inode          blocks
/var/log/  --> (deleted) -->  [block1][block2]
               ^
               nginx descriptor

du doesn't see the file in /var/log/, df records the occupied blocks.

Classic Cases of the Problem

Log rotation: logrotate performs rm on an old file, but nginx or java holds a descriptor from startup. The process writes to the deleted file, df registers increased usage, du does not.

Other scenarios:

• Long-running daemons with open logs.
• Bugs in cleanup scripts where rm precedes descriptor closure.
• Docker containers where internal processes hold host files.

The problem accumulates: gigabytes disappear unnoticed until the disk fills up.

Finding Processes with Deleted Files

Use lsof +L1 for files with NLINK < 1 (deleted from FS but open):

COMMAND  PID  USER  FD  TYPE  DEVICE  SIZE/OFF  NLINK  NODE  NAME
nginx   1234  www   7w  REG   8,1  2147483648  0  12345  /var/log/app.log (deleted)
java    5678  app   3w  REG   8,1   536870912  0  67890  /var/log/service.log (deleted)

Key fields:

• COMMAND/PID: process and ID.
• FD: descriptor (7w — write, number 7).
• SIZE/OFF: size — source of lost GB.
• NLINK=0: confirmation of deletion.
• NAME (deleted): path before rm.

Size filter: lsof +L1 | awk '$7 > 104857600 {print}' (files >100MB).

Freeing Space Without Restart

For process PID 1234, FD 7:

$ > /proc/1234/fd/7

/proc is procfs in memory. The command zeroes the file via the descriptor, the kernel frees blocks instantly. df updates, the process continues writing to fd 7.

Check: lsof +L1 | grep 1234 — SIZE/OFF=0 or empty.

Script automation:

#!/bin/bash
lsof +L1 | awk '$7 > 104857600 && $9 ~ /deleted/ {print $2, $8}' | while read pid fd; do
  echo "Zeroing $pid/$fd"
  > /proc/$pid/fd/$fd
  echo "Freed"
done

Preventing the Problem in logrotate

Avoid rm in rotation. In /etc/logrotate.d/app:

/var/log/app.log {
    daily
    rotate 7
    compress
    copytruncate
}

copytruncate copies the log, then truncates the original (> mechanism). The descriptor doesn't change.

Downside: micro-delay in copying — possible loss of lines.

Alternative for zero-loss:

• logrotate: mv old → new file.
• kill -HUP <pid> (SIGHUP for log reopen).

nginx: nginx -s reopen. systemd: systemctl reload nginx.

Monitoring and Alerts

Check script:

lsof +L1 -F s | awk '/^s/ && substr($0,2)+0 > 104857600 {count++} END {print count+0}'

Outputs count of deleted files >100MB.

Prometheus integration:

• node_filefd_allocated — growth of open fd.
• Compare df/disk_used vs du sum.

Cron every 5 min: alert if discrepancy >10%.

Key Takeaways

• df shows allocated blocks from superblock, du shows files visible in directory.
• lsof +L1 finds processes with (deleted) files by NLINK=0.
• Free space: > /proc/PID/fd/FD — zeroing without restart.
• logrotate copytruncate prevents the issue without HUP.
• Monitoring: scripts with lsof or node_exporter metrics.

— Editorial Team