When "No space left on device" Isn't About Disk Space

The Linux kernel uses the error code ENOSPC (“No space left on device”) for any situation where an internal resource pool is exhausted, not just for low disk capacity. When a process attempts to allocate a kernel‑managed object—such as an inotify watch—and the corresponding limit has been reached, the kernel returns ENOSPC even though the underlying filesystem still reports plenty of free blocks and inodes. This mismatch between the error message and the actual condition is a frequent source of confusion for operators who first inspect df -h or df -i and see no shortage.

Typical scenario

A log‑shipping agent or configuration reloader watches thousands of directories with inotifywait. Each watched directory consumes one entry from the global watch table. The default limit (fs.inotify.max_user_watches) is often 8 192 or 65 536. When the agent monitors a large tree—e.g., 50 000 directories—the watch table overflows, and any subsequent inotify_add_watch() call fails with ENOSPC. The operator sees “No space left on device” and spends time chasing disk usage, while the real bottleneck is the exhausted watch pool.

Diagnostic checklist

• Verify filesystem space: df -h and df -i.
• Search kernel logs for inotify messages: dmesg | grep -i inotify or journalctl -k | grep -i inotify.
• Check current limits: sysctl fs.inotify.max_user_watches and sysctl fs.inotify.max_user_instances.
• Count watches used by a process (e.g., with strace -e trace=add_watch or by inspecting /proc/<pid>/fd).

Remediation

Increasing the watch limit resolves the error without touching the filesystem. A temporary change can be applied with:

sysctl -w fs.inotify.max_user_watches=524288

For a persistent configuration, create a drop‑in file:

echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
sysctl --system

If the application also creates many short‑lived instances, raise fs.inotify.max_user_instances in the same file. Remember that these limits are per‑user; containers sharing the same user namespace can exhaust the quota faster than expected.

Understanding that ENOSPC can indicate a depleted kernel resource pool—most commonly the inotify watch table—prevents unnecessary disk investigations and reduces mean‑time‑to‑resolution for monitoring and log‑collection pipelines.

Inotify is the Linux kernel’s native mechanism for reporting file‑system events such as creation, modification, or deletion. An application creates an inotify instance and then registers one or more watches on specific paths. Each watch consumes an entry from a per‑user pool that the kernel maintains; when the pool is exhausted the kernel returns ENOSPC (“No space left on device”) even though the underlying storage is untouched.

Typical defaults for the watch pool are either 8192 or 65536 entries, controlled by the sysctl variable fs.inotify.max_user_watches. A second limit, fs.inotify.max_user_instances, caps the number of distinct inotify instances a single user may create. Both limits are per‑user, not system‑wide, so a process running as root has its own budget separate from non‑privileged users or containers.

Practical example: a log‑shipping daemon that watches /var/log/app recursively will allocate one watch per directory. If the directory tree contains 50 000 sub‑directories, the daemon needs at least 50 000 watches. When the default of 8192 is in effect, the daemon will fail with ENOSPC and the logs will stop flowing.

• Diagnose the condition
  • Check disk space first (e.g., df -h) – it will usually be fine.
  • Search kernel messages: dmesg | grep -i inotify or journalctl -k | grep -i inotify.
  • Confirm the limit: sysctl fs.inotify.max_user_watches.
• Adjust the limit temporarily

  sysctl -w fs.inotify.max_user_watches=524288

• Make the change persistent

  echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
  sysctl --system

• Consider raising fs.inotify.max_user_instances if many short‑lived watchers are spawned.

When planning capacity, count the directories that will be monitored and size max_user_watches accordingly. Remember that container runtimes share the host’s per‑user limits, so a busy host can exhaust the pool faster than expected. Adjusting the sysctl values is a low‑risk operation that resolves the misleading “No space left on device” error without touching the actual filesystem.

Inotify is the Linux kernel subsystem that reports file‑system events to user‑space programs. Each directory that a tool watches consumes one “watch” from a per‑user pool, and each process can create a limited number of inotify instances. When the pool is exhausted the kernel returns ENOSPC (“No space left on device”), which is misleading because the underlying storage is not full.

Before assuming a disk‑space problem, verify that the kernel has reported an inotify limit breach. The most direct way is to search the kernel log for the phrase “inotify watch limit reached”. Example commands:

# dmesg | grep -i inotify
# journalctl -k | grep -i inotify

If the output contains a line such as inotify watch limit reached, the failure is due to exhausted watches rather than a full filesystem.

Next, inspect the two sysctl parameters that control the limits:

• fs.inotify.max_user_watches – maximum number of watches a single user may allocate (default often 8192 or 65536).
• fs.inotify.max_user_instances – maximum number of inotify instances a user may create.

Query the current values with:

# sysctl fs.inotify.max_user_watches
# sysctl fs.inotify.max_user_instances

If the workload (e.g., a log shipper monitoring a deep directory tree) requires more watches than the current limit, increase the values. A temporary change can be applied with:

# sysctl -w fs.inotify.max_user_watches=524288
# sysctl -w fs.inotify.max_user_instances=1024

For a persistent configuration on systemd‑based distributions, create a drop‑in file:

# echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
# echo "fs.inotify.max_user_instances=1024" >> /etc/sysctl.d/90-inotify.conf
# sysctl --system

Typical diagnostic workflow:

1. Confirm the error is ENOSPC from an application.
2. Search kernel logs for “inotify watch limit reached”.
3. Check current max_user_watches and max_user_instances values.
4. Estimate required watches (roughly one per directory monitored).
5. Adjust sysctl settings temporarily, then persist via a drop‑in file.

Remember that the limits are per‑user; containers or services running under the same UID share the same pool, so a busy host can exhaust the quota faster than expected. Properly diagnosing and tuning these parameters eliminates false “disk full” alarms and restores reliable file‑watching behavior.

In Linux the inotify subsystem notifies user‑space programs about filesystem changes. Each directory that a monitoring tool watches consumes a single “watch” from the kernel‑maintained pool fs.inotify.max_user_watches. When the pool is exhausted the kernel returns ENOSPC (“No space left on device”), even though the underlying disk has ample space. This is a common failure mode for log shippers, configuration reloaders, or container‑based agents that traverse large directory trees.

Temporary fix

sysctl -w fs.inotify.max_user_watches=524288

The command writes the new value directly to /proc/sys/fs/inotify/max_user_watches, making the change effective immediately. It persists only until the next reboot.

Persistent fix – sysctl drop‑in

1. Create a dedicated configuration file in /etc/sysctl.d/:

   echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
   

2. Reload the sysctl database so the new value takes effect without a reboot:

   sysctl --system
   

The --system flag reads all files in /etc/sysctl.d/, /run/sysctl.d/, and /usr/lib/sysctl.d/, applying them in order. Because the file lives in /etc/sysctl.d/, the setting survives every system start and remains clearly separated from the legacy /etc/sysctl.conf.

Additional considerations

• fs.inotify.max_user_instances controls how many separate inotify instances a single user may create. If your workload spawns many short‑lived watchers, increase this value in the same drop‑in file.
• Limits are per‑UID. Running a monitoring agent as root gives it a distinct budget from non‑privileged users, but container runtimes often share the host’s UID space, so a busy host can exhaust the pool faster than expected.
• After applying the change, verify the effective value:

  sysctl -n fs.inotify.max_user_watches
  

By understanding that the “no space” error originates from the inotify watch table and by applying the one‑liner or a persistent drop‑in, engineers can eliminate false disk‑space alarms and keep file‑watching services running reliably across reboots.

Inotify is the Linux kernel subsystem that reports file‑system events to user‑space processes. Each directory that a tool watches consumes one entry from the per‑user watch pool, and each process that creates an inotify instance consumes from the per‑user instance pool. When either pool is exhausted the kernel returns ENOSPC (“No space left on device”), even though the underlying file system has free blocks and inodes. Understanding the relationship between directory structure, watch consumption, and the configurable limits is therefore essential for any service that relies on continuous file monitoring (e.g., log shippers, configuration reloaders, or security agents).

Planning watch counts

A good rule of thumb is one watch per directory in the tree that must be monitored. For a tree containing n directories, the required watch budget is at least n. To estimate n you can use:

find /var/log -type d | wc -l

For example, if a container logs from /var/log/app and the command above returns 48 000, the process will need a minimum of 48 000 watches.

Adjusting kernel limits

Two sysctls control the resources:

• fs.inotify.max_user_watches – total watches a single UID may allocate.
• fs.inotify.max_user_instances – total inotify instances a UID may create.

Typical defaults (e.g., 8 192 watches) are insufficient for large trees. Increase them persistently with a drop‑in file:

# /etc/sysctl.d/90-inotify.conf
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=1024

Apply immediately with sysctl --system. This method respects systemd conventions and keeps the change auditable.

Per‑user budgeting and containers

• Limits are scoped to the UID, not the whole host. Root has its own budget; non‑root users have separate budgets.
• Container runtimes often share the host’s UID namespace. Multiple containers running as the same UID will compete for the same watch pool, potentially hitting the limit faster than expected.
• When deploying agents in containers, either run them under distinct UIDs or configure the host’s fs.inotify.* values to accommodate the aggregate watch count across all containers.

Verification steps

1. Confirm the error source: dmesg | grep -i inotify or journalctl -k | grep -i inotify.
2. Count active watches: cat /proc/sys/fs/inotify/max_user_watches and compare with the number of directories you intend to monitor.
3. After raising limits, re‑run the monitoring tool and re‑check the kernel log for the “watch limit reached” message.

By aligning watch budgets with actual directory counts, adjusting both watch and instance limits, and accounting for per‑user consumption in containerized environments, engineers can eliminate surprise ENOSPC failures without misleading disk‑space investigations.