How to Monitor a Directory for New Files in Bash and Run a Command Automatically
How to Monitor a Directory for New Files in Bash and Run a Command Automatically
You drop a file into a watched folder, nothing happens, and you spend an hour blaming your script when the real problem is that the file was written in two chunks and your handler fired on the first, half-written one. The fix is inotifywait with a close-write event and a settle check. The one-liner that gets you 80% of the way:
inotifywait -m -e close_write --format '%w%f' /srv/incoming | while read -r file; do
/usr/local/bin/process.sh "$file"
doneNow here is what it actually does, why it breaks, and what to run instead when it does.
What inotifywait actually watches for, and why close_write beats create
close_write fires only after a writer closes a file descriptor it had opened for writing, which means the file is complete on disk at the moment your handler runs.
The instinct is to watch -e create. That event fires the instant the inode exists, which is typically before a single byte has been flushed. If your downstream command is gzip, ffmpeg, rsync, or anything that reads the whole file, it will read a truncated version and either fail or silently produce corrupt output. This is the failure mode that costs people hours: the script “works” in manual testing with touch, then fails in production because real uploads arrive over a network socket in 64KB chunks.
The event set worth using in practice:
close_write— file finished being written. Primary trigger.moved_to— file appeared viamvorrename, common with atomic uploads.create— only useful if you want to log arrivals, not process them.deleteandmoved_from— for cleanup hooks or queue bookkeeping.
Note that inotifywait comes from the inotify-tools package, which is not installed by default on most distributions. On Ubuntu 24.04 and Debian 13: apt install inotify-tools. On RHEL 9 and Rocky 9: dnf install inotify-tools. The -m flag is what keeps it running instead of exiting after the first event.
The bare while-loop has three failure modes you will hit within a week
A pipeline into while read runs the loop body in a subshell, so any variable you set inside it is gone when the loop exits, and a slow handler blocks event consumption.
Three specific problems, in the order you will encounter them:
1. Subshell variable loss. If you increment a counter or accumulate a list inside that loop, the value is discarded. Fix it with process substitution so the loop runs in the current shell:
#!/usr/bin/env bash
set -euo pipefail
WATCH_DIR="/srv/incoming"
HANDLER="/usr/local/bin/process.sh"
count=0
while IFS= read -r file; do
count=$((count + 1))
echo "[$(date -Is)] event #${count}: ${file}"
"$HANDLER" "$file" || echo "handler failed for ${file}" >&2
done < <(inotifywait -m -q -e close_write -e moved_to --format '%w%f' "$WATCH_DIR")2. The half-written file race. Some applications, notably Samba and certain Java uploaders, open, write, close, then reopen and append. You get two close_write events for one logical file. Guard against it by waiting for the file size to stop changing before you hand it off:
wait_for_stable() {
local f="$1" prev=-1 curr
for _ in $(seq 1 20); do
curr=$(stat -c %s "$f" 2>/dev/null || echo -1)
[[ "$curr" == "$prev" ]] && return 0
prev="$curr"
sleep 0.5
done
return 1
}Call it as wait_for_stable "$file" || { echo "unstable: $file" >&2; continue; }. Ten seconds of patience is cheaper than re-processing a corrupt archive.
3. Handler blocking the watcher. If process.sh takes 90 seconds and files arrive every 10, your queue grows unbounded in the kernel buffer and events get dropped. The kernel's inotify queue is finite — fs.inotify.max_queued_events, default 16384 on most 2026 kernels — and when it overflows you get an IN_Q_OVERFLOW event and silently lose notifications. Decouple with a spool directory or push work to a queue.
Running the watcher as a systemd service so it survives reboots
A systemd unit gives you restart-on-failure, journald logging, and no nohup guesswork.
Save this as /etc/systemd/system/dirwatch.service:
[Unit]
Description=Watch /srv/incoming for new files
After=network.target
[Service]
Type=simple
User=watcher
Group=watcher
ExecStart=/usr/local/bin/dirwatch.sh
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.targetThen:
sudo systemctl daemon-reload
sudo systemctl enable --now dirwatch.service
journalctl -u dirwatch.service -fVerified against systemd 255 on Ubuntu 24.04 and systemd 257 on Fedora 42. The Restart=always matters because inotifywait exits with a non-zero status if the watched directory is deleted — which happens more often than you would expect when the directory lives on a mount that briefly disappears.
Why polling still wins for network filesystems and containers
inotify does not work across NFS, and it does not work reliably across Docker bind mounts on macOS, because inotify is a Linux kernel facility and the events happen on a different kernel than the one your process runs on.
If your watched directory is an NFS mount, an SMB share, or a macOS Docker bind mount, inotifywait will either report nothing or report events with wrong paths. The symptom is a watcher that works perfectly in a shell on the host and does nothing inside the container. Do not debug this for an hour. Switch to a polling loop:
#!/usr/bin/env bash
set -euo pipefail
WATCH_DIR="/mnt/share/incoming"
STATE="/var/lib/dirwatch/seen.txt"
touch "$STATE"
while true; do
find "$WATCH_DIR" -maxdepth 1 -type f -newer "$STATE" -print0 |
while IFS= read -r -d '' f; do
/usr/local/bin/process.sh "$f"
done
touch "$STATE"
sleep 15
doneThe cost is latency — up to 15 seconds here — and a directory scan every interval. On a directory with 50,000 files that scan is not free. Keep watched directories shallow and small, or move to find -mmin -1 style windows if you need to bound it.
For container workloads specifically, the 2026-era answer is to run the watcher inside the container against a volume, not on the host against a bind mount. Docker's own documentation on bind mounts notes that file events are not propagated on Docker Desktop for Mac and Windows.
Testing the script without waiting for real traffic
Generate the exact event pattern you care about, in a scratch directory, before you point the watcher at production.
mkdir -p /tmp/watchtest && cd /tmp/watchtest
# simulate a slow multi-chunk write
( exec 3>bigfile.bin; for i in $(seq 1 5); do dd if=/dev/urandom bs=1M count=1 2>/dev/null >&3; sleep 1; done; exec 3>&- ) &
# simulate an atomic rename upload
echo "payload" > .tmp.part && mv .tmp.part final.txtRun your watcher against /tmp/watchtest and confirm you see exactly two events, not six. If you see six, you are watching create or modify and you have found your bug before it found you.
For load testing, stress-ng --hdd 1 --hdd-bytes 1G will generate a sustained write pattern; watch cat /proc/sys/fs/inotify/max_queued_events alongside your handler's runtime to see whether you are near the overflow threshold.
What to do next, and the failure mode to watch for
Start with the process-substitution loop and close_write, add wait_for_stable the first time you see a truncated file, and only move to polling when the filesystem forces you to.
The failure mode that bites hardest is silence. A watcher that has crashed, been OOM-killed, or hit an inotify queue overflow looks identical to a watcher with nothing to do. Emit a heartbeat — a log line every five minutes from a timeout 300 inotifywait ... wrapper, or a systemd WatchdogSec= — so that "no output" and "dead" are distinguishable in your logs. A monitoring script nobody monitors is just a script that stops working on a Tuesday and gets noticed on Friday.