The classic dashboard has three gauges: CPU, memory, disk. When a server misbehaves, that’s where everyone looks first. So the worst kind of outage is the one where all three read green and the site is still dropping connections, throwing 500s, or refusing logins at random.
What’s happening is real, it’s just not on that dashboard. A handful of kernel and per-process limits run out independently of CPU, RAM and disk space — and when they do, the symptom shows up two layers away from the cause. Here are the three that catch fleets most often, how to confirm each in one command, and why none of them is visible on the gauges you were watching.
1. The connection table is full
The first one almost nobody monitors. The kernel tracks every connection in a table (nf_conntrack), and that table has a fixed size. On a busy box behind a firewall — which is every box — it fills up. When it does, the kernel stops accepting new connections and writes one line to the log:
nf_conntrack: table full, dropping packet
From the outside this looks insane: the server is up, CPU is at 20%, but new visitors time out while existing connections are fine. Check how close you are:
# current vs maximum tracked connections
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
If count is anywhere near max, you’re one traffic spike away from dropping packets. The fix is to raise net.netfilter.nf_conntrack_max (and the hashsize) in sysctl, or shorten the timeouts that keep dead connections in the table — but the point of this post is upstream of the fix: nothing on a CPU/RAM/disk dashboard would ever have told you this was coming.
2. Out of file descriptors
Every open file, every socket, every pipe costs a file descriptor. There’s a per-process ceiling and a system-wide one, and busy services — a database, an FPM pool, an event-driven proxy — burn through them fast. The application-level symptom is unmistakable once you know it:
accept4(): Too many open files
Two limits to check, because either can be the one you hit:
# system-wide: allocated / unused / max
cat /proc/sys/fs/file-nr
# per-process ceiling for a running service (by PID)
cat /proc/<pid>/limits | grep "open files"
The per-process one bites hardest because it’s usually a stale default. A systemd service inherits LimitNOFILE, and the value that was fine three years ago throttles the same service today. The disk isn’t full, memory is fine — the process simply isn’t allowed to open another socket. Raise LimitNOFILE in the unit (or fs.file-max system-wide) and reload.
3. Out of inodes (with disk space to spare)
This one produces the most confusing error of all: No space left on device while df -h swears the disk is half empty. The filesystem ran out of inodes, not bytes. Every file consumes one inode regardless of size, so a few million tiny files — PHP session files, cache fragments, a maildir, an unrotated log directory — exhaust the inode table while barely touching capacity.
df -h won’t show it. The -i flag will:
df -i
# look at IUse% — it can read 100% while df -h shows 50%
Once you know to look, the fix is usually janitorial (find the directory with millions of files and clean it, or fix whatever keeps creating them). The trap is the half-day you can lose before you know to look, chasing a “disk full” error on a disk that isn’t full.
The pattern, not the three bugs
Conntrack, file descriptors and inodes are just the three most common. The list goes on — ephemeral port exhaustion under high outbound fan-out, the per-cgroup PID limit (pids.max) capping how many tasks a service can spawn, TIME_WAIT sockets piling up. They share a shape:
- They’re hard limits, not gradual degradation. Up to 99% everything is perfectly fine; at 100% it falls off a cliff. There’s no warm-up curve to notice.
- The symptom is dislocated from the cause. “New visitors time out” doesn’t say connection table. “Too many open files” in one service doesn’t say the limit is a
systemddefault. “No space left” actively lies about what’s full. - None of them is CPU, RAM or disk. The three gauges everyone watches will read green through every one of these outages.
That’s exactly why they eat hours: the dashboard says healthy, so you distrust the dashboard before you distrust a limit you didn’t know you had.
Watching the limits, not just the gauges
The fix at the fleet level isn’t memorizing these commands — it’s treating the limits as first-class metrics, alongside CPU and memory, so you find out at 80% instead of at the outage. CentralHost’s agent samples conntrack utilization, file-descriptor pressure and inode usage on every managed host and alerts edge-triggered when one crosses its threshold — the alert fires once on the way up, not every minute while it stays hot.
And when something does break, the AI Assistant is built to reason across exactly this dislocation. You hand it the symptom — “new connections to this box time out intermittently” — and it correlates the network drops with the conntrack counter instead of starting, like a human would, by staring at a CPU graph that has nothing to tell it. The cause is two layers below the error; the whole point is to not spend the afternoon discovering which two.
Next time a server “has nothing wrong with it” and still misbehaves, the first move isn’t the CPU graph. It’s the limit nobody put on the dashboard.