Linux /proc Filesystem: Troubleshoot Any Desktop

TL;DR: /proc is the kernel's live dashboard, and it answers the four questions that eat sysadmin evenings: what killed my apps, why is load high, who's eating the disk, and why the network feels dead. This guide walks symptom-first through OOM kills, phantom load, disk I/O, and dead ports — reading the right file, then fixing it with the right /proc/sys knob. You'll need a Linux desktop (Ubuntu, Mint, Debian, Arch, or Fedora) and a shell; root helps for a handful of files.

Leave /proc unmonitored and you'll miss the kernel's real-time heartbeat — the first sign of OOM kills, runaway processes, and silent filesystem corruption. The desktop that "just froze" was talking the whole time. You weren't reading the right file.

Every Linux box ships with a live dashboard the size of a directory tree. /proc is a virtual filesystem: zero bytes on disk, generated on demand by the kernel, and it answers the questions that turn a three-hour debugging session into a three-minute one. The trick is knowing which file to open when the symptom hits. That's what this guide is for — a symptom-first map, the way we actually use it in the lab, not the way the man page lists it.

First, the Map: What /proc Actually Is

When you ls /proc, the numeric directories are processes (each PID gets one), and the named files are kernel state. Nothing here is a real file on a real disk. Open /proc/meminfo and the kernel computes the answer on the spot, from its own counters. That's why every read is a snapshot and why two reads a second apart can disagree.

Here's the decision table we keep pinned to the lab wall:

It won't tell you anything? Start here. If ls /proc comes back empty, the filesystem isn't mounted — a chroot or container that skipped the mount step. Fix it with mount -t proc proc /proc (root required). If a specific file says Permission denied, that's normal on hardened systems: /proc/PID/io belongs to the process owner or root, and some /proc/sys entries are root-only. Read them with sudo, or read the man page first (man 5 proc) to know which ones need it.

Symptom 1: Apps Die and the Desktop Freezes — OOM

Apps die and the desktop freezes? Start here. The OOM killer fires when the kernel can't find a page to reclaim, and it picks a victim by score. Two files tell you the whole story.

Step one, confirm the killer actually fired. On Ubuntu, Mint, Debian, Arch, and Fedora, dmesg is restricted to root by default, so use sudo:

sudo dmesg -T | grep -iE "out of memory|killed process"

If you see a line like Out of memory: Killed process 1234 (firefox), the killer fired and you have a name. If dmesg is silent, the freeze has a different cause — check the load section below before blaming RAM.

Step two, read the honest memory number. MemFree lies on healthy systems: Linux stuffs free RAM with page cache, so MemFree hovers near zero while the box runs fine. MemAvailable (added in kernel 3.14) estimates what a new allocation could actually reclaim — that's the number that matters:

grep -E "MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree" /proc/meminfo

If MemAvailable sits in the low hundreds of MB and SwapTotal is 0, the box is one allocation away from the killer. free -h reads this same file and formats it for humans — same data, easier eyes.

Step three, find the victim before it dies. The kernel scores every process from 0 to 1000; higher means more likely to be killed. Check the score of the process you care about:

cat /proc/$(pgrep -n firefox)/oom_score

Swap in a real process name if firefox isn't yours. A score above 500 means the kernel will happily sacrifice it. To protect a critical app, write a negative oom_score_adj (range −1000 to +1000, −1000 makes it immune); to make a known leak die first, write a positive one. The pragmatic fix for a desktop that OOMs is usually more swap — zram on modern distros — plus vm.overcommit_memory set to 2 if you want the kernel to refuse impossible allocations instead of killing your editor at 2 AM. All of those are /proc/sys knobs; we'll get to persistence in a minute.

Symptom 2: Load Average Is High, but Nothing Looks Busy

Load average is high but the CPU is idle? Start here. This is the classic phantom-load trap, and it's almost always I/O hiding behind a process in D state.

Step one, read the trend:

cat /proc/loadavg

The line has three numbers — 1, 5, and 15 minute averages — then a fraction like 2/341 (runnable processes / total) and the PID of the last process spawned. If the 5-minute number is above your core count and the CPU looks idle, you're not looking at CPU work. Load counts uninterruptible sleep too, and that's where the phantom lives.

Step two, split the picture with vmstat. It reads /proc/stat, /proc/meminfo, and /proc/vmstat and prints a column per resource:

vmstat 1 5

Watch two columns: r (runnable, waiting for CPU) and b (blocked, waiting on I/O). If b climbs while wa (I/O wait) stays high, the disk is the bottleneck and the CPU is innocent. If r is high instead, you have a CPU problem and the load is real.

Step three, name the D-state process. The STAT column in ps marks it with a D — uninterruptible sleep, usually stuck on a disk that won't answer:

ps -eo pid,stat,wchan,comm | awk '$2 ~ /D/'

The wchan column shows the kernel function the process is parked in — that's your clue to the device. A process stuck in wait_on_page_bit or blkdev_issue_flush is waiting on a disk; one parked in rpc_wait_bit_killable is waiting on a network filesystem. Check cat /proc/mounts for NFS or CIFS mounts that might be hanging, then check the disk itself (smartctl, or the next section). A D-state process that won't die even with SIGKILL is the kernel's problem, not the process's — a reboot is sometimes the only clean exit.

Symptom 3: Disk I/O Crawls — Who's Eating the Disk?

The disk grinds but you can't name the culprit? Start here. Two files, one per-process and one per-device.

Step one, per-process I/O. pidstat -d (from the sysstat package) reads /proc/PID/io for every process and prints kB read/written per second:

pidstat -d 1 5

If a process shows up with a fat kB_wr/s, you have your culprit. Drill into its cumulative counters — note that /proc/PID/io is readable only by the process owner or root, so add sudo if the process isn't yours:

sudo cat /proc/$(pgrep -n firefox)/io

read_bytes and write_bytes are cumulative since the process started; cancelled_write_bytes is the write that never landed (think: a process that wrote, then truncated). If the numbers climb between two reads, that process is the disk hog.

Step two, if pidstat shows nothing but the disk still grinds, look at the device itself:

cat /proc/diskstats

Each line is one device: field 4 is reads completed, field 8 writes completed, field 13 is milliseconds spent doing I/O. Read it twice, five seconds apart. If the time field grows fast but no process shows in pidstat, the writes are coming from the kernel itself — writeback flushing dirty pages, or a kernel thread. That's when you check cat /proc/meminfo | grep Dirty and wait, or find the sync loop you started and forgot about. We've all done it.

Symptom 4: The Network Feels Dead — Ports and Sockets

Ports that should listen don't, and connections stall? Start here. /proc/net is the kernel's socket table, and ss is the tool that makes it readable.

Step one, list what's actually listening:

ss -tulpn

The -t filters TCP, -u UDP, -l listening sockets only, -p shows the process, -n skips DNS lookups. If your service isn't in the list, it isn't listening — check whether it's running at all before you touch any config. Under the hood, ss reads /proc/net/tcp, which is hex-encoded; you never want to parse that by hand, which is exactly why ss exists.

Step two, interface counters. /proc/net/dev tracks every interface with running totals — RX/TX bytes, packets, errors, drops:

cat /proc/net/dev

If errs or drop climb between two reads, the problem is the link, not your app — cable, radio, or the switch port. A stalled connection with zero errors is more likely the backlog or the firewall.

Step three, the backlog wall. When a server accepts connections slowly and clients see refusals under load, check the listen backlog:

sysctl net.core.somaxconn

Modern kernels default to 4096, but older ones shipped 128 — and a proxy or web server that sets its own backlog higher than the kernel allows gets silently clamped. If connections die under load and somaxconn is low, that's your wall. Raise it, test, and persist it (next section).

Tuning Knobs: /proc/sys and sysctl Persistence

Everything under /proc/sys is a kernel parameter wearing a file costume. Read one, write one, and the kernel changes behavior on the spot. The classic:

sysctl -w vm.swappiness=10

That's the same as echo 10 > /proc/sys/vm/swappiness, with syntax checking thrown in. But here's the trap: a runtime write is gone on reboot. Every change you make this way evaporates. Persistence lives in /etc/sysctl.conf and /etc/sysctl.d/*.conf — drop a file there, then reload everything with:

sudo sysctl --system

⚠ Lab rule we never break: no live-prod changes without testing. Tune knobs on a VM or a throwaway box first, one variable at a time, and know your rollback before you write: sysctl -p reloads the persisted files, and a reboot resets everything you didn't persist. A wrong vm.overcommit_memory=2 on a busy server can refuse allocations for hours; a wrong net.core.rmem_max can make every connection in the building crawl. Test first. Persist second. Reboot third.

Some knobs are read-only for a reason (kernel.* mostly, and anything you see in sysctl -a that refuses writes). If a write fails with Read-only file system or Permission denied, that's the kernel or SELinux saying no — respect it, and check man 5 proc before you fight it.

The procps Toolkit: What Reads What

Every monitoring tool you already use is a /proc reader wearing a nicer face. Knowing which file each one reads tells you which tool to grab when a symptom hits:

One last read worth building into muscle memory: grep -c ^processor /proc/cpuinfo (or just nproc) — knowing your core count is what turns a load average of 8 from a panic into a shrug.

Which /proc knob will you tune first? Pick one from the tables above, read it on your own machine tonight, change it on a throwaway VM, and watch what breaks. That's how /proc teaches — by letting you poke it and survive the consequences. Start with vm.swappiness; it's the gentlest lie detector on the box.

Go deeper: Essential Linux Tools After a Fresh Install for the monitoring stack that reads these files for you · DNS & Nameserver Resolution on Linux for the other layer that decides whether your network feels dead · Netcat (nc) Command Examples on Linux for raw port checks when ss isn't enough.