CPU, memory and the kernel's view
Load average
uptime shows load averages over 1, 5 and 15 minutes. On Linux, load counts tasks that are running, runnable (waiting for CPU), or in uninterruptible sleep (D state). So:
- Load 4 on a 4-vCPU box: roughly fully busy.
- Load 12 on 4 vCPUs with high CPU usage: CPU saturation — processes queueing for CPU.
- Load 12 on 4 vCPUs with idle CPU: tasks stuck in D state — storage or NFS problems, not CPU.
Always compare load to the number of CPUs (nproc) and check what the CPUs are actually doing.
CPU time breakdown
top and mpstat -P ALL 1 split CPU time:
| Field | Meaning | High value suggests |
|---|---|---|
us |
User code | The application is busy (profile it) |
sy |
Kernel code | Lots of syscalls, context switches, networking |
wa (iowait) |
Idle while waiting for I/O | Disk bottleneck |
st (steal) |
Hypervisor gave the vCPU to someone else | Noisy neighbour, or burstable T-instance out of CPU credits |
si/hi |
Soft/hard interrupts | Heavy network packet processing |
On AWS, sustained steal on a t3 instance usually means CPU credits are exhausted — check CPUCreditBalance in CloudWatch and move to unlimited mode or a non-burstable type.
Memory: what "free" really means
$ free -h
total used free shared buff/cache available
Mem: 15Gi 6.1Gi 0.5Gi 0.2Gi 8.9Gi 9.0Gi
- free — truly unused RAM. Low is normal: Linux uses spare RAM for the page cache (cached file contents) because unused RAM is wasted RAM.
- buff/cache — page cache and buffers; reclaimable when apps need memory.
- available — the number to watch: an estimate of memory available for new work without swapping.
Process memory columns: RSS (resident set — physical RAM actually used now), VSZ (virtual size — everything mapped, often huge and meaningless). Sort by RSS: ps aux --sort=-rss | head.
The OOM killer
When the kernel can't reclaim enough memory, the OOM killer picks a victim using oom_score (mostly based on memory usage, adjusted by oom_score_adj, −1000 to 1000) and sends SIGKILL. Evidence:
dmesg -T | grep -i -E 'killed process|out of memory'
journalctl -k | grep -i oom
Inside containers, the limit comes from the cgroup: exceeding a container's memory limit triggers an OOM kill of that container even if the host has free RAM — Kubernetes reports OOMKilled, exit code 137.
Fixes: find whether it's a leak (memory grows steadily) or a spike (a large request/batch); right-size limits; for JVMs, set heap relative to the container (-XX:MaxRAMPercentage=75) because the heap isn't the only memory a JVM uses.
cgroups and namespaces — how containers really work
Containers are ordinary Linux processes with two kernel features applied:
- Namespaces control what a process can see:
pid(own PID tree; the app is PID 1),net(own interfaces, routes, ports),mnt(own filesystem view),uts(hostname),ipc,user(UID mapping),cgroup. - cgroups (control groups) control what it can use: CPU, memory, I/O, PIDs. cgroups v2 (unified hierarchy) is the default on modern distros and on EKS AL2023 nodes.
CPU limits in cgroups are quotas per period (e.g. 50 ms of CPU per 100 ms). A bursty app can be throttled even if its average usage is low — visible as nr_throttled in cpu.stat, and as latency spikes. That's the root of the Kubernetes "remove CPU limits for latency-sensitive apps" debate.
You can enter a container's namespaces from the host with nsenter -t <pid> -n ip addr — handy for debugging networking on a node.
Kernel tuning with sysctl
sysctl net.core.somaxconn # read
sysctl -w vm.max_map_count=262144 # set now (Elasticsearch needs this)
echo 'vm.max_map_count=262144' > /etc/sysctl.d/99-es.conf # persist
sysctl --system
Commonly tuned: net.core.somaxconn (listen backlog), net.ipv4.ip_local_port_range (ephemeral ports), net.ipv4.tcp_tw_reuse, fs.file-max, vm.swappiness, net.ipv4.ip_forward (needed for routing/Kubernetes).
File descriptor limits
Each process has soft and hard limits (ulimit -n; /proc/<pid>/limits). Users can raise the soft limit up to the hard limit; only root raises the hard limit. Persistent settings: /etc/security/limits.conf for login sessions, LimitNOFILE= for systemd services, --ulimit nofile= for Docker. "Too many open files" (EMFILE) means raise the limit and check for leaked sockets/files: ls /proc/<pid>/fd | wc -l growing forever is a leak.