Opscraft
Linux · Lesson 6 of 10 · 7 min read

CPU, memory and the kernel's view

Load average

uptime shows load averages over 1, 5 and 15 minutes. On Linux, load counts tasks that are running, runnable (waiting for CPU), or in uninterruptible sleep (D state). So:

  • Load 4 on a 4-vCPU box: roughly fully busy.
  • Load 12 on 4 vCPUs with high CPU usage: CPU saturation — processes queueing for CPU.
  • Load 12 on 4 vCPUs with idle CPU: tasks stuck in D state — storage or NFS problems, not CPU.

Always compare load to the number of CPUs (nproc) and check what the CPUs are actually doing.

Enlarge diagramWAITING FOR CPUCPUS (NPROC = 4)STUCK ON DISK (D)task 5task 6task 7CPU 0task 1CPU 1task 2CPU 2task 3CPU 3task 4task 8task 9Load ≈ 4 running + 3 waiting + 2 on disk = 9on a 4-CPU machine: more work than it can handleLoad 9, CPUs ~100% busyCPU saturation: find the busy processLoad 9, CPUs mostly idletasks stuck on I/O: look at disks and NFSwaiting for EBS or NFS
Figure 1. Load average counts tasks running, waiting for a CPU, and stuck waiting on disk. Always compare it with the number of CPUs.
Swipe sideways to see all of it, or tap to open it full screen.

CPU time breakdown

top and mpstat -P ALL 1 split CPU time:

Field Meaning High value suggests
us User code The application is busy (profile it)
sy Kernel code Lots of syscalls, context switches, networking
wa (iowait) Idle while waiting for I/O Disk bottleneck
st (steal) Hypervisor gave the vCPU to someone else Noisy neighbour, or burstable T-instance out of CPU credits
si/hi Soft/hard interrupts Heavy network packet processing

On AWS, sustained steal on a t3 instance usually means CPU credits are exhausted — check CPUCreditBalance in CloudWatch and move to unlimited mode or a non-burstable type.

Memory: what "free" really means

$ free -h
              total   used   free   shared  buff/cache  available
Mem:           15Gi   6.1Gi  0.5Gi   0.2Gi     8.9Gi      9.0Gi
  • free — truly unused RAM. Low is normal: Linux uses spare RAM for the page cache (cached file contents) because unused RAM is wasted RAM.
  • buff/cache — page cache and buffers; reclaimable when apps need memory.
  • available — the number to watch: an estimate of memory available for new work without swapping.
Enlarge diagram"free" is lownormal: unused RAM is wasted RAM"available" is lowreal memory pressure: swapping or OOM kills nextused 6.1 GB · your programsbuff/cache 8.9 GB · cached filesfree 0.5available ≈ 9.0 GB: free + cache the kernel can hand back
Figure 2. Reading free -h. Linux fills spare RAM with cached files, so "free" is usually low. Watch "available".
Swipe sideways to see all of it, or tap to open it full screen.

Process memory columns: RSS (resident set — physical RAM actually used now), VSZ (virtual size — everything mapped, often huge and meaningless). Sort by RSS: ps aux --sort=-rss | head.

The OOM killer

When the kernel can't reclaim enough memory, the OOM killer picks a victim using oom_score (mostly based on memory usage, adjusted by oom_score_adj, −1000 to 1000) and sends SIGKILL. Evidence:

dmesg -T | grep -i -E 'killed process|out of memory'
journalctl -k | grep -i oom

Inside containers, the limit comes from the cgroup: exceeding a container's memory limit triggers an OOM kill of that container even if the host has free RAM — Kubernetes reports OOMKilled, exit code 137.

Enlarge diagramMemory runs outon the host, or a container hitsits cgroup limitKernel reclaimsdrops cache, swaps if it canStill not enoughthe OOM killer is triggeredPick a victimhighest oom_score, usually thebiggest processSIGKILLno cleanup, no gracefulshutdownYou see itdmesg "Killed process", exit137, OOMKilled in Kubernetes123456
Figure 3. What happens when memory truly runs out, on a server and inside a container.
Swipe sideways to see all of it, or tap to open it full screen.

Fixes: find whether it's a leak (memory grows steadily) or a spike (a large request/batch); right-size limits; for JVMs, set heap relative to the container (-XX:MaxRAMPercentage=75) because the heap isn't the only memory a JVM uses.

cgroups and namespaces — how containers really work

Containers are ordinary Linux processes with two kernel features applied:

  • Namespaces control what a process can see: pid (own PID tree; the app is PID 1), net (own interfaces, routes, ports), mnt (own filesystem view), uts (hostname), ipc, user (UID mapping), cgroup.
  • cgroups (control groups) control what it can use: CPU, memory, I/O, PIDs. cgroups v2 (unified hierarchy) is the default on modern distros and on EKS AL2023 nodes.
Enlarge diagramNAMESPACES · WHAT IT SEESCGROUPS · WHAT IT USESpidown PID tree; the app is PID 1netown IP address, ports, routesmntown view of the filesystemutsown hostnameuserown user IDsnginxan ordinaryprocessCPUe.g. 0.5 coreMemorye.g. 512 MiB maxDisk I/Oread/write limitsPIDsmax number of processesContainer = process + namespaces + cgroups + an image as its root filesystem
Figure 4. A container is an ordinary Linux process with two kernel features applied: namespaces limit what it can see, cgroups limit what it can use.
Swipe sideways to see all of it, or tap to open it full screen.

CPU limits in cgroups are quotas per period (e.g. 50 ms of CPU per 100 ms). A bursty app can be throttled even if its average usage is low — visible as nr_throttled in cpu.stat, and as latency spikes. That's the root of the Kubernetes "remove CPU limits for latency-sensitive apps" debate.

Enlarge diagramAverage use here is only about 20%, yet requests stall. Look for a rising nr_throttled in cpu.stat; forlatency-sensitive apps, a higher limit or no CPU limit often helps.the applimit 0.5 CPU0–100 ms100–200 ms200–300 ms300–400 msruns 50 mspaused30 msa request needing 80 ms of CPU takes 130 msblue = runningred = throttled: work is waitinggrey = idle
Figure 5. A CPU limit of 0.5 core is a quota: 50 ms of CPU time in every 100 ms window. Once a burst uses it up, the app is paused until the next window starts.
Swipe sideways to see all of it, or tap to open it full screen.

You can enter a container's namespaces from the host with nsenter -t <pid> -n ip addr — handy for debugging networking on a node.

Kernel tuning with sysctl

sysctl net.core.somaxconn                   # read
sysctl -w vm.max_map_count=262144           # set now (Elasticsearch needs this)
echo 'vm.max_map_count=262144' > /etc/sysctl.d/99-es.conf   # persist
sysctl --system

Commonly tuned: net.core.somaxconn (listen backlog), net.ipv4.ip_local_port_range (ephemeral ports), net.ipv4.tcp_tw_reuse, fs.file-max, vm.swappiness, net.ipv4.ip_forward (needed for routing/Kubernetes).

File descriptor limits

Each process has soft and hard limits (ulimit -n; /proc/<pid>/limits). Users can raise the soft limit up to the hard limit; only root raises the hard limit. Persistent settings: /etc/security/limits.conf for login sessions, LimitNOFILE= for systemd services, --ulimit nofile= for Docker. "Too many open files" (EMFILE) means raise the limit and check for leaked sockets/files: ls /proc/<pid>/fd | wc -l growing forever is a leak.