Opscraft
Linux · Lesson 1 of 10 · 8 min read

How Linux is put together

Linux is the ground floor of everything you run: EC2 instances, EKS worker nodes, containers (which are just Linux processes with isolation), CI agents and bastion hosts. At senior level, interviewers aren't checking whether you know ls. They want to see that you understand what the kernel is doing — processes, memory, file descriptors, the page cache, cgroups — and that you can take a vague complaint like "the server is slow" and narrow it down methodically. This topic goes from how Linux boots to a full troubleshooting runbook.

Kernel space and user space

Linux is split into two worlds:

  • Kernel space — the kernel manages hardware, memory, scheduling, filesystems, networking and security. It runs with full privileges.
  • User space — every program you run (bash, nginx, java, containerd, kubelet). Programs can't touch hardware directly; they ask the kernel through system calls (open, read, write, fork, execve, mmap, socket, connect...).
Enlarge diagramUSER SPACE · THE PROGRAMS YOU RUNKERNEL SPACE · FULL CONTROL OF THE MACHINEbashyour shellnginxweb serverjava appyour servicekubeletKubernetes agentSystem callsopen · read · write · fork · execve · socket · connectSchedulerwho gets the CPUMemoryRAM, page cacheFilesystemsext4, XFS, /procNetworkingTCP/IP, firewallSecuritypermissionsCPURAMDiskNetwork card
Figure 1. Programs never touch the hardware themselves. They ask the kernel through system calls, and the kernel does the work.
Swipe sideways to see all of it, or tap to open it full screen.

This boundary matters in troubleshooting: strace shows the system calls a process makes, so when an app "hangs", you can see whether it's stuck in read() on a socket, futex() waiting for a lock, or open() failing with EACCES.

A distribution (Ubuntu, RHEL, Amazon Linux, Debian, Alpine) is the kernel plus a userland: package manager, init system, libc (glibc, or musl on Alpine), shell and tools. Alpine's musl is why some binaries built for glibc fail with "not found" inside Alpine containers.

Enlarge diagramUSERLAND · CHOSEN BY THE DISTROLinux kernelthe same core onevery distroPackage managerapt · dnf · apkInit systemsystemd (PID 1)C libraryglibc · musl on AlpineShell and toolsbash · coreutils · sshUbuntuaptDebianaptRHELdnfAmazon LinuxdnfAlpineapk · muslGotchaAlpine uses musl instead of glibc. A binary built on Ubuntu can fail inside an Alpine container with a confusing"not found" error.+together = a distribution
Figure 2. A distribution is the Linux kernel plus a userland. The kernel is the same idea everywhere; the userland is what makes Ubuntu feel different from Alpine.
Swipe sideways to see all of it, or tap to open it full screen.

The boot sequence

Interviewers like "what happens when a Linux server boots?" because it tests the whole stack:

  1. Firmware (BIOS or UEFI) runs hardware checks and finds a boot device.
  2. Bootloader (GRUB2) loads the kernel image (vmlinuz) and the initramfs — a small temporary root filesystem with drivers needed to mount the real root disk (e.g. NVMe, LVM, encryption).
  3. The kernel initialises CPU, memory and devices, mounts the initramfs, finds and mounts the real root filesystem.
  4. The kernel starts PID 1 — on modern distros that's systemd.
  5. systemd reads unit files and starts services in dependency order until it reaches its target (e.g. multi-user.target, the equivalent of the old runlevel 3).
  6. Login prompts / SSH become available.
Enlarge diagramFirmwareBIOS or UEFI checks thehardware and finds the bootdiskBootloader (GRUB2)loads the kernel (vmlinuz) andthe initramfsKernelstarts CPU, memory and drivers,mounts the real root diskPID 1: systemdthe first process; parent of everyother processServices startin dependency order; cloud-initruns here on EC2Readylogin prompt and SSH areavailableWhat is initramfs?initramfs is a tiny temporary filesystem holding just enough drivers (NVMe, LVM, encryption) for the kernel toreach the real root disk.123456
Figure 3. The six stages of a Linux boot. A mistake in /etc/fstab stops the machine at stage 5 and drops it into emergency mode.
Swipe sideways to see all of it, or tap to open it full screen.

On EC2, cloud-init runs during step 5: it reads instance metadata and user data, sets the hostname, installs SSH keys and runs your bootstrap scripts. On EKS nodes the bootstrap (nodeadm on AL2023, bootstrap.sh on older AMIs) joins the node to the cluster at this point — which is why "node failed to join" problems are debugged with /var/log/cloud-init-output.log.

Enlarge diagramEC2 startsboot stages 1 to 4cloud-initreads user dataand instancemetadatanodeadm /bootstrap.shwrites kubeletconfigkubelet joinsregisters with theEKS APINode never became Ready?/var/log/cloud-init-output.log shows the output of every bootstrap command, so it is the first place to look.
Figure 4. How an EC2 instance becomes an EKS worker node during boot. When a node never joins, this is the chain to check.
Swipe sideways to see all of it, or tap to open it full screen.

If a server won't boot after an /etc/fstab mistake, it usually drops to emergency mode at step 5 — on EC2 you'd detach the root volume, attach it to a rescue instance, fix fstab, and reattach. (Using nofail on non-root mounts prevents this.)

Everything is a file

Linux exposes most things through the filesystem:

  • Regular files and directories.
  • Device files in /dev (/dev/nvme0n1, /dev/null, /dev/urandom).
  • Sockets and pipes.
  • Pseudo-filesystems: /proc (process and kernel info — /proc/<pid>/status, /proc/meminfo, /proc/loadavg) and /sys (devices, cgroups under /sys/fs/cgroup).

When a program opens any of these, the kernel gives it a file descriptor (FD) — a small integer. 0, 1, 2 are stdin, stdout and stderr. Network connections are FDs too, which is why a busy proxy can hit "Too many open files".

Enlarge diagramNGINX (PID 812) · OPEN FDS0 · stdin1 · stdout2 · stderr3 · access.log4 · listening socket5 · client conn./dev/nulla device file that throws data awayjournaldcollects the service logs through a pipe/var/log/nginx/access.loga regular file on diskThe networkport 443 listener, plus one connection per clientEvery open file, pipe and network connection uses one FD. A busy proxy with thousands of connections can hitthe limit and fail with "Too many open files".
Figure 5. To a process, almost everything is an open file. Each one gets a number called a file descriptor (FD).
Swipe sideways to see all of it, or tap to open it full screen.

The filesystem hierarchy

Enlarge diagram/the root: everything lives under hereetc/configuration: passwd, hosts, fstab, ssh/var/data that changes while the system runslog/log fileslib/docker/container images and layershome/users' home folders (root's is /root)usr/bin/installed programsopt/self-contained third-party softwaretmp/temporary files, often wiped on rebootproc/ sys/live kernel information, not on any diskdev/device files: disks, /dev/nullboot/kernel, initramfs, GRUB configdata/a second disk mounted into the tree
Figure 6. The Linux directory tree. There are no drive letters: every disk is mounted somewhere inside one tree that starts at /.
Swipe sideways to see all of it, or tap to open it full screen.

Two more worth knowing: /usr/local/bin holds programs you installed yourself, and /root is the root user's home folder.

A filename is just a directory entry pointing to an inode. The inode holds the metadata — owner, permissions, size, timestamps, and pointers to data blocks — but not the name.

  • Hard link (ln target link): a second directory entry for the same inode. Both names are equal; the data survives until the last link is removed and no process has it open. Can't cross filesystems or link directories.
  • Symbolic link (ln -s target link): a small file containing a path. It can cross filesystems and point to directories, but breaks ("dangling") if the target moves.

Each filesystem has a fixed number of inodes. Millions of tiny files (session files, cache shards, unrotated small logs) can exhaust inodes while plenty of bytes remain — df -i reveals it.