How Linux is put together
Linux is the ground floor of everything you run: EC2 instances, EKS worker nodes, containers (which are just Linux processes with isolation), CI agents and bastion hosts. At senior level, interviewers aren't checking whether you know ls. They want to see that you understand what the kernel is doing — processes, memory, file descriptors, the page cache, cgroups — and that you can take a vague complaint like "the server is slow" and narrow it down methodically. This topic goes from how Linux boots to a full troubleshooting runbook.
Kernel space and user space
Linux is split into two worlds:
- Kernel space — the kernel manages hardware, memory, scheduling, filesystems, networking and security. It runs with full privileges.
- User space — every program you run (bash, nginx, java, containerd, kubelet). Programs can't touch hardware directly; they ask the kernel through system calls (
open,read,write,fork,execve,mmap,socket,connect...).
This boundary matters in troubleshooting: strace shows the system calls a process makes, so when an app "hangs", you can see whether it's stuck in read() on a socket, futex() waiting for a lock, or open() failing with EACCES.
A distribution (Ubuntu, RHEL, Amazon Linux, Debian, Alpine) is the kernel plus a userland: package manager, init system, libc (glibc, or musl on Alpine), shell and tools. Alpine's musl is why some binaries built for glibc fail with "not found" inside Alpine containers.
The boot sequence
Interviewers like "what happens when a Linux server boots?" because it tests the whole stack:
- Firmware (BIOS or UEFI) runs hardware checks and finds a boot device.
- Bootloader (GRUB2) loads the kernel image (
vmlinuz) and the initramfs — a small temporary root filesystem with drivers needed to mount the real root disk (e.g. NVMe, LVM, encryption). - The kernel initialises CPU, memory and devices, mounts the initramfs, finds and mounts the real root filesystem.
- The kernel starts PID 1 — on modern distros that's systemd.
- systemd reads unit files and starts services in dependency order until it reaches its target (e.g.
multi-user.target, the equivalent of the old runlevel 3). - Login prompts / SSH become available.
On EC2, cloud-init runs during step 5: it reads instance metadata and user data, sets the hostname, installs SSH keys and runs your bootstrap scripts. On EKS nodes the bootstrap (nodeadm on AL2023, bootstrap.sh on older AMIs) joins the node to the cluster at this point — which is why "node failed to join" problems are debugged with /var/log/cloud-init-output.log.
If a server won't boot after an /etc/fstab mistake, it usually drops to emergency mode at step 5 — on EC2 you'd detach the root volume, attach it to a rescue instance, fix fstab, and reattach. (Using nofail on non-root mounts prevents this.)
Everything is a file
Linux exposes most things through the filesystem:
- Regular files and directories.
- Device files in
/dev(/dev/nvme0n1,/dev/null,/dev/urandom). - Sockets and pipes.
- Pseudo-filesystems:
/proc(process and kernel info —/proc/<pid>/status,/proc/meminfo,/proc/loadavg) and/sys(devices, cgroups under/sys/fs/cgroup).
When a program opens any of these, the kernel gives it a file descriptor (FD) — a small integer. 0, 1, 2 are stdin, stdout and stderr. Network connections are FDs too, which is why a busy proxy can hit "Too many open files".
The filesystem hierarchy
Two more worth knowing: /usr/local/bin holds programs you installed yourself, and /root is the root user's home folder.
Inodes and links
A filename is just a directory entry pointing to an inode. The inode holds the metadata — owner, permissions, size, timestamps, and pointers to data blocks — but not the name.
- Hard link (
ln target link): a second directory entry for the same inode. Both names are equal; the data survives until the last link is removed and no process has it open. Can't cross filesystems or link directories. - Symbolic link (
ln -s target link): a small file containing a path. It can cross filesystems and point to directories, but breaks ("dangling") if the target moves.
Each filesystem has a fixed number of inodes. Millions of tiny files (session files, cache shards, unrotated small logs) can exhaust inodes while plenty of bytes remain — df -i reveals it.