Processes, signals and systemd
How processes are born
A process is created with fork() (copy the parent) followed by exec() (replace the copy's program with a new one). Every process has a PID, a parent PID (PPID), a UID, an environment, a working directory, open FDs, and resource limits. ps -ef --forest or pstree -p shows the family tree.
Threads are execution units inside one process that share its memory and FDs. Java and Go apps run many threads; ps -eLf or top -H shows them. A thread dump (jstack) is how you find a Java thread stuck on a lock.
Process states
In ps / top, the STAT column shows:
| State | Meaning |
|---|---|
R |
Running or runnable (waiting for CPU) |
S |
Interruptible sleep — waiting for an event (most processes) |
D |
Uninterruptible sleep — usually waiting on disk or NFS I/O; can't be killed until the I/O returns |
Z |
Zombie — exited, but parent hasn't collected the exit status |
T |
Stopped (Ctrl-Z or SIGSTOP) or being traced |
Many processes in D state is a strong signal of storage problems (a hung NFS mount, an EBS volume at its throughput limit).
Zombies take no memory, only a PID-table slot. You can't kill them — they're already dead. The parent must call wait(); if the parent is buggy, kill or restart the parent, and the zombies get re-parented to PID 1, which reaps them.
Orphans are live processes whose parent died; they're adopted by PID 1 (or the nearest "subreaper").
This is exactly why containers need a proper PID 1: if your app is PID 1 and never reaps children, zombies accumulate. docker run --init (tini) or init: true in Compose fixes it.
Signals
| Signal | Number | Default | Notes |
|---|---|---|---|
SIGHUP |
1 | Terminate | Often "reload config" for daemons (nginx, sshd) |
SIGINT |
2 | Terminate | Ctrl-C |
SIGKILL |
9 | Kill | Cannot be caught, blocked or ignored; no cleanup |
SIGTERM |
15 | Terminate | The polite default of kill; apps should catch it and shut down gracefully |
SIGSTOP / SIGCONT |
19 / 18 | Pause / resume | SIGSTOP can't be caught |
SIGCHLD |
17 | Ignore | Sent to a parent when a child exits |
Graceful shutdown everywhere — systemd, Docker, Kubernetes — follows the same pattern: send SIGTERM, wait a grace period, then SIGKILL. Kubernetes waits terminationGracePeriodSeconds (30 s default). An app that ignores SIGTERM always gets killed hard after the timeout, dropping in-flight requests.
Exit codes follow a convention: 0 success, 1–125 app errors, 126 not executable, 127 command not found, and 128 + signal number when killed by a signal — so 137 = 128 + 9 (SIGKILL, often the OOM killer) and 143 = 128 + 15 (SIGTERM).
kill -TERM 4321 # ask nicely
kill -9 4321 # last resort
pkill -f 'java.*orders'
kill -HUP $(cat /run/nginx.pid) # reload config
Priorities
The scheduler gives CPU time based on nice values from −20 (highest priority) to 19 (lowest). nice -n 10 ./backup.sh starts a job at lower priority; renice -n 5 -p PID changes a running one. Only root can raise priority (lower the number). For disk I/O priority, ionice -c3 (idle class) keeps a backup from hurting the app.
systemd
systemd is PID 1 and the service manager. Services are units defined in files:
# /etc/systemd/system/orders.service
[Unit]
Description=Orders API
After=network-online.target
Wants=network-online.target
[Service]
User=orders
WorkingDirectory=/opt/orders
EnvironmentFile=/etc/orders/env
ExecStart=/opt/orders/bin/orders --port 8080
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
MemoryMax=1G
[Install]
WantedBy=multi-user.target
systemctl daemon-reload # after editing unit files
systemctl enable --now orders # start now and at boot
systemctl status orders # state, PID, last log lines, exit code
systemctl restart orders
systemctl list-units --failed
systemctl cat orders # effective unit including drop-ins
systemctl edit orders # create an override drop-in safely
Key ideas:
- enable creates the boot-time symlink; start runs it now. They're independent.
Restart=on-failuregives you supervision without extra tools.LimitNOFILEsets the FD limit for the service —ulimitin your shell does not apply to systemd services.MemoryMax,CPUQuotause cgroups, the same mechanism containers use.- Other unit types:
.timer(scheduling),.socket,.mount,.target(groups of units).
journald stores logs from all units:
journalctl -u orders -f # follow one service
journalctl -u orders --since "1 hour ago" -p err
journalctl -b -1 # previous boot (needs persistent journal)
journalctl -k # kernel ring buffer, like dmesg
journalctl --disk-usage
journalctl --vacuum-size=500M # trim
Scheduling: cron and timers
cron uses five fields — minute hour day-of-month month day-of-week:
# m h dom mon dow command
*/15 * * * * /opt/scripts/sync.sh # every 15 minutes
0 2 * * 1-5 /opt/scripts/backup.sh # 02:00 Mon–Fri
30 3 1 * * /opt/scripts/monthly.sh # 03:30 on the 1st
User crontabs via crontab -e; system ones in /etc/crontab and /etc/cron.d/ (which include a user field).
Watch out: cron runs with a minimal environment — a short PATH, no profile, no
~/.bashrc. Scripts that work in your terminal fail in cron. Use absolute paths, set variables explicitly, and redirect output (>> /var/log/job.log 2>&1) so failures aren't silent.
systemd timers are the modern alternative: logging in journald, Persistent=true to run missed jobs after downtime, randomised delays, and dependencies.
To keep a process alive after logging out: nohup cmd & (ignores SIGHUP), tmux/screen, or — properly — a systemd unit.