Opscraft
Linux · Lesson 4 of 10 · 6 min read

Shell scripting and text processing

Streams, redirection and pipes

Enlarge diagramInputkeyboard, file orpipea commandany programOutputscreen, file or pipeErrorsscreen or a filels /var/loglists filesgrep nginxkeeps matching lineswc -lcounts them0 stdin1 stdout2 stderrA pipe joins stdout of one command to stdin of the next:||
Figure 1. Every process starts with three streams. Redirections (> 2> <) and pipes (|) only change where they point.
Swipe sideways to see all of it, or tap to open it full screen.
cmd > out.log            # stdout to file (truncate)
cmd >> out.log           # append
cmd 2> err.log           # stderr only
cmd > all.log 2>&1       # both: redirect stdout first, THEN point stderr at it
cmd &> all.log           # bash shorthand for the same
cmd 2>/dev/null          # discard errors
cmd1 | cmd2              # stdout of cmd1 → stdin of cmd2
cmd | tee out.log        # see it AND save it

Order matters: cmd 2>&1 > file sends stderr to the terminal, because stderr was duplicated before stdout moved.

Enlarge diagramcmd > all.log 2>&1cmd 2>&1 > all.log1. stdout (1) → all.log2. stderr (2) → wherever 1 points now: all.logBoth streams end up in the file1. stderr (2) → wherever 1 points now: the screen2. stdout (1) → all.logErrors still appear on screen
Figure 2. Redirections are applied left to right, so the order changes the result.
Swipe sideways to see all of it, or tap to open it full screen.

Exit codes and safe scripts

Every command returns an exit code; $? holds the last one. && runs the next command only on success, || only on failure.

Start production scripts with strict mode:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
  • -e — exit on any failing command.
  • -u — error on unset variables (catches typos like rm -rf "$BUILDDIR/" with an empty variable).
  • -o pipefail — a pipeline fails if any command fails, not just the last. Without it, curl bad-url | jq . "succeeds" because jq succeeded.

Clean up with a trap:

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

Variables and quoting

name="api"
echo "$name"              # always quote expansions
echo "${name}-svc"        # braces when adjacent text follows
echo "${ENV:-dev}"        # default if unset/empty
echo "${#name}"           # length
files=(a.yaml b.yaml)     # arrays
for f in "${files[@]}"; do kubectl apply -f "$f"; done
out=$(git rev-parse --short HEAD)   # command substitution
export AWS_REGION=ap-south-1        # visible to child processes

Unquoted variables undergo word splitting and globbing — the source of many bugs with spaces in filenames.

Loops, conditions and functions

if [[ -f /etc/os-release ]]; then source /etc/os-release; fi
[[ "$ENV" == "prod" ]] && echo "careful"

for host in web1 web2 web3; do ssh "$host" uptime; done

while read -r line; do echo "got $line"; done < hosts.txt

retry() {               # retry a command with backoff
  local n=0 max=5
  until "$@"; do
    n=$((n+1)); (( n >= max )) && return 1
    sleep $((2**n))
  done
}
retry curl -fsS https://internal/health

Test operators: -f file exists, -d directory, -z empty string, -n non-empty, -eq/-lt/-gt numeric.

The text toolkit

grep -rn "ERROR" /var/log/app/          # recursive, with line numbers
grep -v DEBUG app.log                   # invert match
grep -E 'timeout|refused' app.log       # extended regex
grep -c 500 access.log                  # count matching lines

find /var/log -name '*.log' -mtime +7 -size +100M
find . -type f -name '*.tmp' -delete
find . -name '*.yaml' -print0 | xargs -0 -n1 kubeval   # safe with spaces

cut -d: -f1 /etc/passwd                 # field 1, colon-delimited
awk '{print $1}' access.log             # first whitespace field
awk -F, '$3 > 100 {sum += $3} END {print sum}' data.csv
sed -n '100,120p' big.log               # print lines 100–120
sed -i 's/old-host/new-host/g' app.conf # in-place replace
sort | uniq -c | sort -rn | head        # frequency count
jq -r '.items[].metadata.name' pods.json

Classic interview one-liners:

# Top 10 client IPs in an nginx access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Count of each HTTP status code
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# 5xx errors per minute
awk '$9 ~ /^5/ {print substr($4,2,17)}' access.log | uniq -c

# Largest 10 directories under /var
du -xh /var 2>/dev/null | sort -rh | head -10
Enlarge diagramaccess.logone line per request; the client IP is the first fieldawk '{print $1}'10.0.0.510.0.0.910.0.0.510.0.0.710.0.0.5sort10.0.0.510.0.0.510.0.0.510.0.0.710.0.0.9uniq -c3 10.0.0.51 10.0.0.71 10.0.0.9sort -rn | head3 10.0.0.51 10.0.0.91 10.0.0.7uniq only merges duplicate lines that sit next to each other, which is why sort always comes first.
Figure 3. The classic "top client IPs" one-liner, showing what each stage passes to the next.
Swipe sideways to see all of it, or tap to open it full screen.

uniq only collapses adjacent duplicates — always sort first. find matches names and attributes; grep matches contents.

Watch out: sed 's/a/b/' file > file produces an empty file — the shell truncates the output file before sed reads it. Use sed -i (or write to a temp file and mv).