Production-grade Bash
Most Senior DevOps interviews in Bangalore include a scripting round: "parse this log and find the top 10 IPs", "write a script that cleans up unused EBS volumes", "call an API with retries", or a live Bash/Python exercise. The Linux topic covers shell fundamentals; this topic goes further into production-grade Bash, Python for automation (boto3, the Kubernetes client, HTTP APIs), packaging and testing your tools, and worked solutions to the classic coding-round problems.
A robust script skeleton
#!/usr/bin/env bash
# rotate-logs.sh — compress and prune application logs
set -Eeuo pipefail
IFS=$'\n\t'
readonly SCRIPT_NAME=$(basename "$0")
LOG_DIR="/var/log/app"
RETENTION_DAYS=14
DRY_RUN=false
log() { printf '%s [%s] %s\n' "$(date -u +%FT%TZ)" "$1" "${*:2}" >&2; }
die() { log ERROR "$*"; exit 1; }
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [-d dir] [-r days] [-n]
-d log directory (default: $LOG_DIR)
-r retention in days (default: $RETENTION_DAYS)
-n dry run
EOF
}
while getopts ":d:r:nh" opt; do
case $opt in
d) LOG_DIR=$OPTARG ;;
r) RETENTION_DAYS=$OPTARG ;;
n) DRY_RUN=true ;;
h) usage; exit 0 ;;
:) die "option -$OPTARG requires an argument" ;;
\?) usage; exit 2 ;;
esac
done
[[ -d $LOG_DIR ]] || die "no such directory: $LOG_DIR"
[[ $RETENTION_DAYS =~ ^[0-9]+$ ]] || die "retention must be a number"
tmpdir=$(mktemp -d)
cleanup() { rm -rf "$tmpdir"; }
trap cleanup EXIT
trap 'log ERROR "failed at line $LINENO: $BASH_COMMAND"' ERR
run() { if $DRY_RUN; then log INFO "DRY-RUN: $*"; else "$@"; fi; }
find "$LOG_DIR" -type f -name '*.log' -mtime +0 -print0 |
while IFS= read -r -d '' f; do run gzip -9 "$f"; done
find "$LOG_DIR" -type f -name '*.gz' -mtime "+$RETENTION_DAYS" -print0 |
while IFS= read -r -d '' f; do run rm -f -- "$f"; done
log INFO "done"
What each piece buys you:
| Element | Why |
|---|---|
set -e |
Exit on unhandled command failure (know its gaps: not in if conditions, && chains, or command substitutions in some contexts) |
set -u |
Error on unset variables (catches typos like $LOG_DRI) |
set -o pipefail |
A pipeline fails if any stage fails, not just the last |
set -E |
ERR trap inherited by functions |
trap ... EXIT |
Cleanup always runs (temp files, locks) |
getopts |
Standard option parsing |
-print0 / read -d '' |
Filenames with spaces/newlines handled safely |
-- before filenames |
Prevents names starting with - being treated as options |
| Logging to stderr | stdout stays clean for data piped to other tools |
| Dry-run mode | Safe testing of destructive scripts |
Run ShellCheck on every script (in CI and editor); format with shfmt; test with bats for important scripts.
Idioms worth knowing
# defaults and required variables
: "${AWS_REGION:=ap-south-1}"
: "${CLUSTER:?CLUSTER must be set}"
# arrays (keep arguments intact)
args=(--region "$AWS_REGION" --output json)
aws ec2 describe-volumes "${args[@]}"
# associative arrays (bash 4+)
declare -A count
while read -r ip _; do (( count[$ip]++ )) || true; done < access.log
for ip in "${!count[@]}"; do echo "${count[$ip]} $ip"; done | sort -rn | head
# retry with exponential backoff
retry() {
local max=$1; shift; local n=1 delay=2
until "$@"; do
(( n >= max )) && { log ERROR "failed after $n attempts: $*"; return 1; }
log WARN "attempt $n failed, retrying in ${delay}s"; sleep "$delay"
(( n++ )); (( delay *= 2 ))
done
}
retry 5 curl -fsS https://api.example.com/health
# lock to prevent concurrent runs (e.g. from cron)
exec 9>/var/lock/rotate-logs.lock
flock -n 9 || die "another instance is running"
# parallelism with a limit
printf '%s\n' "${hosts[@]}" | xargs -P 8 -I{} ssh -o BatchMode=yes {} 'uptime'
# wait for a condition with timeout
timeout 300 bash -c 'until kubectl rollout status deploy/orders -n orders; do sleep 5; done'
# here-doc to a file with sudo
sudo tee /etc/sysctl.d/99-net.conf >/dev/null <<'EOF'
net.core.somaxconn = 4096
EOF
jq for JSON (AWS CLI, kubectl, APIs)
# instance IDs and Name tags of running instances
aws ec2 describe-instances --filters Name=instance-state-name,Values=running |
jq -r '.Reservations[].Instances[] |
[.InstanceId, (.Tags // [] | map(select(.Key=="Name"))[0].Value // "-")] | @tsv'
# pods not Running, with namespace
kubectl get pods -A -o json |
jq -r '.items[] | select(.status.phase != "Running") |
"\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"'
# images running in the cluster, counted
kubectl get pods -A -o json |
jq -r '.items[].spec.containers[].image' | sort | uniq -c | sort -rn
# update a JSON file
jq '.replicas = 3' config.json > config.json.tmp && mv config.json.tmp config.json
The AWS CLI also has --query (JMESPath): aws ec2 describe-volumes --query 'Volumes[?State==\available`].VolumeId' --output text`.
When to switch from Bash to Python
Bash is ideal for gluing commands, short operational scripts and container entrypoints. Switch to Python when you need: data structures beyond arrays, JSON/YAML manipulation, API clients with pagination and retries, error handling and testing, concurrency, or anything over ~100 lines. Interviewers like hearing that judgement.