Image building with Packer · Lesson 3 of 5 · 4 min read
Building good images
Hardening and clean-up
A good base image is patched, minimal and generic:
- Patch fully during the build (
dnf -y upgrade), and remove packages you don't need. - Harden to a benchmark such as CIS Level 1: SSH settings (no root login, no passwords), auditd, file permissions, kernel parameters. Pre-hardened CIS AMIs exist on the Marketplace, or apply the rules in your scripts.
- Install agents every server needs: the SSM agent, the CloudWatch agent or OpenTelemetry Collector, your security agent, time sync.
- Configure IMDSv2 required in the launch template, and set
imds_support = "v2.0"on the AMI so instances default to it. - Never bake secrets (passwords, keys, tokens) into an image: AMIs are copied, shared and kept for years. Fetch secrets at boot from Secrets Manager or Parameter Store using the instance role.
- Clean up before the snapshot so every instance starts fresh:
sudo cloud-init clean --logs --seeds # let cloud-init run again on first boot
sudo rm -f /etc/ssh/ssh_host_* # each instance generates its own host keys
sudo truncate -s 0 /etc/machine-id # unique ID per instance
sudo rm -rf /tmp/* /var/tmp/* ~/.bash_history
sudo dnf clean all
Leaving SSH host keys or the machine ID in the image means every instance shares them, which breaks host-key trust and confuses tools that identify machines by ID.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Testing and scanning
An image is software, so test it before anyone uses it:
- Inside the build: a final provisioner that runs checks (services enabled, ports closed, versions correct) with a tool such as Goss or InSpec, failing the build if anything is wrong.
- After the build: launch a test instance (or a test node group) from the new AMI and run smoke tests: it boots, joins the cluster or registers with the load balancer, the agents report in.
- Scan for vulnerabilities with Amazon Inspector (which scans running instances and can scan AMIs in the pipeline) or Trivy against the root filesystem, and fail on critical findings with a fix available.
- Record an SBOM and the build metadata (source AMI, git commit, package versions) as tags or in the manifest, so you can answer "which images contain this vulnerable library?".
Speed and reproducibility
- Builds take 5–20 minutes, mostly instance launch, updates and snapshotting. Keep provisioning scripts idempotent and fail-fast (
set -euo pipefail). - Use layered images: a slow-changing base image (OS, hardening, agents) built weekly, then thinner role images (Java runtime, runner toolchain) built on top of it more often.
- A single
packer buildcan build several sources in parallel (x86 and arm64, or several regions). - Pin plugin versions and package versions where it matters, so the same commit produces the same image.