Opscraft
Image building with Packer · Lesson 1 of 5 · 3 min read

Why build images

Every EC2 instance, EKS node and autoscaled server starts from a machine image. Teams that build those images themselves, patched, hardened and tested, ship faster and more safely than teams that configure servers after they boot. Packer is the standard tool for this, and "how do you patch 300 servers?" is a common interview question whose best answer is "we don't; we build a new image and replace them". This topic covers why to bake images, how Packer templates work, how to build good images, and how to run an image pipeline from build to rollout and clean-up.

Golden images and immutable infrastructure

A golden image is a pre-built, tested machine image (an AMI on AWS) containing the operating system, patches, agents and settings your servers need. Instead of logging into servers to change them, you build a new image and replace the servers. That is immutable infrastructure:

  • Every server built from image v42 is identical, so "works on one server, fails on another" disappears.
  • Rollback is starting instances from the previous image.
  • Nobody needs SSH access to production to install things, which shrinks the attack surface.
  • Configuration drift can't build up, because servers don't live long enough to drift.
Enlarge diagramChange in place (mutable)Replace (immutable)SSH in and run updatesservers drift apart over timerollback means undoing changes by handbuild image v43, test itlaunch new servers, remove old onesrollback: launch from v42 again
Figure 1. Immutable infrastructure: build a new image and replace servers, instead of changing servers in place.
Swipe sideways to see all of it, or tap to open it full screen.

Bake versus fry

There are two places to put setup work:

Bake (build into the image) Fry (do it at boot with user data)
Boot time Fast: everything is already there Slow: installs on every launch
Reliability at boot High: nothing to download Fails if a package mirror or network is down during a scale-out
Consistency Exact, tested Can differ from one boot to the next as packages change
Flexibility Needs a rebuild for every change Change the script, done
Best for OS, patches, agents, runtimes, hardening Environment-specific settings, secrets fetched at runtime, the app version

Most teams bake the slow, stable parts (OS updates, security agents, the container runtime, CIS hardening) and fry the small, environment-specific parts (which cluster to join, which config to load, secrets from Secrets Manager). Scale-out during an incident is exactly when you don't want instances spending five minutes running yum update.

Enlarge diagramat boot (user data)join cluster prod-1, fetch secrets, load configrole imageJava 21 runtime, app dependenciesbase imagepatched OS, CIS hardening, SSM and CloudWatch agentsAWS base AMIAmazon Linux 2023
Figure 2. Bake the slow, stable layers into the image; fry the small, environment-specific parts at boot.
Swipe sideways to see all of it, or tap to open it full screen.

Where images are used

  • Auto Scaling groups behind load balancers: the launch template references the AMI.
  • EKS worker nodes: managed node groups or Karpenter with a custom AMI based on the EKS-optimised Amazon Linux 2023 or Bottlerocket images, when you need extra agents, hardening or pre-pulled images.
  • Build agents and runners: Jenkins agents and self-hosted GitHub runners with toolchains pre-installed start in seconds.
  • Other platforms: the same Packer template structure builds Azure managed images, GCP images, VMware templates and Docker images.