Why build images
Every EC2 instance, EKS node and autoscaled server starts from a machine image. Teams that build those images themselves, patched, hardened and tested, ship faster and more safely than teams that configure servers after they boot. Packer is the standard tool for this, and "how do you patch 300 servers?" is a common interview question whose best answer is "we don't; we build a new image and replace them". This topic covers why to bake images, how Packer templates work, how to build good images, and how to run an image pipeline from build to rollout and clean-up.
Golden images and immutable infrastructure
A golden image is a pre-built, tested machine image (an AMI on AWS) containing the operating system, patches, agents and settings your servers need. Instead of logging into servers to change them, you build a new image and replace the servers. That is immutable infrastructure:
- Every server built from image
v42is identical, so "works on one server, fails on another" disappears. - Rollback is starting instances from the previous image.
- Nobody needs SSH access to production to install things, which shrinks the attack surface.
- Configuration drift can't build up, because servers don't live long enough to drift.
Bake versus fry
There are two places to put setup work:
| Bake (build into the image) | Fry (do it at boot with user data) | |
|---|---|---|
| Boot time | Fast: everything is already there | Slow: installs on every launch |
| Reliability at boot | High: nothing to download | Fails if a package mirror or network is down during a scale-out |
| Consistency | Exact, tested | Can differ from one boot to the next as packages change |
| Flexibility | Needs a rebuild for every change | Change the script, done |
| Best for | OS, patches, agents, runtimes, hardening | Environment-specific settings, secrets fetched at runtime, the app version |
Most teams bake the slow, stable parts (OS updates, security agents, the container runtime, CIS hardening) and fry the small, environment-specific parts (which cluster to join, which config to load, secrets from Secrets Manager). Scale-out during an incident is exactly when you don't want instances spending five minutes running yum update.
Where images are used
- Auto Scaling groups behind load balancers: the launch template references the AMI.
- EKS worker nodes: managed node groups or Karpenter with a custom AMI based on the EKS-optimised Amazon Linux 2023 or Bottlerocket images, when you need extra agents, hardening or pre-pulled images.
- Build agents and runners: Jenkins agents and self-hosted GitHub runners with toolchains pre-installed start in seconds.
- Other platforms: the same Packer template structure builds Azure managed images, GCP images, VMware templates and Docker images.