Opscraft
Jenkins · Lesson 6 of 11 · 5 min read

Jenkins as code, high availability and upgrades

Configuration as Code (JCasC)

Enlarge diagramGIT REPOSITORYjenkins.yamlJCasC: security, cloudsplugins.txtpinned plugin versionsJob DSL seedcreates the jobscontroller imageor Helm valuesJenkinsconfigured on start
Figure 1. Jenkins as code: everything that defines the controller lives in Git, so a lost controller can be rebuilt in minutes.
Swipe sideways to see all of it, or tap to open it full screen.

Everything you'd click in "Manage Jenkins" expressed as YAML:

jenkins:
  systemMessage: "Managed by JCasC — do not edit in the UI"
  numExecutors: 0
  securityRealm:
    oic:                                          # exact keys vary by OIDC plugin version
      clientId: jenkins
      clientSecret: "${OIDC_CLIENT_SECRET}"     # injected from a secret
      wellKnownOpenIDConfigurationUrl: https://sso.example.com/.well-known/openid-configuration
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: admin
            permissions: ["Overall/Administer"]
            entries: [{ group: "platform-admins" }]
          - name: read
            permissions: ["Overall/Read"]
            entries: [{ group: "engineering" }]
  clouds:
    - kubernetes:
        name: eks
        namespace: jenkins-agents
        jenkinsUrl: http://jenkins.jenkins.svc.cluster.local:8080
        webSocket: true
        containerCapStr: "50"
unclassified:
  location:
    url: https://jenkins.example.com/
  globalLibraries:
    libraries:
      - name: platform-lib
        defaultVersion: v2.3.0
        retriever:
          modernSCM:
            scm:
              git: { remote: "https://github.com/acme/jenkins-platform-lib.git" }

With JCasC + pinned plugins in an image + pipelines in Git + jobs generated by Job DSL or Organization Folders, the controller becomes reproducible: you can rebuild it from Git in minutes, and every change is reviewed.

Job DSL

Generates jobs from code instead of clicking:

multibranchPipelineJob('orders-api') {
  branchSources {
    github {
      id('orders-api')
      repoOwner('acme')
      repository('orders-api')
      scanCredentialsId('github-app')
    }
  }
  orphanedItemStrategy { discardOldItems { numToKeep(20) } }
}

A seed job runs the DSL scripts from a repo.

Running Jenkins on Kubernetes

The official Helm chart deploys the controller as a StatefulSet with a PVC for JENKINS_HOME (EBS gp3), JCasC configured via ConfigMaps, an Ingress/ALB in front, and the Kubernetes cloud pre-configured for agents. Give the controller adequate requests/limits and JVM heap (-Xmx around 50–70% of the container memory, or -XX:MaxRAMPercentage).

Enlarge diagramEKS: JENKINS NAMESPACEIngress / ALBTLScontrollerStatefulSet, 1 replicaPVC: JENKINS_HOMEEBS gp3agent pod 1created per buildagent pod 2created per build
Figure 2. Jenkins on Kubernetes with the official Helm chart.
Swipe sideways to see all of it, or tap to open it full screen.

High availability — the honest answer

Open-source Jenkins controllers are single-instance: JENKINS_HOME can't be shared by two active controllers. HA strategies:

  • Fast recovery rather than active-active: controller as a Kubernetes StatefulSet (rescheduled automatically if the node dies, EBS volume reattached in the same AZ), or an EC2 Auto Scaling group of 1 with EFS/EBS — plus JCasC so a rebuild is quick.
  • Backups of JENKINS_HOME (EBS snapshots via AWS Backup/DLM, or ThinBackup), tested restores.
  • Split controllers by team or domain to reduce blast radius instead of one giant controller.
  • CloudBees CI (commercial) offers HA/horizontal scaling features.
  • Pipelines survive controller restarts thanks to durability (CPS checkpoints) — running builds resume where possible.
Enlarge diagramNode failscontroller pod lostRescheduledStatefulSet startsa new podVolumereattachedsame EBS, sameAZBack in minutespipelines resumewhere possible
Figure 3. The honest HA story for open-source Jenkins: one controller, recovered fast, not two running at once.
Swipe sideways to see all of it, or tap to open it full screen.

Upgrading safely

  1. Read the LTS changelog and upgrade guide; check plugin compatibility.
  2. Back up JENKINS_HOME (snapshot).
  3. Build a new controller image with the new LTS core and updated pinned plugins.
  4. Test on a staging controller with a copy of JENKINS_HOME or representative jobs.
  5. Roll out in a maintenance window ("Prepare for shutdown" lets running builds finish), keep the old image for rollback.
Enlarge diagramReadLTS changelog,plugin notesBack upsnapshotJENKINS_HOMEBuild imagenew core +pinned pluginsTeston a stagingcontrollerRoll outkeep the oldimage forrollback12345
Figure 4. Upgrading safely, with a rollback ready.
Swipe sideways to see all of it, or tap to open it full screen.