Architecture: managers, workers and Raft
Node roles
- Managers maintain cluster state, schedule tasks, serve the Swarm API (
docker service ...,docker node ...) and replicate state among themselves using Raft. One manager is the leader; others forward writes to it. Managers also run workloads by default. - Workers only run tasks assigned by managers. They can't run cluster-management commands ("This node is not a swarm manager").
A node can be promoted or demoted at any time:
docker node promote node-4
docker node demote node-2
docker node ls # MANAGER STATUS: Leader / Reachable / Unavailable
Raft and quorum
Raft needs a majority (quorum) of managers to agree on any change. Quorum = ⌊N/2⌋ + 1.
| Managers (N) | Quorum | Managers you can lose |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 (worse than 1 — two points of failure, no tolerance) |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Rules of thumb:
- Use an odd number — 3 for most clusters, 5 for large or critical ones. An even number adds cost and failure points without adding tolerance.
- More than 7 managers slows consensus; Docker recommends no more.
- Spread managers across availability zones so losing one AZ can't take out quorum (3 managers in 3 AZs).
- Managers' Raft state lives in
/var/lib/docker/swarm— keep it on reliable, low-latency disk.
If quorum is lost: running tasks keep running, and the routing mesh keeps working, but you can't change anything — no deploys, no scaling, no rescheduling of failed tasks. Recover by restoring managers or, if they're gone for good, running on a surviving manager:
docker swarm init --force-new-cluster --advertise-addr <this-node-ip>
This creates a single-manager cluster from that node's copy of the state; then promote new managers back to 3 or 5.
Built-in security (PKI)
docker swarm init creates a root CA. Every node gets a certificate, and all manager–node communication uses mutual TLS. Certificates rotate automatically (default every 90 days; docker swarm update --cert-expiry). The CA can be rotated with docker swarm ca --rotate.
Join tokens contain the CA fingerprint plus a secret; there's one for workers and one for managers:
docker swarm init --advertise-addr 10.0.1.10
docker swarm join-token worker # print the join command
docker swarm join-token --rotate manager # invalidate a leaked token
docker swarm join --token SWMTKN-1-... 10.0.1.10:2377
docker swarm leave [--force]
docker node rm node-5
Autolock (docker swarm init --autolock or update --autolock=true) encrypts the Raft logs' keys at rest; after a manager restarts you must run docker swarm unlock with the unlock key. It protects secrets if someone steals a manager's disk.
Ports to open between nodes
| Port | Protocol | Purpose |
|---|---|---|
| 2377 | TCP | Cluster management (to managers) |
| 7946 | TCP + UDP | Node-to-node gossip (membership, network control plane) |
| 4789 | UDP | Overlay network data (VXLAN) |
| IP protocol 50 | ESP | Only if overlay encryption is enabled |
In AWS security groups, a classic mistake is opening 7946 TCP but forgetting UDP, or forgetting 4789/UDP — containers on different nodes then can't talk.