Opscraft
How SSL/TLS works · Lesson 7 of 8 · 5 min read

Operating and troubleshooting TLS

The essential commands

# Full handshake view: protocol, cipher, chain, verification result
openssl s_client -connect shop.example.com:443 -servername shop.example.com -showcerts </dev/null

# Just the expiry dates
echo | openssl s_client -connect shop.example.com:443 -servername shop.example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

# SANs on the served certificate
echo | openssl s_client -connect shop.example.com:443 -servername shop.example.com 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

# Test protocol support
openssl s_client -connect shop.example.com:443 -tls1_1 </dev/null    # should FAIL on a modern config
openssl s_client -connect shop.example.com:443 -tls1_3 </dev/null

# Verbose curl
curl -vI https://shop.example.com
curl --cacert internal-ca.pem https://api.internal:8443/health

# Verify a chain file offline
openssl verify -CAfile root.pem -untrusted intermediate.pem server.pem

External scanners (SSL Labs, testssl.sh) grade configurations and catch chain issues.

Common errors

Error Cause Fix
certificate has expired Renewal failed or wasn't automated Renew/replace and reload now; then automate and alert
unable to get local issuer certificate / unable to verify the first certificate Server doesn't send intermediates, or the client lacks the root (private CA, old image) Serve fullchain; add the CA to the client trust store
certificate verify failed: Hostname mismatch / SSL: no alternative certificate subject name matches Hostname not in SANs; wrong cert served (missing SNI → default cert) Correct SANs; send SNI (-servername)
self signed certificate in certificate chain Corporate TLS-intercepting proxy, or a private CA unknown to the client Install the corporate/private CA in the trust store (image, JVM cacerts)
wrong version number Speaking TLS to a plain HTTP port (or vice versa) Correct port/protocol (https:// vs http://)
handshake failure / no shared cipher / protocol version Client and server share no protocol or cipher (old client vs TLS1.2+ policy; FIPS settings) Upgrade client, or adjust policy deliberately
Works in browser, fails in curl/Java Browser fetched or cached the missing intermediate Serve the full chain
Fails only in containers Minimal/distroless image without ca-certificates, or old CA bundle Install/update CA bundle in the image
Intermittent cert errors One of several backends/LB nodes has an old cert Check each backend individually
certificate is not yet valid Client/server clock skew Fix NTP/chrony

Preventing expiry outages

An expired certificate is a full outage that's 100% preventable:

  1. Automate issuance and renewal — ACM with DNS validation, cert-manager, Let's Encrypt clients.
  2. Inventory every certificate: ACM, imported certs, Kubernetes Secrets, certificates baked into appliances, client certificates used by apps, internal CAs.
  3. Monitor and alert at 30 days, escalate at 14 and 7: ACM DaysToExpiry CloudWatch metric and EventBridge expiry events, cert-manager Prometheus metrics, blackbox exporter probes (probe_ssl_earliest_cert_expiry) against every public endpoint, synthetic checks.
  4. Rotate without downtime — deploy the new certificate alongside (ALB listeners accept multiple certificates; ACM swaps in place), then reload processes gracefully (nginx -s reload, rolling restarts).
  5. Rehearse rotation of internal CAs and client certificates, which are often forgotten.
Enlarge diagramAutomateACM,cert-manager,ACME clientsInventoryevery cert,includingimported andclient certsAlertat 30 days;escalate at 14and 7Rotate safelynew certalongside, reloadgracefullyRehearseinternal CAs andclient certs too12345
Figure 1. Expired certificates cause complete outages and are entirely preventable.
Swipe sideways to see all of it, or tap to open it full screen.

At 2 AM with an expired cert: replace/renew immediately (ACM re-issue, cert-manager cmctl renew, manual upload), reload, verify with openssl s_client from outside, then write the postmortem: why automation didn't renew (e.g. the DNS validation CNAME was deleted, ACME DNS permissions broke, imported cert with no owner) and add alerts.

A leaked private key

Treat it as an incident: revoke the certificate with the CA, generate a new key and certificate, deploy everywhere the old one was used, check CT logs for unexpected issuance, rotate any secrets that might have been exposed with it, and investigate how it leaked (keys in Git, in images, in shared drives). Prefer services where keys never leave a managed boundary (ACM, KMS, HSMs).

Certificate pinning

Mobile apps sometimes pin a certificate or public key to resist interception. Rotation then breaks old app versions unless you pin to a stable intermediate/root or a backup key, plan rotations with app releases, and keep pin sets updated. Many teams now prefer CT monitoring over pinning.

Performance

TLS cost today is small, but at scale: use TLS 1.3, ECDSA certificates (cheaper signatures), session resumption, keep-alive and connection pooling (avoid handshakes per request), HTTP/2, and offload to managed load balancers and CDNs. If an ALB/proxy shows high TLS negotiation counts, look for clients that don't reuse connections.