Kubernetes: 30% nodes failure test

Kubernetes Failover Drills: Taking Down 30% of Nodes in Production

The team took down 30% of nodes in a production OKD-based Kubernetes cluster to uncover real weaknesses in system resilience. This drill tested how the control plane, infrastructure components, and application services responded to a major infrastructure failure. Issues with webhook controllers and replica distribution—previously invisible during normal operation—surfaced clearly.

Cluster Architecture Under Test

The cluster runs in the cloud across three availability zones, each in a separate data center with less than 1ms latency between them. The fully meshed topology mimics local network behavior.

Key components:

• User traffic Load Balancer: Backends are 3 infra-nodes running Ingress pods and Persistent Volume controller.
• API Load Balancer: Backends are 3 master nodes hosting the control plane.
• Worker nodes: Evenly distributed across zones, N total.

This setup allows simulating a full zone failure without losing cluster quorum—etcd can survive up to one master node loss out of three.

Preparing for the Disaster Drill

The plan included 8 steps:

• Verify baseline cluster and service health.
• Take snapshots of master and infra nodes for rapid rollback.
• Disable all nodes in one zone (or 1 master, 1 infra + 30% worker nodes).
• Monitor for 1–2 hours: log failures, collect metrics, support teams.
• Restore disabled nodes.
• Re-validate cluster state.
• Complete the drill.
• Document action items.

Snapshots are preferred over etcd backups for drills—faster recovery, lower risk. Worker nodes aren’t backed up since their state is in etcd. Infra nodes are backed up due to persistent volumes.

30% is a balanced target: realistic failure scenario (testing anti-affinity, PDBs, replication), but not catastrophic. Master node quorum survives up to 33% loss.

Success criteria:

• API remains accessible.
• Pods can be created and deleted.
• Persistent Volumes remain available.
• Services continue functioning.
• Replica recovery within 10–15 minutes.
• Alerts resolved (except those from offline nodes).

Drill #1: Development Cluster

Disabled 1/3: 1 master, 1 infra, ~80 worker cores. The load balancer removed unhealthy backends within a minute. API stayed responsive, but nodes entered Unknown state for 300 seconds (default timeout), then NotReady/Unreachable.

Pods didn’t restart for 10 minutes. When creating new ones:

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://infra-kyverno-svc.infra-kyverno.svc:443/mutate/fail?timeout=30s": dial tcp 10.128.4.174:9443: connect: connection refused

Kyverno (admission/mutation webhook) had one Pod per type, with 2/3 located in the disabled zone—now Terminating. Without active webhooks, changes never reach etcd. System namespaces are excluded from Kyverno to prevent deadlocks.

Increased Kyverno replicas. Events showed scheduling blocked due to resource shortages (implied by context, though not explicitly stated).

Problems and Solutions

Kyverno: Moved to 3+ replicas per webhook type with zone-aware topology spread.

Scheduler block: Checked resource quotas and limits in affected namespaces.

In production (Drills #2 & #3), similar scenarios at scale revealed PDB violations in services lacking proper anti-affinity, plus delays in PV migration.

Application teams received a checklist:

• Minimum 3 replicas.
• PodDisruptionBudget with minAvailable.
• TopologySpreadConstraints or nodeAffinity.
• Resource requests/limits.
• Liveness/readiness probes with sensible timeouts.

What Matters Most

• Taking down 30% of nodes exposes hidden issues with webhooks and scheduling that don’t appear under normal conditions.
• Kyverno requires more than one replica, evenly distributed across zones.
• The 300-second NotReady/Unreachable timeout is standard but demands buffer resources.
• Snapshots of infra/master nodes are essential for safe drills.
• Full recovery: 10–15 minutes for replicas + alert resolution.

Recommendations for Teams

For Service Developers:

• Set PDBs: minAvailable: 50%.
• Use topologySpreadConstraints for zone distribution.
• Set CPU/RAM requests/limits to 80% of observed peaks.
• Probes with 5–10s timeout and 3 retries.

For Kubernetes Admins:

• Ensure Kyverno and other webhooks have >3 replicas with high availability.
• Monitor etcd leader elections.
• Plan capacity with +30% reserve.
• Automate snapshots before every drill.

These drills are a must for any production cluster with over 100 nodes.

— Editorial Team