Keep Django Available During Kubernetes Node Maintenance With PodDisruptionBudgets

Problem statement

During a managed Kubernetes upgrade or a node pool resize, nodes are drained one after another. If both of your Django web replicas happen to sit on the same node, or the drain evicts them in quick succession, the site goes down even though nothing failed. PodDisruptionBudgets (PDBs) tell Kubernetes how many replicas must stay available during voluntary disruptions, and topology spread constraints keep replicas apart so drains rarely threaten them.

1. Run at least two web replicas (three for comfortable drains).
2. Add a PDB with maxUnavailable: 1 (or minAvailable: 1 for two replicas) selecting the web pods.
3. Spread replicas across nodes and zones with topologySpreadConstraints.
4. Set terminationGracePeriodSeconds and a short preStop sleep so evicted pods finish in-flight requests.
5. Do not set PDBs that can never be satisfied (for example minAvailable equal to replicas), or node drains hang.

Step-by-step solution

1. Add the budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: prod
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

maxUnavailable: 1 allows evicting one pod at a time; the drain waits until a replacement is Ready before evicting the next. It scales naturally with replica count, unlike a fixed minAvailable.

2. Spread replicas

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: web } }
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: web } }

Replicas on different nodes and zones mean a single node drain or zone problem affects at most one of them.

3. Make evictions graceful

An eviction sends SIGTERM like a normal pod termination. Give in-flight requests time to finish and let the endpoint removal propagate:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: web
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]

The preStop sleep keeps the pod serving while load balancers and kube-proxy stop routing to it; Gunicorn then receives SIGTERM and finishes current requests within its graceful_timeout. Distroless images lack sleep; newer Kubernetes versions support a native sleep action in preStop.

4. Budgets for workers

Celery workers handle retries, so a percentage budget (maxUnavailable: 25%) keeps throughput while drains proceed. Set terminationGracePeriodSeconds longer than typical task duration and use acks_late with idempotent tasks so interrupted work is redelivered.

5. Avoid blocking drains

A PDB that cannot be satisfied stops node drains indefinitely; managed upgrades then time out or force evictions. Common mistakes: minAvailable: 1 on a single-replica Deployment, minAvailable equal to the replica count, or pods that never become Ready so replacements never count. Check before maintenance:

kubectl get pdb -A

ALLOWED DISRUPTIONS of 0 on a healthy workload means drains will wait.

Explanation

Kubernetes distinguishes involuntary disruptions (node crashes) from voluntary ones (drains, upgrades, autoscaler scale-down). Voluntary evictions go through the Eviction API, which checks matching PDBs and refuses evictions that would violate them, so drains proceed at a safe pace. PDBs do not protect against crashes; spreading replicas across nodes and zones does. Together with graceful termination, they let cluster maintenance happen during business hours without users noticing.

Edge cases / notes

• Cluster autoscaler. It respects PDBs when removing nodes; overly strict budgets keep underused nodes alive and cost money.
• Deployments vs PDBs. Rolling updates are controlled by the Deployment strategy, not PDBs; configure both.
• Unhealthy pods. unhealthyPodEvictionPolicy: AlwaysAllow lets drains evict pods that are not Ready, avoiding stuck drains when pods are broken anyway.

Next steps

• Configure rollouts with Zero-Downtime Rolling Updates for Django on Kubernetes.
• Handle termination with Django Graceful Shutdown on SIGTERM.
• Review the rest with the Django Kubernetes Production Checklist.

FAQ

Do I need a PDB for my Django Deployment?

Yes, if it has several replicas and runs on a cluster with node upgrades or autoscaling.

minAvailable or maxUnavailable?

maxUnavailable: 1 is usually better: it scales with replica count and never blocks a healthy workload.

Why is my node drain stuck?

A PDB allows zero disruptions, often because of a single replica, minAvailable equal to replicas, or pods that are not Ready.

Do PDBs protect against node crashes?

No. They only apply to voluntary evictions. Spread replicas across nodes and zones for crash resilience.