Autoscale Celery Workers on Kubernetes With KEDA

Problem statement

Celery workers on Kubernetes are usually a Deployment with a fixed replica count. That count is wrong most of the time: too many workers sit idle overnight, and too few let queues back up during imports or end-of-month reports. The standard Horizontal Pod Autoscaler scales on CPU and memory, which reflects queue backlog poorly (a worker waiting on an API uses little CPU). KEDA (Kubernetes Event-Driven Autoscaling) scales Deployments on external metrics such as the length of a Redis list or RabbitMQ queue, which is exactly the signal Celery needs.

1. Install KEDA with Helm in its own namespace.
2. Run each Celery queue as its own Deployment (-Q default, -Q reports).
3. Create a ScaledObject per Deployment with the redis (list length) or rabbitmq (queue length) scaler.
4. Set listLength or value to the backlog one worker should handle, plus minReplicaCount and maxReplicaCount.
5. Use minReplicaCount: 0 only for batch queues where a short start delay is acceptable.
6. Give workers a long terminationGracePeriodSeconds and acks_late so scale-down does not lose tasks.

Step-by-step solution

1. Install KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

2. One Deployment per queue

apiVersion: apps/v1
kind: Deployment
metadata: { name: celery-reports }
spec:
  replicas: 1
  selector: { matchLabels: { app: celery-reports } }
  template:
    metadata: { labels: { app: celery-reports } }
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: worker
          image: ghcr.io/you/myapp:SHA
          command: ["celery", "-A", "config", "worker", "-Q", "reports", "--concurrency", "2", "-l", "info"]
          envFrom: [{ secretRef: { name: django-env } }]
          resources: { requests: { cpu: 250m, memory: 512Mi }, limits: { memory: 1Gi } }

Separate Deployments let fast queues and slow queues scale independently.

3. Scale on the Redis list length

With the Redis broker, each Celery queue is a Redis list named after the queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: celery-reports }
spec:
  scaleTargetRef: { name: celery-reports }
  minReplicaCount: 0
  maxReplicaCount: 10
  pollingInterval: 15
  cooldownPeriod: 300
  triggers:
    - type: redis
      metadata:
        address: redis.app.svc.cluster.local:6379
        listName: reports
        listLength: "20"
        databaseIndex: "0"
      authenticationRef: { name: redis-auth }

listLength: 20 means "one replica per 20 waiting tasks". KEDA creates and manages an HPA for the Deployment; cooldownPeriod delays scaling to zero after the queue empties.

For RabbitMQ, use type: rabbitmq with queueName: reports, mode: QueueLength, and value: "20", with the connection string in a TriggerAuthentication.

4. Choose the scaling target per queue

• Latency-sensitive queues (emails, webhooks): minReplicaCount: 1 or more so tasks start immediately.
• Batch queues (reports, imports): minReplicaCount: 0; the first task waits for a pod to start (often 10-30 seconds), which is acceptable.
• maxReplicaCount protects downstream systems: 50 workers can overwhelm the database or a rate-limited API.

Note that the queue length counts only waiting tasks; tasks already prefetched or running are not in the list. With long tasks, keep worker_prefetch_multiplier = 1 so the queue length reflects real backlog.

5. Scale down without losing work

When KEDA reduces replicas, Kubernetes sends SIGTERM. Celery performs a warm shutdown: it stops consuming and finishes running tasks. Make sure:

• terminationGracePeriodSeconds exceeds your longest task.
• CELERY_TASK_ACKS_LATE = True and CELERY_TASK_REJECT_ON_WORKER_LOST = True, so a task killed anyway is redelivered.
• Tasks are idempotent, since redelivery can run them twice.

Verification check: queue 100 test tasks onto reports and watch kubectl get hpa,pods -w: replicas rise towards 5, the queue drains, and after the cooldown replicas return to the minimum. Check that no task was lost by comparing completed counts with queued counts.

Explanation

KEDA runs an operator and a metrics adapter. For each ScaledObject, it polls the configured source (a Redis LLEN, a RabbitMQ queue depth from the management API or AMQP) and exposes the value as an external metric. It creates an HPA that targets the metric divided by the per-replica threshold, and handles the zero-to-one transition itself, which plain HPAs cannot do. Because the signal is backlog rather than resource usage, worker counts track demand directly.

Edge cases / notes

• Beat must not be scaled; keep it a single-replica Deployment outside KEDA.
• Priority queues in Redis use multiple lists (reports, reports\x06\x163, and so on); scale on the main list or sum them.
• Node autoscaling. Pods need nodes; pair KEDA with Cluster Autoscaler or Karpenter, or GKE Autopilot.
• Database connections grow with replicas; use PgBouncer or limit maxReplicaCount.

Next steps

• Split work into queues first with Route Celery Tasks to Separate Queues.
• Size each worker with Tune Celery Workers for Django.

FAQ

Can the HPA scale Celery workers on queue length?

Not by itself; it needs external metrics. KEDA provides them and manages the HPA.

Can Celery workers scale to zero?

Yes, with KEDA's minReplicaCount: 0. The first task then waits for a pod to start.

How do I avoid losing tasks when scaling down?

Use a grace period longer than your tasks, acks_late, and idempotent tasks.

Does KEDA work with RabbitMQ?

Yes, through its RabbitMQ scaler using queue length or message rate.