Celery & background jobs for Django in production
Running work outside the request: Celery workers and Beat, Django tasks, queues, retries, and monitoring.
Celery moves slow work out of the request: emails, reports, image processing, and anything that talks to an external service. Running it reliably takes more than a worker process.
The guides cover the broker, workers and Beat under systemd, queues and tuning, idempotent tasks and retries, monitoring, and the classic failures: tasks stuck pending, run twice, or running old code.
Set it up
Guides that take this from nothing to running in production.
- Configure Redis and Celery for Django Production
Running background jobs in Django production is not just a matter of installing Celery and pointing it at Redis.
- Run Celery Workers with systemd for Django
Running Celery workers manually in production is fragile. If you start a worker from a shell and disconnect, the process may stop.
- Run Celery Beat for Scheduled Tasks in Django Production
Run Celery Beat reliably for Django: schedules, file or database schedulers, a single instance under systemd, time zones, and monitoring.
- Run Django's Built-In Tasks Framework in Production
Deploy background work with Django 6's django.tasks: choosing a production backend, enqueueing after commit, running database workers under systemd, scaling, retries, and monitoring.
- Run Celery With RabbitMQ for Django in Production
Use RabbitMQ as the Celery broker for Django in production: install and secure RabbitMQ, create a vhost and user, configure the broker URL and quorum queues, set prefetch and acks, and monitor queues with the management plugin.
- Autoscale Celery Workers on Kubernetes With KEDA
Scale Celery workers on Kubernetes by queue length with KEDA: install KEDA, define a ScaledObject with the Redis or RabbitMQ scaler, scale to zero for batch queues, and shut workers down gracefully so tasks are not lost.
- Run django-rq Workers in Production
Run django-rq (Redis Queue) workers for Django in production: configure queues in settings, supervise rq workers with systemd, schedule jobs with rq-scheduler or cron, handle failed jobs, and monitor queues in the admin.
- Celery Alternatives for Django Background Jobs
Compare Celery alternatives for Django background jobs: Django’s built-in tasks framework, django-rq, Dramatiq, Huey, django-q2, and database-backed queues, by setup effort, brokers, scheduling, retries, monitoring, and scale.
- Run Celery Workers on Heroku for a Django App
Run Celery on Heroku with a Django app: add a worker process type, use Heroku Key-Value Store or another Redis as the broker with TLS, schedule periodic tasks without running beat on several dynos, size concurrency to dyno memory, and handle SIGTERM restarts.
- Run Celery Workers on Render, Railway, and Fly.io
Run Celery workers and beat for a Django app on Render background workers, Railway services, and Fly.io process groups: share the image and environment, connect a Redis broker, keep beat to one instance, and handle shutdowns and scaling on each platform.
- Use Amazon SQS as the Celery Broker for Django
Use Amazon SQS as a Celery broker for Django: install the SQS transport, authenticate with IAM roles, create queues, set visibility timeout and long polling, handle FIFO limitations, and avoid surprises with ETA tasks, results, and costs.
- Run Dramatiq Workers for Django in Production
Run Dramatiq background workers for Django in production with django-dramatiq: choose Redis or RabbitMQ, configure middleware for retries, time limits, and results, run workers under systemd or containers, schedule periodic jobs, and shut down safely.
- Run Huey as a Lightweight Django Task Queue
Run Huey for Django background tasks and periodic jobs in production: configure the Redis or SQLite storage, start the consumer with workers and the built-in scheduler, avoid duplicate periodic tasks, handle database connections, and supervise it with systemd.
- Run Django-Q2 Workers and Schedules in Production
Run Django-Q2 in production: configure the ORM or Redis broker in Q_CLUSTER, start qcluster workers under systemd or containers, manage schedules in the admin, set timeouts and retries correctly, and avoid running duplicate clusters.
- Run Procrastinate as a PostgreSQL Task Queue for Django
Use Procrastinate, a PostgreSQL-based task queue, with Django in production: install the Django contrib app, define and defer tasks inside transactions, run workers with concurrency, schedule periodic tasks, use locks and queueing locks, and monitor jobs in the admin.
- Run Django and Celery with Supervisor
Supervise Gunicorn and Celery for Django with Supervisor: program configs, environment variables, stop signals and timeouts, log rotation, group restarts, and when to prefer systemd.
- Send Django Email in the Background
Send Django email outside the request cycle: queue emails with Celery after commit, retry provider failures with backoff, use django-celery-email or a custom backend, keep templates rendered in the task, and avoid duplicate sends.
Run it well
Tuning, reliability, and operations once it works.
- Tune Celery Worker Concurrency and Prefetch for Django
Tune Celery workers for Django workloads: pool types, concurrency, prefetch multiplier, acks_late, time and memory limits, and dedicated queues.
- Make Celery Tasks Idempotent and Safe to Retry
Design Django Celery tasks that can run twice without harm: enqueue on commit, idempotency keys and unique constraints, row locks, external API idempotency, and bounded retries with backoff.
- Route Celery Tasks to Separate Queues
Stop slow Celery tasks from delaying urgent ones in Django: define queues, route tasks with CELERY_TASK_ROUTES, run dedicated workers per queue with their own concurrency and limits, and monitor queue depth.
- Monitor Celery with Flower
Run Flower to monitor Celery in a Django deployment: enable task events, run Flower under systemd bound to localhost, protect it with authentication behind Nginx, persist state, and export Prometheus metrics.
- Rate Limit Celery Tasks in Django
Control how fast Celery tasks run in a Django app: per-worker rate_limit on tasks, why it is not global, global limits with a Redis token bucket or dedicated queues and concurrency, handling third-party API 429 responses, and protecting the database from task floods.
- Retry Celery Tasks With Exponential Backoff in Django
Configure Celery retries for Django tasks: autoretry_for with retry_backoff and jitter, manual self.retry with countdown, max_retries and retry_backoff_max, which exceptions to retry, idempotency, and dead-letter handling for tasks that keep failing.
- Monitor Celery With Prometheus Metrics and Grafana
Expose Celery metrics to Prometheus for a Django app: run celery-exporter for task counts, failures, runtimes, and queue lengths, enable task events, build Grafana panels, and alert on growing queues, rising failures, and missing workers.
- Use the Transactional Outbox Pattern With Django and Celery
Stop losing or duplicating side effects when Django saves data and enqueues Celery tasks: compare transaction.on_commit with a transactional outbox table, write outbox rows atomically, relay them with a worker using SKIP LOCKED, and make consumers idempotent.
- Configure Redis Persistence for Django and Celery
Decide which Redis data must survive restarts in a Django stack: AOF versus RDB for Celery queues and sessions, no persistence for caches, fork memory, Docker volumes, and verifying recovery.
- Process Incoming Webhooks Reliably in Django
Build reliable webhook endpoints in Django: verify signatures, store the raw event first, return 2xx quickly, process asynchronously with Celery, deduplicate by event ID, handle out-of-order delivery, and replay failed events safely.
Check it
Review lists to run before launch or after a change.
- Celery Production Checklist for Django
Review Celery before production: broker durability, queues and routing, retries and idempotency, time limits, single Beat, monitoring, deploys, and security.
Fix it
Errors and symptoms, each traced to a cause and a fix.
- Celery Not Running in Django Production: Debugging Guide
A common production failure is that the Django web app is healthy, but Celery tasks stop running.
- Fix Celery Tasks Stuck in PENDING
Fix Celery tasks that stay PENDING in Django: missing result backends, workers not consuming queues, routing mismatches, unregistered tasks, and commit timing.
- Fix Celery Beat Running Tasks Twice
Fix Celery periodic tasks running twice in Django: duplicate Beat processes, -B on workers, overlapping deploys, duplicate schedules, and redelivery.
- Fix Celery WorkerLostError in Django
Fix Celery WorkerLostError 'Worker exited prematurely: signal 9 (SIGKILL)' in Django: find OOM kills and short stop timeouts, cap memory per child, chunk big tasks, and redeliver safely.
- Fix Celery "Received unregistered task" in Django
Fix the Celery KeyError and Received unregistered task of type message in Django: make workers import the same task modules as the web app, align task names, and stop stale queued messages from old releases.
- Fix Celery Connecting to amqp://guest@localhost Instead of Redis
Fix Celery workers that log Cannot connect to amqp://guest@127.0.0.1:5672 when you configured Redis: load Django settings in celery.py, use the CELERY_ namespace correctly, and pass the broker URL to every process.
- Fix Celery SoftTimeLimitExceeded and Hard Time Limits
Fix Celery SoftTimeLimitExceeded and TimeLimitExceeded errors in Django: find why tasks run long, set soft and hard limits that fit the work, clean up on the soft limit, and split slow jobs into chunks.
- Fix a Celery Worker Whose Memory Keeps Growing
Fix Celery worker memory that grows until the OOM killer strikes: recycle children with max-tasks-per-child and max-memory-per-child, find the task that leaks, and avoid DEBUG query logging and large result sets.
- Fix Celery Tasks Executing Twice (Redis Visibility Timeout)
Fix Celery tasks that run two or three times with a Redis broker: understand visibility_timeout, ETA and countdown redelivery, acks_late, and make long or scheduled tasks safe from duplicate execution.
- Fix AppRegistryNotReady in Celery and Django Scripts
Fix Apps not loaded yet and settings not configured errors in Celery workers, cron scripts, and Gunicorn config: set DJANGO_SETTINGS_MODULE, call django.setup() first, and avoid early model imports.
- Fix Celery Beat Not Sending Scheduled Tasks
Fix Celery Beat that runs but never sends periodic tasks in Django: check that Beat is actually running, the scheduler and timezone settings, django-celery-beat database schedules, stale schedule files, and queue routing.
- Fix Celery Tasks Raising DoesNotExist for Newly Created Objects
Fix Celery tasks that raise Model.DoesNotExist for an object the view just created: queue tasks with transaction.on_commit, understand ATOMIC_REQUESTS, read replicas, and pass IDs instead of model instances.
- Fix Celery Tasks Running Old Code After a Deploy
Fix Celery workers that keep executing old task code after a Django deploy: restart every worker gracefully, check which release each worker runs, and keep task signatures compatible across versions.
- Fix Celery Beat "Pidfile Already Exists" in Docker and systemd
Fix Celery beat failing with ERROR: Pidfile (celerybeat.pid) already exists. Seems we’re already running? after container or server restarts: disable the pidfile in Docker, place it in /run under systemd, and make sure only one beat runs.
- Fix Celery "No Result Backend Is Configured" in Django
Fix Celery errors No result backend is configured and DisabledBackend object has no attribute _get_task_meta_for in Django: configure CELERY_RESULT_BACKEND, load settings with the CELERY namespace, or stop calling .get() on tasks that do not need results.
- Fix "SSL Error: Decryption Failed or Bad Record MAC" in Celery and Gunicorn
Fix PostgreSQL errors SSL error decryption failed or bad record mac and SSL SYSCALL error EOF detected in Celery workers or preloaded Gunicorn: stop sharing database connections across fork, close connections before forking, and use per-process connections.
- Fix Redis SSL CERTIFICATE_VERIFY_FAILED With rediss:// in Django and Celery
Fix SSL CERTIFICATE_VERIFY_FAILED errors when Django, Celery, or Channels connect to Redis over rediss:// on Heroku, Upstash, ElastiCache, or Azure: configure ssl_cert_reqs, CA bundles, and broker_use_ssl without disabling verification blindly.
- Fix Django InterfaceError: Connection Already Closed
Fix django.db.utils.InterfaceError connection already closed in Celery tasks, management commands, and long-running scripts: call close_old_connections, set CONN_MAX_AGE and CONN_HEALTH_CHECKS, and handle database restarts and idle timeouts.