Configure Redis and Celery for Django Production

Problem statement

Running background jobs in Django production is not just a matter of installing Celery and pointing it at Redis. Your web app, worker processes, broker, and scheduler all have different failure modes and restart behavior.

A naive setup often breaks in predictable ways:

• Redis is exposed publicly or left unauthenticated
• Celery settings are hardcoded in Django settings
• workers run in a shell session instead of a service manager
• Celery Beat runs more than once and creates duplicate scheduled jobs
• long tasks block short tasks in the same queue
• deploys restart workers at the wrong time and leave queued or in-flight tasks in a bad state

If you want Django, Celery, and Redis to work reliably in production, treat them as a separate service stack: web, worker, broker, optional scheduler, and verification.

Quick answer

A safe default production architecture is:

• Django web app running under Gunicorn or Uvicorn
• Redis on a private interface or managed private service
• one or more Celery worker services
• one Celery Beat service only if you use periodic tasks
• broker URL and related settings loaded from environment variables
• systemd or containers used to keep worker processes running
• JSON-only serialization, task time limits, and intentional queue design

Minimum safe setup:

1. Install and secure Redis
2. Configure Celery in your Django project
3. Run workers with systemd or containers
4. Run Beat once only if needed
5. Verify Redis connectivity, worker registration, and test task execution
6. Define a rollback path before changing worker code or schedules

Step-by-step solution

Choose the production architecture

For small apps, a single Linux server can run:

• Django web service
• Redis bound to localhost or a private IP
• Celery worker
• optional Celery Beat

For larger or more sensitive setups, split Redis onto a private internal host or managed Redis service and keep workers on app hosts.

Docker and non-Docker both work. Use the one you already deploy consistently. If your web app uses systemd and a virtualenv, do not introduce Docker only for Celery unless you want the extra operational overhead.

Install and secure Redis for production

On Ubuntu:

sudo apt update
sudo apt install -y redis-server

Edit /etc/redis/redis.conf:

bind 127.0.0.1
protected-mode yes
port 6379

requirepass change-this-to-a-long-random-secret

appendonly yes
save 900 1
save 300 10
save 60 10000

maxmemory-policy noeviction

Notes:

• bind 127.0.0.1 keeps Redis local on a single-server setup.
• If Redis is on another host, bind it to a private interface only, not 0.0.0.0.
• requirepass is acceptable for straightforward deployments. On newer Redis versions, ACLs are also an option.
• appendonly yes improves recovery after restart, but it does not guarantee zero task loss.
• noeviction is safer than silently evicting broker data under memory pressure.

Restart Redis:

sudo systemctl restart redis-server
sudo systemctl enable redis-server

Verification:

redis-cli -a 'change-this-to-a-long-random-secret' ping

Expected output:

PONG

If Redis is remote, also restrict network access with a firewall or security group so only app and worker hosts can connect. If you use a managed Redis service, prefer TLS if connections leave the local host or private trusted network.

Configure Django and Celery

Install dependencies in your app environment:

pip install celery redis

Create proj/celery.py:

import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings.production")

app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Update proj/__init__.py:

from .celery import app as celery_app

__all__ = ("celery_app",)

Example task in myapp/tasks.py:

from celery import shared_task
import logging

logger = logging.getLogger(__name__)

@shared_task
def healthcheck_task():
    logger.info("Celery test task executed")
    return "ok"

In settings/production.py:

import os

CELERY_BROKER_URL = os.environ["CELERY_BROKER_URL"]

# Only set a result backend if you actually need task results.
CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND")
CELERY_RESULT_EXPIRES = 86400

CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_TIMEZONE = "UTC"
CELERY_ENABLE_UTC = True

CELERY_TASK_ALWAYS_EAGER = False

CELERY_TASK_ACKS_LATE = True  # use only for idempotent tasks; a task may run again after worker failure
CELERY_WORKER_PREFETCH_MULTIPLIER = 1

CELERY_TASK_TIME_LIMIT = 1800
CELERY_TASK_SOFT_TIME_LIMIT = 1500

CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True

Environment file example:

DJANGO_SETTINGS_MODULE=proj.settings.production
CELERY_BROKER_URL=redis://:change-this-to-a-long-random-secret@127.0.0.1:6379/0
CELERY_RESULT_BACKEND=redis://:change-this-to-a-long-random-secret@127.0.0.1:6379/1

Verification:

python manage.py shell -c "from myapp.tasks import healthcheck_task; print(healthcheck_task.delay().id)"

If this errors before workers are started, it still confirms Django can publish to Redis.

Configure production Celery settings

For safer defaults in production:

• CELERY_TASK_ACKS_LATE = True helps avoid losing work if a worker dies mid-task, but only use it for idempotent tasks
• CELERY_WORKER_PREFETCH_MULTIPLIER = 1 reduces one worker reserving too many tasks
• set soft and hard time limits for untrusted or long-running jobs
• use separate queues for long-running tasks

Example queue routing:

CELERY_TASK_ROUTES = {
    "myapp.tasks.generate_report": {"queue": "long"},
}

Then run a separate worker for the long queue if needed.

Do not enable a result backend just by habit. If your app only fires jobs and logs side effects, skipping results reduces Redis load. If you do use Redis as a result backend, set an expiry so old task results do not accumulate indefinitely.

Run Celery workers in production

Create /etc/default/celery-proj:

DJANGO_SETTINGS_MODULE=proj.settings.production
CELERY_BROKER_URL=redis://:change-this-to-a-long-random-secret@127.0.0.1:6379/0
CELERY_RESULT_BACKEND=redis://:change-this-to-a-long-random-secret@127.0.0.1:6379/1
PATH=/srv/proj/venv/bin

Create /etc/systemd/system/celery-worker.service:

[Unit]
Description=Celery Worker for proj
After=network.target redis-server.service

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/srv/proj/current
EnvironmentFile=/etc/default/celery-proj
ExecStart=/srv/proj/venv/bin/celery -A proj worker --loglevel=INFO --concurrency=2
Restart=always
RestartSec=5
TimeoutStopSec=600

[Install]
WantedBy=multi-user.target

If you use periodic tasks, create /etc/systemd/system/celery-beat.service:

[Unit]
Description=Celery Beat for proj
After=network.target redis-server.service

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/srv/proj/current
EnvironmentFile=/etc/default/celery-proj
ExecStart=/srv/proj/venv/bin/celery -A proj beat --loglevel=INFO
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Load and start:

sudo systemctl daemon-reload
sudo systemctl enable --now celery-worker
sudo systemctl status celery-worker

If using Beat:

sudo systemctl enable --now celery-beat
sudo systemctl status celery-beat

Log checks:

journalctl -u celery-worker -n 100 --no-pager
journalctl -u celery-beat -n 100 --no-pager

Environment-safe verification:

set -a
. /etc/default/celery-proj
set +a
/srv/proj/venv/bin/celery -A proj inspect ping
/srv/proj/venv/bin/celery -A proj inspect active

Docker production variant

If you deploy with Compose, define separate worker and beat services:

services:
  worker:
    image: your-app-image:latest
    command: celery -A proj worker --loglevel=INFO --concurrency=2
    env_file:
      - .env
    restart: always
    depends_on:
      - redis

  beat:
    image: your-app-image:latest
    command: celery -A proj beat --loglevel=INFO
    env_file:
      - .env
    restart: always
    depends_on:
      - redis

  redis:
    image: redis:7
    restart: always
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]

Example redis.conf:

bind 0.0.0.0
protected-mode yes
appendonly yes
requirepass your-long-random-password
maxmemory-policy noeviction

In container deployments, make sure Redis is reachable only on the internal container network or private network path you intend to use. Do not publish Redis publicly unless you have a specific reason and matching network controls.

Important: depends_on does not guarantee application readiness. Your worker still needs retry behavior on startup, which Celery supports with CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True.

Run Beat only once in the whole environment.

Connect periodic tasks safely

Use Celery Beat only if you have scheduled jobs. Never run multiple Beat instances against the same schedule unless you are intentionally using a scheduler with distributed locking.

If schedules are simple and version-controlled, define them in code. If non-developers need to manage schedules, django-celery-beat can be appropriate, but it adds another database-backed component to operate.

Verification checks

After deployment, verify in order:

1. Redis connectivity:

   redis-cli -a "$REDIS_PASSWORD" ping
   

2. Worker registration:

   set -a
   . /etc/default/celery-proj
   set +a
   celery -A proj inspect ping
   

3. Task execution:

   python manage.py shell -c "from myapp.tasks import healthcheck_task; r=healthcheck_task.delay(); print(r.id)"
   

4. Logs:

   journalctl -u celery-worker -n 100 --no-pager
   

Look for connection failures, permission errors, serializer errors, or import errors.

Explanation

Redis is common for Celery because it is simple to operate and fast enough for many Django background job workloads. For small and medium apps, it is usually enough as a broker and sometimes as a result backend.

However, Redis is not the right answer for every queueing problem. If you need more complex broker semantics, stronger delivery guarantees, or very large-scale queueing patterns, another broker may fit better. For many Django teams, though, Redis gives a practical balance between operational complexity and reliability.

Running workers separately from the web process matters because background tasks have different concurrency and timeout behavior. A task worker should be able to restart, scale, and fail independently from Gunicorn. Beat also must be independent because it is a scheduler, not an HTTP process.

Rollback and recovery

If a worker deploy is bad:

1. Stop Beat first so it does not enqueue new scheduled jobs.
2. Check whether workers still have active or reserved tasks before restarting them.
3. Restart workers back to the previous code or image.
4. Verify workers can consume the expected queues.
5. Re-enable Beat only after the queue and logs look healthy.

Systemd example:

sudo systemctl stop celery-beat

set -a
. /etc/default/celery-proj
set +a
/srv/proj/venv/bin/celery -A proj inspect active
/srv/proj/venv/bin/celery -A proj inspect reserved

sudo systemctl restart celery-worker

A few deploy rules matter here:

• if new code enqueues tasks that old workers do not understand, rollback can still fail
• if new workers expect a schema change, apply migrations before they consume migration-dependent tasks
• if you use acks_late, interrupted tasks may run again after worker failure or restart
• tasks should be idempotent so retries or duplicate execution do not corrupt data

TimeoutStopSec=600 gives workers time to finish work during shutdown, but it is not a substitute for task design. Very long-running tasks may still need separate queues, longer shutdown windows, or a different execution strategy.

If Redis restarts and transient tasks are lost, recovery depends on your persistence mode and whether tasks can be recreated safely. For critical workflows, tasks should be idempotent and externally recoverable where possible.

When to automate this setup

Once you repeat this process across environments, convert the Celery files and service definitions into reusable templates. Good candidates are celery.py, hardened production settings, systemd unit files, environment file generation, and post-deploy smoke tests. It is also worth scripting Beat pause and resume around deploys.

Edge cases and notes

• Long-running jobs: keep them on a separate queue so they do not block short tasks.
• CPU-bound tasks: Celery concurrency helps less for heavy CPU work; tune worker count carefully based on available cores.
• Migrations during deploys: if a task depends on a new schema, apply migrations before restarting workers onto code that enqueues or consumes those tasks.
• Same server as Gunicorn: acceptable for smaller apps, but cap worker concurrency so Celery does not starve the web app of CPU or memory.
• Static files: Celery does not change static handling directly, but your deploy order still matters: collect static, migrate, restart app, restart workers, then verify.
• Multi-instance deployments: only one Beat should run, but many worker instances can share the same queues.
• Remote Redis: private networking is the baseline; add TLS when traffic crosses hosts or leaves a trusted network.
• Result backend growth: if you keep task results in Redis, use expiry and monitor memory usage.

Internal links

To build the full production picture, also read:

• Django Deployment Checklist for Production
• Deploy Django with Gunicorn and Nginx on Ubuntu
• Deploy Django ASGI with Uvicorn and Nginx
• How to debug stuck Celery tasks in production

FAQ

Can I run Celery and Gunicorn on the same server?

Yes, for small apps. Keep Redis private, use systemd or containers, and limit Celery concurrency so the worker does not consume all CPU or memory.

Do I need Celery Beat in production?

Only if you run scheduled tasks. If you do, run exactly one Beat instance for that environment.

Should Redis be used as both broker and result backend?

It can be, but only enable the result backend if your app actually needs task state or returned values. Many production setups use Redis only as a broker.

How many Celery workers should I run?

Start with one worker service and a conservative concurrency value such as 2. Increase based on task type, CPU, memory, and queue backlog. Separate long-running tasks into their own queue before scaling blindly.

What happens to queued tasks during a deploy?

Queued tasks usually remain in Redis if the broker stays up. In-progress tasks may be retried depending on acknowledgment timing and worker shutdown behavior. That is why tasks should be idempotent and deploys should include worker restart and verification steps.