Configure Redis Persistence for Django and Celery

Problem statement

Redis keeps data in memory. What happens to it on restart depends entirely on its persistence settings, and Django stacks use Redis for very different kinds of data: a cache (losing it is harmless), sessions (losing them logs everyone out), and the Celery broker (losing it silently drops queued tasks: emails never sent, payments never reconciled).

Many deployments run Redis with whatever the package or container image defaults to, and discover the consequences after a reboot or a container recreation. This guide chooses persistence per role, configures it, accounts for its memory and disk costs, and verifies that data actually survives a restart.

1. Separate roles: one Redis (or at least one configuration) for the cache, another for the broker and sessions.
2. Cache: disable persistence (save "", appendonly no) and use an eviction policy.
3. Celery broker and sessions: enable AOF with appendfsync everysec; optionally keep RDB snapshots for backups.
4. Store data on a persistent disk or Docker volume, and make sure the directory is writable.
5. Set vm.overcommit_memory = 1 and leave memory headroom for background saves and rewrites.
6. Restart Redis and confirm queued tasks and sessions are still there.

Step-by-step solution

1. Understand the two mechanisms

AOF with everysec is the usual choice for data you do not want to lose; RDB is a compact format for backups. Redis can use both. Since Redis 7, the AOF is stored as multiple files in an appendonlydir directory.

2. Configure a cache instance

# /etc/redis/redis-cache.conf (excerpt)
port 6379
save ""
appendonly no
maxmemory 1gb
maxmemory-policy allkeys-lru

A cache that starts empty after a restart is expected behaviour. Persistence would only add disk I/O and slow restarts.

3. Configure a broker and sessions instance

# /etc/redis/redis-queue.conf (excerpt)
port 6380
dir /var/lib/redis-queue
appendonly yes
appendfsync everysec
save 900 1
save 300 100
maxmemory 1gb
maxmemory-policy noeviction

• appendonly yes with everysec persists every write within about a second.
• The save lines keep periodic RDB snapshots as well, handy for off-server backups.
• noeviction ensures Redis rejects writes rather than silently deleting queued tasks when memory is full.

Point Django and Celery at the right instance:

CACHES = {"default": {"BACKEND": "django.core.cache.backends.redis.RedisCache", "LOCATION": "redis://10.0.0.8:6379/0"}}
CELERY_BROKER_URL = "redis://10.0.0.8:6380/0"
CELERY_RESULT_BACKEND = "redis://10.0.0.8:6380/1"

If sessions use the cache backend, give them a cache alias on the persistent instance, or use cached_db sessions so the database is the source of truth.

4. Persistent storage in containers

services:
  redis-queue:
    image: redis:7
    command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec", "--maxmemory-policy", "noeviction"]
    volumes:
      - redisqueue:/data
    restart: unless-stopped

volumes:
  redisqueue:

The official image writes to /data. Without a volume, persistence files live in the container's filesystem and disappear when the container is recreated, which makes the persistence settings pointless.

5. Memory and kernel settings

RDB snapshots and AOF rewrites fork the Redis process. Copy-on-write means the child shares memory with the parent, but pages modified during the save are duplicated. Under heavy writes, memory use can approach double the dataset.

echo "vm.overcommit_memory = 1" | sudo tee /etc/sysctl.d/99-redis.conf
sudo sysctl --system

Redis logs a warning at start-up if overcommit is disabled, because the fork can fail. Keep maxmemory well below the server's or container's memory limit to leave room for the fork.

6. Managed Redis

Managed services handle persistence differently. Many rely on replicas and automatic failover plus periodic snapshots rather than AOF; some offer AOF or durable modes as options. Read your provider's documentation for what survives a node replacement, and do not assume queued Celery messages survive a failover.

7. Check what is configured and what is written

redis-cli -p 6380 CONFIG GET appendonly
redis-cli -p 6380 CONFIG GET appendfsync
redis-cli -p 6380 INFO persistence | grep -E "aof_enabled|aof_last_write_status|rdb_last_bgsave_status|rdb_last_save_time"
ls -l /var/lib/redis-queue/

aof_last_write_status:ok and rdb_last_bgsave_status:ok confirm writes succeed. A failing background save (for example because the disk is full or the directory is not writable) causes Redis, by default, to refuse writes to protect you from silent data loss (stop-writes-on-bgsave-error yes).

Verification check:

# queue a task that no worker consumes yet
sudo systemctl stop celery-worker
python manage.py shell -c "from notifications.tasks import send_digest; send_digest.delay(1)"
redis-cli -p 6380 LLEN celery

sudo systemctl restart redis-queue
redis-cli -p 6380 LLEN celery          # same count as before the restart

sudo systemctl start celery-worker      # the task runs

Explanation

Celery's Redis transport stores pending tasks as list entries. With persistence off, a Redis restart (reboot, upgrade, OOM kill, container recreation) empties those lists, and tasks disappear without any error in Django or Celery. AOF records each write so Redis can rebuild its dataset on start-up, limiting loss to about a second of writes. Caches deliberately trade durability for speed and should stay non-persistent. Choosing per role, preferably with separate instances, gives each kind of data the right guarantee.

Edge cases / notes

Unacknowledged tasks

Tasks already delivered to a worker but not yet acknowledged are tracked in Redis too (in the unacked structures used by Celery's Redis transport) and are restored on restart if Redis persisted them. Idempotent tasks make any resulting re-execution harmless.

RabbitMQ as broker

If losing queued tasks is unacceptable and volumes are high, RabbitMQ with durable queues and persistent messages is designed for message durability. Redis with AOF is adequate for many Django projects.

When to automate this

Put the Redis configuration for each role in version control and provisioning, back up RDB snapshots off the server if you rely on them, and alert on aof_last_write_status or rdb_last_bgsave_status not being ok. Include the restart test above in your disaster recovery drills.

Next steps

• Set up the broker with Configure Redis and Celery for Django Production.
• Review everything else with the Redis Production Checklist for Django.
• If Redis refuses writes, see Fix Redis "OOM command not allowed" in Django.
• If Redis will not accept connections after a restart, see Fix Redis Connection Refused in Django.

FAQ

Do I need Redis persistence for Celery?

If you use Redis as the Celery broker and cannot afford to lose queued tasks on a restart, yes. Use AOF with appendfsync everysec.

Should a Django cache Redis have persistence?

No. A cache can start empty. Disable persistence and use an eviction policy.

What is the difference between RDB and AOF?

RDB takes periodic snapshots; AOF logs every write. AOF loses at most about a second of data with everysec, RDB loses everything since the last snapshot.

Why did my Redis data disappear after recreating the Docker container?

The data directory was not on a volume. Mount a volume at /data for the official Redis image.