Fix a Celery Worker Whose Memory Keeps Growing

Problem statement

A Celery worker starts at 150 MB per child and a few hours later each child holds 800 MB, then 1.5 GB. Eventually the kernel kills one, the task fails with WorkerLostError, and dmesg shows an OOM kill. Restarting the worker "fixes" it until the next day.

Sometimes this is a true leak, such as a module-level list that grows forever. More often it is Python's normal behaviour: a task loads 200,000 rows into memory once, the process grows to hold them, and the memory is never returned to the operating system even after the objects are freed. Either way, a long-lived worker process accumulates the worst case of every task it has run.

1. Recycle children: --max-tasks-per-child=200 and --max-memory-per-child=400000 (KiB).
2. Make sure DEBUG = False on workers; with DEBUG on, Django stores every SQL query in memory.
3. Iterate large querysets with .iterator(chunk_size=2000) and select only needed columns.
4. Find the leaking task by logging RSS before and after each task.
5. Remove module-level caches that grow without bounds.
6. Lower concurrency if the steady state does not fit in RAM.

Step-by-step solution

1. Confirm the growth pattern

Watch per-child memory over time:

ps -o pid,ppid,rss,etime,cmd -C celery --sort=-rss | head

rss is in KiB. If every child grows steadily regardless of which tasks it runs, suspect a general cause such as DEBUG. If one child jumps after particular work, suspect a specific task. Note the parent process stays small; it is the children that grow.

2. Recycle children before they get large

These two options are the immediate fix and belong in every production worker:

celery -A config worker \
  --concurrency=4 \
  --max-tasks-per-child=200 \
  --max-memory-per-child=400000

Or in settings:

CELERY_WORKER_MAX_TASKS_PER_CHILD = 200
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 400_000   # KiB, about 400 MB

max-memory-per-child replaces a child after the task that pushed it over the limit finishes, so the task still completes. Pick a value well below the memory available per child so that a spike does not reach the OOM killer first.

3. Turn off DEBUG on workers

With DEBUG = True, Django appends every executed query to connection.queries. In a web request that list is reset per request; in a Celery task loop it is not reset unless you do it. Workers must use production settings:

python -c "import django, os; os.environ.setdefault('DJANGO_SETTINGS_MODULE','config.settings.production'); django.setup(); from django.conf import settings; print(settings.DEBUG)"

If a worker must run with DEBUG for some reason, call django.db.reset_queries() at the start of each task.

4. Stop loading whole tables

The single biggest source of growth is code like:

for order in Order.objects.all():
    process(order)

The queryset cache holds every model instance until the loop ends. Stream instead, and fetch only the columns you need:

qs = Order.objects.filter(status="paid").only("id", "total", "email")
for order in qs.iterator(chunk_size=2000):
    process(order)

For pure data export, .values_list(...) avoids building model instances at all. Also avoid returning large objects from tasks: the result is serialised and kept in memory and in the result backend.

5. Find the task that leaks

Log memory around each task with Celery signals:

import resource, logging
from celery.signals import task_prerun, task_postrun

log = logging.getLogger("celery.memory")
_before = {}

@task_prerun.connect
def _pre(task_id, task, **kw):
    _before[task_id] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

@task_postrun.connect
def _post(task_id, task, **kw):
    peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    grew = peak - _before.pop(task_id, peak)
    if grew > 50_000:
        log.warning("task %s raised peak RSS by %s KiB", task.name, grew)

ru_maxrss is the peak, in KiB on Linux. A task that repeatedly raises the peak is your candidate. Profile it locally with tracemalloc and realistic data.

6. Fit concurrency to memory

After the fixes, measure the steady per-child size and do the arithmetic: children times peak size, plus the parent, must fit in RAM alongside everything else on the host. If not, reduce --concurrency or move workers to their own machine.

Verification check: run the worker for a full day and graph per-child RSS. It should rise and fall as children are replaced, never climbing past the max-memory-per-child value plus one task's growth, and no OOM kills should appear in journalctl -k.

Explanation

CPython allocates small objects from arenas that are only released when completely empty. After a task creates and frees a million objects, the arenas are fragmented and mostly stay mapped, so RSS remains high even though the memory is free inside the process. The next large task reuses that space, which is why growth eventually plateaus at the size of the largest task. A real leak never plateaus. Recycling children handles both, because a new child starts small.

Edge cases / notes

• Prefork only. max-tasks-per-child and max-memory-per-child apply to the prefork pool. With threads or gevent, the whole worker grows; restart it periodically or with a supervisor memory limit.
• Container limits. In Docker or Kubernetes, the container limit applies to the sum of all children. Set max-memory-per-child so that concurrency times the limit stays under it.
• C extensions. Libraries like image processing or PDF rendering can hold memory outside Python's allocator. Running those tasks in a dedicated queue with low max-tasks-per-child contains the damage.
• Prefetch. A high worker_prefetch_multiplier holds many messages in memory. With long tasks, use 1.

Next steps

• Choose concurrency and pool settings with Tune Celery Workers for Django.
• If children are already being killed, read Fix Celery WorkerLostError.
• On small servers, Add Swap to a Small Django Server gives headroom during spikes.

FAQ

Is this a Celery bug?

Rarely. It is usually how Python manages memory combined with tasks that load large data sets.

What max-tasks-per-child value should I use?

Something between 100 and 1,000 is common. Lower values cost a little startup time per child; pair it with a memory limit for safety.

Does gc.collect() help?

It frees unreachable objects but rarely returns memory to the OS. Streaming data and recycling children are more effective.

Why does the worker log WorkerLostError?

The kernel killed a child for using too much memory. Fix the growth or reduce concurrency.