Run Celery Beat for Scheduled Tasks in Django Production

Problem statement

Celery Beat is the scheduler that sends periodic tasks (nightly reports, cleanup jobs, subscription renewals) to your Celery queue. It is a separate process from the workers, and treating it like "just another worker" causes production problems:

• running Beat inside every worker with -B sends each scheduled task once per worker
• a Beat process on every server during a scale-out duplicates every schedule
• the schedule state file is written somewhere the service user cannot write, so Beat crashes on start
• tasks run at the wrong hour because Celery's time zone differs from Django's
• after an outage, a backlog of stale scheduled tasks all run at once
• nobody notices that Beat stopped, because missing tasks produce no errors

This guide runs a single Celery Beat process under systemd next to your workers, with schedules defined in code or in the database, and adds safeguards and monitoring.

Quick answer

1. Define periodic tasks in CELERY_BEAT_SCHEDULE (or use django-celery-beat to manage them in the admin).
2. Set CELERY_TIMEZONE to match Django's TIME_ZONE, and prefer UTC for both.
3. Run exactly one Beat process in the whole deployment, as its own systemd service, never with -B on workers.
4. Give Beat a writable schedule file path (file scheduler) or use django_celery_beat.schedulers:DatabaseScheduler.
5. Add expires to periodic tasks so stale runs are discarded, and make tasks idempotent.
6. Monitor Beat with a heartbeat task that updates a timestamp you alert on.

Step-by-step solution

1. Celery app configuration

A typical config/celery.py:

import os

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

With namespace="CELERY", Celery reads settings prefixed with CELERY_ from Django settings, so CELERY_BEAT_SCHEDULE maps to Celery's beat_schedule and CELERY_TIMEZONE to timezone.

2. Define schedules in settings

from celery.schedules import crontab

TIME_ZONE = "UTC"
USE_TZ = True
CELERY_TIMEZONE = "UTC"

CELERY_BEAT_SCHEDULE = {
    "send-daily-digest": {
        "task": "notifications.tasks.send_daily_digest",
        "schedule": crontab(hour=6, minute=0),
        "options": {"expires": 60 * 60},
    },
    "clear-expired-sessions": {
        "task": "core.tasks.clear_expired_sessions",
        "schedule": crontab(hour=3, minute=30),
        "options": {"expires": 60 * 60 * 2},
    },
    "beat-heartbeat": {
        "task": "core.tasks.beat_heartbeat",
        "schedule": 60.0,
        "options": {"expires": 50},
    },
}

• crontab() fields follow cron semantics; unspecified fields default to "every".
• A number (seconds) or timedelta schedules fixed intervals.
• options.expires (seconds) tells workers to discard the message if it has not started within that time. After a worker outage, a digest queued yesterday is skipped instead of being sent late.

Using UTC for both Django and Celery avoids daylight saving time surprises: a job scheduled at 02:30 local time does not run twice or not at all when clocks change. If business rules require local time, set CELERY_TIMEZONE to that zone and test the transition dates.

3. Write idempotent tasks

Beat guarantees a task is sent on schedule; it does not guarantee it runs exactly once. Retries, redeliveries, and operator mistakes can all run a task twice. Design for that:

# core/tasks.py
from celery import shared_task
from django.core.cache import cache
from django.core.management import call_command
from django.utils import timezone


@shared_task
def clear_expired_sessions():
    call_command("clearsessions")


@shared_task
def beat_heartbeat():
    cache.set("celery-beat-heartbeat", timezone.now().isoformat(), timeout=None)

clearsessions is naturally idempotent. For tasks such as sending digests, record what has been sent (for example a DigestRun row with a unique date) and skip if it already exists.

4. Choose a scheduler

File scheduler (default). Beat stores last run times in a local file (celerybeat-schedule). Schedules come from CELERY_BEAT_SCHEDULE. Simple and reliable when Beat runs on one known host.

Database scheduler (django-celery-beat). Schedules and last run times live in the Django database and can be edited in the admin:

pip install django-celery-beat

INSTALLED_APPS += ["django_celery_beat"]

python manage.py migrate django_celery_beat

Beat then runs with --scheduler django_celery_beat.schedulers:DatabaseScheduler. Choose this when non-developers need to change schedules, or when Beat may move between hosts. Entries in CELERY_BEAT_SCHEDULE are synced into the database when Beat starts.

5. Run Beat as its own systemd service

/etc/systemd/system/celery-beat.service:

[Unit]
Description=Celery Beat for myapp
After=network.target redis-server.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp/app
EnvironmentFile=/etc/myapp/env
StateDirectory=myapp-beat
ExecStart=/srv/myapp/venv/bin/celery -A config beat \
    --loglevel=INFO \
    --schedule=/var/lib/myapp-beat/celerybeat-schedule \
    --pidfile=
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

• StateDirectory=myapp-beat creates /var/lib/myapp-beat owned by the service user, so the schedule file is writable and survives restarts.
• An empty --pidfile= disables the pid file; systemd already tracks the process, and a stale pid file from a crash can otherwise block restarts.
• For the database scheduler, replace --schedule=... with --scheduler django_celery_beat.schedulers:DatabaseScheduler.

sudo systemctl daemon-reload
sudo systemctl enable --now celery-beat
journalctl -u celery-beat -f

Workers keep running as their own service, for example celery -A config worker --loglevel=INFO.

6. Guarantee a single Beat instance

Beat has no built-in leader election. Two Beat processes send every task twice. Common ways to keep exactly one:

• Single-server or dedicated scheduler host: enable celery-beat.service only on one host, and keep it out of the image or role used for scaled web and worker servers.
• Docker Compose: a separate beat service with no replicas setting, never scaled.
• Kubernetes: a Deployment with replicas: 1 and the Recreate strategy, so a rollout stops the old pod before starting the new one.
• Platforms like Heroku or Render: one dedicated process type scaled to exactly one instance.

If your environment cannot guarantee a single instance, add a distributed lock inside critical tasks (for example cache.add() with a short timeout) as a second line of defense.

7. Monitor Beat

Missing scheduled runs are silent. The heartbeat task from step 3 records a timestamp every minute; check it from your monitoring system or a health endpoint:

# core/views.py
from datetime import timedelta

from django.core.cache import cache
from django.http import JsonResponse
from django.utils import timezone
from django.utils.dateparse import parse_datetime


def beat_health(request):
    value = cache.get("celery-beat-heartbeat")
    last = parse_datetime(value) if value else None
    healthy = last is not None and timezone.now() - last < timedelta(minutes=5)
    return JsonResponse({"healthy": healthy}, status=200 if healthy else 503)

This endpoint fails if Beat stops, if workers stop consuming, or if the broker is down, which are all conditions you want to know about. Protect or obscure it if it reveals operational details you do not want public.

Dead man's switch services (for example Healthchecks.io or Cronitor) work well too: have the heartbeat task ping a URL, and get alerted when pings stop.

8. Verify

journalctl -u celery-beat -n 20 --no-pager

Look for lines such as Scheduler: Sending due task beat-heartbeat (core.tasks.beat_heartbeat). Then confirm workers ran it:

journalctl -u celery-worker -n 50 --no-pager | grep beat_heartbeat
python manage.py shell -c "from django.core.cache import cache; print(cache.get('celery-beat-heartbeat'))"

Verification check:

• exactly one Beat process exists across all hosts (pgrep -af "celery.*beat" on each)
• Beat logs "Sending due task" at the expected times
• worker logs show each scheduled task received once per interval
• the heartbeat timestamp is less than two minutes old
• stopping workers for longer than a task's expires causes that task to be revoked, not run late

Deploying changes

Restart Beat after changing CELERY_BEAT_SCHEDULE or task names:

sudo systemctl restart celery-beat

With the database scheduler, admin edits are picked up automatically, but code-defined entries still sync on restart. If you rename a task, remove the old periodic task entry so Beat does not keep sending a task name that no longer exists.

Explanation

Beat is a clock, not a worker. It reads schedules, decides what is due, and publishes messages to the broker; workers consume and execute them. That separation is why Beat must be a singleton while workers can scale freely, and why task-level safeguards (expiry, idempotency) matter: Beat cannot know whether a message it sent was delayed, retried, or executed twice.

Edge cases / notes

Beat started with -B in development

celery worker -B embeds Beat into a worker. It is convenient locally but produces duplicates as soon as you run more than one worker process in production.

Very frequent schedules

Beat checks schedules on an interval (beat_max_loop_interval, which is shorter by default for the database scheduler than for the file scheduler). Sub-second precision is not realistic; for near-real-time work, use a queue consumer instead of a schedule.

Alternative: systemd timers

If you have only a few jobs and no Celery, running management commands from systemd timers avoids a broker and a long-running scheduler process.

Troubleshooting Beat

Inspecting the schedule

With the file scheduler, list what Beat will send by reading the configuration in a shell:

python manage.py shell -c "
from config.celery import app
for name, entry in app.conf.beat_schedule.items():
    print(name, entry['task'], entry['schedule'])
"

With django-celery-beat, check the PeriodicTask table, including disabled entries and last run times:

python manage.py shell -c "
from django_celery_beat.models import PeriodicTask
for t in PeriodicTask.objects.order_by('name'):
    print(t.name, t.task, t.enabled, t.last_run_at)
"

Deploy ordering with Beat

When a release adds a new periodic task, the worker must know the task before Beat starts sending it, otherwise workers log unregistered task errors and discard the messages. Restart workers first, then Beat. When removing a task, remove it from the schedule and restart Beat before deploying workers without the task code.

Clock synchronization

Beat decides when tasks are due from its host clock, and workers evaluate expires against their own clocks. Make sure all hosts run time synchronization:

timedatectl status | grep -E "synchronized|NTP service"

Both lines should report yes and active.

Scaling considerations

Beat itself is lightweight; one process can handle hundreds of schedule entries. Load comes from the tasks it sends. If many schedules fire at the same minute (for example everything at midnight), spread them out with different minute values so workers are not flooded at once.

Next steps

• Run workers correctly with Run Celery Workers with systemd for Django.
• If tasks run twice, see Fix Celery Beat Running Tasks Twice.
• Compare with Schedule Django Management Commands with Cron or systemd Timers.

FAQ

Should I run Celery Beat with the -B flag in production?

No. -B embeds the scheduler inside a worker, so every worker instance also schedules tasks and each job is sent multiple times. Run one separate Beat process.

Where does Celery Beat store its schedule?

The default scheduler writes last run times to a local celerybeat-schedule file, so point --schedule at a writable, persistent path. django-celery-beat stores schedules and run state in the Django database.

Why do my Celery periodic tasks run at the wrong time?

Celery uses its own timezone setting, which may differ from Django's TIME_ZONE. Set CELERY_TIMEZONE explicitly, ideally UTC for both.

How do I know if Celery Beat stopped?

Schedule a heartbeat task every minute that records a timestamp or pings a monitoring service, and alert when it is older than a few minutes.