Monitor Django Scheduled Jobs With Heartbeat Checks

Problem statement

Nightly backups, invoice generation, data syncs, and cleanup jobs are critical and invisible. When a job crashes, someone might notice an error email. When it simply stops running (Beat died after a deploy, a cron entry was lost in a server move, a Kubernetes CronJob was suspended, a timer was never enabled), nothing reports anything, and you find out weeks later when the backup is missing. Heartbeat monitoring flips the model: each successful run pings a monitor, and the monitor alerts when an expected ping does not arrive.

1. Create a check per job in a heartbeat service (Healthchecks.io, Cronitor, Better Stack, or self-hosted Healthchecks) with the job's schedule and a grace period.
2. Ping the check URL at the end of a successful run; ping /start at the beginning to measure duration and catch hangs.
3. Ping /fail (or exit non-zero in wrapper scripts) on errors for immediate alerts.
4. Add pings to management commands, Celery Beat tasks, systemd timers, and Kubernetes CronJobs alike.
5. Route alerts to the same channel as other production alerts.
6. Keep ping calls short with timeouts so monitoring never breaks the job.

Step-by-step solution

1. Create checks

For each job, define the expected schedule (cron expression or period) and a grace time longer than the job's normal duration. For example, send_invoices at 0 6 * * * with a 30-minute grace period. Self-hosting the open-source Healthchecks project works if you prefer not to use a SaaS, but host it outside the infrastructure it monitors.

2. Ping from management commands

# core/heartbeat.py
import contextlib, logging, os
import requests

log = logging.getLogger(__name__)

def ping(url, suffix=""):
    if not url:
        return
    try:
        requests.get(f"{url}{suffix}", timeout=5)
    except requests.RequestException:
        log.warning("heartbeat ping failed", extra={"url": url})

@contextlib.contextmanager
def heartbeat(env_var):
    url = os.environ.get(env_var)
    ping(url, "/start")
    try:
        yield
    except Exception:
        ping(url, "/fail")
        raise
    ping(url)

class Command(BaseCommand):
    def handle(self, *args, **opts):
        with heartbeat("HC_SEND_INVOICES"):
            send_invoices()

Monitoring failures are logged, never raised, so an outage of the heartbeat service cannot break the job.

3. Celery Beat tasks

@shared_task(bind=True)
def nightly_sync(self):
    with heartbeat("HC_NIGHTLY_SYNC"):
        run_sync()

This catches both task failures and Beat not scheduling the task at all, which is the classic silent failure after deploys or when Beat runs with the wrong scheduler.

4. systemd timers and Kubernetes CronJobs

For jobs you would rather not modify, wrap them:

# systemd service for a timer
ExecStartPre=-/usr/bin/curl -fsS -m 5 https://hc-ping.com/<uuid>/start
ExecStart=/srv/app/.venv/bin/python manage.py clearsessions
ExecStartPost=/usr/bin/curl -fsS -m 5 https://hc-ping.com/<uuid>

# Kubernetes CronJob container
command: ["sh", "-c", "python manage.py clearsessions && wget -q -T 5 -O- https://hc-ping.com/$HC_UUID || wget -q -T 5 -O- https://hc-ping.com/$HC_UUID/fail"]

Store ping URLs as secrets or environment variables; they act as write tokens for the check.

5. Alert routing

Send heartbeat alerts to the same on-call channel as uptime alerts. Review the dashboard periodically for checks that are paused or never received a ping (a new job someone forgot to wire up).

Verification check: run each job manually and see its check turn green with a duration; temporarily disable one job's schedule and confirm an alert arrives after the grace period.

Explanation

Most monitoring detects events: an error is raised, a request fails, a metric crosses a threshold. A job that never starts produces no event to detect. Heartbeat monitoring, also called a dead man's switch, inverts the signal: the job reports success, and the monitor alerts on the absence of a report within an expected window. Start pings add duration tracking and catch hung jobs. Because the check is independent of how the job is scheduled, the same approach covers cron, systemd timers, Celery Beat, and Kubernetes CronJobs.

Edge cases / notes

• Very frequent jobs (every minute) can ping every run or every N runs; set grace accordingly.
• Jobs on several servers should ping only from one, or use separate checks per host.
• Long-running jobs benefit from start pings so hangs past the expected duration alert.
• Backups deserve heartbeats plus periodic restore tests; a successful ping does not prove the backup is restorable.

Next steps

• Schedule jobs reliably with Schedule Django Management Commands With systemd Timers.
• Watch the site itself with Django Uptime Monitoring and Alerts.

FAQ

How do I know if a Django cron job stopped running?

Use heartbeat monitoring: the job pings a monitor on success, and the monitor alerts if a ping is missing.

What is a dead man's switch?

A monitor that alerts when an expected signal does not arrive, rather than when something bad happens.

Will a heartbeat outage break my job?

Not if pings use short timeouts and errors are caught and logged.

Can I monitor Celery Beat tasks this way?

Yes. Ping from the task, which also detects Beat failing to schedule it.