Monitor Celery with Flower

Problem statement

Celery works invisibly until it does not: tasks pile up, a worker silently disconnects, one task type starts failing, or a queue stops draining. Without monitoring, you find out from users ("I never got the email") or from a full broker.

Flower is a web dashboard and API for Celery. It shows workers and their status, tasks with arguments, results, runtimes and tracebacks, and can export Prometheus metrics. It can also revoke tasks and shut down workers, which makes access control essential. This guide runs Flower safely in production next to a Django app.

1. pip install flower into the same environment as Celery.
2. Enable task events: CELERY_WORKER_SEND_TASK_EVENTS = True and CELERY_TASK_SEND_SENT_EVENT = True (or start workers with -E).
3. Run celery -A config flower --address=127.0.0.1 --port=5555 --url-prefix=flower under systemd.
4. Put Nginx in front with HTTPS and authentication (Flower's basic auth, OAuth options, or an access proxy), ideally with an IP allowlist.
5. Enable --persistent with a state database if you want history across restarts, and cap memory with --max-tasks.
6. Scrape /metrics with Prometheus for alerting; use the dashboard for investigation.

Step-by-step solution

1. Enable events

Flower learns about tasks from events that workers publish. Enable them in settings:

CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True

Restart workers after changing this. Without events, Flower shows workers but task lists stay empty. Events add some broker traffic; on very high-volume systems, consider sampling via metrics instead of full task tracking.

2. Run Flower under systemd

/srv/myapp/current/.venv/bin/pip install flower

# /etc/systemd/system/flower.service
[Unit]
Description=Flower for Celery
After=network.target

[Service]
User=myapp
WorkingDirectory=/srv/myapp/current
EnvironmentFile=/etc/myapp.env
EnvironmentFile=/etc/flower.env
ExecStart=/srv/myapp/current/.venv/bin/celery -A config flower \
  --address=127.0.0.1 --port=5555 --url-prefix=flower \
  --persistent=True --db=/var/lib/flower/flower.db --max-tasks=20000
Restart=on-failure
StateDirectory=flower

[Install]
WantedBy=multi-user.target

# /etc/flower.env (mode 600)
FLOWER_BASIC_AUTH=ops:use-a-long-random-password

• --address=127.0.0.1 keeps Flower off the public network; only Nginx reaches it.
• --url-prefix=flower serves it under /flower/ behind the proxy.
• --persistent with --db stores state across restarts; --max-tasks limits how many tasks are kept in memory.
• FLOWER_BASIC_AUTH enables HTTP basic authentication; Flower reads options from FLOWER_-prefixed environment variables. Flower also supports OAuth providers (Google, GitHub, and others) through its auth options.

sudo systemctl daemon-reload
sudo systemctl enable --now flower
curl -s -u ops:... http://127.0.0.1:5555/flower/healthcheck

3. Put it behind Nginx with HTTPS

location /flower/ {
    allow 203.0.113.0/24;    # office or VPN
    deny all;

    proxy_pass http://127.0.0.1:5555;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Basic auth over plain HTTP exposes the password; always serve Flower through HTTPS. The IP restriction is a second layer: Flower's API can revoke and terminate tasks and control worker pools, so treat access as administrative.

4. Use the dashboard

What to look at during an incident:

• Workers tab: which workers are online, their pool size, and active tasks. A missing worker explains a queue that does not drain.
• Tasks tab: filter by state (FAILURE, RETRY) and by task name; open a task for its arguments, traceback, runtime, and worker.
• Broker tab: messages per queue. A growing queue with healthy workers means throughput is too low or tasks are stuck.

Task arguments are visible in Flower. If tasks receive personal data or tokens as arguments, anyone with Flower access can read them; pass IDs instead.

5. Export metrics for alerting

Flower exposes Prometheus metrics at /metrics (under the URL prefix when set):

# prometheus.yml (excerpt)
scrape_configs:
  - job_name: flower
    metrics_path: /flower/metrics
    static_configs:
      - targets: ["127.0.0.1:5555"]
    basic_auth:
      username: ops
      password_file: /etc/prometheus/flower_password

Useful alerts built from its metrics: task failure rate by task name, workers online below the expected count, and task runtime percentiles above your limits. Queue length itself is best measured from the broker (for example with a Redis exporter), because Flower only reports what it has observed.

6. Know the limits

Flower keeps its view in memory (plus the optional state database). It is not a long-term task history or an audit log. For long-term analysis, rely on metrics in Prometheus and structured logs in your log system; for errors, rely on your error tracker capturing task exceptions.

Verification check:

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/flower/                 # expect 401 or 403 without auth
curl -s -u ops:... https://example.com/flower/api/workers | head -c 300              # workers listed
python manage.py shell -c "from notifications.tasks import ping; ping.delay()"

The task should appear in Flower's task list within a few seconds, with state SUCCESS.

Explanation

Celery workers publish events (task sent, started, succeeded, failed, worker heartbeats) to the broker when events are enabled. Flower subscribes to that event stream, builds an in-memory model of workers and tasks, and serves it as a dashboard, an HTTP API, and Prometheus metrics. Because the same API can send control commands to workers, Flower is both a monitoring tool and an administrative interface, which is why it belongs behind authentication, HTTPS, and network restrictions.

Edge cases / notes

Multiple Flower instances

Each Flower instance keeps its own state. Run one per environment; for high availability of monitoring, rely on Prometheus rather than several Flower instances.

Upgrading Flower

Flower's options and API have changed between major versions (for example around authentication for API access). Pin the version and read release notes before upgrading.

When to automate this

Include Flower's unit, Nginx location, and credentials in provisioning, and wire its metrics into the same alerting system as the rest of the app. Dashboards are for investigating; alerts are what tell you to look.

Next steps

• Build metrics and alerts with Monitor Django with Prometheus and Grafana.
• Make queues meaningful to monitor with Route Celery Tasks to Separate Queues.
• Use Flower during incidents alongside Celery Not Running in Django Production: Debugging Guide.
• Review worker settings with the Celery Production Checklist for Django.

FAQ

Why does Flower show workers but no tasks?

Task events are disabled. Set CELERY_WORKER_SEND_TASK_EVENTS = True (and CELERY_TASK_SEND_SENT_EVENT = True) and restart the workers.

How do I secure Flower in production?

Bind it to localhost, serve it through Nginx over HTTPS, enable authentication, and restrict access by IP or an access proxy.

Can Flower alert me when tasks fail?

Use its Prometheus metrics with alert rules. The dashboard itself does not send alerts.

Does Flower keep task history after a restart?

Only with --persistent and a state database, and only up to --max-tasks. Use logs and metrics for long-term history.