Ship Django Logs to Grafana Loki

Problem statement

When something goes wrong, the evidence is scattered: Django errors in the journal of one server, Nginx access logs in /var/log/nginx on two others, Celery output in a third place, and some of it already rotated away. Searching means SSHing into machines and running grep under pressure.

Grafana Loki stores logs centrally and indexes only a few labels per stream, which keeps it cheap to run. Combined with Grafana, you can search all services at once, filter by request ID, and graph error rates from log lines. This guide makes Django's logs structured, ships them with Grafana Alloy, and sets up useful queries.

1. Log JSON from Django, Gunicorn, and Celery to stdout (systemd captures it in the journal) with fields such as level, logger, message, and request ID.
2. Configure Nginx to write access logs as JSON.
3. Install Grafana Alloy on each host; read the journal for your units and Nginx log files, and forward to Loki.
4. Use only low-cardinality labels: service, env, host, and optionally level.
5. Query with LogQL in Grafana: filter by label, parse with | json, and aggregate with count_over_time.
6. Configure retention in Loki and keep logs free of secrets and unnecessary personal data.

Step-by-step solution

1. Structured logging in Django

pip install python-json-logger

# settings/production.py
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.json.JsonFormatter",
            "fmt": "%(asctime)s %(levelname)s %(name)s %(message)s %(request_id)s",
            "rename_fields": {"levelname": "level", "name": "logger", "asctime": "time"},
        },
    },
    "filters": {
        "request_id": {"()": "config.logging.RequestIDFilter"},
    },
    "handlers": {
        "stdout": {
            "class": "logging.StreamHandler",
            "formatter": "json",
            "filters": ["request_id"],
        },
    },
    "root": {"handlers": ["stdout"], "level": "INFO"},
    "loggers": {
        "django.request": {"handlers": ["stdout"], "level": "WARNING", "propagate": False},
    },
}

Older python-json-logger releases use pythonjsonlogger.jsonlogger.JsonFormatter. The request ID filter adds a per-request identifier from a contextvar set by middleware (or - outside requests), so every line from one request can be found together:

# config/logging.py
import contextvars
import logging

request_id_var = contextvars.ContextVar("request_id", default="-")


class RequestIDFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id_var.get()
        return True

Set request_id_var in a middleware from the X-Request-ID header (or a new UUID), and have Nginx forward its $request_id as that header so proxy and application logs share the ID.

2. JSON access logs in Nginx

log_format json_access escape=json
  '{"time":"$time_iso8601","remote_addr":"$remote_addr","method":"$request_method",'
  '"uri":"$request_uri","status":$status,"bytes":$body_bytes_sent,'
  '"request_time":$request_time,"upstream_time":"$upstream_response_time",'
  '"request_id":"$request_id","user_agent":"$http_user_agent"}';

access_log /var/log/nginx/access.json.log json_access;

Pass proxy_set_header X-Request-ID $request_id; to Django so both logs carry the same ID.

3. Install and configure Grafana Alloy

Grafana Alloy is Grafana's collector (it replaces Promtail, which has reached end of life). Install it from Grafana's package repository, then configure:

// /etc/alloy/config.alloy

loki.write "default" {
  endpoint {
    url = "https://loki.example.com/loki/api/v1/push"
    basic_auth {
      username = "logs"
      password = sys.env("LOKI_PASSWORD")
    }
  }
  external_labels = { env = "production", host = constants.hostname }
}

// Django, Gunicorn, and Celery via journald
loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "service"
  }
}

loki.source.journal "apps" {
  matches       = "_SYSTEMD_UNIT=gunicorn.service _SYSTEMD_UNIT=celery-worker.service"
  relabel_rules = loki.relabel.journal.rules
  forward_to    = [loki.write.default.receiver]
}

// Nginx JSON access log
local.file_match "nginx" {
  path_targets = [{"__path__" = "/var/log/nginx/access.json.log", "service" = "nginx"}]
}

loki.source.file "nginx" {
  targets    = local.file_match.nginx.targets
  forward_to = [loki.write.default.receiver]
}

The loki.relabel component here only defines rules; the journal source applies them while the internal __journal__* fields are still available, turning the unit name into a service label. Journal matches for different units in one string are combined as alternatives. The Alloy service user needs read access to the journal (membership in systemd-journal) and to the Nginx log directory (adm on Debian and Ubuntu).

sudo usermod -aG systemd-journal,adm alloy
sudo systemctl restart alloy
sudo journalctl -u alloy -f

For Docker hosts, loki.source.docker reads container logs in the same way.

4. Keep labels low-cardinality

Loki indexes labels, not content. Every unique label combination creates a separate stream; high-cardinality labels (user IDs, request IDs, URLs, IP addresses) create millions of tiny streams and make Loki slow and expensive. Good labels: service, env, host, and at most level. Everything else stays in the JSON line and is extracted at query time.

5. Query with LogQL

In Grafana's Explore view with the Loki data source:

{service="gunicorn.service", env="production"} | json | level="ERROR"

{service="nginx"} | json | status >= 500

{env="production"} |= "3f2c9a1e"

The last query finds every line containing a request ID across all services. Graph error rates from logs:

sum by (status) (count_over_time({service="nginx"} | json | status >= 500 [5m]))

Grafana alert rules can use such queries, for example to notify when 5xx responses exceed a threshold.

6. Retention and hygiene

Configure retention in Loki (for example 30 days) through its compactor and limits configuration, or your hosted provider's settings. Before shipping logs off the server, review what they contain: never log passwords, tokens, full request bodies, or session cookies, and minimise personal data such as email addresses.

Verification check:

logger -t test "loki pipeline check"   # appears only if you also ship this unit
curl -s -o /dev/null https://example.com/does-not-exist-check

In Grafana, query {service="nginx"} |= "does-not-exist-check" and a recent Django log line by level. Lines should appear within seconds, with the expected labels.

Explanation

Loki stores log lines in compressed chunks grouped into streams by label set, and only indexes the labels. Queries first select streams by label, then filter and parse line content, which keeps storage cheap but makes label choice critical. Structured JSON logs let LogQL extract fields at query time without indexing them. Alloy on each host reads the sources you already have (journald and log files), attaches a few labels, and pushes to Loki, so applications need no Loki-specific code.

Edge cases / notes

Multi-line tracebacks

JSON logging puts a traceback inside one JSON field, so it stays one log entry. Plain-text logging splits tracebacks across lines, which then need multi-line handling in the collector.

Local logs still matter

Keep journald and log rotation configured. If the network or Loki is down, local logs are the fallback, and Alloy resumes from its saved position when the connection returns.

When to automate this

Install and configure Alloy through your provisioning tool with the same labels on every host, keep the Django LOGGING dictionary in shared settings, and add a Grafana dashboard with error rate, top error loggers, and 5xx by endpoint as the default incident starting point.

Next steps

• Build the logging configuration with Django Logging Setup for Production.
• Add metrics alongside logs with Monitor Django with Prometheus and Grafana.
• Keep local log files bounded with How to Rotate Django and Gunicorn Logs on Linux.
• Check coverage with the Django Monitoring and Alerting Checklist.

FAQ

Should I use Promtail or Grafana Alloy to send logs to Loki?

Use Grafana Alloy for new setups. Promtail is deprecated and has reached end of life.

Which labels should Django logs have in Loki?

A few low-cardinality ones: service, environment, host, and possibly level. Keep request IDs and user data in the log line.

How do I find all logs for one request?

Add a request ID to every log line (and Nginx logs), then search for it with a line filter such as |= "<id>".

Can Loki alert on Django errors?

Yes. Grafana alert rules can evaluate LogQL metric queries, such as the rate of error-level lines or 5xx responses.