Ship Django Logs With Vector

Problem statement

Your Django deployment writes logs from several places: Django and Gunicorn to stdout or journald, Nginx to files, Celery to its own stream, all on several servers. You want them in one searchable place with consistent fields, without running a heavy agent on every host. Vector is a lightweight, high-performance pipeline that reads logs from many sources, transforms them with its VRL language, and sends them to almost any destination.

1. Make Django log JSON to stdout (for example with python-json-logger or structlog).
2. Install Vector on each host and add sources: journald for systemd services, file for Nginx, or docker_logs for containers.
3. Parse JSON with a remap transform (parse_json!), add fields like service and env, and drop health-check noise.
4. Redact secrets and personal data before logs leave the host.
5. Send to your backend with the matching sink (Loki, Elasticsearch, Datadog, S3), with buffering for outages.

Step-by-step solution

1. Emit structured logs from Django

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {"()": "pythonjsonlogger.json.JsonFormatter",
                 "fmt": "%(asctime)s %(levelname)s %(name)s %(message)s"},
    },
    "handlers": {"console": {"class": "logging.StreamHandler", "formatter": "json"}},
    "root": {"handlers": ["console"], "level": "INFO"},
}

JSON logs are parsed reliably downstream; free-form text needs fragile regexes.

2. Configure Vector sources

# /etc/vector/vector.yaml
sources:
  django:
    type: journald
    include_units: ["shop-web.service", "shop-worker.service"]
  nginx_access:
    type: file
    include: ["/var/log/nginx/access.log"]

For Docker hosts, use type: docker_logs with label filters; for Kubernetes, the kubernetes_logs source reads pod logs with metadata.

3. Parse, enrich, and filter

transforms:
  django_parsed:
    type: remap
    inputs: ["django"]
    source: |
      parsed, err = parse_json(.message)
      if err == null { . = merge(., parsed) }
      .service = "shop"
      .env = "production"
      .host = get_hostname!()
  nginx_parsed:
    type: remap
    inputs: ["nginx_access"]
    source: |
      . = merge(., parse_nginx_log!(.message, "combined"))
  drop_health:
    type: filter
    inputs: ["nginx_parsed"]
    condition: '!starts_with(string!(.path), "/healthz")'

parse_nginx_log understands Nginx's combined format; adjust if you use a custom log_format.

4. Redact sensitive data

  redacted:
    type: remap
    inputs: ["django_parsed", "drop_health"]
    source: |
      .message = redact(string!(.message), filters: [r'(?i)(password|token|secret)=\S+', "us_social_security_number"])
      del(.authorization)

Do this before data leaves the host, so third-party log services never receive secrets.

5. Send to the backend

sinks:
  loki:
    type: loki
    inputs: ["redacted"]
    endpoint: "http://loki.internal:3100"
    labels: { service: "{{ service }}", env: "{{ env }}", host: "{{ host }}" }
    encoding: { codec: json }
    buffer: { type: disk, max_size: 268435488, when_full: block }

A disk buffer keeps logs during backend outages. Keep labels low-cardinality (service, env, host); put request IDs and user IDs in the log body, not labels.

Explanation

Vector is a single Rust binary that models log processing as a graph of sources, transforms, and sinks. Its Vector Remap Language (VRL) is a small, fail-safe language for parsing and reshaping events, so pipelines catch errors at configuration time. Running it on each host means Django needs to do nothing more than log JSON to stdout, and the choice of log backend can change without touching the application.

Edge cases / notes

• Multiline tracebacks. With JSON logging, tracebacks are a field inside one event; with plain text, configure multiline aggregation in the source.
• Resource use. Vector is light, but disk buffers need space; set sizes and monitor them.
• Kubernetes. Run Vector as a DaemonSet with the kubernetes_logs source to get pod metadata automatically.

Next steps

• Emit useful logs with Django Structured JSON Logging.
• Store them with Ship Django Logs to Grafana Loki.
• Keep costs down with Reduce Django Log Volume and Costs.

FAQ

Does Django need a special library for Vector?

No. Log JSON to stdout or a file; Vector collects it.

Can Vector redact secrets from logs?

Yes, with the redact function and field deletions in a remap transform, before logs leave the host.

What happens if the log backend is down?

With a disk buffer, Vector stores logs locally and sends them when the backend recovers.

Vector or Fluent Bit?

Both work. Vector's VRL makes complex transformations easier; Fluent Bit has a smaller footprint.