Send Django Logs to Amazon CloudWatch

Problem statement

A Django app on AWS writes logs to stdout in containers or to the journal on EC2, where they disappear when instances are replaced and cannot be searched across hosts. CloudWatch Logs collects them centrally, keeps them for a retention period you choose, and lets you query them with Logs Insights, turn patterns into metrics, and alarm on them. Getting there involves the right shipping mechanism for your compute, structured logs, retention settings so costs do not grow forever, and a few useful alarms.

1. Log JSON to stdout from Django, Gunicorn, and Celery.
2. ECS/Fargate: use the awslogs log driver in the task definition (log group per service).
3. EC2: install the CloudWatch agent and configure it to collect the journal or log files.
4. EKS: use Fluent Bit (the AWS for Fluent Bit image) as a DaemonSet.
5. Set retention on every log group (for example 30 or 90 days); the default is to keep logs forever.
6. Create metric filters and alarms for error rates, and use Logs Insights for investigations.

Step-by-step solution

1. Structured logs from Django

Configure a JSON formatter (see Structured JSON Logging for Django) so each line is an object with level, logger, message, request_id, status, and duration_ms. Logs Insights discovers JSON fields automatically.

2. ECS and Fargate

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/shop/web",
    "awslogs-region": "eu-west-1",
    "awslogs-stream-prefix": "web",
    "awslogs-create-group": "true",
    "mode": "non-blocking",
    "max-buffer-size": "25m"
  }
}

mode: non-blocking prevents a slow CloudWatch API from blocking the app's stdout writes (at the risk of dropping logs if the buffer fills). The task execution role needs logs:CreateLogStream and logs:PutLogEvents (and CreateLogGroup if auto-creating).

3. EC2 with the CloudWatch agent

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          { "file_path": "/var/log/nginx/access.log", "log_group_name": "/ec2/shop/nginx-access", "retention_in_days": 30 },
          { "file_path": "/var/log/shop/app.log", "log_group_name": "/ec2/shop/app", "retention_in_days": 90 }
        ]
      }
    }
  }
}

For apps logging to the journal, either write a log file as well or use a Fluent Bit or Vector agent that reads systemd-journald. The instance role needs the CloudWatchAgentServerPolicy.

4. Set retention

aws logs put-retention-policy --log-group-name /ecs/shop/web --retention-in-days 30

New log groups keep data indefinitely by default, and storage costs accumulate. Set retention in your infrastructure code for every group; export to S3 for long-term archives if needed.

5. Query with Logs Insights

fields @timestamp, level, logger, message, request_id, status, duration_ms
| filter level = "ERROR"
| sort @timestamp desc
| limit 50

filter logger = "http" | stats pct(duration_ms, 95) as p95, count(*) as requests by bin(5m)

Save useful queries for the team.

6. Metric filters and alarms

Create a metric filter on { $.level = "ERROR" } that counts errors into a CloudWatch metric, then an alarm when errors exceed a threshold for five minutes, notifying through SNS to email, Slack, or PagerDuty.

Verification check: log lines appear in the log group within a minute of a request, a Logs Insights query by request_id returns a request's lines, every log group shows a retention period, and a test error triggers the alarm.

Explanation

CloudWatch Logs organises data into log groups (one per application component) containing streams (one per container or instance). Shippers such as the awslogs driver, the CloudWatch agent, and Fluent Bit read stdout or files and call the PutLogEvents API. Logs Insights runs queries over the stored events and parses JSON fields automatically, which is why structured logs matter. Metric filters convert matching log events into metrics, letting you alarm on log patterns without a separate monitoring system.

Edge cases / notes

• Cost. Ingestion is the main cost; sample or drop verbose logs at the source and avoid logging full request bodies.
• Multiline tracebacks are kept together when logged as one JSON field; plain-text tracebacks may split into several events.
• Cross-account central logging can use subscription filters to stream to a logging account.
• Sensitive data in logs is subject to your compliance controls; consider data protection policies on log groups.

Next steps

• Make logs structured with Structured JSON Logging for Django.
• An alternative stack: Ship Django Logs to Grafana Loki.

FAQ

How do I send Django logs to CloudWatch?

Log to stdout and use the awslogs driver on ECS, the CloudWatch agent on EC2, or Fluent Bit on EKS.

Why are my CloudWatch costs growing?

Log groups default to never expiring, and verbose logs increase ingestion. Set retention and reduce noise.

Can I search Django logs by field in CloudWatch?

Yes, with Logs Insights, if logs are JSON.

How do I alert on Django errors in CloudWatch?

Create a metric filter on error-level log lines and an alarm on that metric.