Django Incident Response Runbook

Problem statement

When the site goes down at 22:00, the damage comes as much from confusion as from the fault: three people SSH in and restart different things, nobody tells customers, the fix that worked last time is in someone's memory, and after recovery no one writes down what happened, so it recurs.

A runbook replaces improvisation with a short, rehearsed procedure: who does what, which checks to run in which order, how to mitigate before diagnosing, how to communicate, and how to learn afterwards. This one is written for small teams running Django with Nginx, Gunicorn, PostgreSQL, Redis, and Celery, and can be adapted to containers or PaaS.

1. Declare the incident, assign one incident lead, open one channel, and start a timeline.
2. Assess impact: what is broken, for whom, since when, and did anything change (deploy, config, traffic)?
3. Triage in order: edge and proxy → app servers → workers → database → cache and broker → external dependencies.
4. Mitigate first: roll back the last deploy, disable the failing feature, scale, or put up maintenance mode.
5. Communicate on a schedule (status page, customers, internal) even when there is no news.
6. After recovery, verify, keep monitoring, and hold a blameless review with owned follow-up actions.

Checklist

1. Declare

• One person is the incident lead: they coordinate and decide, not necessarily fix. Others investigate on their request.
• One channel for the incident; decisions and findings are posted there.
• A timeline starts immediately: time of first alert, actions taken, observations, with timestamps (UTC).
• Severity is set: for example SEV1 (site down or data at risk), SEV2 (major feature broken), SEV3 (degraded).

2. Assess impact and recent changes

• What do users see (errors, slowness, wrong data), which pages or APIs, which regions or customers?
• When did it start? Match against the deploy log, configuration changes, cron schedules, traffic spikes, and provider status pages.
• Is data at risk (corruption, loss, exposure)? If yes, protect data first: stop writes or take a snapshot before experimenting.

3. Triage, outside in

Edge and proxy

curl -sI https://example.com/ | head -5
curl -sI https://example.com/healthz/
sudo tail -n 50 /var/log/nginx/error.log

Certificate expiry, DNS, CDN, or load balancer errors show up here. 502 points at the app being down or unreachable; 504 at slowness.

Application servers

sudo systemctl status gunicorn --no-pager
sudo journalctl -u gunicorn --since "30 min ago" | tail -50
uptime; free -h; df -h

Look for crash loops, OOM kills (journalctl -k | grep -i oom), full disks, and CPU saturation. Check the error tracker for new exceptions since the incident started.

Workers

sudo systemctl status celery-worker celery-beat --no-pager
celery -A config inspect ping --timeout 5
redis-cli -u "$CELERY_BROKER_URL" LLEN celery

Database

SELECT count(*), state FROM pg_stat_activity GROUP BY state;
SELECT pid, now() - query_start AS runtime, wait_event_type, left(query, 80)
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY runtime DESC LIMIT 10;

Connection exhaustion, lock chains, long queries, replication lag, and disk space are the usual database causes.

Cache and broker

redis-cli -u "$REDIS_URL" INFO memory | grep -E "used_memory_human|maxmemory_human"
redis-cli -u "$REDIS_URL" PING

External dependencies: payment providers, email services, storage, and APIs. Check their status pages and your outbound error rates.

4. Mitigate before diagnosing

Choose the fastest action that is safe and reversible:

• Recent deploy? Roll back the code (previous release, image tag, or revision). Keep backward-compatible migrations in place.
• One feature failing? Disable it with a feature flag or configuration switch.
• Overloaded? Scale web or worker capacity, enable caching for anonymous pages, rate limit abusive traffic, or block a bad client at the edge.
• Database overwhelmed? Cancel runaway queries (pg_cancel_backend), pause heavy background jobs, move reports off the primary.
• Disk full? Free safe space (journal, rotated logs, build caches) before restarting services.
• Nothing safe works quickly? Maintenance mode with an honest message is better than a half-working site.

Record every action in the timeline before or as you take it.

5. Communicate

• Post a first update within minutes: what is affected, that you are working on it, when the next update comes.
• Update on a fixed interval (for example every 30 minutes for SEV1), even to say "still investigating".
• Tell internal stakeholders (support, sales, management) what to say to customers.
• Announce resolution, and later a short summary for affected customers when appropriate.

6. Recover and verify

• Confirm recovery with the same signals that detected the problem (error rate, latency, health checks, queue lengths).
• Check for side effects: backlog of Celery tasks, failed payments or emails that need replaying, data that needs correction.
• Keep watching for at least an hour; many incidents return once traffic or a scheduled job comes back.

7. Review

• Within a few days, write a blameless review: timeline, impact, root cause and contributing factors, what went well, what was hard.
• Follow-up actions have owners and dates: monitoring gaps, missing runbook steps, tests, capacity changes, process fixes.
• Update this runbook with anything that would have made the incident shorter.

Explanation

Incidents are stressful and time-critical, which is exactly when memory and judgement are least reliable. A runbook moves decisions to calm moments: roles are predefined, checks come in a fixed order, and mitigation options are listed with their risks. Mitigating before root-causing shortens user impact, and a structured review turns each incident into specific improvements instead of a vague resolution to "be more careful".

Edge cases / notes

Security incidents

If you suspect a compromise (unknown admin users, unexpected outbound traffic, leaked credentials), switch to a security response: preserve evidence (snapshots, logs) before cleaning up, rotate credentials, and follow data breach obligations.

Single-person teams

Solo operators still benefit: the checklist and timeline prevent tunnel vision, and a status update template saves time. Set a timer to post updates.

When to automate this

Link alerts directly to the relevant runbook section, script the triage commands into one "incident snapshot" script that collects the outputs above, and automate rollback so the most common mitigation is one command.

Next steps

• Make sure incidents are detected quickly with the Django Monitoring and Alerting Checklist.
• Practise the most common mitigation with How to Roll Back a Django Release Safely.
• Prepare for worst cases with Write a Disaster Recovery Plan for a Django App.
• For a typical first symptom, see Fix Django 500 Internal Server Error in Production.

FAQ

What should I do first when a Django site goes down?

Declare an incident, assign a lead, check what changed recently, and apply the fastest safe mitigation (often rolling back the last deploy).

Should I find the root cause before restoring service?

No. Restore service first with a safe mitigation, then investigate. Preserve logs and evidence while doing so.

What goes into an incident review?

A timeline, impact, root cause and contributing factors, what went well and badly, and follow-up actions with owners and dates.

How often should incident updates be posted?

On a fixed schedule appropriate to severity, for example every 30 minutes for a major outage, even without new information.