Fix High CPU Usage on a Django Server

Problem statement

The server sits at 100% CPU, pages take seconds, and the cloud dashboard sends alerts. Adding a bigger instance helps for a week. Before paying for more cores, find out which process is using the CPU and why: on a typical single-server Django stack the answer is one of Gunicorn workers running slow Python, PostgreSQL executing bad queries, Celery tasks doing heavy work, bots hammering expensive pages, or something unrelated such as a cryptominer on a compromised host.

1. Run top -c or htop sorted by CPU and identify the process type.
2. Gunicorn: attach py-spy top --pid <pid> to see which functions are hot.
3. PostgreSQL: check pg_stat_activity for running queries and pg_stat_statements for total time.
4. Celery: find which tasks are running and how often they are scheduled.
5. Check access logs for bursts from a few IPs or user agents and rate-limit them.
6. Unknown processes with random names: treat the server as compromised.

Step-by-step solution

1. Identify the process

top -c -o %CPU

Note the command lines: gunicorn: worker [config.wsgi], postgres: app appdb 10.0.0.5 SELECT, celery worker, nginx: worker process, or something unfamiliar. Check load against cores with nproc; a load average of 4 on a 4-core machine is full, not overloaded.

2. Gunicorn workers: profile live

py-spy samples a running Python process without restarting it:

pip install py-spy
sudo py-spy top --pid $(pgrep -f "gunicorn: worker" | head -1)
sudo py-spy record -o profile.svg --pid <pid> --duration 30

The flame graph shows where time goes. Typical findings:

• Template rendering of large lists, or many {% include %} calls in loops.
• Serialisation of big API responses without pagination.
• Python-side filtering of querysets that should be filtered in SQL.
• Password hashing on every API request with basic auth; use tokens or sessions.
• N+1 queries, which also show as many small database round trips; see Fix Django N+1 Queries.

3. PostgreSQL: find the expensive queries

SELECT pid, now() - query_start AS runtime, state, left(query, 120)
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY runtime DESC;

For the cumulative picture, pg_stat_statements ranks queries by total execution time; see Find Slow Django Queries With pg_stat_statements. Sequential scans on large tables (EXPLAIN ANALYZE shows Seq Scan) usually mean a missing index. Hundreds of identical fast queries per request mean N+1 in Django.

4. Celery: find heavy or runaway tasks

celery -A config inspect active
celery -A config inspect scheduled

Look for tasks that run far more often than intended (a Beat schedule of 1.0 seconds instead of crontab(minute=1)), retry storms from a failing external API, or CPU-heavy work such as image processing that should run with lower concurrency or on a separate machine.

5. Traffic and abuse

sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

A few IPs making thousands of requests, crawlers paging through search results, or credential-stuffing on the login view can saturate CPU. Rate-limit in Nginx (Rate Limit Django With Nginx), cache anonymous pages, and block abusive ranges.

6. Check for compromise

Processes with random names, binaries running from /tmp or /dev/shm, unknown cron entries, or outbound connections to mining pools mean the host is compromised, often via an exposed Redis, a weak SSH password, or DEBUG = True leaking secrets. Do not just kill the process: take the server off the network, rotate secrets, and rebuild from a clean image.

Verification check: after each change, compare CPU usage and response times at the same traffic level (same hour of the weekday). py-spy top should no longer show the hot function at the top, and pg_stat_statements totals for the fixed queries should drop.

Explanation

CPU time on a Django server is spent in three places: Python executing views and templates, the database executing queries, and the operating system handling network and TLS. Because they often share one machine, it is easy to blame the wrong one. Sampling profilers like py-spy attach to a live process and show exactly which Python functions consume time, without code changes or noticeable overhead, which makes them the fastest route from "CPU is high" to a specific line of code.

Edge cases / notes

• Steal time. On shared cloud instances, %st in top shows CPU taken by the hypervisor. Burstable instances (t-class) throttle when credits run out, which looks like high CPU with low throughput.
• Worker count. More Gunicorn workers than cores for CPU-bound views increases context switching without throughput gains.
• Compression. Gzip or Brotli at high levels on large dynamic responses costs CPU; use moderate levels and precompress static files.
• py-spy in containers needs SYS_PTRACE capability or running from the host.

Next steps

• Go deeper into endpoint profiling with Profile Slow Django Endpoints in Production.
• Rank database work with Find Slow Django Queries With pg_stat_statements.

FAQ

Is 100% CPU always bad?

Briefly during deploys or batch jobs, no. Sustained 100% during normal traffic means requests are queuing.

Should I add more Gunicorn workers?

Not if CPU is already saturated. More workers compete for the same cores.

Can I use py-spy safely in production?

Yes. It samples from outside the process with low overhead and does not modify it.

Why does PostgreSQL use so much CPU?

Usually sequential scans from missing indexes, or many repeated queries from N+1 patterns.