Fix Docker "No Space Left on Device" When Building or Running Django

Problem statement

A deploy that has worked for months suddenly fails during the build or the pull:

ERROR: failed to solve: failed to copy files: copy file range failed: no space left on device

or while running:

OSError: [Errno 28] No space left on device: '/tmp/tmpk2j3...'
write /var/lib/docker/overlay2/.../merged/app/media/...: no space left on device

Docker keeps every image, build cache layer, stopped container, and volume until you remove them. Each deploy adds a new image of a few hundred megabytes, BuildKit stores cache for every build, and container logs grow without limit by default. On a 25 GB server, a few months of deploys is enough to fill /var/lib/docker.

1. See what uses space: docker system df -v.
2. Remove build cache: docker builder prune -f (add --all to remove everything).
3. Remove unused images: docker image prune -a --filter "until=168h" -f keeps the last week.
4. Check logs: du -sh /var/lib/docker/containers/*/*-json.log and set log rotation.
5. Never run docker volume prune without checking: it deletes volumes that hold database data or media.

Step-by-step solution

1. Measure first

df -h /var/lib/docker
docker system df

Typical output on a server that deploys daily:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          63        3         21.4GB    20.6GB (96%)
Containers      5         3         1.2GB     40MB (3%)
Local Volumes   4         3         6.1GB     0B (0%)
Build Cache     812       0         9.8GB     9.8GB

Here almost everything is reclaimable images and build cache. If Containers is large, logs or files written inside containers are the cause.

2. Prune build cache and old images

docker builder prune -f --filter "until=72h"
docker image prune -a -f --filter "until=168h"
docker container prune -f

image prune -a removes every image not used by a container, so the currently running release stays. The until filter keeps recent images for quick rollbacks.

3. Leave volumes alone unless you know what they are

docker volume ls
docker volume ls -f dangling=true

A dangling volume is not attached to any container, but that includes your Postgres data volume if the database container happens to be stopped. Inspect each one (docker volume inspect) before removing it by name. Never put docker system prune --volumes in a cron job.

4. Limit container logs

Without limits, the json-file driver writes forever. Set a default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "20m", "max-file": "5" }
}

Restart Docker and recreate containers so the limit applies. See Fix Docker Logs Filling the Disk for details.

5. Automate cleanup

Add a scheduled cleanup to servers and CI runners:

# /etc/cron.daily/docker-cleanup
#!/bin/sh
docker builder prune -f --filter "until=72h" >/dev/null
docker image prune -a -f --filter "until=168h" >/dev/null

Self-hosted CI runners need it most, because every pipeline builds images. Also check that your Django app is not writing uploads or temporary files inside the container's writable layer; those belong in a volume or object storage.

Explanation

Docker's storage driver keeps each image layer as a directory under /var/lib/docker/overlay2. New releases share base layers but add new ones for dependencies and code, so every deploy costs disk. BuildKit caches intermediate layers and downloaded packages to speed up builds, which is valuable but unbounded unless you configure garbage collection. Logs are plain files per container. None of this is cleaned up automatically, which is a deliberate choice: Docker cannot know which images you might want to roll back to.

Edge cases / notes

• Inode exhaustion. df -i may show 100 percent inodes with free space; many small files in layers or caches cause it. Pruning fixes it the same way.
• Separate disk. Moving /var/lib/docker to its own volume (data-root in daemon.json) protects the OS when Docker fills up.
• BuildKit GC. Docker Engine supports builder.gc settings in daemon.json to cap cache size automatically.

Next steps

• Reduce the size of each release with Optimize Django Docker Image Size.
• Keep builds fast after pruning with Speed Up Django Docker Builds With Cache.
• For non-Docker disk problems, see Fix Django Server Disk Full.

FAQ

Is docker system prune safe?

Without --volumes, it removes stopped containers, unused networks, dangling images, and build cache. It is reasonably safe but also removes images you might want for rollbacks.

Why does the build fail even after deleting files in the project?

The build uses Docker's storage, not your project folder. Prune Docker's images and cache.

How much disk should a Docker host have?

Enough for a handful of releases, build cache, and volumes, plus headroom. 40 GB or more is comfortable for a single Django app with daily deploys.

Will pruning slow down my next build?

The first build after removing the cache is slower. Keeping recent cache with an until filter limits the impact.