Speed Up Django collectstatic to S3

Problem statement

With STORAGES["staticfiles"] pointing at S3, collectstatic takes ten minutes or more on every deploy. Django checks each file's modification time with a remote request, post-processing reads every CSS file back from S3 to rewrite URLs, and hashed manifest storage uploads multiple copies. Deploys slow down, and running it on every server multiplies the cost. Most of that time is avoidable.

1. Collect to a local directory first, with a manifest storage that does post-processing on local disk.
2. Upload with aws s3 sync (or rclone sync), which skips unchanged files and runs uploads in parallel.
3. Or use a backend that skips unchanged files by comparing content hashes (django-storages with AWS_S3_FILE_OVERWRITE and preload metadata patterns, or packages like collectfasta).
4. Run collectstatic once per release in CI, not on every server or container start.
5. Set long cache headers on hashed files and keep old files for in-flight pages.

Step-by-step solution

1. Understand the slowness

For each static file, Django's collector asks the storage whether the file exists and when it was modified. With S3, each check is an HTTP request, so thousands of files mean thousands of round trips. Manifest storage then post-processes CSS and JS, reading files and saving hashed copies, which with remote storage means more downloads and uploads. All of this runs sequentially.

2. Collect locally, then sync

Keep STATIC_ROOT on local disk during the build and upload the result:

# settings/build.py
STORAGES = {
    "default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage"},
}
STATIC_ROOT = BASE_DIR / "staticfiles"

DJANGO_SETTINGS_MODULE=config.settings.build python manage.py collectstatic --noinput
aws s3 sync staticfiles/ s3://shop-static/static/ \
  --cache-control "public, max-age=31536000, immutable" \
  --exclude "staticfiles.json"
aws s3 cp staticfiles/staticfiles.json s3://shop-static/static/staticfiles.json --cache-control "no-cache"

aws s3 sync compares sizes and timestamps and uploads in parallel. Production settings then point STATIC_URL at the bucket or CDN, and use ManifestStaticFilesStorage with the manifest available to every app instance (shipped in the image is simplest).

3. Or keep a direct backend but skip work

If you must collect straight to S3, reduce per-file requests:

• Use a backend or package that compares file hashes or ETags in bulk instead of checking modification times one by one (for example collectfasta, which supports django-storages backends).
• Enable --ignore patterns for source maps or vendor files you do not serve.
• Avoid --clear on every deploy; it deletes everything and uploads again.

4. Run it once per release

Running collectstatic in each container's entrypoint means every replica uploads the same files on every start. Run it once in CI after building the image, or as a one-off release job. Hashed file names make concurrent releases safe: old pages keep using old files, new pages use new ones.

5. Keep old files and set headers

Do not delete previous releases' hashed files immediately; users with cached HTML from the previous release still request them. Clean up with an S3 lifecycle rule on old prefixes, or a periodic script that removes files not in the last few manifests. Long Cache-Control headers on hashed files and no-cache on the manifest let CDNs cache aggressively.

Explanation

Django's staticfiles app was designed around local file systems, where existence and timestamp checks are nearly free. With remote storage, each check becomes a network request, and manifest post-processing multiplies reads and writes. Doing the collection and hashing on local disk keeps that work fast, and tools built for bulk synchronization handle the upload efficiently with parallelism and change detection. Running it once per release removes duplicated work across servers.

Edge cases / notes

• Manifest location. With ManifestStaticFilesStorage, app instances need staticfiles.json to resolve names; shipping it in the image avoids a network fetch at startup.
• Content types. aws s3 sync guesses MIME types; check .js and .css get correct Content-Type, or set them explicitly.
• ACLs. New buckets disable ACLs; do not pass --acl public-read, use a bucket policy or CDN origin access instead.

Next steps

• Serve from a CDN with Serve Django Static Files With CloudFront.
• Understand hashed names in Django Static Files Cache Busting.
• Fix upload permission errors with Fix django-storages AccessControlListNotSupported.

FAQ

Why is collectstatic so slow with S3?

It makes a remote request for each file check and post-processes files remotely, all sequentially.

Can I collect locally and still serve from S3?

Yes. Collect with local storage, then sync the directory to the bucket and point STATIC_URL at it.

Should collectstatic run on every server?

No. Run it once per release in CI or a release job.

Should I use --clear?

Not on every deploy. It deletes and re-uploads everything and breaks pages still referencing old files.