Canary Releases for Django

Problem statement

Tests and staging catch many bugs, but some only appear with real traffic: a query that is slow on production data, an edge case in real user input, a memory leak under sustained load. With an all-at-once deploy, those bugs hit every user at the same moment.

A canary release sends a small share of real traffic (say 5 percent) to the new version while the rest stays on the current one. If the canary's error rate and latency stay healthy, you shift more traffic until it takes everything; if not, you send everyone back to the stable version, having affected only a few users. This guide sets up canaries for Django on servers with Nginx and outlines the equivalents on load balancers and Kubernetes.

1. Make the release backward compatible: migrations that both versions can run against, and no incompatible task or cache formats.
2. Run the new version next to the current one (a second Gunicorn service or container on another port or socket).
3. Split traffic in Nginx with split_clients, keeping each user on one version with a cookie.
4. Tag responses and logs with the version so you can compare error rates and latency per version.
5. Increase the canary share in steps (5 → 25 → 50 → 100 percent) while metrics stay within limits.
6. Roll back by setting the canary weight to zero; keep the stable version running until the canary is fully promoted.

Step-by-step solution

1. Prepare a compatible release

During a canary, two versions of your code run against the same database, cache, and task queue:

• Apply migrations before starting the canary, and make sure the stable version works with the migrated schema (add columns as nullable, keep old columns until a later release).
• Keep Celery task signatures compatible in both directions; stable web servers may enqueue tasks that canary workers run and vice versa.
• Version cache keys whose format changes, so the two versions do not read each other's incompatible cached data.

Releases that cannot meet these rules are not canary candidates; use a maintenance window or a feature flag instead.

2. Run two versions side by side

With release directories on one or more servers:

# /etc/systemd/system/gunicorn-canary.service
[Service]
User=myapp
WorkingDirectory=/srv/myapp/canary
EnvironmentFile=/etc/myapp.env
Environment=APP_VERSION=1.5.0
ExecStart=/srv/myapp/canary/.venv/bin/gunicorn config.wsgi:application --bind 127.0.0.1:8001 --workers 2
Restart=on-failure

The stable service keeps running from /srv/myapp/current on port 8000 with APP_VERSION=1.4.2. With containers, run two services from different image tags.

Expose the version in every response:

# config/middleware.py
import os

APP_VERSION = os.environ.get("APP_VERSION", "unknown")


class VersionHeaderMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        response["X-App-Version"] = APP_VERSION
        return response

3. Split traffic in Nginx

upstream django_stable { server 127.0.0.1:8000; keepalive 16; }
upstream django_canary { server 127.0.0.1:8001; keepalive 16; }

# Assign new visitors a bucket deterministically
split_clients "${remote_addr}${http_user_agent}${msec}" $canary_bucket {
    5%   canary;
    *    stable;
}

# Returning visitors keep their bucket; staff can force the canary with a cookie
map $cookie_release_bucket $release_pool {
    "canary"  canary;
    "stable"  stable;
    default   $canary_bucket;
}

server {
    # ...
    location / {
        add_header Set-Cookie "release_bucket=$release_pool; Path=/; Max-Age=3600; Secure; HttpOnly; SameSite=Lax" always;
        proxy_pass http://django_$release_pool;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

• split_clients hashes the key and assigns percentages. Including $msec randomises the first assignment; the cookie then keeps the visitor on the same version, so a user does not bounce between old and new behaviour on every click.
• Staff can test the canary deliberately by setting release_bucket=canary in their browser.
• To change the share, edit the percentage and reload Nginx (nginx -t && systemctl reload nginx). Existing cookies keep users in their bucket for the cookie lifetime.

Log the pool for analysis:

log_format canary '$remote_addr "$request" $status rt=$request_time pool=$release_pool';
access_log /var/log/nginx/access.log canary;

4. Compare the versions

Watch, per version, for 15 to 60 minutes at each step:

awk '{for(i=1;i<=NF;i++) if($i ~ /^pool=/) p=$i; if ($9 ~ /^5/) err[p]++; total[p]++} END {for (p in total) printf "%s total=%d 5xx=%d rate=%.3f%%\n", p, total[p], err[p], 100*err[p]/total[p]}' /var/log/nginx/access.log

(The status field position depends on your log format.) Better still, use your monitoring system: group error rate and latency percentiles by the X-App-Version header or by the release tag in your error tracker (Sentry's release option). Define limits before starting, for example: canary 5xx rate no more than 0.1 percentage points above stable, p95 latency no more than 20 percent higher, and no new error types.

5. Promote or roll back

• Promote: raise the percentage in steps. At 100 percent, point the stable service at the new release (switch current, restart stable), then stop the canary service and remove the split.
• Roll back: set the canary percentage to 0 and delete the bucket cookie by changing its name (or map canary to stable), reload Nginx, and stop the canary service. Only the canary share of users saw the problem.

Celery workers can follow the same pattern: run a small number of workers on the new version first, or keep workers on the stable version until the web canary is promoted, as long as task compatibility holds.

6. Platform equivalents

• AWS ALB: weighted target groups in a listener rule (for example 95/5), with target group stickiness.
• Kubernetes: two Deployments behind one Service in proportion to replica counts, ingress controller canary annotations (such as ingress-nginx's canary weight), or a progressive delivery controller (Argo Rollouts, Flagger) that shifts weights and rolls back automatically based on metrics.
• Cloud Run, App Engine, Container Apps: traffic splitting between revisions is built in.

Verification check:

for i in $(seq 1 200); do curl -s -o /dev/null -D - https://example.com/ | grep -i "^x-app-version"; done | sort | uniq -c
curl -s -o /dev/null -D - -b "release_bucket=canary" https://example.com/ | grep -i "^x-app-version"

Without cookies, roughly 5 percent of responses should come from the canary version; with the canary cookie, all of them.

Explanation

A canary is a controlled experiment on production: the same traffic mix, split between two versions, compared on the metrics that matter. Sticky assignment keeps each user's experience consistent, and version tagging makes the comparison possible. The approach limits the blast radius of bugs that only real traffic reveals, at the cost of requiring every release to be compatible with its predecessor, which is also what makes rolling updates and fast rollbacks safe.

Edge cases / notes

Low traffic

With a few requests per minute, 5 percent of traffic may produce too few requests to judge. Increase the share, extend the observation time, or use internal users (the forced cookie) as the first canary audience.

Canary versus feature flags

Canaries test the whole release (code, dependencies, configuration). Feature flags control individual features inside one release. They complement each other: canary the release, then roll out risky features gradually with flags.

When to automate this

Script the steps (start canary, set weight, check metrics against thresholds, promote or revert) so the rollout runs the same way every time. A progressive delivery tool automates the same loop on Kubernetes.

Next steps

• Compare with full-environment switching in Blue-Green Deployments for Django.
• Roll out features gradually with Use Feature Flags for Safer Django Releases.
• Keep releases compatible with Django Migration Strategy for Safe Deployments.
• Prepare the reverse path with How to Roll Back a Django Release Safely.

FAQ

What is a canary release?

Deploying a new version to a small share of real traffic first, comparing it with the current version, and expanding it only if it performs well.

How do I split traffic between two Django versions with Nginx?

Run both versions on different ports or sockets, use split_clients to assign a percentage to each, and pin visitors with a cookie.

Can database migrations run during a canary release?

Yes, if they are backward compatible, because both versions use the same database. Apply them before starting the canary.

When should I roll back a canary?

When its error rate, latency, or error types exceed the limits you defined before the rollout.