Run Django Migrations as a Kubernetes Job

Problem statement

On a single server, python manage.py migrate runs once in the deploy script. On Kubernetes, the obvious places to put it are wrong: in the container's command, every replica runs it on every start; in an init container, every new pod of a rolling update runs it, several at the same time. Django does not take a global lock around migrate, so concurrent runs can race to apply the same migration and fail halfway.

The reliable pattern is a Kubernetes Job that runs the new image's migrations once, a deploy step that waits for it to succeed, and only then a rollout of the new pods. This guide builds that pattern with plain manifests and kubectl, and shows the same idea in Helm and Argo CD.

1. Create a Job named after the release (for example myapp-migrate-3f9c2a1) using the new image and the same configuration as the web pods.
2. Set restartPolicy: Never, backoffLimit: 0, activeDeadlineSeconds, and ttlSecondsAfterFinished.
3. Set a PostgreSQL lock_timeout so a blocked migration fails fast instead of stalling traffic.
4. In CI, apply the Job, wait for completion, print logs, and fail the pipeline if it fails.
5. Only after success, update the Deployment's image.
6. Keep migrations backward compatible, because old pods serve traffic against the new schema during the rollout.

Step-by-step solution

1. The Job manifest

# k8s/migrate-job.yaml (rendered by CI with the image tag)
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp-migrate-${TAG}
  namespace: myapp
  labels: { app: myapp, component: migrate }
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 86400
  template:
    metadata:
      labels: { app: myapp, component: migrate }
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/example/myapp:${TAG}
          command: ["python", "manage.py", "migrate", "--noinput"]
          envFrom:
            - configMapRef: { name: myapp-config }
            - secretRef: { name: myapp-secrets }
          env:
            - name: PGOPTIONS
              value: "-c lock_timeout=5s -c statement_timeout=0"
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }

• One Job per release, named with the tag. Job templates are immutable, so reusing one name for different images requires deleting the old Job first; unique names also leave a readable history.
• backoffLimit: 0: a failed migration should not retry automatically. Rerunning a partially applied non-atomic migration can make things worse; a human should look at it.
• activeDeadlineSeconds caps runtime; ttlSecondsAfterFinished cleans up finished Jobs after a day.
• PGOPTIONS is read by libpq (and therefore psycopg) and sets session parameters. lock_timeout=5s makes a migration fail if it cannot acquire a table lock within five seconds, instead of queueing behind a long transaction while every request on that table queues behind the migration.

2. Apply and wait in CI

#!/usr/bin/env bash
set -euo pipefail
TAG="$1"
NS=myapp
JOB="myapp-migrate-${TAG}"

export TAG
envsubst < k8s/migrate-job.yaml | kubectl apply -f -

# Wait for completion or failure, whichever comes first
for _ in $(seq 1 180); do
  succeeded=$(kubectl -n "$NS" get job "$JOB" -o jsonpath='{.status.succeeded}')
  failed=$(kubectl -n "$NS" get job "$JOB" -o jsonpath='{.status.failed}')
  if [ "${succeeded:-0}" -ge 1 ]; then break; fi
  if [ "${failed:-0}" -ge 1 ]; then
    kubectl -n "$NS" logs "job/$JOB" --all-containers || true
    echo "Migration job failed" >&2
    exit 1
  fi
  sleep 5
done
[ "${succeeded:-0}" -ge 1 ] || { echo "Migration job timed out" >&2; exit 1; }

kubectl -n "$NS" logs "job/$JOB"
kubectl -n "$NS" set image deployment/myapp web="ghcr.io/example/myapp:${TAG}"
kubectl -n "$NS" rollout status deployment/myapp --timeout=10m

kubectl wait --for=condition=complete alone would wait for the full timeout when the Job fails, because a failed Job never becomes complete. Polling both counters fails fast. Newer kubectl versions can express this with a JSONPath condition, but the loop works everywhere.

3. Why not an init container?

An init container on the web Deployment runs in every new pod. During a rolling update with several replicas, two or more pods start at once and each runs migrate. Django checks the django_migrations table before applying each migration, but that check and the apply are not protected by a cluster-wide lock, so two processes can apply the same migration concurrently. Scaling up or pods being rescheduled also reruns it. It can work for single-replica apps, but a Job is the correct model.

If you want an extra safety net against accidental concurrent runs, wrap migrate in a command that takes a PostgreSQL advisory lock first:

# myapp/management/commands/migrate_locked.py
from django.core.management import call_command
from django.core.management.base import BaseCommand
from django.db import connection

LOCK_ID = 727274  # any constant shared by all migration runs


class Command(BaseCommand):
    def handle(self, *args, **options):
        with connection.cursor() as cursor:
            cursor.execute("SELECT pg_advisory_lock(%s)", [LOCK_ID])
            try:
                call_command("migrate", interactive=False)
            finally:
                cursor.execute("SELECT pg_advisory_unlock(%s)", [LOCK_ID])

4. Helm and Argo CD equivalents

• Helm: the same Job with helm.sh/hook: pre-install,pre-upgrade and a hook delete policy. Helm waits for it and aborts the upgrade on failure.
• Argo CD: the same Job with argocd.argoproj.io/hook: PreSync. The sync stops if it fails.

Both run the Job before the Deployment changes, which is the property that matters.

5. Handle failures

When the Job fails:

1. Read the logs: kubectl -n myapp logs job/myapp-migrate-<tag>.
2. Check which migrations applied: run python manage.py showmigrations in a pod with the new image, for example with a debug Job or kubectl run --rm -it.
3. On PostgreSQL, most schema migrations run in a transaction and roll back completely on failure. Migrations with atomic = False or operations like CREATE INDEX CONCURRENTLY can leave partial state that must be cleaned up before a retry.
4. Fix forward with a corrected migration, or revert the offending commit. The old pods are still running the previous version, so users are unaffected while you investigate.

Verification check:

kubectl -n myapp get jobs -l component=migrate
kubectl -n myapp get deployment myapp -o jsonpath='{.spec.template.spec.containers[0].image}'

The latest Job should show 1/1 completions for the same tag the Deployment is running.

Explanation

Kubernetes Deployments are designed for interchangeable, repeatable processes; migrations are the opposite: a one-time, ordered change to shared state. A Job gives migrations their own lifecycle (run once, report success or failure) and lets the deploy pipeline express the dependency "schema first, then code" explicitly. The lock timeout protects the running application from a migration that would otherwise block production queries while waiting for a lock.

Edge cases / notes

Long-running data migrations

Backfills over millions of rows do not belong in the release-blocking Job. Ship them as separate management commands run as their own Jobs, in batches, after the schema change is deployed.

Database credentials

The migration Job may need more privileges (DDL) than the web pods. Some teams give it a separate Secret with an owner role, while the application runs with a role limited to data changes.

When to automate this

The script above is the automation; keep it in the repository and run it from CI for every environment. Add a notification with the Job logs on failure, so the person on call sees the error without opening a cluster shell.

Next steps

• Review the full cluster setup in Deploy Django on Kubernetes.
• Package the Job as a hook with Deploy Django to Kubernetes with a Helm Chart.
• Write migrations that survive rolling updates with Django Migration Strategy for Safe Deployments.
• If migrations stall on locks, see Fix Django Migrations Hanging on a Database Lock.

FAQ

Should Django migrations run in an init container?

Not with more than one replica. Every new pod runs the init container, so migrations can run concurrently. Use a Job that runs once per release.

How do I make CI wait for a Kubernetes migration Job?

Apply the Job, then poll its succeeded and failed status (or use kubectl wait with a condition) and fail the pipeline if it fails or times out.

Why set backoffLimit to 0 for migrations?

Automatic retries of a failed migration can repeat a partial change. A failure should stop the release and be investigated.

Is it safe to run migrate while old pods are still serving?

Yes, if migrations are backward compatible, for example adding nullable columns or new tables, and removals happen in a later release.