Run a Django Data Migration on a Large Table

Problem statement

You need to fill a new column for 40 million orders, normalize phone numbers in all customers, or move data from one model to another. Written as a RunPython migration that loops over Model.objects.all(), it holds a transaction open for an hour, locks rows the app needs, generates huge WAL, blocks the deploy pipeline, and if it fails at 95 percent, rolls everything back. Large data changes need to run in batches, outside the deploy's critical path, and be resumable.

1. Ship schema changes that are cheap and backward compatible first (add a nullable column, a new table).
2. Update the code so new and changed rows are written correctly from now on.
3. Backfill old rows in batches by primary key, in a management command or Celery task that records progress and can resume.
4. Verify completion with a count, then add constraints (NOT NULL, unique) in a later migration.
5. Keep RunPython in migrations only for small tables, and mark migrations atomic = False if they batch.

Step-by-step solution

1. Split the change into phases

For "add country_code to customers computed from phone":

1. Migration: add country_code as nullable.
2. Code: compute and save country_code whenever a customer is created or the phone changes.
3. Backfill: compute it for existing customers in batches.
4. Migration: make it NOT NULL and add an index if needed.

Each phase is deployable on its own, and the app works at every point.

2. Write a resumable backfill command

# customers/management/commands/backfill_country_code.py
import time
from django.core.management.base import BaseCommand
from django.db import transaction
from customers.models import Customer
from customers.phones import country_from_phone

class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument("--batch", type=int, default=2000)
        parser.add_argument("--start-id", type=int, default=0)
        parser.add_argument("--sleep", type=float, default=0.1)

    def handle(self, batch, start_id, sleep, **opts):
        last_id = start_id
        while True:
            rows = list(Customer.objects.filter(pk__gt=last_id, country_code__isnull=True)
                        .order_by("pk").only("pk", "phone")[:batch])
            if not rows:
                break
            for c in rows:
                c.country_code = country_from_phone(c.phone)
            with transaction.atomic():
                Customer.objects.bulk_update(rows, ["country_code"], batch_size=batch)
            last_id = rows[-1].pk
            self.stdout.write(f"updated up to id {last_id}")
            time.sleep(sleep)

Filtering on country_code__isnull=True makes reruns skip finished rows, so the command can be stopped and restarted anytime. Log last_id so you can resume from it after interruptions.

3. Batch inside migrations when you must

For moderately large tables where a migration is convenient, batch inside RunPython with atomic = False:

class Migration(migrations.Migration):
    atomic = False
    dependencies = [("customers", "0031_customer_country_code")]

    def backfill(apps, schema_editor):
        Customer = apps.get_model("customers", "Customer")
        qs = Customer.objects.filter(country_code__isnull=True).order_by("pk")
        while ids := list(qs.values_list("pk", flat=True)[:5000]):
            Customer.objects.filter(pk__in=ids).update(country_code="US")

    operations = [migrations.RunPython(backfill, migrations.RunPython.noop)]

Use historical models from apps.get_model, not imports from your current code, because the code may change later while the migration must keep working.

4. Use SQL for simple transformations

When the new value can be computed in SQL, a batched UPDATE is much faster than loading rows into Python:

from django.db import connection
with connection.cursor() as cur:
    cur.execute("""
        UPDATE orders_order SET total_cents = (total * 100)::bigint
        WHERE id BETWEEN %s AND %s AND total_cents IS NULL
    """, [low, high])

Loop over id ranges of a fixed size rather than LIMIT, so each batch touches a predictable number of rows.

5. Watch the database while it runs

Monitor replica lag, WAL generation, autovacuum activity, and lock waits. Increase the sleep between batches if replicas fall behind. Run ANALYZE on the table after a large backfill so the planner knows about the new values.

Explanation

PostgreSQL holds row locks and keeps old row versions until a transaction ends, so one giant data migration transaction blocks concurrent updates to those rows and prevents vacuum from cleaning up. Batching commits frequently, releasing locks and letting vacuum and replication keep pace. Moving the backfill out of the migration means deploys stay fast and a failure halfway leaves finished batches committed, ready to resume. Separating the constraint step ensures the database only enforces rules once every row satisfies them.

Edge cases / notes

• Signals and save(). bulk_update and update() bypass save() and signals; if other logic depends on them, call it explicitly or use a slower path.
• Concurrent edits. Rows edited by users during the backfill already get correct values from the new code; the IS NULL filter skips them.
• Large moves between tables. Copy in batches, switch reads, then switch writes, keeping both in sync during the transition.

Next steps

• Finish with constraints safely using Fix Django Column Contains Null Values.
• Change schemas without locks with Run Django Schema Changes on Large Tables.
• Speed up writes with Django bulk_create and bulk_update Performance.

FAQ

Should data migrations run inside Django migrations?

For small tables, yes. For large tables, run batched management commands separately from the deploy.

Why use apps.get_model in RunPython?

It gives the historical model as of that migration, so the migration keeps working when the current model changes.

How do I make a backfill resumable?

Filter on rows that still need the change, process in primary key order, and log the last processed ID.

When can I add NOT NULL to the backfilled column?

After verifying that no NULL rows remain and that the code writes the value for every new row.