Copy Production Data to Staging Safely (Anonymised)

Problem statement

Bugs that only appear with real data, slow queries that only show at production scale, and migrations that only fail on production's messy rows all argue for testing on production data. Copying the production database to staging as-is, though, puts customers' names, emails, addresses, and order history on a less protected system, where it may violate GDPR and your own policies, and where staging can accidentally email real customers. The safe middle ground is a copy that keeps the shape and volume of production data while replacing personal information.

1. Restore a production backup into an isolated, short-lived database (not staging itself).
2. Run an anonymisation step that overwrites personal fields with fake but valid values, deterministically per row.
3. Remove or scramble secrets: API tokens, OAuth tokens, payment customer IDs, password hashes (set a known staging password).
4. Optionally subset huge tables (last 90 days of orders, a sample of users) while keeping foreign keys consistent.
5. Dump the anonymised database and restore it into staging; destroy the intermediate copy.
6. In staging, disable real email, SMS, payments, and webhooks; use sandbox credentials.

Step-by-step solution

1. Decide what is personal

List models and fields with personal or sensitive data: User (name, email, last login IP), profiles, addresses, orders (shipping details), support messages, uploaded documents, audit logs, and free-text fields where people type anything. Also secrets: API keys, tokens, Stripe customer IDs, webhook secrets stored in the database.

2. Anonymise in an isolated database

# core/management/commands/anonymize.py
import hashlib
from django.conf import settings
from django.contrib.auth import get_user_model
from django.contrib.auth.hashers import make_password
from django.core.management.base import BaseCommand, CommandError
from django.db import transaction
from faker import Faker

class Command(BaseCommand):
    def handle(self, *args, **opts):
        if settings.ENVIRONMENT == "production":
            raise CommandError("refusing to anonymise production")
        fake = Faker()
        staging_password = make_password("staging-only")
        User = get_user_model()
        with transaction.atomic():
            for user in User.objects.iterator(chunk_size=2000):
                Faker.seed(int(hashlib.sha256(str(user.pk).encode()).hexdigest()[:8], 16))
                user.first_name, user.last_name = fake.first_name(), fake.last_name()
                user.email = f"user{user.pk}@example.test"
                user.password = staging_password
                user.save(update_fields=["first_name", "last_name", "email", "password"])
            Address.objects.update(street="1 Example Street", phone="")
            SupportMessage.objects.update(body="[redacted]")
            ApiToken.objects.all().delete()

Seeding per primary key makes results deterministic, so repeated refreshes produce the same fake names. Use the .test or example.com domains for emails so nothing can be delivered. The production guard prevents disasters if the command is run in the wrong place.

3. Subset large tables

Full copies of huge tables slow staging and cost money. Keep recent data and maintain integrity:

DELETE FROM analytics_event WHERE created_at < now() - interval '30 days';
DELETE FROM orders_order WHERE created_at < now() - interval '180 days';

Delete in dependency order (children first) or rely on ON DELETE CASCADE where defined. Tools such as pg_sample or Jailer can extract consistent subsets.

4. Make staging harmless

# staging settings
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"   # or Mailpit SMTP
STRIPE_API_KEY = env("STRIPE_TEST_KEY")
CELERY_TASK_ROUTES = {...}            # disable tasks that call partners

Even anonymised data can contain leftovers (an email in a free-text note); a staging email backend that never delivers externally is the backstop.

5. Automate the refresh

Run the pipeline weekly in CI or a scheduled job with access to backups: restore, anonymise, dump, upload, then restore into staging and run migrations for the current staging code. Record when staging data was refreshed.

Verification check: after a refresh, search staging for a few real customer emails from production (User.objects.filter(email__iendswith="@gmail.com").exists() should be false), confirm no API tokens remain, and send a test email to verify it is captured rather than delivered.

Explanation

Realistic data improves testing because volume, distributions, and edge cases matter for performance and correctness. Personal data, however, is regulated and risky outside production's controls. Anonymisation keeps the structure (row counts, relationships, field lengths, distributions) while replacing identifying values, ideally deterministically so data is stable across refreshes. Doing it in an isolated intermediate database means raw production data never lands on the less protected staging system.

Edge cases / notes

• Pseudonymisation vs anonymisation. Deterministic fakes keyed by ID are pseudonymous if someone can link them back; avoid copying production IDs into systems with weaker access if that matters.
• Uploaded files (media) contain personal data too; replace them with placeholders rather than copying buckets.
• Search indexes and caches built from staging data must be rebuilt after a refresh.
• Tooling such as PostgreSQL Anonymizer or Neon branches with masking can do this at the database level.

Next steps

• Build the environment with Set Up a Django Staging Environment.
• Check your obligations with the Django Privacy and GDPR Production Checklist.

FAQ

Is it OK to copy production data to staging?

Only after removing or replacing personal data and secrets, and with staging locked down.

How do I anonymise data in Django?

Restore into an isolated database and run a management command that overwrites personal fields with fake values.

Should anonymisation be deterministic?

It helps: the same user gets the same fake name each refresh, which keeps test scenarios stable.

How do I stop staging emailing real users?

Use a non-delivering email backend in staging and fake email addresses in the data.