Rotate the Django Database Password Without Downtime

Problem statement

Your security policy requires rotating the database password periodically, an employee with access left, or the password appeared somewhere it should not. Changing the password with ALTER ROLE instantly breaks every Django process still configured with the old one when it next opens a connection: web workers, Celery, cron jobs, and scripts. Rotation without downtime needs a moment where both old and new credentials work.

1. Use two database roles that alternate (shop_app_a, shop_app_b) and share privileges through a group role.
2. Set a new password on the inactive role, update the secret to that role, and roll all processes.
3. When no connections use the old role (check pg_stat_activity), change or disable its password.
4. Include every consumer: Gunicorn, Celery workers and beat, cron, PgBouncer's auth file, monitoring, and migrations jobs.
5. Automate with AWS Secrets Manager's alternating-users rotation, or avoid passwords entirely with IAM or Entra authentication.

Step-by-step solution

1. Set up alternating roles

CREATE ROLE shop_app NOLOGIN;                      -- group with privileges
GRANT CONNECT ON DATABASE shop TO shop_app;
GRANT USAGE ON SCHEMA public TO shop_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO shop_app;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO shop_app;

CREATE ROLE shop_app_a LOGIN PASSWORD '...' IN ROLE shop_app;
CREATE ROLE shop_app_b LOGIN PASSWORD '...' IN ROLE shop_app;

Ownership matters for migrations: objects should be owned by a role that both login roles can act as (for example shop_owner, with migrations run as a member using SET ROLE), or new tables created by one login role will not be accessible to the other. Set ALTER DEFAULT PRIVILEGES accordingly.

2. Rotate to the inactive role

ALTER ROLE shop_app_b PASSWORD 'new-strong-password';

Update the secret (Secrets Manager, Vault, Kubernetes Secret, EnvironmentFile) so DATABASE_URL uses shop_app_b. Nothing breaks yet: running processes keep their existing connections and credentials.

3. Roll every process

Restart or redeploy everything that reads the secret:

sudo systemctl restart shop-web shop-celery shop-beat
kubectl -n prod rollout restart deployment/web deployment/worker

Cron jobs and one-off scripts pick up the new value on their next run if they read the same secret source. PgBouncer with auth_file needs its file updated and a RELOAD; with auth_query, it looks passwords up in the database automatically.

4. Confirm and revoke

SELECT usename, application_name, client_addr, count(*)
FROM pg_stat_activity
WHERE usename IN ('shop_app_a', 'shop_app_b')
GROUP BY 1, 2, 3;

When no sessions remain for shop_app_a, set it a random password nobody knows, or ALTER ROLE shop_app_a NOLOGIN until the next rotation. Any process that still fails is one you missed; find it from the connection error logs.

5. Automate or remove passwords

AWS Secrets Manager offers an "alternating users" rotation strategy for RDS that implements this pattern with a Lambda function on a schedule. Better still, remove static passwords: RDS IAM authentication, Cloud SQL IAM, and Azure Entra authentication issue short-lived tokens tied to the workload identity, so there is nothing to rotate.

Explanation

PostgreSQL checks passwords only when a connection is established; existing connections stay open after a password change. With a single login role, a password change immediately breaks any process that opens a new connection with the old credential, which happens constantly with connection churn and restarts. Two login roles with the same privileges let the old credential stay valid while processes switch, so the change becomes a rolling update instead of a cutover.

Edge cases / notes

• MySQL. MySQL 8.0.14+ supports dual passwords per account (ALTER USER ... RETAIN CURRENT PASSWORD), which achieves the same without a second user.
• Persistent connections. With CONN_MAX_AGE, processes keep old-role connections until restart; that is fine as long as you restart before revoking.
• Leaked password. For an active leak, rotate immediately and revoke the old credential quickly, accepting brief errors.

Next steps

• Rotate the Django key too with Rotate the Django SECRET_KEY Safely.
• Follow the incident playbook in Respond to Leaked Django Secrets.
• Eliminate passwords with Connect Django to AWS RDS With IAM Authentication.

FAQ

Why does changing the password break the app?

New connections with the old password are rejected immediately. Processes open new connections all the time.

Do existing connections drop after ALTER ROLE PASSWORD?

No. Only new connections are checked. That is why rolling processes before revoking works.

Can MySQL rotate without a second user?

Yes. MySQL 8.0.14+ supports dual passwords with RETAIN CURRENT PASSWORD.

How do I avoid rotating passwords at all?

Use IAM or Entra database authentication with short-lived tokens tied to the workload identity.