Test and Deploy Django With Buildkite

Problem statement

Your team uses Buildkite: the orchestration is hosted, but builds run on your own agents, inside your network, on hardware you choose. A Django pipeline needs a database for tests, Docker image builds with good caching, a manual approval before production, a migration step, and secrets that stay on the agents rather than in YAML.

1. Keep .buildkite/pipeline.yml in the repo, uploaded by a minimal pipeline step (buildkite-agent pipeline upload).
2. Run tests with the Docker Compose plugin and a docker-compose.ci.yml that includes PostgreSQL.
3. Build and push the image tagged with $BUILDKITE_COMMIT, using registry cache.
4. Add a block step before production, then a deploy step that runs migrations and updates the service.
5. Inject secrets with agent environment hooks or a secrets plugin, never in pipeline.yml.

Step-by-step solution

1. Define the pipeline in the repo

# .buildkite/pipeline.yml
steps:
  - label: ":python: tests"
    plugins:
      - docker-compose#v5.5.0:
          run: web
          config: docker-compose.ci.yml
    command: ["python", "manage.py", "test", "--parallel", "--noinput"]

  - label: ":docker: build"
    command: |
      docker buildx build --push \
        --cache-from type=registry,ref=ghcr.io/acme/shop:buildcache \
        --cache-to type=registry,ref=ghcr.io/acme/shop:buildcache,mode=max \
        -t ghcr.io/acme/shop:${BUILDKITE_COMMIT} .
    branches: main

  - wait

  - block: ":rocket: Deploy to production?"
    branches: main

  - label: ":shipit: deploy"
    command: scripts/deploy.sh "${BUILDKITE_COMMIT}"
    branches: main
    concurrency: 1
    concurrency_group: shop-prod-deploy

concurrency: 1 with a group ensures two deploys never run at once. Pin plugin versions.

2. Test with PostgreSQL

# docker-compose.ci.yml
services:
  db:
    image: postgres:17
    environment: { POSTGRES_PASSWORD: postgres, POSTGRES_DB: shop }
    healthcheck: { test: ["CMD-SHELL", "pg_isready -U postgres"], interval: 2s, retries: 20 }
  web:
    build: .
    environment:
      DATABASE_URL: postgres://postgres:postgres@db:5432/shop
      DJANGO_SETTINGS_MODULE: config.settings.test
    depends_on: { db: { condition: service_healthy } }

Testing against PostgreSQL rather than SQLite catches database-specific bugs before production.

3. Handle secrets

Buildkite agents run hooks before each job. Put secrets in an agent environment hook or fetch them from a secrets manager there, scoped by pipeline:

# /etc/buildkite-agent/hooks/environment
if [[ "$BUILDKITE_PIPELINE_SLUG" == "shop" ]]; then
  export DEPLOY_TOKEN="$(aws secretsmanager get-secret-value --secret-id shop/deploy --query SecretString --output text)"
fi

Buildkite also offers hosted secrets and plugins for AWS Secrets Manager and Vault. Avoid env: blocks with secrets in pipeline.yml, since the file lives in git and appears in the build UI.

4. Deploy script

#!/usr/bin/env bash
set -euo pipefail
TAG="$1"
# run migrations with the new image, then update the service
kubectl -n prod create job "migrate-${TAG:0:12}" --image="ghcr.io/acme/shop:${TAG}" -- python manage.py migrate --noinput
kubectl -n prod wait --for=condition=complete "job/migrate-${TAG:0:12}" --timeout=600s
kubectl -n prod set image deployment/web web="ghcr.io/acme/shop:${TAG}"
kubectl -n prod rollout status deployment/web --timeout=600s

Adapt the commands to your platform (ECS, a VPS via SSH, a PaaS CLI). The structure stays the same: migrate, roll out, wait for health.

5. Speed and reliability

Run tests in parallel with parallelism: 4 and Django's --parallel, or split by test labels. Use persistent agents with warm Docker caches, or registry cache for ephemeral agents. Add retry: automatic for flaky infrastructure steps, not for tests.

Explanation

Buildkite separates the control plane (pipelines, UI, scheduling) from the agents that execute jobs. Pipelines are YAML lists of steps; wait and block steps create ordering and approvals, and plugins add reusable behavior like Docker Compose integration. Because agents run in your infrastructure, they can reach private databases, registries, and clusters directly, and secrets can stay entirely within your network.

Edge cases / notes

• Dynamic pipelines. Generating steps with a script before upload lets you skip tests for docs-only changes or fan out test shards.
• Artifacts. Upload JUnit XML from Django tests (--junitxml with pytest) and use the Test Analytics features to track flaky tests.
• Rollback. Deploy the previous commit's image with the same script; keep images for recent releases.

Next steps

• Compare with Deploy Django With GitHub Actions CI/CD.
• Review gates with the Django CI/CD Pipeline Checklist.
• Fix database test failures with Fix Django Tests Failing in CI With PostgreSQL.

FAQ

Where do Buildkite builds run?

On agents you host: VMs, Kubernetes, or the Elastic CI Stack for AWS.

How do I require approval before production?

Add a block step before the deploy step.

How should secrets reach the pipeline?

Through agent hooks or secrets plugins that fetch from a secrets manager, not through pipeline YAML.

How do I stop two deploys from overlapping?

Use concurrency: 1 with a concurrency_group on the deploy step.