Fix Certbot "Challenge failed" When Issuing a Certificate

Problem statement

Running Certbot for a new Django site fails:

Certbot failed to authenticate some domains (authenticator: nginx).
  Domain: example.com
  Type:   unauthorized
  Detail: 203.0.113.10: Invalid response from http://example.com/.well-known/acme-challenge/kX3...: 404

or Type: connection with Timeout during connect (likely firewall problem), or Type: dns with no valid A records found. For the HTTP-01 challenge, Let's Encrypt fetches a token from http://<your domain>/.well-known/acme-challenge/ over port 80. Any of DNS, firewalls, proxies, or your Nginx and Django config can stop that fetch. The Type and Detail lines say which.

1. Type: dns → the A/AAAA records are missing or wrong; fix DNS and wait for it to resolve publicly.
2. Type: connection / timeout → open port 80 in ufw and the cloud firewall.
3. Type: unauthorized with 404 → the request reached a server that does not serve the token: wrong IP, wrong server_name, Django handling the path, or a CDN in front.
4. Test the path yourself: create a file under /.well-known/acme-challenge/ and curl it from outside.
5. With Cloudflare proxy on, temporarily set the record to DNS-only or use the DNS-01 challenge.

Step-by-step solution

1. Read the Type and Detail lines

Certbot's summary names the failure class. Re-run with more output if needed:

sudo certbot --nginx -d example.com -d www.example.com -v

Let's Encrypt validates every name separately, so example.com may pass while www.example.com fails.

2. Fix DNS problems

dig +short example.com A @1.1.1.1
dig +short example.com AAAA @1.1.1.1
curl -4 ifconfig.me    # this server's public IPv4

The A record must equal the server's public IP. Let's Encrypt prefers IPv6 when an AAAA record exists; a stale or wrong AAAA record makes validation fail even if IPv4 is perfect. Remove it or make the server listen on IPv6. New records may take time to appear at the resolvers Let's Encrypt uses.

3. Open port 80

HTTP-01 always uses port 80, even if your site will be HTTPS-only.

sudo ufw allow 80/tcp
sudo ss -ltnp | grep ':80 '

Also check the provider firewall or security group. Port 80 must stay open after issuance too, because renewals use the same challenge.

4. Make the ACME path reachable

For Type: unauthorized with a 404, create a test file and fetch it from outside:

sudo mkdir -p /var/www/letsencrypt/.well-known/acme-challenge
echo ok | sudo tee /var/www/letsencrypt/.well-known/acme-challenge/test

Add an explicit location to the port 80 server block, before any redirect:

server {
    listen 80;
    server_name example.com www.example.com;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
        default_type text/plain;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

sudo nginx -t && sudo systemctl reload nginx
curl -i http://example.com/.well-known/acme-challenge/test

When curl from another machine returns ok, use the webroot authenticator, which is predictable with custom configs:

sudo certbot certonly --webroot -w /var/www/letsencrypt -d example.com -d www.example.com

Common reasons for the 404: the domain is missing from server_name so the default server answers; location / proxies everything to Django, which has no such URL; or a return 301 at server level runs before any location.

5. Proxies and CDNs

With Cloudflare's orange-cloud proxy, Let's Encrypt reaches Cloudflare, not you. HTTP-01 can still work if Cloudflare passes the path through, but "Always Use HTTPS" or page rules often interfere. Either set the records to DNS-only during issuance, use a Cloudflare origin certificate, or use DNS-01:

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.cloudflare.ini -d example.com -d '*.example.com'

Verification check: sudo certbot certificates lists the certificate with all names, and sudo certbot renew --dry-run succeeds, which proves renewals will also pass.

Explanation

The HTTP-01 challenge proves you control a domain by showing you can serve a specific file at a specific URL on that domain over port 80. Let's Encrypt fetches it from several network locations, follows redirects (including to HTTPS), and compares the content. Anything that changes where the request lands or what it returns breaks the proof. Certbot's Nginx plugin injects a temporary location, but complex configs with server-level redirects, catch-all proxies, or multiple matching server blocks can bypass it, which is why the webroot method with an explicit location is the most reliable option for Django setups.

Edge cases / notes

• Rate limits. Five failed validations per hostname per hour trigger a temporary block. Use --dry-run (staging) while debugging.
• Docker. If Nginx runs in a container, the webroot must be a volume shared between Certbot and Nginx.
• Geo-blocking or fail2ban that blocks foreign IPs can block Let's Encrypt's validation servers.
• Internal hosts that are not reachable from the internet must use DNS-01.

Next steps

• Complete the setup with Configure HTTPS With Let's Encrypt for Django.
• Once issued, keep it working with Fix Certbot Renewal Failed.

FAQ

Why does Let's Encrypt need port 80 if my site is HTTPS-only?

HTTP-01 validation is defined on port 80. Keep it open and redirect everything except the challenge path.

Can Django serve the ACME challenge?

It can, but letting Nginx serve it from a webroot is simpler and does not depend on the app running.

What is the difference between unauthorized and connection errors?

Connection means Let's Encrypt could not reach port 80. Unauthorized means it reached something, but the response was not the token.

When should I use DNS-01?

For wildcard certificates, hosts behind proxies, or servers that are not reachable from the internet.