Fix UnicodeEncodeError from Locale Settings on Django Servers

Problem statement

Something that works on your machine fails on the server when it meets a non-ASCII character, such as an accented name, a currency symbol, or an emoji:

UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 12: ordinal not in range(128)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Typical places: a management command printing output under cron, writing a CSV export with open(), saving an upload with a non-ASCII filename, or a subprocess call. The code is the same as in development; the environment is not. Python chooses several default encodings from the process locale, and services started by systemd, cron, or minimal containers often run with a different locale than your login shell. This guide checks which encodings a process uses and makes them UTF-8 everywhere.

1. Check the encodings inside the failing context (service, cron job, or container), not in your shell.
2. Set a UTF-8 locale for the process: Environment=LANG=C.UTF-8 in the systemd unit, ENV LANG=C.UTF-8 in the Dockerfile, and a LANG line in the crontab.
3. Or enable Python's UTF-8 mode with PYTHONUTF8=1, which makes Python use UTF-8 regardless of locale.
4. Pass encoding="utf-8" explicitly to open() and similar calls in your code.
5. Remove overrides like LC_ALL=C or LANG=POSIX inherited from provisioning scripts.
6. If the error comes from the database, check the database and connection character sets instead.

Step-by-step solution

1. Print the encodings where it fails

python3 -c "import sys, locale; print('fs:', sys.getfilesystemencoding(), '| preferred:', locale.getpreferredencoding(False), '| stdout:', sys.stdout.encoding, '| utf8_mode:', sys.flags.utf8_mode)"

Run it in each context:

# as the service sees it
sudo systemd-run --uid=myapp --pipe --property=EnvironmentFile=/etc/myapp.env \
  /srv/myapp/current/.venv/bin/python -c "import locale, sys; print(locale.getpreferredencoding(False), sys.flags.utf8_mode)"

# inside the container
docker compose exec web python -c "import locale, sys; print(locale.getpreferredencoding(False), sys.flags.utf8_mode)"

# the environment of the running Gunicorn master
sudo cat /proc/$(pgrep -f "gunicorn.*master" | head -1)/environ | tr '\0' '\n' | grep -E "^(LANG|LC_|PYTHONUTF8|PYTHONIOENCODING)"

Any value other than UTF-8 (for example ANSI_X3.4-1968, which is ASCII, or ISO-8859-1) explains the error. Modern Python (3.7 and newer) switches to UTF-8 automatically when the locale is exactly C or POSIX, but not for other non-UTF-8 locales, and not for code paths in libraries or subprocesses that make their own choices.

2. Set a UTF-8 locale for services

systemd services do not inherit your shell's locale. Set it in the unit:

# /etc/systemd/system/gunicorn.service
[Service]
Environment=LANG=C.UTF-8
Environment=PYTHONUTF8=1

sudo systemctl daemon-reload
sudo systemctl restart gunicorn celery-worker

C.UTF-8 is available on current Debian, Ubuntu, and RHEL-family systems without generating additional locales. If you need a specific language locale (for example de_DE.UTF-8 for date formatting in a library), generate it first (locale-gen or localedef); an ungenerated locale falls back to C with warnings.

Also check /etc/default/locale or /etc/locale.conf for the system-wide default, and look for LC_ALL=C in environment files. LC_ALL overrides every other locale variable.

3. Set it for cron and timers

Cron runs jobs with a minimal environment. Add variables at the top of the crontab:

LANG=C.UTF-8
PYTHONUTF8=1
15 3 * * * cd /srv/myapp/current && .venv/bin/python manage.py send_digests

systemd timers use the service unit they trigger, so the Environment= lines in that unit apply.

4. Set it in Docker images

Official Python images set LANG=C.UTF-8. Custom base images may not. Set it explicitly:

ENV LANG=C.UTF-8 \
    PYTHONUTF8=1

Setting PYTHONIOENCODING=utf-8 fixes standard streams only; PYTHONUTF8=1 covers streams, open() defaults, and filesystem encoding.

5. Be explicit in code

Defaults depend on the environment; explicit encodings do not:

# Writing an export
with open(path, "w", encoding="utf-8", newline="") as fh:
    writer = csv.writer(fh)
    ...

# Reading a fixture or template-like file
text = Path("data/countries.txt").read_text(encoding="utf-8")

# Subprocess output
result = subprocess.run(["some-tool"], capture_output=True, text=True, encoding="utf-8", check=True)

Tools like ruff can flag open() calls without an encoding argument (the unspecified-encoding rule), which is worth enabling in CI.

6. Filenames of uploads

Django's FileSystemStorage keeps Unicode in filenames. Saving Résumé.pdf requires the filesystem encoding to handle it, which it does with UTF-8. If you prefer ASCII-only names on disk (simpler for backups and tools), normalise in upload_to:

from django.utils.text import get_valid_filename, slugify
from pathlib import Path


def ascii_upload_to(instance, filename):
    stem, suffix = Path(filename).stem, Path(filename).suffix
    return f"uploads/{slugify(stem) or 'file'}{get_valid_filename(suffix)}"

slugify without allow_unicode=True transliterates what it can and drops the rest.

7. When it is the database, not the locale

Errors such as character with byte sequence 0xf0 0x9f ... has no equivalent in encoding "LATIN1" (PostgreSQL) or Incorrect string value: '\xF0\x9F...' (MySQL) come from database character sets. PostgreSQL databases should use UTF8 encoding; MySQL tables and connections must use utf8mb4 (not the 3-byte utf8/utf8mb3) to store emoji. These need database changes, not locale changes.

Verification check:

sudo cat /proc/$(pgrep -f "gunicorn.*master" | head -1)/environ | tr '\0' '\n' | grep -E "^(LANG|PYTHONUTF8)"
sudo -u myapp /srv/myapp/current/.venv/bin/python manage.py shell -c "print('café ✓')"

Then repeat the original failing action (the export, the upload, the cron job) with non-ASCII data.

Explanation

Python derives default text encodings from the locale: for open() without an encoding, for standard input and output, and for converting file names between bytes and strings. Interactive shells usually have a UTF-8 locale; services, cron jobs, and some containers may not. When the default is ASCII, any non-ASCII character in those operations raises an encode or decode error. Setting the locale (or UTF-8 mode) for every process type, and writing explicit encodings in code, removes the dependency on the environment.

Edge cases / notes

Python's future default

Newer Python releases are moving towards UTF-8 as the default for open() regardless of locale. Explicit encodings remain correct on every version and make the intent clear.

Log files

Logging handlers that write files use the default encoding unless you pass encoding="utf-8" to FileHandler (or "encoding" in the LOGGING dict). A non-ASCII log message can otherwise raise an error inside the logging system.

When to automate this

Put LANG=C.UTF-8 and PYTHONUTF8=1 in your systemd unit templates, crontab templates, and Dockerfiles, and enable a linter rule for open() without an encoding. Add a test that exports and uploads data containing non-ASCII characters.

Next steps

• Set environment lines in your units with How to Configure systemd for Gunicorn and Django.
• Give scheduled jobs a proper environment with Schedule Django Management Commands with Cron or systemd Timers.
• For other production-only differences, see Django Works with DEBUG=True but Fails with DEBUG=False.
• For character sets on MySQL, see Configure Django with MySQL in Production.

FAQ

Why do I get 'ascii' codec errors only on the server?

The process runs with a non-UTF-8 locale, common for systemd services, cron jobs, and minimal containers. Your shell uses UTF-8, so the problem does not appear locally.

What does PYTHONUTF8=1 do?

It enables Python's UTF-8 mode, making UTF-8 the default for file I/O, standard streams, and filesystem names, regardless of the locale.

Should I use C.UTF-8 or en_US.UTF-8?

C.UTF-8 is available by default on modern distributions and is enough for encoding. Use a language-specific locale only if something needs its formatting rules, and generate it first.

Why can't MySQL store emoji from Django?

The table or connection uses the 3-byte utf8 charset. Convert to utf8mb4 and set the connection charset accordingly.