Python — UnicodeEncodeError: charmap codec can't encode character

Fix UnicodeEncodeError on Windows where the console and files default to cp1252, pin UTF-8 on every boundary, and stop replacing errors with question marks.

beginnerdebuggingpythonunicodewindows

Symptoms

UnicodeEncodeError: 'charmap' codec can't encode character '\u2603' in position 42: character maps to <undefined> when printing or writing files on Windows
The same script runs unchanged on Linux or macOS and only fails on Windows machines or Windows CI runners
Output shows ?? or ? instead of the expected emoji, CJK characters, or box-drawing symbols

Possible causes

On Windows the default locale encoding is cp1252 (or cp936/cp949/cp1250 by locale), which cannot represent most non-Latin characters
open() without encoding= uses the locale encoding, so writing UTF-8 content lands in a legacy codepage
print() to a redirected pipe uses the ANSI codepage, unlike the interactive console that may handle Unicode better
A library writes to sys.stdout with the default strict error handler during redirected output

Diagnosis steps

Check the active default encoding

python -c "import sys,locale; print(sys.stdout.encoding, locale.getpreferredencoding())"

Interpretation

cp1252 on a machine that processes CJK or emoji guarantees encode failures; UTF-8 here would shift suspicion to explicit encode() calls.

Reproduce with the exact character

python -c "print('\u2603 snowman')" > out.txt

Interpretation

A minimal print redirected to a file reproduces the boundary: console may work while redirected output raises.

Audit encoding-less IO boundaries

grep -Rn "open(\|to_csv(\|write(" src/ | grep -v "encoding=" | head -20

Interpretation

Every open() without encoding= inherits the locale codec — each is a latent UnicodeEncodeError on a different machine.

Root cause

Windows defaults text IO to a legacy ANSI codepage such as cp1252, which can represent only 256 code points. Writing characters outside that set with the default strict error handler raises UnicodeEncodeError; on Linux the same code runs because the platform default is UTF-8.

Fix

Pass encoding="utf-8" to every open(), set PYTHONUTF8=1 or PYTHONIOENCODING=utf-8 for the process, and configure libraries (csv, json.dump, pandas to_csv) explicitly. On Windows 10+ enable the system-wide UTF-8 beta option for a deployment-wide fix.

Risk note

errors="replace" turns data corruption into silent question marks — the failure disappears while the data degrades. Reserve it for logs, never for data files.

Verify the fix

Write a file containing emoji and CJK through the fixed path, read it back as UTF-8, and assert byte equality; run the suite on a Windows runner with PYTHONUTF8=1 unset to prove the code alone is correct.

Symptom walkthrough

The error names the codec and the offending code point, which is most of the diagnosis:

UnicodeEncodeError: 'charmap' codec can't encode character
'\u2603' in position 42: character maps to <undefined>

The codec field is the verdict: charmap means a Windows ANSI codepage was asked to encode something outside its 256 slots. Confirm the environment defaults:

>>> import sys
>>> sys.stdout.encoding
'cp1252'

The platform asymmetry — Linux green, Windows red, same code — is the fingerprint of locale-dependent encoding rather than a data problem.

Fix in practice

Pin the codec at every boundary. For files, always pass the encoding explicitly:

with open("out.txt", "w", encoding="utf-8") as f:
    f.write("\u2603 snowman \u4e2d\u6587")

For the process, prefer the UTF-8 mode over per-stream patches:

set PYTHONUTF8=1
python app.py

For third-party boundaries — subprocess pipes, csv writers, pandas exports — pass encoding through their APIs (to_csv(path, encoding=“utf-8-sig”) when Excel must open the file). A repo-wide guard catches regressions in review:

grep -Rn "open(" --include="*.py" . | grep -v "encoding=" && echo "FOUND encoding-less open()"