Skip to content

fix(output): write --json as UTF-8 regardless of the console codepage (#546) - #549

Merged
padak merged 2 commits into
mainfrom
claude/json-utf8-issue-546
Aug 2, 2026
Merged

fix(output): write --json as UTF-8 regardless of the console codepage (#546)#549
padak merged 2 commits into
mainfrom
claude/json-utf8-issue-546

Conversation

@padak

@padak padak commented Aug 2, 2026

Copy link
Copy Markdown
Member

Fixes #546.

The bug

kbagent --json flow list crashed with UnicodeEncodeError on a default
Czech/Polish/Hungarian Windows console (cp1250) whenever the output contained a
non-ASCII character. An arrow in a flow name was enough:

File "keboola_agent_cli/output.py", line 57, in output
    sys.stdout.write(response.model_dump_json(indent=2) + "\n")
UnicodeEncodeError: 'charmap' codec can't encode character '→'

--json exists to be piped into another program, so its bytes must not depend on the
terminal's active codepage.

Which writer actually crashes

Worth being precise, because it decides what the tests are worth:

Writer Non-ASCII handling Crashes on cp1250
response.model_dump_json() (pydantic) — output(), success() emits raw UTF-8 yes
json.dumps(..., indent=2)error() escapes to \uXXXX (ensure_ascii default) no

So only the two pydantic paths reproduce the report. The error envelope is routed
through the same helper anyway for consistency, and its test says so plainly rather
than posing as a regression test for a bug it cannot hit.

The fix

One write_machine_output helper behind all three call sites. It writes UTF-8 through
sys.stdout.buffer, bypassing the text layer's encoder entirely, so the codepage never
gets a say. The text layer is flushed first so anything already written through it keeps
its place in the stream; a stdout with no binary buffer (captured or replaced streams,
e.g. under test) has no encoder to bypass and takes the plain write.

Chosen over sys.stdout.reconfigure(encoding="utf-8") because reconfigure is not
available on every stream kind the CLI runs under, and over PYTHONUTF8=1 because the
reporter should not have to set an environment variable to get valid JSON.

Testing

Five tests in tests/test_output.py::TestMachineOutputIsAlwaysUtf8. They simulate the
cp1250 encoder with io.TextIOWrapper(io.BytesIO(), encoding="cp1250"), so they run on
every platform rather than needing a Windows box.

I verified they are load-bearing by reverting the fix: the two covering the crash
(output, success) fail with the reporter's exact UnicodeEncodeError; the other
three pass either way and are documented as invariant guards, not bug reproductions.

Also covered: the no-binary-buffer fallback, and that a human-mode write issued before a
machine-output write keeps its order (the reason for the flush).

make check equivalent is green — ruff, ty, and 4688 tests.

Reproduced end to end through the real CLI as well, not just the formatter. Forcing the
console encoding and asking for output that contains box-drawing characters (the v0.76.1
changelog entry):

PYTHONIOENCODING=cp1250 kbagent --json changelog --limit 8 --full
  • before the fix: exit=1, UnicodeEncodeError: 'charmap' codec can't encode characters in position 4481-4482 — the reporter's failure;
  • after: exit=0, output parses as UTF-8 JSON with ─ └ ├ intact.

(Worth noting for anyone repeating this: a small --limit proves nothing here. The most
recent changelog entries happen to be pure ASCII, so the command exits 0 either way.)

Notes for merge

  • Side effect on Windows: JSON lines now end LF rather than CRLF, because the
    binary buffer does no newline translation. For machine-consumed output that is the
    better default and no parser cares, but it is a real behaviour change worth knowing.

  • No version bump or changelog entry, matching how fix(update): make self-update safe on Windows #530 / fix(semantic-layer): support snapshot export on Windows #531 landed before the
    chore(release): 0.76.2 Windows update and export fixes #533 release PR bundled them. The line for whichever release picks this up:

    Fix (#546): kbagent --jsonno longer crashes withUnicodeEncodeErroron Windows consoles with a non-UTF-8 codepage (cp1250 on Czech/Polish/Hungarian Windows). Any non-ASCII character in the data — an arrow in a flow name was the report — made machine-readable output unusable, because pydantic'smodel_dump_jsonemits raw UTF-8 andsys.stdoutencoded it through the console codepage. JSON is now written as UTF-8 bytes throughsys.stdout.buffer, independent of the console. The PYTHONUTF8=1 workaround is no longer needed. Thanks to @MichalProchazka for the report.

  • Independent of fix(update): defer the Windows self-update out of the running environment (#528) #543 (the Windows self-update fix); different subsystem, no overlapping
    files.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment thread src/keboola_agent_cli/output.py
Comment thread tests/test_output.py Outdated
padak added 2 commits August 2, 2026 10:00
…#546)

`kbagent --json flow list` crashed with UnicodeEncodeError on a default
Czech/Polish/Hungarian Windows console (cp1250) whenever the data held a
non-ASCII character -- an arrow in a flow name was enough.

`--json` exists to be piped into another program, so its bytes must not
depend on the terminal's active codepage. The three JSON writers now go
through `write_machine_output`, which writes UTF-8 through
`sys.stdout.buffer` and so bypasses the text layer's encoder entirely.
The text layer is flushed first so anything written through it keeps its
place; a stdout with no binary buffer (captured or replaced streams) has
no encoder to bypass and takes the plain write.

Only the two pydantic paths actually crashed: `model_dump_json` emits raw
non-ASCII, whereas `json.dumps` escapes it to \uXXXX under its
ensure_ascii default. The error envelope is routed through the same
helper anyway, and its test says plainly that it pins the invariant
rather than reproducing the bug.

The regression tests simulate the cp1250 encoder, so they run on every
platform; the two covering the crash were confirmed to fail without this
change.

Side effect on Windows: JSON lines now end LF rather than CRLF, since
the binary buffer does no newline translation. Machine output is the
better place for that, and no parser cares.
… too (#546)

Review catch: the fix covered the three OutputFormatter call sites but not
`_render_stream_event`, which builds its own NDJSON line and wrote it
straight to stdout. It uses `json.dumps(..., ensure_ascii=False)` --
deliberately, so event text stays readable -- which is exactly the
property that crashes on cp1250. `kbagent --json agent run --stream` was
therefore still broken on a non-UTF-8 Windows console.

`kbagent http`'s `_print_json` moves to the same helper. That one uses
the `ensure_ascii` default and so cannot crash today; it is routed for
the invariant, and its docstring says which of the two it is rather than
implying it was a bug.

Both new tests confirmed load-bearing by reverting the change: the
non-ASCII one fails with the reporter's `UnicodeEncodeError`.

Also adds the type hints CONTRIBUTING requires to the tests added in the
previous commit.
@padak
padak force-pushed the claude/json-utf8-issue-546 branch from 99b81b8 to 88ccb5c Compare August 2, 2026 08:00
@padak
padak merged commit 866e80c into main Aug 2, 2026
4 checks passed
@padak
padak deleted the claude/json-utf8-issue-546 branch August 2, 2026 08:16
padak added a commit that referenced this pull request Aug 2, 2026
…#550)

v0.77.1 was merged but never tagged, and the next release will be 0.78.0
bundling several PRs -- so the changelog key and pyproject version are
renumbered rather than leaving an entry for a version that will not exist.

More importantly, #549 landed with no changelog entry at all, matching how
#530/#531 landed before their release PR. That is the failure mode that
left 0.67.0-0.70.1 unpublished and undocumented, so the #546 fix is
written up now rather than at tag time.

Also clears the two `redundant-cast` warnings my #549 tests introduced.
Deleting the casts turned them into `unresolved-attribute` ERRORS --
`TextIOWrapper.buffer` is declared as `_WrappedBuffer`, which has no
`getvalue` -- so the fix is to stop reaching through `.buffer` at all: a
small frozen `Cp1250Stdout` holds the `BytesIO` directly. That also gives
the streaming tests in the other class access to the same helper, which
they did not have.
padak added a commit that referenced this pull request Aug 2, 2026
…son fix

Release audit for 0.78.0 turned up two gaps in the agent-facing docs.

Five reference files existed but nothing linked them, so the skill could never
load them: config-metadata, storage-describe, lineage-deep, permissions and
kai workflows. They are now rows in SKILL.md's workflow table (the
hand-maintained one below the auto-generated command block, so
`make skill-check` stays green).

#549 (--json written as UTF-8 regardless of the console codepage) merged with
no changelog entry and no gotcha. It changes what an agent parsing --json on
Windows sees -- the payload is no longer encoder-dependent, and JSON lines end
LF rather than CRLF there -- so it gets both: a (since v0.78.0) gotcha and a
changelog bullet under 0.78.0, the release that ships it.

Not documented here: the Apache 2.0 LICENSE added by #544. pyproject.toml,
nfpm.yaml and the Homebrew formula all still declare MIT, so which licence
kbagent ships under is a decision to settle before it is announced anywhere.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--json output crashes with UnicodeEncodeError on Windows when data contains non-ASCII characters

1 participant