fix(output): write --json as UTF-8 regardless of the console codepage (#546) - #549
Merged
Conversation
…#546) `kbagent --json flow list` crashed with UnicodeEncodeError on a default Czech/Polish/Hungarian Windows console (cp1250) whenever the data held a non-ASCII character -- an arrow in a flow name was enough. `--json` exists to be piped into another program, so its bytes must not depend on the terminal's active codepage. The three JSON writers now go through `write_machine_output`, which writes UTF-8 through `sys.stdout.buffer` and so bypasses the text layer's encoder entirely. The text layer is flushed first so anything written through it keeps its place; a stdout with no binary buffer (captured or replaced streams) has no encoder to bypass and takes the plain write. Only the two pydantic paths actually crashed: `model_dump_json` emits raw non-ASCII, whereas `json.dumps` escapes it to \uXXXX under its ensure_ascii default. The error envelope is routed through the same helper anyway, and its test says plainly that it pins the invariant rather than reproducing the bug. The regression tests simulate the cp1250 encoder, so they run on every platform; the two covering the crash were confirmed to fail without this change. Side effect on Windows: JSON lines now end LF rather than CRLF, since the binary buffer does no newline translation. Machine output is the better place for that, and no parser cares.
… too (#546) Review catch: the fix covered the three OutputFormatter call sites but not `_render_stream_event`, which builds its own NDJSON line and wrote it straight to stdout. It uses `json.dumps(..., ensure_ascii=False)` -- deliberately, so event text stays readable -- which is exactly the property that crashes on cp1250. `kbagent --json agent run --stream` was therefore still broken on a non-UTF-8 Windows console. `kbagent http`'s `_print_json` moves to the same helper. That one uses the `ensure_ascii` default and so cannot crash today; it is routed for the invariant, and its docstring says which of the two it is rather than implying it was a bug. Both new tests confirmed load-bearing by reverting the change: the non-ASCII one fails with the reporter's `UnicodeEncodeError`. Also adds the type hints CONTRIBUTING requires to the tests added in the previous commit.
padak
force-pushed
the
claude/json-utf8-issue-546
branch
from
August 2, 2026 08:00
99b81b8 to
88ccb5c
Compare
This was referenced Aug 2, 2026
padak
added a commit
that referenced
this pull request
Aug 2, 2026
…#550) v0.77.1 was merged but never tagged, and the next release will be 0.78.0 bundling several PRs -- so the changelog key and pyproject version are renumbered rather than leaving an entry for a version that will not exist. More importantly, #549 landed with no changelog entry at all, matching how #530/#531 landed before their release PR. That is the failure mode that left 0.67.0-0.70.1 unpublished and undocumented, so the #546 fix is written up now rather than at tag time. Also clears the two `redundant-cast` warnings my #549 tests introduced. Deleting the casts turned them into `unresolved-attribute` ERRORS -- `TextIOWrapper.buffer` is declared as `_WrappedBuffer`, which has no `getvalue` -- so the fix is to stop reaching through `.buffer` at all: a small frozen `Cp1250Stdout` holds the `BytesIO` directly. That also gives the streaming tests in the other class access to the same helper, which they did not have.
padak
added a commit
that referenced
this pull request
Aug 2, 2026
…son fix Release audit for 0.78.0 turned up two gaps in the agent-facing docs. Five reference files existed but nothing linked them, so the skill could never load them: config-metadata, storage-describe, lineage-deep, permissions and kai workflows. They are now rows in SKILL.md's workflow table (the hand-maintained one below the auto-generated command block, so `make skill-check` stays green). #549 (--json written as UTF-8 regardless of the console codepage) merged with no changelog entry and no gotcha. It changes what an agent parsing --json on Windows sees -- the payload is no longer encoder-dependent, and JSON lines end LF rather than CRLF there -- so it gets both: a (since v0.78.0) gotcha and a changelog bullet under 0.78.0, the release that ships it. Not documented here: the Apache 2.0 LICENSE added by #544. pyproject.toml, nfpm.yaml and the Homebrew formula all still declare MIT, so which licence kbagent ships under is a decision to settle before it is announced anywhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #546.
The bug
kbagent --json flow listcrashed withUnicodeEncodeErroron a defaultCzech/Polish/Hungarian Windows console (cp1250) whenever the output contained a
non-ASCII character. An arrow in a flow name was enough:
--jsonexists to be piped into another program, so its bytes must not depend on theterminal's active codepage.
Which writer actually crashes
Worth being precise, because it decides what the tests are worth:
response.model_dump_json()(pydantic) —output(),success()json.dumps(..., indent=2)—error()\uXXXX(ensure_asciidefault)So only the two pydantic paths reproduce the report. The error envelope is routed
through the same helper anyway for consistency, and its test says so plainly rather
than posing as a regression test for a bug it cannot hit.
The fix
One
write_machine_outputhelper behind all three call sites. It writes UTF-8 throughsys.stdout.buffer, bypassing the text layer's encoder entirely, so the codepage nevergets a say. The text layer is flushed first so anything already written through it keeps
its place in the stream; a stdout with no binary buffer (captured or replaced streams,
e.g. under test) has no encoder to bypass and takes the plain write.
Chosen over
sys.stdout.reconfigure(encoding="utf-8")because reconfigure is notavailable on every stream kind the CLI runs under, and over
PYTHONUTF8=1because thereporter should not have to set an environment variable to get valid JSON.
Testing
Five tests in
tests/test_output.py::TestMachineOutputIsAlwaysUtf8. They simulate thecp1250 encoder with
io.TextIOWrapper(io.BytesIO(), encoding="cp1250"), so they run onevery platform rather than needing a Windows box.
I verified they are load-bearing by reverting the fix: the two covering the crash
(
output,success) fail with the reporter's exactUnicodeEncodeError; the otherthree pass either way and are documented as invariant guards, not bug reproductions.
Also covered: the no-binary-buffer fallback, and that a human-mode write issued before a
machine-output write keeps its order (the reason for the flush).
make checkequivalent is green — ruff, ty, and 4688 tests.Reproduced end to end through the real CLI as well, not just the formatter. Forcing the
console encoding and asking for output that contains box-drawing characters (the v0.76.1
changelog entry):
exit=1,UnicodeEncodeError: 'charmap' codec can't encode characters in position 4481-4482— the reporter's failure;exit=0, output parses as UTF-8 JSON with─ └ ├intact.(Worth noting for anyone repeating this: a small
--limitproves nothing here. The mostrecent changelog entries happen to be pure ASCII, so the command exits 0 either way.)
Notes for merge
Side effect on Windows: JSON lines now end
LFrather thanCRLF, because thebinary buffer does no newline translation. For machine-consumed output that is the
better default and no parser cares, but it is a real behaviour change worth knowing.
No version bump or changelog entry, matching how fix(update): make self-update safe on Windows #530 / fix(semantic-layer): support snapshot export on Windows #531 landed before the
chore(release): 0.76.2 Windows update and export fixes #533 release PR bundled them. The line for whichever release picks this up:
Independent of fix(update): defer the Windows self-update out of the running environment (#528) #543 (the Windows self-update fix); different subsystem, no overlapping
files.