Skip to content

fix: don't split session JSONL on Unicode line separators#354

Merged
alejandro-ao merged 1 commit into
huggingface:mainfrom
rian-dolphin:fix/unicode-line-separators-in-jsonl
Jul 19, 2026
Merged

fix: don't split session JSONL on Unicode line separators#354
alejandro-ao merged 1 commit into
huggingface:mainfrom
rian-dolphin:fix/unicode-line-separators-in-jsonl

Conversation

@rian-dolphin

@rian-dolphin rian-dolphin commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Session files are JSONL, but the reader split them with str.splitlines(), which also breaks on U+2028/U+2029/U+0085. These are characters pydantic writes unescaped into JSON strings. One message containing such a character (e.g. from a file read into a tool result) splits a valid line into two malformed fragments, so every subsequent session load/resume raises SessionJsonlError — the session is permanently bricked. Fix: split on \n only, which also retroactively repairs already-broken files; same bug fixed in the session index reader.

To reproduce, run this snippet on main (fails with SessionJsonlError) and on this branch (prints one entry). Note the append succeeds either way — the corruption is only visible on the next read, and typing/pasting the character into the TUI won't trigger it because the input path sanitizes it; the script drives the storage API directly, as a tool result would:

uv run python - <<'EOF'
import asyncio, tempfile
from pathlib import Path
from tau_agent.messages import UserMessage
from tau_agent.session.entries import MessageEntry
from tau_agent.session.storage import JsonlSessionStorage

content = "Here is the error:" + chr(0x2028) + "TypeError: cannot read properties of undefined"

async def main():
    with tempfile.TemporaryDirectory() as tmp:
        storage = JsonlSessionStorage(Path(tmp) / "session.jsonl")
        await storage.append(MessageEntry(message=UserMessage(content=content)))
        print("append: OK (write never fails)")
        entries = await storage.read_all()  # raises SessionJsonlError before this fix
        print(f"read_all: OK, {len(entries)} entry")

asyncio.run(main())
EOF

@rian-dolphin
rian-dolphin marked this pull request as ready for review July 13, 2026 14:24
Pydantic's dump_json escapes only C0 control characters, so U+2028,
U+2029, and U+0085 in message content are written literally to session
files. The readers used str.splitlines(), which treats those characters
as line boundaries, splitting one valid JSON line into two malformed
fragments — after which every load of the session raised
SessionJsonlError.

Split on \n only, the actual JSONL delimiter. Unescaped U+2028 is
valid JSON, so this read-side fix also repairs already-affected session
files. The same pattern in SessionManager._read_index is fixed too,
where a session title containing such a character broke the index.
@alejandro-ao
alejandro-ao force-pushed the fix/unicode-line-separators-in-jsonl branch from aa7cb21 to 2386293 Compare July 19, 2026 15:36

@alejandro-ao alejandro-ao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great. thanks!!

@alejandro-ao
alejandro-ao merged commit 87dce35 into huggingface:main Jul 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants