Skip to content

fix(cost): serialize ledger records with json.dumps so ids can't corrupt JSON (#165)#241

Open
minion1227 wants to merge 2 commits into
mini-router:mainfrom
minion1227:sn74-minion1227-165
Open

fix(cost): serialize ledger records with json.dumps so ids can't corrupt JSON (#165)#241
minion1227 wants to merge 2 commits into
mini-router:mainfrom
minion1227:sn74-minion1227-165

Conversation

@minion1227

Copy link
Copy Markdown
Contributor

Fixes #165

Problem

_ledger_append in src/trinity/llm/openai_compatible_pool.py built each cost-ledger line by hand with an f-string:

f.write(
    f'{{"provider":"{provider}","m":"{model}","p":{int(prompt_tokens)},'
    f'"c":{int(completion_tokens)}}}\n'
)

model is route.model_id, taken verbatim from configs/models*.yaml. If an id contains a double quote, a backslash, or a control character, the emitted line is not valid JSON:

{"provider":"openrouter","m":"weird"model","p":1000,"c":2000}   <- invalid
{"provider":"openrouter","m":"back\slash","p":10,"c":20}        <- invalid escape

costing.ledger_cost_report does try: json.loads(line) except JSONDecodeError: continue, so those lines are silently skipped — their tokens never reach cost_usd, cost_calls, or the per-model breakdown. That is silent under-reporting of API spend, with no error and no warning, and it corrupts the JSONL file for any other consumer.

Reproduced before the fix: appending 3 records where 2 have metacharacter ids gives cost_calls == 1.

Fix

Serialize the record with json.dumps(..., ensure_ascii=False) instead of hand-formatting, so the writer and reader agree on escaping. ensure_ascii=False keeps non-ASCII ids human-readable in the ledger.

After the fix the same 5 records all parse, and cost_calls == 5 with exact token totals and ids preserved (weird"model, back\slash, ctrl\tchar, uniécode/model).

Tests

Adds tests/test_cost_ledger_json.py (14 cases, pure stdlib — no network/GPU/torch):

  • quote / backslash / tab / newline / unicode ids each round-trip as valid JSON with the id preserved
  • the provider field is escaped too
  • no call is dropped from cost_calls / cost_prompt_tokens / cost_completion_tokens
  • tricky ids still appear in the per-model breakdown
  • a newline in an id no longer splits one record into two lines
  • the unset-TRINITY_COST_LEDGER path stays a no-op

Full root suite: 260 passed; ruff clean.

minion1227 and others added 2 commits July 22, 2026 21:47
…upt JSON (mini-router#165)

`_ledger_append` built each cost-ledger line by hand with an f-string, so a
provider or model id containing a double quote, a backslash, or a control
character emitted a line that is not valid JSON. `costing.ledger_cost_report`
skips unparseable lines silently, so every call for such a model vanished from
`cost_usd`, `cost_calls`, and the per-model breakdown — silent under-reporting
of API spend, with no error and no warning, and a corrupted JSONL file for any
other consumer.

Before (model id `weird"model`):

    {"provider":"openrouter","m":"weird"model","p":1000,"c":2000}   <- invalid

Serialize the record with `json.dumps(..., ensure_ascii=False)` instead, so the
writer and the reader agree on escaping. Non-ASCII ids stay human-readable.

Adds tests/test_cost_ledger_json.py: quote / backslash / tab / newline / unicode
ids each round-trip as valid JSON with the id preserved, the provider is escaped
too, no call is dropped from the cost report totals, tricky ids survive in the
per-model breakdown, a newline no longer splits one record into two lines, and
the unset-env path stays a no-op.

Fixes mini-router#165
…outer#165)

AGENTS.md §6 requires a dated JOURNAL entry for every mistake/fix; the
original commit changed openai_compatible_pool.py and tests but skipped
the lab notebook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@minion1227

Copy link
Copy Markdown
Contributor Author

CI: register_submission failure is not from this PR

The red check is repo-wide infrastructure, not this diff. actions/checkout now refuses to check out fork code under pull_request_target (pwn-request guard), so the job fails at checkout on every fork PR — currently #112, #226, #232, #233, #239, #240 and this one. Because pull_request_target runs the workflow definition from base main, it cannot be fixed from a PR branch. #233 already proposes the fix (dropping the unused checkout step from that job).

All substantive checks pass here: test-router, test-validator, web, and check_submission_body.

Added since opening

A docs/JOURNAL.md entry per AGENTS.md §6, which the original commit omitted — recording that the corruption was silent (ledger_cost_report skips unparseable lines rather than raising, so affected calls simply vanished from the spend total) and why ensure_ascii=False is safe here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cost ledger: model ids with a quote or backslash write malformed JSON and are dropped from the cost total

1 participant