feat(traces): record model, reasoning effort, and token usage per session event#320
Merged
Conversation
New shared module src/notifications/model-usage.ts with two best-effort parsers that read the on-disk transcript the capture hooks already receive a path to: - parseClaudeTurnMeta: last assistant turn's model + usage from a Claude Code transcript. Reasoning effort left null (no per-message field). - parseCodexTurnMeta: model + reasoning_effort (turn_context) + per-turn and cumulative token usage (token_count) from a Codex rollout. Both normalize onto a shared NormalizedUsage shape (cache_read_tokens, cache_creation_tokens, reasoning_output_tokens, total_tokens) so a per-model rollup is a single GROUP BY over the sessions table. Absent fields stay absent; any read/parse failure returns null. Covered by tests/shared/notifications-model-usage.test.ts (real transcript shapes: reverse-scan, omitted fields, malformed lines, model fallback, last-vs-total usage).
Enrich the assistant_message trace row (Stop event) with model, reasoning_effort (null for Claude) and per-turn token_usage, read from the transcript's last assistant turn via parseClaudeTurnMeta. Best-effort: the row is written unchanged when the transcript is unreadable.
Enrich the Codex trace meta with reasoning_effort, per-turn token_usage and cumulative token_usage_total from the rollout (parseCodexTurnMeta). Codex emits no assistant event, so the enrichment rides on every user/tool row; the running total rolls up per model with MAX, not SUM. The turn_context model refines the payload model when present.
Four correctness fixes from the codex review of this branch: - SubagentStop: parse agent_transcript_path when present. transcript_path points at the parent session, so the subagent's assistant event was recording the parent model/usage (capture.ts). - Codex per-turn usage: clear it on each turn_context so a new turn's model never inherits the previous turn's tokens. total stays cumulative. - token_usage_total: documented as whole-session cumulative (across all models), not per-model; per-model totals sum per-turn usage deduped by turn_id. Fixed the misleading MAX-per-model note. - Best-effort contract: skip non-object JSONL roots (a bare `null` line) before dereferencing, in both parsers. Tests extended: turn-boundary model change omits stale usage; bare-null line survives in both parsers.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds best-effort model, token-usage, billing, quota, and stop-reason extraction for Claude, Codex, Pi, and OpenClaw captures. Adds unit and end-to-end validation, and prevents recognized provider model identifiers from being redacted as secrets. ChangesModel usage capture and preservation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant TraceScript
participant TranscriptFiles
participant CaptureHooks
participant ModelUsageParsers
participant SessionsTable
TraceScript->>TranscriptFiles: Discover representative model transcripts
TraceScript->>CaptureHooks: Invoke compiled hooks
CaptureHooks->>ModelUsageParsers: Parse transcript or SDK usage metadata
ModelUsageParsers-->>CaptureHooks: Return normalized model metadata
CaptureHooks->>SessionsTable: Insert enriched capture rows
TraceScript->>SessionsTable: Poll and validate stored metadata
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Coverage ReportScope: files changed in this PR. Enforced threshold: 90% per metric (per file via
File Coverage — 5 files changed
Generated for commit d73ed74. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/notifications/model-usage.ts`:
- Around line 72-78: Update readLines to avoid synchronously rereading and
splitting the entire transcript on every event. Maintain a per-transcript
cursor/cache or otherwise read only bytes appended since the previous call,
while preserving the missing-transcript handling and returning the newly
available lines needed by callers.
- Around line 62-64: Update toNum in model-usage.ts to return a value only when
v is a finite, non-negative safe integer; return undefined for negative,
fractional, unsafe, or otherwise invalid inputs before they reach usage
aggregation persistence.
- Around line 204-211: Update the turn_context handling in the model-usage
parser to reset model and reasoningEffort for every new turn, using the current
hook model only as the fallback when the new context’s model is missing or
malformed and clearing invalid or absent effort instead of retaining the
previous value. Preserve last/total behavior, and add a regression test covering
a valid first context followed by a context with missing or malformed
model/effort.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7de3d604-2726-4324-b4f3-4cebcf231d7a
📒 Files selected for processing (4)
src/hooks/capture.tssrc/hooks/codex/capture.tssrc/notifications/model-usage.tstests/shared/notifications-model-usage.test.ts
…effort per turn - toNum: accept only non-negative safe integers, so negative / fractional / unsafe values never enter token aggregates. - turn_context: reset both model and reasoning effort per turn (hook model as fallback) so a later turn that omits effort no longer inherits the previous turn's effort — same scoping fix already applied to per-turn usage. Tests: per-turn model/effort reset with hook-model fallback; negative and fractional token counts dropped.
Collaborator
Author
|
CodeRabbit triage:
|
normalizeSdkUsage maps the pi/openclaw usage shape
{input, output, cacheRead, cacheWrite, totalTokens} onto the shared
NormalizedUsage keys; sdkTurnMeta builds the {model, token_usage} enrichment
from an in-process SDK message (these runtimes expose no reasoning effort).
Same non-negative-safe-integer filtering as the transcript parsers.
Tests cover the real Pi and OpenClaw usage shapes (incl. cacheWrite ->
cache_creation_tokens) and the model-only / empty cases.
Storing the model on every trace row exposed a latent false positive: the high-entropy backstop masked long dated slugs like `claude-haiku-4-5-20251001` -> `********`, which would silently break per-model token rollups. Add a model-identifier guard to looksLikeSecret (claude/gpt/o*/gemini/... optionally provider-prefixed) so these survive redaction. Regression test asserts common ids are preserved.
The agent_end auto-capture loop now enriches assistant rows with model and normalized token_usage from the SDK message (via the shared sdkTurnMeta). User rows are unchanged.
message_end now enriches the assistant row with model and normalized token_usage from the SDK message. Pi ships as raw .ts with no shared-module imports, so the normalizer is inlined in lockstep with src/notifications/model-usage.ts (same pattern as the inlined JWT helpers).
scripts/trace-model-usage-e2e.mjs drives the real capture hooks (Claude Code, Codex, Cursor, Hermes) against every model found in the local transcripts, writing to a throwaway table and reading the rows back to show what landed in the JSONB message column. Pi and OpenClaw capture in-process (not via stdin hooks), so it proves their enriched entry from a real transcript message via the shared sdkTurnMeta. Suppresses side effects (HIVEMIND_WIKI_WORKER=1), disables embeddings, and cleans up its rows.
Address codex review [P1]: the earlier exemption used a case-insensitive prefix match with an open charset, so a random key wearing a model prefix (`gpt-AbCd…30chars`) would be exempted and stored unmasked. Narrow it to lowercase-only ids with each dot/dash segment capped at 12 chars, so no 24+ high-entropy run (mixed-case or long) can satisfy it. Added a negative test asserting prefixed high-entropy secrets are still masked.
… agent Address codex review [P2] plus the sessions-table write behaviour observed while running it: - The hooks swallow insert errors and exit 0, so exit code is not proof. Read the rows back and assert each agent's capture path landed rows, with token_usage present for the token-bearing agents (claude_code, codex). - The table drops rapid same-table bursts (200 OK, rows=0, row never lands) and is read-after-write lagged. Pace inserts (SLEEP_MS), poll the read-back, and retry stragglers; report invoked-vs-visible per agent instead of silently treating a hidden tail as success. - Derive the agent from the session_id prefix (the SQL `agent` column reads back unreliably here).
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/trace-model-usage-e2e.mjs`:
- Around line 290-301: Strengthen scripts/trace-model-usage-e2e.mjs at lines
290-301 by retaining expected parser metadata per session and comparing every
stored model, reasoning-effort, usage, and cumulative-usage field for each
capture path, rather than checking only row presence; update lines 312-318 to
exercise Pi’s inline normalizer and both integrations’ real entry/write paths
with assertions, ensuring logging or missing transcripts cannot count as
success.
- Around line 297-301: Update the validation loops in the trace-model usage
check to require rows only for agents recorded in invokedByAgent. Gate both
modelsByAgent and tokenRowsByAgent checks on whether each agent was actually
invoked, while preserving the existing missing-row problems for invoked agents.
In `@src/hooks/shared/redact.ts`:
- Around line 70-74: Update the model-identifier whitelist regex in the
redaction logic to recognize un-hyphenated version digits for the affected
families, such as optional single digits after llama and qwen, while preserving
existing matching behavior. Extend the accepted prefix alternatives to include
common Bedrock/provider prefixes—anthropic, amazon, cohere, meta, us, and eu—so
provider-qualified identifiers remain unredacted after colon splitting.
In `@tests/shared/redact.test.ts`:
- Around line 223-251: Replace the generic toContain assertions in the
provider-model and prefixed-secret tests with exact toBe assertions against the
complete expected redacted or unchanged payload. Keep redactSecrets and MASK
behavior unchanged, and ensure each assertion verifies the entire JSON or token
string rather than only a substring.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0aba9947-0d09-4f5b-a85f-cf481a1a56e1
📒 Files selected for processing (7)
harnesses/openclaw/src/index.tsharnesses/pi/extension-source/hivemind.tsscripts/trace-model-usage-e2e.mjssrc/hooks/shared/redact.tssrc/notifications/model-usage.tstests/shared/notifications-model-usage.test.tstests/shared/redact.test.ts
- redact: broaden the model-id guard to cover cloud/region-prefixed slugs (us.anthropic.claude-3-5-sonnet-20241022-v2) and digit-fused families (qwen2.5-coder-32b-instruct, llama3.1-405b-instruct) so they aren't over-redacted, while keeping the lowercase + <=12-char-segment invariants that stop a secret from slipping through. Tests use exact assertions. - e2e: require only agents that were actually invoked (a CI box may lack a given agent's transcripts); assert token-bearing rows carry positive integer input+output tokens (correctness, not mere presence); and assert the Pi/OpenClaw sdkTurnMeta proof yields valid positive token_usage.
Collaborator
Author
|
CodeRabbit round-2 triage (all in
|
Expand the trace schema beyond raw token counts to everything each agent's
transcript actually exposes:
- NormalizedUsage gains `cost` (fractional dollars; Pi/OpenClaw report it).
- TraceModelMeta gains `stop_reason` (Claude stop_reason / SDK stopReason)
and `usage_extra`, a per-harness bag for fields that don't generalize:
* Claude: service_tier, speed, inference_geo, cache-creation TTL split
(ephemeral 1h vs 5m — priced differently), server_tool_use
(web_search/web_fetch requests).
* Codex: model_context_window + rate_limits (primary/secondary windows).
- parseClaudeTurnMeta / parseCodexTurnMeta / normalizeSdkUsage / sdkTurnMeta
extract all of the above; costs use a finite-non-negative filter (integers
would drop them), token counts stay non-negative safe integers.
Verified against real transcripts for every agent (Claude billing detail,
Codex quota, Pi/OpenClaw dollar cost). Tests cover each new field.
Extend the inlined piModelMeta to pull message.stopReason and the usage `cost` object (input/output/cacheRead/cacheWrite/total), normalized in lockstep with src/notifications/model-usage.ts. Verified live: a real `pi --print` turn wrote token_usage.cost (dollars) + stop_reason to the sessions table.
…hared normalizer) Pass msg.stopReason into sdkTurnMeta so OpenClaw rows record why the turn ended; cost flows through the shared normalizer. Bump the skillify-wiring proximity assertion 4000→4400 to accommodate the added tagging block between `agent_end` and `Auto-captured` (same pattern as the prior 3500→4000 bump).
Codex quota (model_context_window + rate_limits) was retained across a new turn_context and a later token_count without extras couldn't clear it, so a new model's row could be tagged with the previous turn's quota. Reset extra on turn_context and assign unconditionally on each token_count (latest wins, may clear) — matching the existing per-turn reset for last_token_usage.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/shared/notifications-model-usage.test.ts (1)
322-369: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding a negative-case test for invalid quota values.
These tests cover valid
model_context_window/rate_limitsextraction and the turn-boundary reset, but don't exercisecodexUsageExtra's numeric guards (e.g., negativeused_percent, fractionalwindow_minutes/resets_at). A quick case confirming invalid fields are dropped rather than silently included would directly validate that defensive path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/shared/notifications-model-usage.test.ts` around lines 322 - 369, Add a negative-case test in the parseCodexTurnMeta usage_extra suite that supplies invalid codexUsageExtra quota fields, including negative used_percent and fractional window_minutes or resets_at. Assert the resulting usage_extra omits those invalid fields while retaining any valid quota fields, directly exercising the numeric guards.src/notifications/model-usage.ts (1)
304-328: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify
rate_limitsconstruction to avoid repeated casts.Building
extra.rate_limits = {}then repeatedly casting it back toRecord<string, unknown>to assignprimary/secondaryworks but is clunkier than building the object first.♻️ Suggested simplification
- const primary = pickWin(rateLimits?.primary); - const secondary = pickWin(rateLimits?.secondary); - if (primary || secondary) { - extra.rate_limits = {}; - if (primary) (extra.rate_limits as Record<string, unknown>).primary = primary; - if (secondary) (extra.rate_limits as Record<string, unknown>).secondary = secondary; - } + const primary = pickWin(rateLimits?.primary); + const secondary = pickWin(rateLimits?.secondary); + if (primary || secondary) { + const rl: Record<string, unknown> = {}; + if (primary) rl.primary = primary; + if (secondary) rl.secondary = secondary; + extra.rate_limits = rl; + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/notifications/model-usage.ts` around lines 304 - 328, In codexUsageExtra, simplify rate_limits construction by assembling a local object from the available primary and secondary windows, then assign it to extra.rate_limits once. Remove the repeated Record<string, unknown> casts while preserving omission of undefined windows and the existing empty-result behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/claude-code/skillify-session-start-injection.test.ts`:
- Around line 297-299: Replace the character-count-based source-order regex in
the skillify session-start test with structural anchors for the actual agent_end
hook, Auto-captured log, and spawnOpenclawSkillifyWorker({ ... }) call. Preserve
the required ordering while removing the permissive {0,4400} and {0,1500} gaps.
---
Nitpick comments:
In `@src/notifications/model-usage.ts`:
- Around line 304-328: In codexUsageExtra, simplify rate_limits construction by
assembling a local object from the available primary and secondary windows, then
assign it to extra.rate_limits once. Remove the repeated Record<string, unknown>
casts while preserving omission of undefined windows and the existing
empty-result behavior.
In `@tests/shared/notifications-model-usage.test.ts`:
- Around line 322-369: Add a negative-case test in the parseCodexTurnMeta
usage_extra suite that supplies invalid codexUsageExtra quota fields, including
negative used_percent and fractional window_minutes or resets_at. Assert the
resulting usage_extra omits those invalid fields while retaining any valid quota
fields, directly exercising the numeric guards.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ede1af65-1eab-4632-b7b5-9e1ba5638797
📒 Files selected for processing (8)
harnesses/openclaw/src/index.tsharnesses/pi/extension-source/hivemind.tsscripts/trace-model-usage-e2e.mjssrc/hooks/shared/redact.tssrc/notifications/model-usage.tstests/claude-code/skillify-session-start-injection.test.tstests/shared/notifications-model-usage.test.tstests/shared/redact.test.ts
🚧 Files skipped from review as they are similar to previous changes (5)
- tests/shared/redact.test.ts
- src/hooks/shared/redact.ts
- harnesses/openclaw/src/index.ts
- harnesses/pi/extension-source/hivemind.ts
- scripts/trace-model-usage-e2e.mjs
…t char gap
Address CodeRabbit: replace the brittle `agent_end{0,4400}Auto-captured...`
magic-gap regex with source-order index assertions on the real
`hook("agent_end")`, `Auto-captured`, and `spawnOpenclawSkillifyWorker(`
constructs. Robust to unrelated code landing between them (no future bumps).
…ition The order assertion matched the earlier function definition instead of the call after `Auto-captured`, failing the check. Search for the call from the Auto-captured index. Verified: suite passes.
Claude's effort (low/medium/high, ultrathink=high) is a user setting, not a per-message transcript field — so parseClaudeTurnMeta now reads `effortLevel` from settings (project settings.local.json > project settings.json > user ~/.claude/settings.json), falling back to null. Corrects the earlier claim that Claude has no capturable reasoning effort. Verified: a real transcript now yields reasoning_effort="medium" from settings. Tests cover precedence, missing, and malformed settings.
Hermes' pre_llm_call / post_llm_call hooks DO send `model` (and `platform`), nested in `extra` — everything that isn't tool_name/args/session_id lands there (NousResearch/hermes-agent agent/shell_hooks.py). Read extra.model onto the trace row and extra.platform into usage_extra. Token usage / cost are genuinely absent from the Hermes hook payload. Verified end-to-end: the real hermes capture hook wrote model + platform to the table.
Settings files are untrusted input — restrict effortLevel to Claude's supported levels (low/medium/high/xhigh, case-insensitive) so a malformed or oversized value can't be uploaded as reasoning_effort. Falls through otherwise. Tests cover rejection of a bogus value and case normalization.
…r rate_limits - Add a negative-case test: negative used_percent, fractional window_minutes / model_context_window are dropped by codexUsageExtra's numeric guards while valid fields are retained. - Simplify rate_limits assembly into a single local object (drop the repeated inline Record casts).
Resolve the last open CodeRabbit finding (O(n^2) re-read). Model / usage / stop_reason all live in the most recent turn at the END of the transcript, so parseClaudeTurnMeta / parseCodexTurnMeta now read only the tail (default 1 MiB) via a positioned read, dropping the partial leading line so callers only see whole lines. This bounds each capture event to a constant instead of O(filesize) — and O(n^2) across a Codex session that parses on every user/tool row. Falls back to null on any IO error (unchanged contract). Verified: a real 4.28 MB Claude transcript still extracts model/effort/tokens correctly from the 1 MiB tail. Tests cover the window boundary (partial first line dropped, newest line always present) and end-of-file extraction.
Three fixes on the tail reader: - Whole-file fallback: if a single record exceeds the tail window (giant Claude assistant line, or a Codex turn with >window bytes after its turn_context / token_count), the parsers re-scan the whole file so usage/effort/quota aren't silently lost. scanCodexLines reports foundData so a fallback-model-only result still triggers the widen. - Exact line boundary: read one extra leading byte and cut at the first newline, so a window that starts exactly on a line boundary keeps that complete line instead of dropping it. - Short read: Buffer.alloc (zeroed) + decode only the bytes actually read, so a truncated/racing read can't decode uninitialized memory. Tests: boundary-aligned window keeps the whole line; a >1 MiB assistant record is recovered via the whole-file fallback.
Address codex re-review: turn_context (model/effort) and token_count (usage/total/quota) carry different halves of a turn's metadata, so trusting the tail on "either seen" could drop half. scanCodexLines now reports sawTurnContext + sawTokenCount separately, and parseCodexTurnMeta only trusts the tail when BOTH are present — otherwise it re-scans the whole file (whose result is authoritative). Test: a >1 MiB record between turn_context and token_count still recovers model + effort via the whole-file fallback.
Two more edges from codex re-review: - Trust the tail only when it also carries the cumulative total_token_usage (whole-file scan preserves it across turns; an in-tail token_count may omit it if an earlier one set it). Otherwise widen. - If the whole-file re-read fails (file removed/truncated between reads), keep the tail's already-extracted usage instead of clobbering it with a model-only fallback. Test: a total set before the window, followed by a >1 MiB record and a token_count without total, is still recovered via the whole-file scan.
Close the file-mutation race class: if the whole-file reread comes back empty or event-less (file truncated/removed between reads), keep the tail's already-extracted usage instead of overwriting it with a model-only fallback. Trust the reread only when it actually saw a turn_context or token_count.
…n-race class) Replace the two independent readTailLines() opens with one openTranscript() that opens the file once, captures size once, and serves both the tail and the fallback whole-file read from the SAME fd/snapshot. This structurally closes the reopen/truncate-between-reads race the review kept surfacing: the fallback can no longer see a different (older/truncated) file than the tail. readTailLines stays as a thin wrapper for callers/tests. No behavior change on the happy path; verified on real Claude + Codex transcripts.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Root-level reference so agents/contributors can discover what each trace row records (model, reasoning effort, stop reason, token usage incl. cache tokens and cost, per-harness usage_extra), the source of each field per harness, the capture matrix, aggregation queries, and the design notes (best-effort, tail read, redaction guard). Linked from README Architecture.
…race) CodeQL flagged the existsSync()->openSync()/readFileSync() pre-check as a file-system race (2 high alerts on model-usage.ts). Open/read directly and handle the error (incl. ENOENT) instead of checking first — same behavior, no TOCTOU. Tail-read/O(1) design unchanged.
Lives alongside the per-agent integrations it documents. Fixed the internal link (../src/...) and the README pointer.
Update the README pointer to harnesses/TRACE_MODEL_USAGE.md and the doc's internal link to ../src/notifications/model-usage.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Records model, reasoning effort, and token consumption on captured session traces, so per-model token usage is a simple query over the
sessionstable. Covers every code assistant the plugin captures.Neither Claude Code nor Codex passes this in the hook payload, but both write it to the on-disk transcript the capture hooks already receive a path to. Pi and OpenClaw expose model+usage on the in-process SDK message. A shared module reads/normalizes it and the capture paths enrich the JSONB
messagecolumn — no schema migration.Coverage (verified end-to-end against real transcripts)
Stored shape (JSONB
message){ "model": "claude-opus-4-8", "reasoning_effort": null, "token_usage": { "input_tokens": 131, "output_tokens": 403, "cache_read_tokens": 357059, "cache_creation_tokens": 2102 } }Codex rows additionally carry
reasoning_effortandtoken_usage_total(whole-session cumulative).How
src/notifications/model-usage.ts:parseClaudeTurnMeta,parseCodexTurnMeta, andnormalizeSdkUsage/sdkTurnMeta(Pi/OpenClaw). Normalized keys across agents; absent fields stay absent; non-negative-safe-integer filtering; best-effort (null on any failure).src/hooks/capture.ts(Claude),src/hooks/codex/capture.ts(Codex),harnesses/openclaw/src/index.ts(import),harnesses/pi/extension-source/hivemind.ts(inlined — pi ships raw .ts).src/hooks/shared/redact.ts: guard so the entropy backstop doesn't shred provider model ids now stored in every row (would break per-model rollups) — narrow enough that a secret wearing a model prefix is still masked.Testing
scripts/trace-model-usage-e2e.mjsdrives the real capture hooks (Claude/Codex/Cursor/Hermes) across every model in the local transcripts, writes to a throwaway table, reads back, and asserts each agent's rows landed with token_usage for the token-bearing agents. Pi/OpenClaw capture in-process, so it proves their enriched entry from a real transcript message. Passes.tests/shared/notifications-model-usage.test.ts, redaction guard tests intests/shared/redact.test.ts.tscclean; full suite green except one pre-existing, unrelatedcli-bundle-runtimetree-sitter test (environmental; my diff touches no graph/CLI code).Review
codex reviewrun twice (initial + expanded scope); all findings fixed — SubagentStop transcript, Codex turn-boundary scoping, cumulative-vs-per-model semantics, non-object JSONL guard, redaction guard tightening, and e2e capture-verification. CodeRabbit findings (invalid token counts, per-turn effort reset) also fixed.Summary by CodeRabbit