feat(codex): add tool-excluded active Tok/s metrics - #805
Conversation
|
During code review, model found more and more discrepancies. This might be too big a merge but these models get really out of hand and I'm not expert enough to know what I should or should not include. Perhaps just have a model review the changes. The tok/s column is really useful and if you dont want to incorporate these bugfixes then feel free to just reimplement the tok/s column and dump my PR. I worked about an hour on it manually and the code review / bug bash lasted an hour and 8 minutes haha |
…hboard width Review fixes on top of getagentseal#805. Discovery: getCachedCodexProject short-circuits again without opening the file. The PR read the first line of every rollout to collect session_id, which cost +129ms on an 827-file session directory with a warm cache. Archived duplicates are now filtered by basename, which needs no file I/O. Timing: on oversized mcp_tool_call_end lines a `duration_ms` key inside invocation.arguments outranked the payload-level duration, inflating tool wait. The depth-aware payload value wins; the naive scan stays as the fallback for task_complete. Layout: MIN_WIDE goes back to 90 so 90-129 column terminals keep the two-column dashboard. The By Model panel drops the Tok/s column (and its footnote) when the panel has fewer than 61 inner columns or when no model recorded active timing, so narrow terminals do not truncate and non-Codex users get no dead column.
Active/tool-wait timing used to be back-patched onto calls that had already been appended to the result list. Buffer the calls decoded since the last task_started instead, stamp them in place on task_complete, and append them at the next task_started or at end of stream, so a task is only emitted once its timing is final. Single-pass and split decodes then agree instead of depending on where a decode boundary falls. Output is unchanged: parsing 827 real rollouts (4.2 GB, 51224 calls) with and without this change produces byte-identical results. The buffer holds one task's calls; the deepest observed over that corpus was 924 against a result list of 8196. Attribution still spans everything since the last task_started, which matters when a mid-file session_meta re-arms the fork-replay cutoff and swallows a task_started while its task_complete lands past the cutoff. The added test pins that case.
|
Thanks for this contribution — the throughput model held up well under a deep review (the active-time arithmetic verified end-to-end on real rollouts, the bounded reader is sound, and the cache versioning discipline was exactly right). I've pushed two maintainer commits on top rather than round-tripping review comments, since the fixes were small and surgical:
Also added the CHANGELOG entry, including disclosure of the parse-behavior corrections riding along (fork-replay double-count fix, Full suite green (2,480 passed; the 24 |
2 participants
Summary
This contribution adds a low-CPU, retrospective throughput view for Codex rollouts. CodeBurn now reports generated tokens per second (
Tok/s) from persisted JSONL timing while excluding recorded tool wait, so the estimate reflects active model work more closely than wall-clock turn duration.What changed
codeburn codex-tpswith one-shot text output,--json, bounded--limit, and incremental--watchmode.durationfields are not mistaken for the payload duration.Tok/simmediately after1-shotin the TUI’s By Model panel, with a concise explanation of the active-time denominator and excluded tool wait.Presentation
The repository’s generic dashboard image shows the existing CodeBurn visual language and is included here as a neutral reference:
This is an offline JSONL estimate—not live decoder speed. Server, prompt, and other unrecorded latency can still be included when the rollout does not expose a more precise interval.
Validation
npx tsc --noEmitpasses.npm run build:clipasses.npm run buildpasses for the CLI and dashboard; Vite reports its existing large-chunk advisory only.find-bugsandcode-reviewerpasses at GPT-5.6 Sol High approved the staged tree, including 5 MiB MCP argument/result collision fixtures for object and string durations.The unrelated full repository renderer harness remains environment-limited because its current setup lacks the Testing Library Chai matchers and has teardown noise; the feature-focused suite is green.
npm audit --omit=dev --audit-level=highstill reports the repository’s existing transitive advisories;npm audit fix --forcewould require a breaking MCP SDK change and was not applied.Thank you for maintaining CodeBurn and for making room for focused contributions like this—your project made it possible to diagnose the gap from real rollout data.