fix(eval): correct trajectory token/cost accounting + open with the user step#737
Merged
Conversation
…ser step Comparing the chess-best-move trajectories (clawcodex vs claude-code) in the Harbor viewer surfaced two gaps in the clawcodex trajectory: 1. TOKEN ACCOUNTING WAS WRONG: total_prompt_tokens=6 vs claude-code's 412649; cost=None. Root cause: the adapter read token totals from the stream-json `result.usage`, which clawcodex builds for live-CONTEXT measurement — `input_tokens` is the running sum of NON-cached input and it drops cumulative cache tokens (only a `last_*` snapshot survives). On a prompt-cached opus run that under-reports by ~5 orders of magnitude. Fix: read the authoritative BILLING totals from the persisted session's cost block `model_usage` (input + cache_read + cache_creation, summed per turn — the same convention claude-code reports), for BOTH the trajectory final_metrics AND the leaderboard token/cost columns (context.n_*_tokens / cost_usd). Stream-json usage stays the fallback only when no session cost block was synced. Verified end-to-end: a fresh deepseek trial went 208196 -> 1280580 prompt tokens (the 1.07M cache-read tokens are now counted) with cost 0.0377 (was None). 2. NO OPENING USER STEP in the stream-json (reconstructed) path: claude-code's trajectory opens with the task instruction; clawcodex's didn't. Fix: capture the instruction in run() and prepend it as step 1 in the fallback path (the rich/conversation path already carries it, so no double-inject). The backfill script recovers the instruction from the Harbor task cache (config.json task.ref -> instruction.md), so past jobs get it too — including 3 watchdog-crash trials that previously had no trajectory at all (now a 1-step "attempted" trace). Re-backfilled the 2026-07-19 opus job: 89/89 valid ATIF (was 86). No src/ changes — adapter + backfill only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion, drop vestigial arg, fix docstring Critic-tok APPROVE follow-ups (all non-gating): - _load_session_data now selects messages and billing INDEPENDENTLY (each by most messages), so a conversation-bearing session without a cost block still contributes its narration (was: skipped entirely). Unreachable today since Session.save writes the cost block unconditionally, but removes the latent footgun. - Drop the vestigial _instruction_step(_n) parameter. - backfill docstring: crashed trials now get a 1-step "attempted" trace (recovered instruction), not "no trajectory". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comparing the chess-best-move ATIF
trajectory.json(clawcodex vs harbor's built-in claude-code agent) surfaced two gaps in the clawcodex trajectory. Eval-tooling only (eval/harbor/).1. Token/cost accounting was wrong. The adapter set the trajectory
final_metricsAND the leaderboard columns (n_input_tokens/n_cache_tokens/n_output_tokens/cost_usd) from the stream-jsonresult.usage— which clawcodex builds for live-context measurement (input_tokens= running sum of NON-cached input; cumulative cache tokens dropped, only alast_*snapshot survives). On a prompt-cached opus run this reported total_prompt_tokens=6 vs claude-code's 412649, costNone. Fix: read the authoritative billing totals from the persisted session's cost blockmodel_usage(input + cache_read + cache_creationsummed per turn — the exact convention claude-code reports), for both the trajectory and the leaderboard. Stream-json usage stays the fallback only when no session cost block was synced.2. No opening user step. claude-code's trajectory opens with the task instruction; clawcodex's stream-json (reconstructed) path had none. Fix:
run()captures the instruction and the fallback path prepends it as step 1 (rich/conversation path already carries it — no double-inject). The backfill script recovers the instruction from the Harbor task cache so past jobs get it too.Test plan
total_prompt_tokens165812→1082036 (the 916224 cache-read tokens now counted), cost None→0.0306; cross-checks exactly against the synced sessionmodel_usage; leaderboard columns == final_metrics; reward 1.0; rich path step1=user, 39/39 narrated+timestampsclaude_code.py:_build_metrics(prompt=input+cache_read+cache_creation; cached=cache_read); no double-count (model_usageis per-turn-summed)Known remaining gap (documented, follow-up): per-step token metrics — claude-code records per-message usage in its session log; clawcodex's persisted conversation doesn't, so trajectory steps carry aggregate metrics only. Requires a production persistence change.
🤖 Generated with Claude Code