feat: per-step trajectory token metrics + list-price cost estimate on subscription#738
Merged
Merged
Conversation
… subscription
Closes the two documented trajectory-parity gaps vs the built-in
claude-code agent.
FOLLOWUP 1 — per-step token metrics. claude-code's trajectory carries
per-turn token usage on every agent step; clawcodex's didn't, because the
persisted conversation dropped it. AssistantMessage already holds the
turn's usage (query.py sets usage=response.usage), but:
- headless _persist called add_message(role, content), dropping usage;
- message_to_dict never serialized usage (message_from_dict already
loaded it).
Fix (production src): thread usage through add_message → create_message →
create_assistant_message (which already accepts it), have _persist pass
msg.usage, and serialize usage in message_to_dict. Round-trips now. The
Harbor adapter builds per-step ATIF Metrics (prompt=input+cache_read+
cache_creation, cached=cache_read, completion=output) from it.
FOLLOWUP 2 — cost on subscription runs. record_api_usage zeroes billed
cost under a subscription (plan allowance, not metered credits — correct
for the live /cost display), so the session cost block reported $0 while
claude-code reports a list-price cost. Fix: build_cost_block now also
emits estimated_cost_usd — the always-computed list-price figure (via
compute_cost, ignoring billing_mode) — purely additive, never touching
/cost or the budget gate. The adapter uses it for the leaderboard/
trajectory cost column when billed cost is 0.
Verified: fresh headless run persists per-message usage → 4/5 steps get
Metrics (user step none, correct); subscription opus tokens → billed $0 /
estimated $1.10 (== compute_cost); metered → billed wins. 1348 tests pass
across message/conversation/cost/headless/session + new round-trip and
estimated-cost tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two documented trajectory-parity gaps vs harbor's built-in claude-code agent (surfaced by the chess-best-move comparison).
1. Per-step token metrics. claude-code's ATIF trajectory carries per-turn token usage on each agent step; clawcodex's didn't because the persisted conversation dropped per-message usage.
AssistantMessagealready holds the turn's usage (query.pysetsusage=response.usage), but headless_persistdropped it andmessage_to_dictnever serialized it. Fix (production src): threadusagethroughadd_message→create_message→create_assistant_message, persist it in_persist, serialize it inmessage_to_dict(message_from_dictalready loaded it). The adapter builds per-step ATIFMetricsfrom it.2. Cost on subscription runs.
record_api_usagezeroes billed cost under a subscription (correct for live/cost— plan allowance, not metered credits), so the cost block reported $0 while claude-code reports a list-price cost. Fix:build_cost_blockalso emitsestimated_cost_usd— always computed viacompute_cost(ignores billing_mode). Purely additive; never touches/costor the budget gate. The adapter uses it for the cost column when billed cost is 0.Test plan
total_cost_usd=0.0(billed) +estimated_cost_usd=0.4906(list-price) → trajectory/leaderboardcost_usd=0.4906(was 0.0/None). Cost reconciles exactly with opus pricing.🤖 Generated with Claude Code