feat(eval): emit ATIF trajectory.json from the clawcodex harbor adapter#736
Merged
Conversation
Harbor's built-in claude-code agent writes a rich step-by-step
trajectory.json (ATIF) that its viewer/leaderboard render; the clawcodex
adapter emitted none. Add it:
- eval/harbor/clawcodex_agent.py: SUPPORTS_ATIF=True; populate_context_
post_run now builds an ATIF-v1.7 Trajectory (Agent header from the
system:init event; Steps with per-turn message/reasoning/tool_calls and
observations; FinalMetrics from the stream-json result usage) and writes
trajectory.json. Two sources, preferred order:
* the persisted session conversation (RICH — per-turn narration,
reasoning, tool calls AND results), synced under <logs>/sessions/;
* the stream-json log (always present) — every tool call+result plus
the final answer, and the authoritative token usage.
Best-effort throughout: a build failure never fails the trial. Token
backfill behavior is unchanged.
- src/entrypoints/headless.py: persist the session (session.save()) at the
end of a headless run. The print path previously left no session on
disk, so `-p` runs were un-resumable and carried no structured
transcript; this both enables --resume for headless and gives the
adapter its rich trajectory source. Best-effort (never changes exit
code/output).
Verified: rich path yields user→agent(tool+result)→agent steps with
narration; fallback path yields 53 tool steps + metrics from a real
stream-json log; a live headless run persists the conversation; 669
headless/session tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ill script (critic follow-ups) - Read total_cost_usd from the persisted session's cost block (the --print result event has no cost field) and set both FinalMetrics.total_cost_usd AND context.cost_usd — leaderboard cost parity with the claude-code agent (critic's substantive finding). - Fallback (stream-json-only) trajectories carry a Trajectory.notes marker explaining per-step narration is unavailable. - Dedup _load_session_data's rglob (was scanning the sessions/ subtree twice via nested roots). - Add eval/harbor/backfill_trajectories.py: reproduce trajectory.json for a PAST job from its stream-json logs, no re-run. Idempotent (skips existing unless --force); reports trials that crashed before any event. Backfilled 86/89 of the 2026-07-19 opus job (3 misses = watchdog-crash trials whose 5-line logs have no events to reconstruct). Critic verdict on the branch: APPROVE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Harbor's built-in
claude-codeagent writes a rich step-by-steptrajectory.json(ATIF) that its viewer/leaderboard render; the clawcodex adapter emitted none. This adds it, at parity.eval/harbor/clawcodex_agent.py:SUPPORTS_ATIF=True;populate_context_post_runbuilds an ATIF-v1.7Trajectory(Agent header from thesystem:initevent;Steps with per-turn message/reasoning/tool_calls + observations;FinalMetricsfrom token usage;total_cost_usd/context.cost_usdfrom the session cost block). Two sources:<logs>/sessions/;notesmarker noting narration is unavailable.Best-effort throughout — a build failure never fails the trial. Token backfill unchanged.
src/entrypoints/headless.py: persist the session (session.save()) at the end of a--printrun (best-effort). The print path left no session on disk, so-pruns were un-resumable and carried no transcript; this enables--resumefor headless and gives the adapter its rich source. Verified no secret is serialized (session json holds provider/model names, conversation, timestamps, cost — no keys/tokens).eval/harbor/backfill_trajectories.py: reproducetrajectory.jsonfor a past job from its stream-json logs, no re-run.Test plan
trajectory.jsonappears; rich path — 43 steps, 43/43 narrated, multi-tool turns bundled; 62 observations, 0 cross-step mismatches, sequential step_ids; trial reward 1.0, token backfill intactFinalMetrics.total_cost_usd+context.cost_usdsession.save()change🤖 Generated with Claude Code