eval(harbor): stop the adapter injecting a deadline into clawcodex's prompt#751
Merged
Conversation
…prompt
The Harbor adapter appended Harbor's wall-clock deadline to clawcodex's
system prompt via ``settings.append_system_prompt``:
"This run has a hard external execution deadline at <ts>. By <ts>,
preserve the best valid deliverable, stop broad exploration, and
switch to the narrowest checks needed for the explicit requirements
... You can use `date -u` to compare the current time."
The built-in claude-code adapter appends nothing (claude_code_subscription
is a thin subclass that touches only credentials), so this was
harness-shaped guidance one agent received and the other did not. It also
told the model to narrow its own verification, which is exactly the
behavior a trajectory comparison is trying to measure. Whatever is worth
doing under a deadline belongs in the product's prompt for every task, not
in an eval adapter for one agent. Official terminal-bench submissions may
not modify task timeouts, so it would not exist on a scored run either.
It never took effect
--------------------------------------------------------------------
Worth recording precisely, because the obvious guess is wrong.
``resolve_agent_timeout_seconds`` DID resolve a real deadline on every
task -- re-running the deleted resolver against the recorded lock files
and the cached task definitions returns 900s for cancel-async-tasks,
regex-log, fix-code-vulnerability and sparql-university, 1200s for
large-scale-text-editing, 3600s for circuit-fibsqrt and
distribution-search -- and the adapter did write it into the container's
config.
The value was then silently dropped: ``settings.append_system_prompt`` is
a dead config key. It parses into the settings object
(src/settings/types.py:261) and nothing ever reads it. The headless chain
ends at src/entrypoints/headless.py:504-513, which passes only the
output-style prompt into build_effective_system_prompt. Verified with a
sentinel under a fake $HOME: the value loads into settings and does not
appear in the assembled prompt. (``settings.effort`` IS live via
src/query/query.py:575-586, which is what made the sibling key look
plausible.)
So past clawcodex-vs-Claude-Code measurements were not confounded by this
-- by accident, not by design. Follow-up worth filing separately:
``settings.append_system_prompt`` and ``settings.custom_system_prompt``
are silently ignored on both the headless and TUI surfaces.
Adds eval/harbor/compare_trajectories.py
--------------------------------------------------------------------
The trajectory comparisons driving recent prompt work were ad-hoc and
several were wrong, all in the same way: clawcodex figures computed over a
7-task job were compared against Claude Code figures computed over an
89-task job. Restricted to the tasks the two jobs actually share, two
conclusions invert outright -- exact-repeat rate is 5.3% vs 0.0% (Claude
Code repeats nothing, not more), heredoc-authored scripts are 12.1% vs
23.4% (Claude Code does it about twice as often, not less).
The script only ever computes over the task intersection, reports
harness-killed trials separately and excludes them from step means (iter2's
large-scale-text-editing died on a TimeoutError; averaging a truncated run
against completed ones inflated the reported gap from +3.2 to +5.5 mean
steps), states that "steps" means steps carrying a tool call, and always
prints rewards -- a step delta on tasks where both sides already score 1.0
is a cost difference, not a quality one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Harbor adapter appended Harbor's wall-clock deadline to clawcodex's system prompt via
settings.append_system_prompt:The built-in claude-code adapter appends nothing (
claude_code_subscriptionis a thin subclass touching only credentials), so this was harness-shaped guidance one agent got and the other didn't — and it told the model to narrow its own verification, which is precisely the behavior a trajectory comparison is trying to measure.eval/harbor/README.mdalready notes official submissions may not modify task timeouts, so it wouldn't exist on a scored run.It never took effect — but not for the reason you'd guess
I initially wrote that the deadline resolved to
Nonebecause hub trials don't materialize atask.toml. That was wrong. Re-running the deleted resolver against the recorded lock files and cached task definitions returns a real deadline for every task:The adapter did write it. The value was then silently dropped:
settings.append_system_promptis a dead config key — it parses into the settings object (src/settings/types.py:261) and nothing reads it. The headless chain ends atsrc/entrypoints/headless.py:504-513, which passes only the output-style prompt intobuild_effective_system_prompt. Verified with a sentinel under a fake$HOME: it loads into settings and never reaches the assembled prompt. (settings.effortis live viasrc/query/query.py:575-586, which is what made the sibling key look plausible.)So prior clawcodex-vs-Claude-Code measurements weren't confounded by this — by accident, not design.
Follow-up worth filing:
settings.append_system_promptandsettings.custom_system_promptare silently ignored on both headless and TUI.Adds
eval/harbor/compare_trajectories.pyThe comparisons driving recent prompt work were ad-hoc, and several were wrong the same way: clawcodex figures over a 7-task job compared against Claude Code figures over an 89-task job. Restricted to the shared tasks, several conclusions invert:
The script only computes over the task intersection; reports harness-killed trials separately and excludes them from step means (iter2's
large-scale-text-editingdied on aTimeoutError— averaging a truncated run against completed ones inflated the gap from +3.2 to +5.5); states that "steps" means steps carrying a tool call (rawlen(steps)is 14.7 vs 9.4); and always prints rewards, because all 14 trials on this set scored 1.0 — a step delta where both sides are already perfect is a cost difference, not a quality one.Tests: 79 passing in the affected area; both adapters import under Harbor's interpreter. No orphan references to
time_budgetanywhere (src, tests, CI, Makefile, pyproject, or either eval README).🤖 Generated with Claude Code