Skip to content

eval(harbor): stop the adapter injecting a deadline into clawcodex's prompt#751

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/harbor-deadline-injection
Jul 26, 2026
Merged

eval(harbor): stop the adapter injecting a deadline into clawcodex's prompt#751
ericleepi314 merged 1 commit into
mainfrom
fix/harbor-deadline-injection

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

The Harbor adapter appended Harbor's wall-clock deadline to clawcodex's system prompt via settings.append_system_prompt:

"This run has a hard external execution deadline at <ts>. By <ts>, preserve the best valid deliverable, stop broad exploration, and switch to the narrowest checks needed for the explicit requirements … You can use date -u to compare the current time."

The built-in claude-code adapter appends nothing (claude_code_subscription is a thin subclass touching only credentials), so this was harness-shaped guidance one agent got and the other didn't — and it told the model to narrow its own verification, which is precisely the behavior a trajectory comparison is trying to measure. eval/harbor/README.md already notes official submissions may not modify task timeouts, so it wouldn't exist on a scored run.

It never took effect — but not for the reason you'd guess

I initially wrote that the deadline resolved to None because hub trials don't materialize a task.toml. That was wrong. Re-running the deleted resolver against the recorded lock files and cached task definitions returns a real deadline for every task:

task resolved
cancel-async-tasks, regex-log, fix-code-vulnerability, sparql-university 900s
large-scale-text-editing 1200s
circuit-fibsqrt, distribution-search 3600s

The adapter did write it. The value was then silently dropped: settings.append_system_prompt is a dead config key — it parses into the settings object (src/settings/types.py:261) and nothing reads it. The headless chain ends at src/entrypoints/headless.py:504-513, which passes only the output-style prompt into build_effective_system_prompt. Verified with a sentinel under a fake $HOME: it loads into settings and never reaches the assembled prompt. (settings.effort is live via src/query/query.py:575-586, which is what made the sibling key look plausible.)

So prior clawcodex-vs-Claude-Code measurements weren't confounded by this — by accident, not design.

Follow-up worth filing: settings.append_system_prompt and settings.custom_system_prompt are silently ignored on both headless and TUI.

Adds eval/harbor/compare_trajectories.py

The comparisons driving recent prompt work were ad-hoc, and several were wrong the same way: clawcodex figures over a 7-task job compared against Claude Code figures over an 89-task job. Restricted to the shared tasks, several conclusions invert:

metric as reported actual (shared 7)
exact-repeat rate 5.3% vs 6.8% — CC repeats more 5.3% vs 0.0% — CC repeats nothing
heredoc-authored 12.1% vs 7.4% — clawcodex more 12.1% vs 23.4% — CC ~2× more
Bash statements/call 8.7 vs 9.3 ≈ same 4.4 vs 5.5 — CC packs more
env-probe share 13.3% vs 11.5% ≈ same 13.6% vs 8.5%
mean steps 12.9 vs 7.4 10.67 vs 7.43

The script only computes over the task intersection; reports harness-killed trials separately and excludes them from step means (iter2's large-scale-text-editing died on a TimeoutError — averaging a truncated run against completed ones inflated the gap from +3.2 to +5.5); states that "steps" means steps carrying a tool call (raw len(steps) is 14.7 vs 9.4); and always prints rewards, because all 14 trials on this set scored 1.0 — a step delta where both sides are already perfect is a cost difference, not a quality one.

Tests: 79 passing in the affected area; both adapters import under Harbor's interpreter. No orphan references to time_budget anywhere (src, tests, CI, Makefile, pyproject, or either eval README).

🤖 Generated with Claude Code

…prompt

The Harbor adapter appended Harbor's wall-clock deadline to clawcodex's
system prompt via ``settings.append_system_prompt``:

    "This run has a hard external execution deadline at <ts>. By <ts>,
     preserve the best valid deliverable, stop broad exploration, and
     switch to the narrowest checks needed for the explicit requirements
     ... You can use `date -u` to compare the current time."

The built-in claude-code adapter appends nothing (claude_code_subscription
is a thin subclass that touches only credentials), so this was
harness-shaped guidance one agent received and the other did not. It also
told the model to narrow its own verification, which is exactly the
behavior a trajectory comparison is trying to measure. Whatever is worth
doing under a deadline belongs in the product's prompt for every task, not
in an eval adapter for one agent. Official terminal-bench submissions may
not modify task timeouts, so it would not exist on a scored run either.

It never took effect
--------------------------------------------------------------------
Worth recording precisely, because the obvious guess is wrong.
``resolve_agent_timeout_seconds`` DID resolve a real deadline on every
task -- re-running the deleted resolver against the recorded lock files
and the cached task definitions returns 900s for cancel-async-tasks,
regex-log, fix-code-vulnerability and sparql-university, 1200s for
large-scale-text-editing, 3600s for circuit-fibsqrt and
distribution-search -- and the adapter did write it into the container's
config.

The value was then silently dropped: ``settings.append_system_prompt`` is
a dead config key. It parses into the settings object
(src/settings/types.py:261) and nothing ever reads it. The headless chain
ends at src/entrypoints/headless.py:504-513, which passes only the
output-style prompt into build_effective_system_prompt. Verified with a
sentinel under a fake $HOME: the value loads into settings and does not
appear in the assembled prompt. (``settings.effort`` IS live via
src/query/query.py:575-586, which is what made the sibling key look
plausible.)

So past clawcodex-vs-Claude-Code measurements were not confounded by this
-- by accident, not by design. Follow-up worth filing separately:
``settings.append_system_prompt`` and ``settings.custom_system_prompt``
are silently ignored on both the headless and TUI surfaces.

Adds eval/harbor/compare_trajectories.py
--------------------------------------------------------------------
The trajectory comparisons driving recent prompt work were ad-hoc and
several were wrong, all in the same way: clawcodex figures computed over a
7-task job were compared against Claude Code figures computed over an
89-task job. Restricted to the tasks the two jobs actually share, two
conclusions invert outright -- exact-repeat rate is 5.3% vs 0.0% (Claude
Code repeats nothing, not more), heredoc-authored scripts are 12.1% vs
23.4% (Claude Code does it about twice as often, not less).

The script only ever computes over the task intersection, reports
harness-killed trials separately and excludes them from step means (iter2's
large-scale-text-editing died on a TimeoutError; averaging a truncated run
against completed ones inflated the reported gap from +3.2 to +5.5 mean
steps), states that "steps" means steps carrying a tool call, and always
prints rewards -- a step delta on tasks where both sides already score 1.0
is a cost difference, not a quality one.
@ericleepi314
ericleepi314 merged commit 02fbcc9 into main Jul 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant