perf(prompt): restore the reference's skip conditions on the task tool#754
Merged
Conversation
Iteration 5. First one derived from classifying trajectory steps by PURPOSE
rather than by tool, which is what finally located the gap.
Seven shared terminal-bench 2.1 tasks, same model (claude-opus-5), same
effort (xhigh), per trial, two clawcodex runs against the latest Claude
Code:
clawcodex Claude Code
explore:inspect 1.71 / 1.57 2.43 <- Claude Code inspects MORE
explore:read 0.86 / 1.14 0.57
author:inline 2.71 / 2.29 1.14
author:deliverable 1.29 / 2.43 1.14
run:exec 2.29 / 3.57 1.14 <- the biggest single gap
meta:memory 0.29 / 0.57 0.00
meta:tasks 0.14 / 0.29 0.00
TOTAL 12.14 / 15.43 8.43
Claude Code buys about 0.7 extra inspection steps and saves 1.2-2.4
execution steps. It looks at the actual data, then writes something that
works first time. clawcodex writes from assumptions, runs, and pays in
write-run-fix cycles -- run:exec and author:inline are each roughly double
Claude Code's, while inspection is LOWER.
Every tool-level metric already matches (Bash 78.8% vs 79.7%, Write 11.8%
vs 11.9%, per-call density 5.2 vs 5.5 statements, heredoc 20.9% vs 23.4%,
exact-repeat 2.4% vs 0.0%). Purpose-level classification is what separated
"same actions, more of them" into "fewer looks, more retries".
This also explains a failure. Iteration 3 added the opposite instruction --
act once you have enough information, do not re-check what the evidence
supports -- and made both step count (+3.2 -> +7.2) and reward (1.000 ->
0.857) worse. It was pushing clawcodex further toward acting before
looking, when under-inspection was the actual deficit. That bullet was
withdrawn; this is the corrected direction, and the earlier failure is
evidence for it.
Scope is deliberately narrow: the assumptions the NEXT action rests on, not
a survey of the environment. #749 had to remove exhaustive-audit machinery
that fired on ordinary English, so the limiter is load-bearing and pinned
by the test.
Not claimed: a verified step reduction. Same-code replicates on this task
set swing up to 11 steps per task and 3.43 in the mean, so at k=1 nothing
under ~3 mean steps is resolvable. Claude Code replicates are in flight to
establish a k=3 baseline; this change gets measured against it, not against
a single run.
The port kept "Break down and manage your work with the TaskCreate tool"
and dropped the reference's gating, which reads: "Skip using this tool
when: There is only a single, straightforward task; the task is trivial and
tracking it provides no organizational benefit." What survived is an
unconditional imperative.
Measured over 14 clawcodex and 21 Claude Code trials on seven shared
terminal-bench 2.1 tasks, same model (claude-opus-5) and effort (xhigh),
steps per trial by purpose:
clawcodex Claude Code delta
run:exec 3.36 2.52 +0.83
other 1.43 0.67 +0.76
author:inline 2.50 1.90 +0.60
meta:memory 0.43 0.00 +0.43
explore:env 1.07 0.67 +0.40
meta:tasks 0.21 0.00 +0.21
explore:inspect 1.64 2.29 -0.64
author:file 2.00 2.19 -0.19
TOTAL 13.79 11.38 +2.40
Claude Code spends ZERO steps on the task tool across 21 trials and zero on
memory writes. clawcodex spends 0.64 per trial on the two together -- 27% of
the whole remaining gap, on bookkeeping rather than the task. This commit
addresses the task-tool half, where the fix is a dropped reference qualifier
rather than a judgment call.
The memory half is deliberately untouched. Its "build up this memory system
over time" imperative IS in the reference, and memory is a shipped feature
whose quality this eval cannot see -- changing it on step-count evidence is
what made iteration 3 regress reward.
Also of note in the table: "Claude Code inspects more" (-0.64) survives
Claude Code's replication, so the iteration-5 premise was sound even though
the change built on it measured null and was dropped.
Not claimed: a verified step reduction. 0.21 steps/trial is far inside the
same-code noise floor (median 2, max 11 per task). This lands because the
dropped qualifier is a real port defect, not because the number is
measurable.
ericleepi314
added a commit
that referenced
this pull request
Jul 26, 2026
#754 was branched while HEAD still sat on the iteration-5 commit, so the squash carried in a change I had measured as null and said I would not merge. This removes it. The task-tool skip conditions that #754 was actually for are untouched. The bullet ("Before running something you just wrote, confirm the specific assumptions it depends on...") rested on a purpose-level classification showing Claude Code inspecting more than clawcodex (2.43 vs 1.57-1.71 steps per trial). That result was a CLASSIFIER ARTIFACT: the regex anchored the verb at the start of the command, so clawcodex's `cd /app && grep ...` inspection calls fell into an "other" bucket instead. Stripping `cd`/`&&` prefixes before classifying reverses it: clawcodex Claude Code delta explore:inspect 4.93 3.95 +0.98 <- clawcodex inspects MORE meta 0.64 0.00 +0.64 author:inline 2.93 2.52 +0.40 run:exec 1.86 1.57 +0.29 So the premise was false, which explains the null measurement (4 of 5 tasks worse, mean 9.9 -> 10.2). Anchor a command classifier on the first real verb, not the first token. What survives from that analysis is the `meta` row: memory writes plus task-tool calls cost clawcodex 0.64 steps per trial where Claude Code spends ZERO across 21 trials -- 26% of the remaining gap. #754's skip-condition restoration addresses the task-tool half and stands on the dropped-qualifier argument, independent of the misclassified rows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The port kept "Break down and manage your work with the TaskCreate tool" and dropped the reference's gating — "Skip using this tool when: There is only a single, straightforward task; the task is trivial and tracking it provides no organizational benefit." What survived is an unconditional imperative.
Measured
14 clawcodex trials and 21 Claude Code trials on seven shared terminal-bench 2.1 tasks, same model (
claude-opus-5), same effort (xhigh). Steps per trial, classified by purpose:Claude Code spends zero steps on the task tool across 21 trials, and zero on memory writes. clawcodex spends 0.64/trial on the two combined — 27% of the remaining step gap, on bookkeeping rather than the task. This addresses the task-tool half, where the fix is a dropped reference qualifier rather than a judgment call.
The memory half is deliberately untouched. Its "build up this memory system over time" imperative is in the reference, and memory is a shipped feature whose quality this eval cannot observe. Trading invisible quality for measurable steps is what made an earlier iteration regress reward.
Also visible in the table: "Claude Code inspects more" (−0.64) survives Claude Code's replication, so that premise was sound even though the change built on it measured null and was dropped.
Not claimed
A verified step reduction. 0.21 steps/trial is far inside the same-code noise floor (median 2, max 11 per task). This lands because the dropped qualifier is a real port defect, not because the effect is measurable.
Tests: 8842 passed, 3 skipped, 0 failures.
🤖 Generated with Claude Code