M3-5: run arm 2 (three-way agent cost comparison) — closes M3 - #70
Merged
Conversation
Arm 2 measures tokens, tool calls, error rate, and wall clock for the same task suite (clamp, titlecase) completed three ways: GitHub + gh (real GitHub, DeDuva/adp-bench-arm2-scratch), ADP via gh (the compat plane, unmodified gh through GH_HOST), and ADP-MCP (the native plane's own MCP tool surface, server/src/mcp/server.ts, run as a real subprocess). The driver (bench/arms/three-way-cost.mjs) uses the claude CLI non-interactively as the real agent, scoped per method via --allowedTools/--mcp-config. Pilot slice: 12/12 trials landed, $1.25 total spend. adp-gh costs about the same as github-gh; adp-mcp costs ~1.7x more, traced to a real gap (no proposal-open tool in the current ~8-tool native MCP surface, so an agent using it pays a REST round trip gh bundles into one command) rather than anything inherent to the native plane. Full writeup: bench/report/three-way-cost.md. This closes M3: all three benchmark arms are now published (arm 1 deterministic/CI-enforced, arm 2 here, arm 3 in squad's duva-bench track — github.com/DeDuva/squad/pull/119). ROADMAP.md and docs/pragmatic_mvp.md updated accordingly.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs M3-5 arm 2 — the last piece of M3 — and closes the milestone.
Arm 2 measures tokens, tool calls, error rate, and wall clock for the same
task suite completed three ways: GitHub +
gh(real GitHub repo,DeDuva/adp-bench-arm2-scratch), ADP viagh(the compat plane,unmodified
ghthroughGH_HOST— the MVP's own success criterion), andADP-MCP (the native plane's own MCP tool surface,
server/src/mcp/server.ts, run as a real subprocess the agent talks to).bench/arms/three-way-cost.mjs: one trial (method × task × rep) perinvocation, same granularity as arm 1's driver. The real agent is the
claudeCLI, non-interactive, tool-scoped per method via--allowedTools/--mcp-config.bench/tasks/arm2/: two small, deterministic-to-grade tasks (clamp,titlecase) — new to this arm, not reused from squad-lab, so ADP's ownbench owns its own task suite per the M3-5 layout.
bench/report/three-way-cost.md: generated frombench/runs/, samereproducibility contract as arm 1 —
npm run report(now regenerates bothreports) reproduces it byte-for-byte from committed records.
Result
Pilot slice (12 trials: 3 methods × 2 tasks × 2 reps): 12/12 landed, $1.25
total spend.
adp-ghcosts about the same asgithub-gh($0.0848 vs $0.0850avg/trial) — once GH_HOST-vs-git-remote host matching and this instance's
one_approvalland policy were handled, unmodifiedghpays no realagent-cost tax against ADP versus GitHub itself.
adp-mcpcosts ~1.7x more($0.1435 avg/trial), traced to a specific, closeable gap: the ~8-tool native
MCP surface has no proposal-open tool, so an agent using it pays a REST round
trip
ghbundles into one command — a finding about today's MCP toolcoverage, not a verdict on the native plane's design. Full writeup and every
number's derivation:
bench/report/three-way-cost.md.Closes M3
All three benchmark arms are now published:
squad PR #119.
ROADMAP.md: M3 → complete 2026-08-10, Now/Next moved to M4 (not started),the M3-5 budget item removed from Blockers.
docs/pragmatic_mvp.md: theM3-5 "not run" sentence replaced with the landed status and links, no other
narrative changes.
Scope note
The one implementation choice worth recording: the native MCP tool surface
has candidate-set open/select/resolve but no tool to open the underlying
proposal itself (that verb only exists on the compat-plane REST route,
/repos/{owner}/{repo}/pulls). Theadp-mcparm's agent is given one scopedcurlfor exactly that one call — everything else (workspace/branch viagit, landing via the native MCP tools) goes through the native plane. This
is called out in the report's own "what this shows" section as a real
finding, not smoothed over.
Test plan
make checkgreen (had to stop a leftover long-running dev serverfrom earlier in this session that was sharing the test Postgres with
make test-alland intermittently failing the mirror-mode tests — not aregression from this diff, confirmed by re-running
test/e2e-mirror.test.tsin isolation both before and after)
npm run report --prefix benchregenerates both reports byte-identicalto what's committed (
git diff --exit-code bench/report/clean)bench/runs/reconcile with the numbers inbench/report/three-way-cost.md