Skip to content

M3-5: run arm 2 (three-way agent cost comparison) — closes M3 - #70

Merged
DeDuva merged 1 commit into
mainfrom
bench/m3-5-arm2-three-way-cost
Aug 10, 2026
Merged

M3-5: run arm 2 (three-way agent cost comparison) — closes M3#70
DeDuva merged 1 commit into
mainfrom
bench/m3-5-arm2-three-way-cost

Conversation

@DeDuva

@DeDuva DeDuva commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Summary

Runs M3-5 arm 2 — the last piece of M3 — and closes the milestone.

Arm 2 measures tokens, tool calls, error rate, and wall clock for the same
task suite completed three ways: GitHub + gh (real GitHub repo,
DeDuva/adp-bench-arm2-scratch), ADP via gh (the compat plane,
unmodified gh through GH_HOST — the MVP's own success criterion), and
ADP-MCP (the native plane's own MCP tool surface,
server/src/mcp/server.ts, run as a real subprocess the agent talks to).

  • bench/arms/three-way-cost.mjs: one trial (method × task × rep) per
    invocation, same granularity as arm 1's driver. The real agent is the
    claude CLI, non-interactive, tool-scoped per method via
    --allowedTools/--mcp-config.
  • bench/tasks/arm2/: two small, deterministic-to-grade tasks (clamp,
    titlecase) — new to this arm, not reused from squad-lab, so ADP's own
    bench owns its own task suite per the M3-5 layout.
  • bench/report/three-way-cost.md: generated from bench/runs/, same
    reproducibility contract as arm 1 — npm run report (now regenerates both
    reports) reproduces it byte-for-byte from committed records.

Result

Pilot slice (12 trials: 3 methods × 2 tasks × 2 reps): 12/12 landed, $1.25
total spend. adp-gh costs about the same as github-gh ($0.0848 vs $0.0850
avg/trial) — once GH_HOST-vs-git-remote host matching and this instance's
one_approval land policy were handled, unmodified gh pays no real
agent-cost tax against ADP versus GitHub itself. adp-mcp costs ~1.7x more
($0.1435 avg/trial), traced to a specific, closeable gap: the ~8-tool native
MCP surface has no proposal-open tool, so an agent using it pays a REST round
trip gh bundles into one command — a finding about today's MCP tool
coverage, not a verdict on the native plane's design. Full writeup and every
number's derivation: bench/report/three-way-cost.md.

Closes M3

All three benchmark arms are now published:

  • Arm 1 (merge contention) — deterministic, CI-enforced, unchanged by this PR.
  • Arm 2 (this PR) — pilot scale, 12 trials.
  • Arm 3 (fan-out vs serial) — run in squad's duva-bench track:
    squad PR #119.

ROADMAP.md: M3 → complete 2026-08-10, Now/Next moved to M4 (not started),
the M3-5 budget item removed from Blockers. docs/pragmatic_mvp.md: the
M3-5 "not run" sentence replaced with the landed status and links, no other
narrative changes.

Scope note

The one implementation choice worth recording: the native MCP tool surface
has candidate-set open/select/resolve but no tool to open the underlying
proposal itself (that verb only exists on the compat-plane REST route,
/repos/{owner}/{repo}/pulls). The adp-mcp arm's agent is given one scoped
curl for exactly that one call — everything else (workspace/branch via
git, landing via the native MCP tools) goes through the native plane. This
is called out in the report's own "what this shows" section as a real
finding, not smoothed over.

Test plan

  • make check green (had to stop a leftover long-running dev server
    from earlier in this session that was sharing the test Postgres with
    make test-all and intermittently failing the mirror-mode tests — not a
    regression from this diff, confirmed by re-running test/e2e-mirror.test.ts
    in isolation both before and after)
  • npm run report --prefix bench regenerates both reports byte-identical
    to what's committed (git diff --exit-code bench/report/ clean)
  • All 12 run records in bench/runs/ reconcile with the numbers in
    bench/report/three-way-cost.md

Arm 2 measures tokens, tool calls, error rate, and wall clock for the same
task suite (clamp, titlecase) completed three ways: GitHub + gh (real
GitHub, DeDuva/adp-bench-arm2-scratch), ADP via gh (the compat plane,
unmodified gh through GH_HOST), and ADP-MCP (the native plane's own MCP
tool surface, server/src/mcp/server.ts, run as a real subprocess). The
driver (bench/arms/three-way-cost.mjs) uses the claude CLI non-interactively
as the real agent, scoped per method via --allowedTools/--mcp-config.

Pilot slice: 12/12 trials landed, $1.25 total spend. adp-gh costs about the
same as github-gh; adp-mcp costs ~1.7x more, traced to a real gap (no
proposal-open tool in the current ~8-tool native MCP surface, so an agent
using it pays a REST round trip gh bundles into one command) rather than
anything inherent to the native plane. Full writeup:
bench/report/three-way-cost.md.

This closes M3: all three benchmark arms are now published (arm 1
deterministic/CI-enforced, arm 2 here, arm 3 in squad's duva-bench track —
github.com/DeDuva/squad/pull/119). ROADMAP.md and docs/pragmatic_mvp.md
updated accordingly.
@DeDuva
DeDuva merged commit d697d89 into main Aug 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant