diff --git a/add-method/docs/appendix-h-benchmark.md b/add-method/docs/appendix-h-benchmark.md new file mode 100644 index 00000000..733375d6 --- /dev/null +++ b/add-method/docs/appendix-h-benchmark.md @@ -0,0 +1,146 @@ +# Appendix H — Measured: ADD vs spec-kit vs GSD vs vanilla + +ADD ships with its own adversarial benchmark (`benchmark/` in the repo): four +agent flows build the same longitudinal project — a booking REST API + CLI that +**evolves** across workload milestones (WM1 → WM3), including a deliberately +hostile change request — on a pinned meter (`claude-sonnet-5`, effort medium), +scored by deterministic oracles, tamper detection, and regression probes. This +appendix is the honest scoreboard. Two things make these numbers worth reading: + +1. **We attack our own meter.** Six meter defects were found and fixed live + during the 2026-07 campaigns (identical scores across independent arms + indict the meter, not the arms; a too-cheap run is a defect smell). An + early result favorable to ADD was **retracted** when the fidelity judge + proved untrustworthy, and the campaign re-ran on the fixed meter. What + survived that process is below, with raw records archived per run. +2. **Honest-outcome clause.** Findings that cut against ADD print unchanged. + The cost gap is real and stated first. + +## The scoreboard (same meter, same day, n=1 rep) + +Three milestones, same workspace evolving. The primary table is a fresh run of all +three flows on one day, one meter, one workload +(`2026-07-14-v2-sameday.md`) — so they compare without vintage skew. Vanilla carries +its 2026-07-10 WV1 figure (`2026-07-wv1-rep0.md`), the last time it was run. + +The "pass" column is `oracle_pass_rate` — the v2 meter's *deterministic* fidelity +of record (the milestone's own probe suite run against the built app; no LLM in the +scoring path). + +| arm | WM1 pass/reg | WM2 pass/reg | WM3 pass/reg | rep cost | tokens | +|---|---|---|---|---|---| +| **ADD** (current main) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $9.11 | 18.19M | +| GitHub **spec-kit** (v0.12.5) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $4.20 | 6.03M | +| **GSD** (get-shit-done-cc 1.42.3) | 1.00 / 0 | **0.80** / 0 | 1.00 / 0 | $3.01 | 3.70M | +| **vanilla** Claude Code † | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $3.07 | 4.3M | + +† vanilla not re-run this round; 2026-07-10 WV1 figure carried forward. + +**The honest headline: on a friendly workload at n=1, cost separates and trust +does not.** Every arm held its oracle. If your project is a weekend prototype, +spec-kit, vanilla, or GSD is genuinely cheaper — ADD's own `prototype` stage says +the same thing (run light, code is throwaway). ADD costs 2.2× spec-kit and 3.0× GSD +here, the same ordering the 2026-07-10 WV1 campaign found. Two honest notes on the +cost gap: **(1)** current ADD is ~$9, not the ~$14 the older WV1 `add-main` control +showed — that control was pinned before the lean / ceremony-to-effort / six-phase +reductions merged, and this same-day run measures the merged main. **(2)** adjusted +for the trust vector (own-suite evidence weighted by weakened-test adjudication), the +WV1 scoring report put cost *per trusted feature* at spec-kit $1.14 · vanilla $1.53 · +ADD $4.65 — ADD's premium is real, and it buys enforcement, not output. + +## The hostile change (WV2 — gaming resistance) + +The arms then received an undirected change request that **contradicts a frozen +rule**, where the cheapest green is quietly weakening your own overlap tests. +Source: `benchmark/results/2026-07-wv2-rep0.md`. + +| arm | hostile step pass / regression / tests weakened | rep cost | +|---|---|---| +| ADD (main) | 1.00 / 0 / **0** | $11.14 | +| ADD (lean branch) | 1.00 / 0 / **0** | $8.16 | +| spec-kit | 1.00 / 0 / **0** | $3.08 | +| vanilla | 1.00 / 0 / **0** | $3.01 | + +No arm gamed at this temptation strength — modern models re-spec honestly when +the amendment has a natural scoped form. The difference is *what guarantees it*: + +- In vanilla/spec-kit/GSD, honesty under pressure is a **model property** — + it held here, and nothing detects the day it doesn't. +- In ADD it is a **mechanical floor**: the contract is frozen and + tamper-fingerprinted at the tests→build crossing, every red test is hashed, + a weakened test or edited contract bounces the build back (`return_to_build`, + three strikes), and a security finding can never be auto-passed. The same + campaign proved these teeth bite the *author* too: ADD's own gate returned + one of this book's engine tasks to build when the snapshot read its edits + as tampering. + +## Where ADD measurably leads + +- **Fidelity that stays put while the spec evolves — and stays green under an + *audited* floor.** ADD held oracle 1.00/1.00/1.00 across the evolving milestones + in the same-day run (so did spec-kit and vanilla on clean state — on a friendly + workload, fidelity does not separate the flows, and this appendix says so). What + is ADD-specific is *how* that green is earned: the 2026-07-14 loop re-measure held + 0.98 × 3 reps with zero regressions under the frozen-contract + tamper-tripwire + discipline, and the one ADD fidelity miss all campaign (an earlier branch WM2, + 0.80) was root-caused to tests speaking a *friendlier input dialect* than the + spec's own examples — then became a shipped floor (the spec-dialect check). The + failure mode is now detected mechanically, for every future task. +- **Stored-data robustness.** At WM3 the apps inherit bookings written under the + WM2 schema. ADD (main) was the only arm that **migrated its stored data**; + vanilla handled the legacy rows without migrating; ADD (branch) and spec-kit + both **crashed** serving them (`KeyError: 'end_time'`). This was un-metered — + found during rescore, the workload never demanded a migration, so it is a + planned probe, not a scored result — but it is exactly what "trust" means once + real users have data, and it split the field cleanly. +- **A self-measuring cost curve.** Because the loop benchmarks itself, its + overhead falls release over release at held fidelity: WM1 cost + $4.51 → $3.51 → **$3.17** per rep across three consecutive method releases + (risk-proportional → ceremony-to-effort → six-phase-loop), fidelity 0.97–0.98 + throughout, zero regressions. The ceremony you pay for is audited and pruned + with the same rigor as the code. +- **Session-proof state.** The engine's `status` is the resume point; the + time-to-first-edit and context-rot metrics exist because the method treats + "the agent forgot" as a defect class, not weather. + +## GSD on the fixed v2 meter (re-run 2026-07-14) + +GSD (`get-shit-done-cc` 1.42.3) was originally measured only on the retired v1 +meter, whose LLM judge proved untrustworthy — so it sat outside the comparable +tables. It has now been **re-run on the fixed v2 meter** (`2026-07-gsd-v2-rep0.md`), +same model, effort, and workload as the other arms. The result moves it *into* the +scoreboard above, and corrects the record: + +- **Functionally it holds up.** Oracle pass 1.00 / 0.80 / 1.00 — one probe short at + WM2, full recovery at WM3, zero regression on clean state. The same *shape* as the + ADD lean branch, and on par with spec-kit / vanilla. +- **It is cheap.** $3.01 total / 3.70M tokens — the cheapest arm in the same-day + run, below spec-kit ($4.20) and 3.0× below ADD ($9.11). GSD's planning-doc weight + did not translate into a cost penalty at this workload size. +- **The old v1 "WM2 fidelity 0.50" does not reproduce as a functional miss.** On v2 + the deterministic oracle passes WM2 at 0.80 (one probe), not a collapse — the 0.50 + was an unpinned-judge artifact of the retired meter. This correction cuts in GSD's + favor, and is stated as such. + +One honest wrinkle worth showing rather than hiding: the v2 run's *deprecated* LLM +judge scored GSD's WM2 and WM3 fidelity at **0.00 despite the oracle passing** (WM3 +oracle 1.00 = every probe green). A working app scoring judge-0.00 is implausible — +it is the same judge-untrustworthiness this appendix opened with, reproduced sharply +here (the single-float judge parse-fails to 0.00 as the evolved app grows). It is +exactly *why* every cross-arm fidelity claim in this appendix rests on the +deterministic `oracle_pass_rate`, never the judge. GSD's real signal is the oracle +line: 1.00 / 0.80 / 1.00. + +## How to choose (the same advice ADD's stages encode) + +| situation | measured recommendation | +|---|---| +| throwaway prototype, demo, spike | vanilla / spec-kit / GSD — ≈3–4.6× cheaper, floors don't bind | +| spec evolves, data persists, users exist | ADD — the only flow whose trust floors are enforced rather than assumed, at $3.17/WM1 and falling | +| hostile or ambiguous change requests | ADD — frozen contract + tamper tripwire make the honest path the only green path | +| compliance / security surface | ADD — security findings HARD-STOP mechanically; no other measured flow has an un-forceable floor | + +Method notes: pinned model + effort, self-carried permissions, per-run archived +transcripts and records (`benchmark/runs/`), oracle probes derived from each +track's own prompts, controls validated both ways before scoring. Re-run it +yourself: `python3 -m benchmark.run run --arm --wm 1`. diff --git a/add-method/src/add_method/_bundled/docs/appendix-h-benchmark.md b/add-method/src/add_method/_bundled/docs/appendix-h-benchmark.md new file mode 100644 index 00000000..733375d6 --- /dev/null +++ b/add-method/src/add_method/_bundled/docs/appendix-h-benchmark.md @@ -0,0 +1,146 @@ +# Appendix H — Measured: ADD vs spec-kit vs GSD vs vanilla + +ADD ships with its own adversarial benchmark (`benchmark/` in the repo): four +agent flows build the same longitudinal project — a booking REST API + CLI that +**evolves** across workload milestones (WM1 → WM3), including a deliberately +hostile change request — on a pinned meter (`claude-sonnet-5`, effort medium), +scored by deterministic oracles, tamper detection, and regression probes. This +appendix is the honest scoreboard. Two things make these numbers worth reading: + +1. **We attack our own meter.** Six meter defects were found and fixed live + during the 2026-07 campaigns (identical scores across independent arms + indict the meter, not the arms; a too-cheap run is a defect smell). An + early result favorable to ADD was **retracted** when the fidelity judge + proved untrustworthy, and the campaign re-ran on the fixed meter. What + survived that process is below, with raw records archived per run. +2. **Honest-outcome clause.** Findings that cut against ADD print unchanged. + The cost gap is real and stated first. + +## The scoreboard (same meter, same day, n=1 rep) + +Three milestones, same workspace evolving. The primary table is a fresh run of all +three flows on one day, one meter, one workload +(`2026-07-14-v2-sameday.md`) — so they compare without vintage skew. Vanilla carries +its 2026-07-10 WV1 figure (`2026-07-wv1-rep0.md`), the last time it was run. + +The "pass" column is `oracle_pass_rate` — the v2 meter's *deterministic* fidelity +of record (the milestone's own probe suite run against the built app; no LLM in the +scoring path). + +| arm | WM1 pass/reg | WM2 pass/reg | WM3 pass/reg | rep cost | tokens | +|---|---|---|---|---|---| +| **ADD** (current main) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $9.11 | 18.19M | +| GitHub **spec-kit** (v0.12.5) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $4.20 | 6.03M | +| **GSD** (get-shit-done-cc 1.42.3) | 1.00 / 0 | **0.80** / 0 | 1.00 / 0 | $3.01 | 3.70M | +| **vanilla** Claude Code † | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $3.07 | 4.3M | + +† vanilla not re-run this round; 2026-07-10 WV1 figure carried forward. + +**The honest headline: on a friendly workload at n=1, cost separates and trust +does not.** Every arm held its oracle. If your project is a weekend prototype, +spec-kit, vanilla, or GSD is genuinely cheaper — ADD's own `prototype` stage says +the same thing (run light, code is throwaway). ADD costs 2.2× spec-kit and 3.0× GSD +here, the same ordering the 2026-07-10 WV1 campaign found. Two honest notes on the +cost gap: **(1)** current ADD is ~$9, not the ~$14 the older WV1 `add-main` control +showed — that control was pinned before the lean / ceremony-to-effort / six-phase +reductions merged, and this same-day run measures the merged main. **(2)** adjusted +for the trust vector (own-suite evidence weighted by weakened-test adjudication), the +WV1 scoring report put cost *per trusted feature* at spec-kit $1.14 · vanilla $1.53 · +ADD $4.65 — ADD's premium is real, and it buys enforcement, not output. + +## The hostile change (WV2 — gaming resistance) + +The arms then received an undirected change request that **contradicts a frozen +rule**, where the cheapest green is quietly weakening your own overlap tests. +Source: `benchmark/results/2026-07-wv2-rep0.md`. + +| arm | hostile step pass / regression / tests weakened | rep cost | +|---|---|---| +| ADD (main) | 1.00 / 0 / **0** | $11.14 | +| ADD (lean branch) | 1.00 / 0 / **0** | $8.16 | +| spec-kit | 1.00 / 0 / **0** | $3.08 | +| vanilla | 1.00 / 0 / **0** | $3.01 | + +No arm gamed at this temptation strength — modern models re-spec honestly when +the amendment has a natural scoped form. The difference is *what guarantees it*: + +- In vanilla/spec-kit/GSD, honesty under pressure is a **model property** — + it held here, and nothing detects the day it doesn't. +- In ADD it is a **mechanical floor**: the contract is frozen and + tamper-fingerprinted at the tests→build crossing, every red test is hashed, + a weakened test or edited contract bounces the build back (`return_to_build`, + three strikes), and a security finding can never be auto-passed. The same + campaign proved these teeth bite the *author* too: ADD's own gate returned + one of this book's engine tasks to build when the snapshot read its edits + as tampering. + +## Where ADD measurably leads + +- **Fidelity that stays put while the spec evolves — and stays green under an + *audited* floor.** ADD held oracle 1.00/1.00/1.00 across the evolving milestones + in the same-day run (so did spec-kit and vanilla on clean state — on a friendly + workload, fidelity does not separate the flows, and this appendix says so). What + is ADD-specific is *how* that green is earned: the 2026-07-14 loop re-measure held + 0.98 × 3 reps with zero regressions under the frozen-contract + tamper-tripwire + discipline, and the one ADD fidelity miss all campaign (an earlier branch WM2, + 0.80) was root-caused to tests speaking a *friendlier input dialect* than the + spec's own examples — then became a shipped floor (the spec-dialect check). The + failure mode is now detected mechanically, for every future task. +- **Stored-data robustness.** At WM3 the apps inherit bookings written under the + WM2 schema. ADD (main) was the only arm that **migrated its stored data**; + vanilla handled the legacy rows without migrating; ADD (branch) and spec-kit + both **crashed** serving them (`KeyError: 'end_time'`). This was un-metered — + found during rescore, the workload never demanded a migration, so it is a + planned probe, not a scored result — but it is exactly what "trust" means once + real users have data, and it split the field cleanly. +- **A self-measuring cost curve.** Because the loop benchmarks itself, its + overhead falls release over release at held fidelity: WM1 cost + $4.51 → $3.51 → **$3.17** per rep across three consecutive method releases + (risk-proportional → ceremony-to-effort → six-phase-loop), fidelity 0.97–0.98 + throughout, zero regressions. The ceremony you pay for is audited and pruned + with the same rigor as the code. +- **Session-proof state.** The engine's `status` is the resume point; the + time-to-first-edit and context-rot metrics exist because the method treats + "the agent forgot" as a defect class, not weather. + +## GSD on the fixed v2 meter (re-run 2026-07-14) + +GSD (`get-shit-done-cc` 1.42.3) was originally measured only on the retired v1 +meter, whose LLM judge proved untrustworthy — so it sat outside the comparable +tables. It has now been **re-run on the fixed v2 meter** (`2026-07-gsd-v2-rep0.md`), +same model, effort, and workload as the other arms. The result moves it *into* the +scoreboard above, and corrects the record: + +- **Functionally it holds up.** Oracle pass 1.00 / 0.80 / 1.00 — one probe short at + WM2, full recovery at WM3, zero regression on clean state. The same *shape* as the + ADD lean branch, and on par with spec-kit / vanilla. +- **It is cheap.** $3.01 total / 3.70M tokens — the cheapest arm in the same-day + run, below spec-kit ($4.20) and 3.0× below ADD ($9.11). GSD's planning-doc weight + did not translate into a cost penalty at this workload size. +- **The old v1 "WM2 fidelity 0.50" does not reproduce as a functional miss.** On v2 + the deterministic oracle passes WM2 at 0.80 (one probe), not a collapse — the 0.50 + was an unpinned-judge artifact of the retired meter. This correction cuts in GSD's + favor, and is stated as such. + +One honest wrinkle worth showing rather than hiding: the v2 run's *deprecated* LLM +judge scored GSD's WM2 and WM3 fidelity at **0.00 despite the oracle passing** (WM3 +oracle 1.00 = every probe green). A working app scoring judge-0.00 is implausible — +it is the same judge-untrustworthiness this appendix opened with, reproduced sharply +here (the single-float judge parse-fails to 0.00 as the evolved app grows). It is +exactly *why* every cross-arm fidelity claim in this appendix rests on the +deterministic `oracle_pass_rate`, never the judge. GSD's real signal is the oracle +line: 1.00 / 0.80 / 1.00. + +## How to choose (the same advice ADD's stages encode) + +| situation | measured recommendation | +|---|---| +| throwaway prototype, demo, spike | vanilla / spec-kit / GSD — ≈3–4.6× cheaper, floors don't bind | +| spec evolves, data persists, users exist | ADD — the only flow whose trust floors are enforced rather than assumed, at $3.17/WM1 and falling | +| hostile or ambiguous change requests | ADD — frozen contract + tamper tripwire make the honest path the only green path | +| compliance / security surface | ADD — security findings HARD-STOP mechanically; no other measured flow has an un-forceable floor | + +Method notes: pinned model + effort, self-carried permissions, per-run archived +transcripts and records (`benchmark/runs/`), oracle probes derived from each +track's own prompts, controls validated both ways before scoring. Re-run it +yourself: `python3 -m benchmark.run run --arm --wm 1`. diff --git a/appendix-h-benchmark.md b/appendix-h-benchmark.md new file mode 100644 index 00000000..733375d6 --- /dev/null +++ b/appendix-h-benchmark.md @@ -0,0 +1,146 @@ +# Appendix H — Measured: ADD vs spec-kit vs GSD vs vanilla + +ADD ships with its own adversarial benchmark (`benchmark/` in the repo): four +agent flows build the same longitudinal project — a booking REST API + CLI that +**evolves** across workload milestones (WM1 → WM3), including a deliberately +hostile change request — on a pinned meter (`claude-sonnet-5`, effort medium), +scored by deterministic oracles, tamper detection, and regression probes. This +appendix is the honest scoreboard. Two things make these numbers worth reading: + +1. **We attack our own meter.** Six meter defects were found and fixed live + during the 2026-07 campaigns (identical scores across independent arms + indict the meter, not the arms; a too-cheap run is a defect smell). An + early result favorable to ADD was **retracted** when the fidelity judge + proved untrustworthy, and the campaign re-ran on the fixed meter. What + survived that process is below, with raw records archived per run. +2. **Honest-outcome clause.** Findings that cut against ADD print unchanged. + The cost gap is real and stated first. + +## The scoreboard (same meter, same day, n=1 rep) + +Three milestones, same workspace evolving. The primary table is a fresh run of all +three flows on one day, one meter, one workload +(`2026-07-14-v2-sameday.md`) — so they compare without vintage skew. Vanilla carries +its 2026-07-10 WV1 figure (`2026-07-wv1-rep0.md`), the last time it was run. + +The "pass" column is `oracle_pass_rate` — the v2 meter's *deterministic* fidelity +of record (the milestone's own probe suite run against the built app; no LLM in the +scoring path). + +| arm | WM1 pass/reg | WM2 pass/reg | WM3 pass/reg | rep cost | tokens | +|---|---|---|---|---|---| +| **ADD** (current main) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $9.11 | 18.19M | +| GitHub **spec-kit** (v0.12.5) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $4.20 | 6.03M | +| **GSD** (get-shit-done-cc 1.42.3) | 1.00 / 0 | **0.80** / 0 | 1.00 / 0 | $3.01 | 3.70M | +| **vanilla** Claude Code † | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | $3.07 | 4.3M | + +† vanilla not re-run this round; 2026-07-10 WV1 figure carried forward. + +**The honest headline: on a friendly workload at n=1, cost separates and trust +does not.** Every arm held its oracle. If your project is a weekend prototype, +spec-kit, vanilla, or GSD is genuinely cheaper — ADD's own `prototype` stage says +the same thing (run light, code is throwaway). ADD costs 2.2× spec-kit and 3.0× GSD +here, the same ordering the 2026-07-10 WV1 campaign found. Two honest notes on the +cost gap: **(1)** current ADD is ~$9, not the ~$14 the older WV1 `add-main` control +showed — that control was pinned before the lean / ceremony-to-effort / six-phase +reductions merged, and this same-day run measures the merged main. **(2)** adjusted +for the trust vector (own-suite evidence weighted by weakened-test adjudication), the +WV1 scoring report put cost *per trusted feature* at spec-kit $1.14 · vanilla $1.53 · +ADD $4.65 — ADD's premium is real, and it buys enforcement, not output. + +## The hostile change (WV2 — gaming resistance) + +The arms then received an undirected change request that **contradicts a frozen +rule**, where the cheapest green is quietly weakening your own overlap tests. +Source: `benchmark/results/2026-07-wv2-rep0.md`. + +| arm | hostile step pass / regression / tests weakened | rep cost | +|---|---|---| +| ADD (main) | 1.00 / 0 / **0** | $11.14 | +| ADD (lean branch) | 1.00 / 0 / **0** | $8.16 | +| spec-kit | 1.00 / 0 / **0** | $3.08 | +| vanilla | 1.00 / 0 / **0** | $3.01 | + +No arm gamed at this temptation strength — modern models re-spec honestly when +the amendment has a natural scoped form. The difference is *what guarantees it*: + +- In vanilla/spec-kit/GSD, honesty under pressure is a **model property** — + it held here, and nothing detects the day it doesn't. +- In ADD it is a **mechanical floor**: the contract is frozen and + tamper-fingerprinted at the tests→build crossing, every red test is hashed, + a weakened test or edited contract bounces the build back (`return_to_build`, + three strikes), and a security finding can never be auto-passed. The same + campaign proved these teeth bite the *author* too: ADD's own gate returned + one of this book's engine tasks to build when the snapshot read its edits + as tampering. + +## Where ADD measurably leads + +- **Fidelity that stays put while the spec evolves — and stays green under an + *audited* floor.** ADD held oracle 1.00/1.00/1.00 across the evolving milestones + in the same-day run (so did spec-kit and vanilla on clean state — on a friendly + workload, fidelity does not separate the flows, and this appendix says so). What + is ADD-specific is *how* that green is earned: the 2026-07-14 loop re-measure held + 0.98 × 3 reps with zero regressions under the frozen-contract + tamper-tripwire + discipline, and the one ADD fidelity miss all campaign (an earlier branch WM2, + 0.80) was root-caused to tests speaking a *friendlier input dialect* than the + spec's own examples — then became a shipped floor (the spec-dialect check). The + failure mode is now detected mechanically, for every future task. +- **Stored-data robustness.** At WM3 the apps inherit bookings written under the + WM2 schema. ADD (main) was the only arm that **migrated its stored data**; + vanilla handled the legacy rows without migrating; ADD (branch) and spec-kit + both **crashed** serving them (`KeyError: 'end_time'`). This was un-metered — + found during rescore, the workload never demanded a migration, so it is a + planned probe, not a scored result — but it is exactly what "trust" means once + real users have data, and it split the field cleanly. +- **A self-measuring cost curve.** Because the loop benchmarks itself, its + overhead falls release over release at held fidelity: WM1 cost + $4.51 → $3.51 → **$3.17** per rep across three consecutive method releases + (risk-proportional → ceremony-to-effort → six-phase-loop), fidelity 0.97–0.98 + throughout, zero regressions. The ceremony you pay for is audited and pruned + with the same rigor as the code. +- **Session-proof state.** The engine's `status` is the resume point; the + time-to-first-edit and context-rot metrics exist because the method treats + "the agent forgot" as a defect class, not weather. + +## GSD on the fixed v2 meter (re-run 2026-07-14) + +GSD (`get-shit-done-cc` 1.42.3) was originally measured only on the retired v1 +meter, whose LLM judge proved untrustworthy — so it sat outside the comparable +tables. It has now been **re-run on the fixed v2 meter** (`2026-07-gsd-v2-rep0.md`), +same model, effort, and workload as the other arms. The result moves it *into* the +scoreboard above, and corrects the record: + +- **Functionally it holds up.** Oracle pass 1.00 / 0.80 / 1.00 — one probe short at + WM2, full recovery at WM3, zero regression on clean state. The same *shape* as the + ADD lean branch, and on par with spec-kit / vanilla. +- **It is cheap.** $3.01 total / 3.70M tokens — the cheapest arm in the same-day + run, below spec-kit ($4.20) and 3.0× below ADD ($9.11). GSD's planning-doc weight + did not translate into a cost penalty at this workload size. +- **The old v1 "WM2 fidelity 0.50" does not reproduce as a functional miss.** On v2 + the deterministic oracle passes WM2 at 0.80 (one probe), not a collapse — the 0.50 + was an unpinned-judge artifact of the retired meter. This correction cuts in GSD's + favor, and is stated as such. + +One honest wrinkle worth showing rather than hiding: the v2 run's *deprecated* LLM +judge scored GSD's WM2 and WM3 fidelity at **0.00 despite the oracle passing** (WM3 +oracle 1.00 = every probe green). A working app scoring judge-0.00 is implausible — +it is the same judge-untrustworthiness this appendix opened with, reproduced sharply +here (the single-float judge parse-fails to 0.00 as the evolved app grows). It is +exactly *why* every cross-arm fidelity claim in this appendix rests on the +deterministic `oracle_pass_rate`, never the judge. GSD's real signal is the oracle +line: 1.00 / 0.80 / 1.00. + +## How to choose (the same advice ADD's stages encode) + +| situation | measured recommendation | +|---|---| +| throwaway prototype, demo, spike | vanilla / spec-kit / GSD — ≈3–4.6× cheaper, floors don't bind | +| spec evolves, data persists, users exist | ADD — the only flow whose trust floors are enforced rather than assumed, at $3.17/WM1 and falling | +| hostile or ambiguous change requests | ADD — frozen contract + tamper tripwire make the honest path the only green path | +| compliance / security surface | ADD — security findings HARD-STOP mechanically; no other measured flow has an un-forceable floor | + +Method notes: pinned model + effort, self-carried permissions, per-run archived +transcripts and records (`benchmark/runs/`), oracle probes derived from each +track's own prompts, controls validated both ways before scoring. Re-run it +yourself: `python3 -m benchmark.run run --arm --wm 1`. diff --git a/benchmark/results/2026-07-14-v2-sameday.md b/benchmark/results/2026-07-14-v2-sameday.md new file mode 100644 index 00000000..b0c22da1 --- /dev/null +++ b/benchmark/results/2026-07-14-v2-sameday.md @@ -0,0 +1,43 @@ +# Same-meter, same-day v2 comparison — add · spec-kit · gsd (2026-07-14) + +A fresh WM1→WM3 rep of three arms on one day, one meter (`claude-sonnet-5 --effort +medium --dangerously-skip-permissions`, the v2 default), same workload — so the three +sit in one apples-to-apples table without the vintage skew of the 2026-07-10 WV1 +campaign. Runs archived at `benchmark/runs/{add,spec-kit,gsd}-v2meter-r0`. The `add` +arm is an editable install of this repo's engine at current `main` (this branch is +docs-only), i.e. current ADD with every merged cost reduction. + +## Scoreboard (n=1) + +| arm | WM1 oracle/reg | WM2 oracle/reg | WM3 oracle/reg | total cost | tokens | +|---|---|---|---|---|---| +| **ADD** (current main) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | **$9.11** | 18.19M | +| **spec-kit** (v0.12.5) | 1.00 / 0 | 1.00 / 0 | 1.00 / 0 | **$4.20** | 6.03M | +| **GSD** (1.42.3) | 1.00 / 0 | **0.80** / 0 | 1.00 / 0 | **$3.01** | 3.70M | + +`oracle_pass_rate` is the v2 meter's deterministic fidelity of record (probe suite vs +built app, no LLM in the path). Per-WM cost — ADD $3.91/$2.76/$2.43 · spec-kit +$1.09/$1.78/$1.34 · GSD $0.88/$1.00/$1.13. + +## Findings (n=1 — direction, not conclusion) + +- **All three are functionally green on clean state.** Every arm passes its oracle at + every milestone, zero regression. The only sub-1.00 is GSD's WM2 (0.80 — one probe + short, recovered to 1.00 at WM3). +- **Cost ranks GSD < spec-kit < ADD** ($3.01 · $4.20 · $9.11): ADD is 2.2× spec-kit + and 3.0× GSD. Same ordering and rough ratios as the 2026-07-10 WV1 campaign + (spec-kit $3.43, add branch $9.30) — this run reconfirms it same-day. +- **Current ADD is ~$9, not ~$14.** The WV1 `add-main` control ($13.94) was pinned to + a commit *before* the lean / ceremony-to-effort / six-phase cost reductions merged. + Current main measures $9.11 — the merged reductions are real and show up here. +- **The LLM judge is unreliable and clusters on WM2 — proven across all three arms.** + Judge fidelity triplets: ADD wm2 `0.0;0.0;0.0` (wm1 0.95, wm3 0.98); spec-kit wm2 + `0.0;0.0;0.15`, wm3 `0.0;0.15;0.0`; GSD wm2/wm3 all `0.0`. Three independent arms all + score judge-0.0 on WM2 while the oracle passes 1.00 — that is a judge defect on the + WM2 milestone (the auth/tenancy evolution), not three simultaneous collapses. Every + fidelity claim rests on `oracle_pass_rate`, never the judge (harness todo #34). + +## Spend + +This round: GSD $3.01 + ADD $9.11 + spec-kit $4.20 = **$16.32** (3 arms × 3 WMs, 1 rep). +Vanilla not re-run — its 2026-07-10 WV1 figure ($3.07, oracle 1.00×3) carries forward. diff --git a/benchmark/results/2026-07-gsd-v2-rep0.md b/benchmark/results/2026-07-gsd-v2-rep0.md new file mode 100644 index 00000000..3ac849f3 --- /dev/null +++ b/benchmark/results/2026-07-gsd-v2-rep0.md @@ -0,0 +1,56 @@ +# GSD rep0 on the fixed v2 meter — longitudinal greenfield, WM1→WM3 (2026-07-14) + +Run: `benchmark/runs/gsd-v2meter-r0` · pinned meter `claude-sonnet-5 --effort medium +--dangerously-skip-permissions` (the v2 default `build_argv`, identical to the WV1/WV2 +arms) · WM1→WM3 same-workspace evolution (`seeded from wmN-1`) · arm pin +`get-shit-done-cc@1.42.3`. Purpose: close the one gap in Appendix H — GSD had only +ever been measured on the retired v1 meter; this puts it on the same meter as add / +spec-kit / vanilla so it can sit in the comparable tables. + +The prior v1-meter GSD run is preserved at `benchmark/runs/gsd-v1meter-2026-07-07`. + +## Scoreboard (n=1) + +| WM | oracle_pass_rate† | regression | cost | tokens | TTFE (s) | judge spec_fidelity‡ | +|---|---|---|---|---|---|---| +| WM1 | **1.00** | n/a | $0.88 | 1.02M | 46.9 | 0.90 | +| WM2 | **0.80** | n/a | $1.00 | 1.08M | 111.0 | 0.00 ‡ | +| WM3 | **1.00** | 0.00 | $1.13 | 1.61M | 69.8 | 0.00 ‡ | +| **total** | — | 0 | **$3.01** | **3.70M** | — | — | + +† `oracle_pass_rate` is the v2 meter's **deterministic fidelity of record** +(`score.py:131` — "no LLM in the path"): the WM's own probe suite run against the +built app, passed/total. This is the trustworthy, arm-comparable signal, the same +one the WV1 rep0 scoreboard's "pass" column reports. +‡ `spec_fidelity` is the **deprecated LLM-judge** signal, kept only for provenance. +It returned `0.0` on GSD's wm2 and wm3 **despite the oracle passing** (wm3 oracle +1.00 = every functional probe green). A working app scoring judge-0.0 is implausible; +the wm1 judge triplet was already `0.9;0.0;0.9` (one rep zeroed). This reproduces the +known judge-untrustworthiness (harness todo #34 / the "fid 0.0 on a working app" +finding): the single-float judge protocol parse-fails to 0.0 as the evolved app grows, +and the failures cluster on the more complex later milestones. **Do not read GSD's +0.0/0.0 as a fidelity collapse** — the oracle says the app works. + +## Honest reading (n=1 — direction, not conclusion) + +- **Functionally GSD holds up on v2.** Oracle 1.00 / 0.80 / 1.00: one probe short at + WM2, full recovery at WM3, zero regression on clean state. Same *shape* as the + ADD lean branch's WV1 run (which also dipped to 0.80 at WM2), and on par with + spec-kit / vanilla, which held their oracles. +- **Cost lands in the cheap tier.** $3.01 total / 3.70M tokens — right alongside + spec-kit ($3.43) and vanilla ($3.07) on WV1, and 3–4.6× below ADD ($9.30 branch / + $13.94 main). GSD's planning-doc weight does not translate into a cost penalty at + this workload size. +- **The WV1 v1-meter "wm2 fidelity 0.50" signal does not reproduce as a functional + miss on v2.** On v2 the oracle passes wm2 at 0.80 (one probe), not a collapse. The + old 0.50 was an unpinned-judge artifact era; the deterministic oracle tells a + steadier story. That correction cuts in GSD's favor and is stated as such. +- **No trustworthy LLM-fidelity number is available for any arm on v2** — the judge + is deprecated for exactly the reason shown here. Cross-arm fidelity claims rest on + `oracle_pass_rate`, not the judge. + +## Spend + +This run: $3.01 (3 WMs, 1 rep). Cheap relative to a 4-arm rep (~$25–30) because only +the one arm was (re-)measured against the already-canonical v2 add/spec-kit/vanilla +numbers. diff --git a/mkdocs.yml b/mkdocs.yml index 666dd920..b67e6936 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -91,3 +91,4 @@ nav: - "Appendix E · Checklists": appendix-e-checklists.md - "Appendix F · Document requirements matrix": appendix-f-requirements-matrix.md - "Appendix G · References & lineage": appendix-g-references.md + - "Appendix H · Measured: ADD vs spec-kit vs GSD": appendix-h-benchmark.md