docs(add-method): Appendix H — measured comparison ADD vs spec-kit vs GSD#153
Open
pilotspacex-byte wants to merge 3 commits into
Open
docs(add-method): Appendix H — measured comparison ADD vs spec-kit vs GSD#153pilotspacex-byte wants to merge 3 commits into
pilotspacex-byte wants to merge 3 commits into
Conversation
… GSD
Add a benchmark-sourced appendix to the AIDD book comparing ADD against
spec-kit, GSD, and vanilla Claude Code on the repo's own adversarial harness
(WV1 greenfield evolution + WV2 hostile change), written under the
honest-outcome clause.
Framing follows the retraction history and Rule-1 honesty: the cost gap is
stated first (ADD is 2.7–4.5x pricier on friendly workloads; spec-kit/vanilla
are the right call for throwaway work), and ADD's measured advantages are
scoped to the trust vector — mechanically-enforced floors (frozen contract,
tamper tripwire at tests->build, earned-green, security HARD-STOP), fidelity
that holds while the spec evolves, stored-data migration robustness, and a
self-measuring cost curve ($4.51 -> $3.51 -> $3.17/WM1 at fid 0.97-0.98 across
three releases). GSD carries an explicit v1-meter-only caveat (never re-run on
the fixed meter). Every number cites benchmark/results/2026-07-wv{1,2}-rep0.md.
Wired into all three book trees byte-identical (canonical add-method/docs,
repo-root mirror, _bundled/docs) with a Part IV nav entry; test_book_parity +
test_bundle_parity + docs-accord suite green.
author: Tin Dang <tindang.ht97@gmail.com>
Re-ran the GSD arm (get-shit-done-cc 1.42.3) on the same pinned v2 meter and workload as the add/spec-kit/vanilla arms (WM1->WM3, 1 rep), closing the one gap called out in Appendix H: GSD previously had only retired-v1-meter numbers and sat outside the comparable tables. Result (benchmark/results/2026-07-gsd-v2-rep0.md): oracle_pass_rate 1.00/0.80/1.00 (the deterministic fidelity of record — no LLM in the scoring path), zero regression, $3.01 total / 3.70M tokens. GSD drops into the WV1 scoreboard right alongside spec-kit ($3.43) and vanilla ($3.07), 3-4.6x below ADD. Honest wrinkle kept in the doc rather than hidden: the run's DEPRECATED LLM judge scored GSD's wm2/wm3 fidelity 0.00 despite the oracle passing (wm3 oracle 1.00 = every probe green) — the same judge-untrustworthiness the appendix opens with, reproduced. Every cross-arm fidelity claim rests on oracle_pass_rate, not the judge; GSD's real signal is the oracle line 1.00/0.80/1.00. The old v1 "wm2 0.50" does not reproduce as a functional miss on v2 — a correction stated in GSD's favor. Synced byte-identical across all three book trees; test_book_parity + test_bundle_parity green. Prior v1 run preserved at runs/gsd-v1meter-2026-07-07. author: Tin Dang <tindang.ht97@gmail.com>
Ran add, spec-kit, and gsd on one meter, one day, one workload (WM1->WM3), so
Appendix H's primary scoreboard is apples-to-apples without the 2026-07-10
vintage skew. Records at benchmark/runs/{add,spec-kit,gsd}-v2meter-r0;
summary at benchmark/results/2026-07-14-v2-sameday.md.
Same-day oracle_pass_rate / cost:
- ADD (current main): 1.00/1.00/1.00, $9.11 / 18.19M
- spec-kit v0.12.5: 1.00/1.00/1.00, $4.20 / 6.03M
- GSD 1.42.3: 1.00/0.80/1.00, $3.01 / 3.70M
(vanilla carried from WV1: 1.00x3, $3.07)
Honest corrections this run forced:
- Dropped the "ADD is the only arm holding 1.00/1.00/1.00" overclaim — on a
friendly workload spec-kit and vanilla hold it too; fidelity does not
separate the flows there, and the doc now says so. ADD's differentiator is
the audited/enforced floor, not the clean-state score.
- Current ADD measures ~$9, not the older WV1 add-main control's ~$14 (that
control predates the lean/ceremony/six-phase reductions now on main).
- The LLM judge's wm2=0.0 artifact reproduced across ALL THREE arms while the
oracle passed 1.00 — three arms can't collapse simultaneously; it's a judge
defect on the wm2 milestone, restated as why every fidelity claim rests on
oracle_pass_rate.
Synced byte-identical across all three book trees; test_book_parity +
test_bundle_parity green.
author: Tin Dang <tindang.ht97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Appendix H — Measured: ADD vs spec-kit vs GSD vs vanilla to the AIDD book: a benchmark-sourced comparison drawn from the repo's own adversarial harness, written under the honest-outcome clause.
Framing (honest by construction)
Per the retraction history (we once claimed "ADD beats spec-kit" on cost and retracted it) and Rule 1, the doc leads with the cost gap and scopes ADD's wins to what the numbers actually support:
Every figure cites
benchmark/results/2026-07-wv1-rep0.md/-wv2-rep0.md.Trees + tests
Wired byte-identical into all three book trees (canonical
add-method/docs/, repo-root mirror,_bundled/docs/) plus a Part IV nav entry inmkdocs.yml.test_book_parity✅test_bundle_parity✅Docs-only; no engine/contract change.