diff --git a/README.md b/README.md index e74c037..e4f944b 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,25 @@ Code invariants: - Tests and typecheck green, one runnable check for new logic - No dead code, stale comments, or debug residue +## Does it actually help? (measured) + +Fair skepticism: a strong model already fixes plenty of UI bugs on its own. +So the skill was A/B tested - the same audit run with it and without it, scored by a script that renders the output and measures pixels, not by opinion. + +The headline result is on the exact bug class this plugin exists for: a layout shift that only appears in a state the default view never shows. +An agent is handed a toolbar and told only "audit this" - the bug is never named - across 8 trials per condition: + +![ocd vs baseline on a hidden-state layout shift](eval/ablation/shots/d2_real_comparison.png) + +- **Without the skill: 0/8 found it.** Every run fixed a *different*, visible issue (the buttons' font) and never simulated the changing count, so the shift sailed through. +- **With the skill: 8/8 found it.** Its "simulate every state" rule renders the count at 3 and at 9999, sees the buttons drift, and reserves the width. + +That is the skill doing the one thing a fast read cannot: catching the bug you only see when you render a state that isn't on screen. + +Honest scope, because the same study measured it: when a bug is named, visible on sight, or readable in the source - even buried in a real, heavy landing page - a strong base model already fixes it, and the skill adds no correctness there while costing 2-3x more tokens. +Its edge is specifically the source-invisible, state-dependent bug. +Full write-up, all five rounds, both render-and-measure scorers, and every number including the ties: [`eval/ablation`](eval/ablation). + ## Install As a plugin (recommended, gets updates): @@ -88,6 +107,25 @@ The natural move after changing a button is to screenshot the button - a frame i It uses whatever browser tooling the session already has, or falls back to a bundled ~50-line puppeteer-core script that reuses your installed Chrome (no browser download, <2s per shot). The heuristic handles shadow DOM, inner scroll containers, `display:contents`, and `position:fixed`, and returns why it picked the frame. +### Verified, not asserted + +Those handling claims are backed by an eval + bench, not just prose - fittingly, given the skill's own "measurements, not impressions" rule. +The suite lives in [`skills/wideshot/scripts/test`](skills/wideshot/scripts/test) (12 correctness cases in real headless Chrome, plus a perf bench): `npm test`, `npm run bench`. + +The heuristic is a *container-selection* step that runs before a screenshot, so the fair comparison is against the framing other tools give you. +Measured on one engine (puppeteer-core + Chrome), same page and target, so the only variable is what gets framed (Apple M5, arm64 - ratios matter, not absolute ms): + +| framing strategy | time (median) | output | note | +|------------------|---------------|--------|------| +| element-only (Playwright / shot-scraper / capture-website `--element`) | 25.1 ms | 40 x 30 px | no siblings -> alignment/shift unverifiable | +| **wideshot** | 24.5 ms | 920 x 50 px | frames the pane, same speed as element-only | +| full-page | 41.9 ms | 1280 x 4400 px | captures everything, 1.7x slower, unfocused | + +The heuristic itself adds < 0.1 ms (p95), below the `performance.now()` floor. +So wideshot costs the same as a plain element shot but returns a frame you can actually verify in - where element-only deleted the siblings and full-page buried the change. +Every other tool screenshots exactly the selector you hand it or the whole page; only wideshot walks to the meaningful container. +Full matrix and how-to in the [eval README](skills/wideshot/scripts/test/README.md). + ## Structure ``` @@ -98,6 +136,7 @@ skills/ ocd/SKILL.md the audit-and-fix pass wideshot/SKILL.md context-framed screenshots for visual verification wideshot/scripts/ container heuristic + fallback capture tool + wideshot/scripts/test/ eval (12 cases) + perf bench for the heuristic ``` Skills under `skills/` are auto-discovered by the plugin. diff --git a/eval/ablation/README.md b/eval/ablation/README.md new file mode 100644 index 0000000..d48ce65 --- /dev/null +++ b/eval/ablation/README.md @@ -0,0 +1,129 @@ +# ocd skill ablation + +Does the ocd *skill* (a prompt) change what Claude produces? The wideshot eval proves the *tool* works; this asks a different question about the *skill*, and answers it with a controlled A/B, scored by a deterministic script - not by model judgment. + +## Method + +- **Corpus** (`tasks/`): 3 single-file bugs with objective ground truth a script can measure. + - `t1` - layout shift: a TUI progress row whose `%` label isn't padded, so the `|` column drifts as the percent widens. + - `t2` - drift: a caret marker at a hardcoded column instead of one computed from the count's width. + - `t3` - root cause vs symptom: `pct()` divides by zero on an empty list; the shared bug also reaches `bar()`. A symptom-only patch of `summary()` leaves the sibling caller broken. + - `a1`/`a2`/`a3` - adversarial versions: clamp both ends (not just >100), one shared bug reached by 3 named-only-once callers, column width from max (not the first row). + - `va`/`vb`/`vc` - visual (HTML), scored by rendered bounding-rects: layout shift, box-model width trap, off-center card. + - `d1`/`d2`/`d3` - discovery (HTML): same measurement, but the bug is NEVER named to the agent ("audit this, may or may not have issues"). Tests detection, not fixing. +- **Conditions**: `base` = agent gets only the symptom + "no slash-command skills"; `ocd` = agent told to read and follow `skills/ocd/SKILL.md`. Same tools, same model. The only variable is the method. +- **Trials**: 2 per (task x condition) = 12 runs. Symptom prompts state the symptom only, never the full check the scorer applies (so a lazy fix that only addresses the named case fails). +- **Score** (`score.mjs`): imports each edited module and measures `{fixed, regression}` across all states / callers. `fixed` = defect gone everywhere, normal cases intact. `regression` = new breakage OR a symptom-only patch that left a sibling caller broken. +- **Scorer is self-tested first** (`selftest.mjs`): buggy -> not fixed; reference fixes -> fixed/no-regression; a symptom-only patch -> fixed BUT regression. A wrong scorer would invalidate everything, so it is validated before any agent runs. + +## Results + +### Round 1 - easy tasks (`t1`-`t3`, n=2) + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~26.8k | 2.0 | +| ocd | 6 / 6 | 0 | ~30.5k | 4.7 | + +Ceiling effect. Baseline `t3` runs guarded the shared `pct()` (root cause), not just `summary()` (symptom) - the exact discipline the skill teaches, applied unprompted. + +### Round 2 - adversarial tasks (`a1`-`a3`, n=2) + +Tasks redesigned so the lazy fix passes the *named* case but fails a deeper check: clamp both ends (not one), trace 3 callers (not the one named), size to max (not a sampled row). + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~27.0k | 2.2 | +| ocd | 6 / 6 | 0 | ~30.7k | 4.7 | + +Same ceiling. Baseline clamped **both** ends (`Math.max(0, Math.min(10, ...||0))`), sized to `Math.max` over all rows, and guarded the shared `fmtPct` - every trap, unprompted. + +### Round 3 - visual tasks, scored by headless-browser rects (`va`-`vc`, n=2) + +The skill's home turf: bugs invisible-or-misleading in source, measurable only by rendering. `va` layout shift (a `%` label with no reserved width moves the Cancel button); `vb` box-model trap (two cards both `width:320px`, one `border-box`, render 38px apart); `vc` off-center card (hardcoded `left:120px`, centered needs 150). Scored by `visual-score.mjs` (puppeteer bounding-rects). + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~26.9k | 2.2 | +| ocd | 6 / 6 | 0 | ~34.3k | 8.3 | + +Same outcome - but here the **process** finally diverged: ocd-condition agents actually **rendered and measured** (headless CDP - "button left edge holds at 246.53px", "verified A=B=320"), while base agents reasoned from source in 2 tool-uses. The skill did exactly what it promises (measure, don't eyeball). The base model's source reasoning simply reached the same correct fix without rendering - so measuring was right but redundant, at +27% tokens and ~4x the tool calls. + +### Round 4 - DISCOVERY: bug not named (`d1`-`d3`, n=2) + +The decisive change: the agent is told only "audit this, it may or may not have issues" - the defect is never named. This tests *detection*, the skill's actual headline. Planted defects that don't scream in source: `d1` one stat card silently wider (padding shorthand), `d2` a counter with no reserved width **that looks perfect at its default value and only shifts siblings when the number grows**, `d3` a flex-centered modal nudged off-center by a stray `margin-left`. + +| task | base found | ocd found | note | +|------|-----------|-----------|------| +| d1 (card bulge, visible in one render) | 2/2 | 2/2 | ceiling | +| **d2 (shift invisible at default state)** | **0/2** | **2/2** | **the signal** | +| d3 (off-center, visible in one render) | 2/2 | 2/2 | ceiling | +| **total** | **4/6** | **6/6** | | + +**First real difference in the whole study.** On `d2`, base audited what it could see in the default render - it even found and fixed a *different, unplanted* real defect (buttons not inheriting the monospace font) - but never varied the count, so it missed the layout shift entirely, both trials. Verified in the file: base left `.count { }` untouched; ocd added `min-width: 4ch` after rendering across counts. The skill's "simulate the states - layout bugs live in the transitions" discipline is exactly what closed the gap. It cost ~46% more tokens on that task. + +`d1`/`d3` stayed at ceiling because those defects are visible in a single default render - no state simulation needed, so base catches them too. + +### Bottom line (48 runs, 12 bug types) + +- **Fixing a *named* bug (rounds 1-3, code + visual): no measurable difference.** Baseline Opus already clamps both ends, traces callers, sizes to max, reasons box-model. The skill changed the process (made agents render+measure) but not the outcome, at +14-27% cost. +- **Discovering an *unnamed* bug (round 4): a real, mechanistic gap - but only for defects invisible in the default view.** base 4/6 vs ocd 6/6, the entire delta on the one state-dependent bug. For bugs visible on sight, still ceiling. + +So the skill earns its keep in one specific place: **finding the bug that isn't visible until you simulate a state the default render doesn't show.** That is a narrow but genuine slice of what it claims - and the slice where "render it, count the cells, across every state" is doing work a strong model skips when it isn't told to. + +### Round 5 - hardening d2 (n=8) + a real heavy page + +**d2 rerun at n=8 per condition** (discovery mode, bug never named), split into *detection* (found the shift at all) and *robust fix* (holds at count=9999): + +| condition | detected the shift | robust fix (holds @9999) | +|-----------|--------------------|--------------------------| +| base | **0 / 8** | 0 / 8 | +| ocd | **8 / 8** | 5 / 8 | + +The 0/8 vs 8/8 detection split holds up. All 8 base runs instead found and fixed a *different* real defect (buttons not inheriting the monospace font) and never simulated the count. 3 of the 8 ocd runs detected the shift but sized the reserve to `3ch`, too small for 4-digit counts - detection is the robust signal; fix-completeness varies. Before/after screenshots (fixture and real agent outputs) in `shots/` - `d2_comparison.png`, `d2_real_comparison.png`. + +**Real heavy landing page** (HTML5 UP "Stellar", full template with parallax/animations, CCA 3.0). Clean template has zero horizontal overflow at any width. Planted one realistic state-dependent bug - an inline `min-width: 720px` on the stats row that renders perfectly on desktop but forces ~385px of horizontal overflow on mobile (375px). Scored by `heavy-score.mjs` (overflow across 1280/1024/768/414/375), discovery mode, n=4: + +| condition | overflow fixed | mean cost | +|-----------|----------------|-----------| +| base | **4 / 4** | ~32k tok, 4-5 tools | +| ocd | **4 / 4** | ~77k tok, 12-21 tools | + +**Ceiling - both found it, on a real cluttered page.** The reason matters: this bug was an *inline style*, a visible source smell, so base spotted it by reading even buried in a heavy template. Contrast d2, whose bug is an *empty `.count {}` rule* - nothing to see in source, only behavior across states. Screenshots: `shots/heavy_buggy_375.png` (stats overflow the viewport) vs `shots/heavy_fixed_375.png` (stats stack cleanly). + +**Refined conclusion: the discriminator is not page heaviness or bug difficulty - it is source-visibility.** A bug base can *read* (inline style, odd value, box-model math), it fixes with or without the skill, even amid real-page clutter. A bug base cannot read - one that exists only as behavior in a non-default state (d2's shift) - base misses and the skill's "simulate every state" catches. That is the narrow, real slice where the skill earns its 2-3x cost. + +## Caveats that still stand + +- **Small n.** d2 is now 16 runs (8+8) and holds 0/8 vs 8/8; other tasks are still n=2. Solid for d2, suggestive elsewhere. +- **Self-authored corpus**, scored objectively. The d2 result is only as good as the claim that an empty-rule layout shift is a fair, representative "invisible-in-source" bug (it is a common real one). +- **Opus only.** On a weaker model the gap is plausibly wider on every round, not just discovery. +- **The tail.** Even where the mean ties, the skill may cut the rare miss. +- **Cost scales with the skill's effort.** It rendered/measured far more (up to ~100k tokens, 20+ tool calls on the heavy page) - 2-3x baseline - whether or not that changed the outcome. + +## Screenshots + +In `shots/` (generated by `shots.mjs`): `d2_comparison.png` (fixture before/after), `d2_real_comparison.png` (real base-missed vs ocd-fixed agent outputs), `heavy_buggy_375.png` / `heavy_fixed_375.png` (real landing-page stats overflowing vs stacked on mobile). + +The heavy page (HTML5 UP "Stellar", CCA 3.0) is fetched to `/tmp`, not vendored: `curl -sSL https://html5up.net/stellar/download -o s.zip && unzip s.zip`, then plant `min-width:720px` on `