Skip to content

test: eval + bench for wideshot heuristic, ocd skill ablation study#1

Merged
undeemed merged 1 commit into
mainfrom
add-evals
Jul 4, 2026
Merged

test: eval + bench for wideshot heuristic, ocd skill ablation study#1
undeemed merged 1 commit into
mainfrom
add-evals

Conversation

@undeemed

@undeemed undeemed commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Adds test/eval infrastructure and an empirical study of whether the ocd skill actually changes outcomes. No changes to the shipped skills themselves.

What's in it

1. skills/wideshot/scripts/test/ - heuristic eval + bench

  • 12-case correctness eval for pick-container.js, one case per claim/guardrail (landmark match, 3x-area rung, sibling requirement, viewport ceiling, shadow DOM, display:contents, position:fixed, scrollport clamp, page coords, bad selector, no-ancestor fallback, generic fallback).
  • Runs in real headless Chrome via puppeteer-core - jsdom has no layout engine, so its rects are 0x0 and every geometric assertion would be vacuous.
  • Perf bench: heuristic p50/p95 (sub-ms) + wideshot.mjs end-to-end (<2s claim) + a framing-strategy comparison (element-only vs wideshot vs full-page).

2. eval/ablation/ - controlled A/B study of the ocd skill

  • Same audit task run with and without the skill, scored by deterministic render-and-measure scripts, not model judgment. Scorers are self-tested (buggy fails, reference-fix passes, symptom-patch is caught) before any agent runs.
  • 5 rounds, 48+ subagent runs, code + visual + discovery + a real heavy landing page (HTML5 UP "Stellar").
  • Robust finding: no correctness lift when a bug is named, visible on sight, or readable in source - even buried in a real heavy page - at 2-3x the token cost. The one real win is discovery of a source-invisible, state-dependent layout shift: base 0/8 vs ocd 8/8, with before/after screenshots.

3. README.md

  • wideshot perf/comparison table.
  • "Does it actually help? (measured)" section presenting the ablation evidence with honest scope.

🤖 Generated with Claude Code

- skills/wideshot/scripts/test: 12-case correctness eval + perf bench for
  pick-container.js, run in real headless Chrome (jsdom has no layout engine,
  so its rects are 0x0 and every geometric assertion would be vacuous).
- eval/ablation: controlled A/B study of the ocd skill vs baseline, scored by
  deterministic render-and-measure scripts, not model judgment. 5 rounds,
  48+ subagent runs. Robust finding: no lift when a bug is named or readable
  in source (even on a real heavy landing page), 2-3x cost; the one real win
  is discovery of a source-invisible, state-dependent layout shift
  (base 0/8 vs ocd 8/8), with before/after screenshots.
- README: wideshot perf/comparison table and a "Does it actually help?
  (measured)" section presenting the ablation evidence with honest scope.
@undeemed undeemed merged commit 05e1fc8 into main Jul 4, 2026
1 check passed
@undeemed undeemed deleted the add-evals branch July 4, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant