diff --git a/AGENTS.md b/AGENTS.md index eff11c0..db43acd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ Before handing `tasks.md` to `ralph-run` (whether you just authored it or just e 3. The verifier's actual reach matches the `Scope:` statement; broaden scope or narrow the verifier when they disagree. 4. Multi-file gates that may hit pre-existing failures enumerate them in a "Pre-existing unrelated failures" sub-section with file:line references and a "do not stop on these" clause. 5. `Stop and hand off if:` conditions are objective (grep-able evidence, exact class/selector names) — never subjective ("looks wrong", "cannot be explained"). -6. Any task requiring human-in-browser verification, deployed-URL checks, or visual judgment is tagged `[manual]` in its title with an explicit "manual verification required — emit BLOCKED_HANDOFF with verification template" stop condition. +6. No `- [ ]` checkbox in a numbered implementer section requires human-in-browser verification, a deployed-URL check, screen-reader judgement, design-review approval, or any other operator-only evidence. Such items either (a) get an automated verifier (Playwright / Puppeteer / Chrome DevTools MCP / `axe-core` / `lighthouse` headless), or (b) move to a dedicated `## Manual verification (operator-only, post-loop)` section that uses `- [op]` markers instead of `- [ ]`. `ralph-run` does not iterate on the operator-only section. Tagging a task `[manual]` and keeping it in the loop is forbidden — every `[manual]` checkbox is, by construction, a guaranteed mid-loop `BLOCKED_HANDOFF`. See the **Human-loop boundary** section in `OPENSPEC-RALPH-BP.md` for the full rule and refactor examples. 7. No two pending tasks claim ownership of the same file/route/symbol; no task's `Stop and hand off if:` would trigger on the normal completion of a later task. Remediate every finding by editing `tasks.md` directly, then re-run `openspec validate ` before starting the loop. If you cannot remediate a finding (it requires a product/policy decision), surface it to the user instead of starting the loop. diff --git a/OPENSPEC-RALPH-BP.md b/OPENSPEC-RALPH-BP.md index 0b0fb65..a2b37a7 100644 --- a/OPENSPEC-RALPH-BP.md +++ b/OPENSPEC-RALPH-BP.md @@ -114,6 +114,7 @@ Baseline artifact compatibility repair template: - Soft verbs without observables (`ensure X`, `support Y`, `validate Z`) - Unresolved policy as tasks ("decide whether X or Y") - Mixing implementation + rollout + manual validation in one checkbox +- **Manual / human-in-the-loop steps as `- [ ]` checkboxes inside the implementer task list.** See [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop). Every `[manual]`-tagged checkbox is, by construction, a guaranteed mid-loop `BLOCKED_HANDOFF`. The runner cannot drive Chrome/DevTools/eyeballs, so the iteration is wasted: the agent reads the task, immediately stops, and the operator now has to context-switch into the loop instead of receiving a single batched verification list at the end. - File chores (separate tasks for imports, renames, tiny follow-through) - Tasks whose only proof is "the next task worked" - `Done when` that only checks unit tests when real behavior is end-to-end @@ -122,6 +123,127 @@ Baseline artifact compatibility repair template: - Repo-wide or slow validators for a narrow task when a focused verifier exists (`npm test`, `make all`, full browser/e2e suites) - Ambiguous package-manager forwarding such as `npm test -- event-schema` unless confirmed to execute only the intended test scope +## Human-loop boundary (no manual steps mid-loop) + +`ralph-run` drives **only** what an agent in a fresh session can execute non-interactively: shell commands, file edits, test runners, static analyzers, scripted browsers (e.g. Playwright, Puppeteer, Chrome DevTools MCP). Anything that needs a human in front of a real browser, eyes on a deployed URL, a touch device, an authenticated SSO session, screen-reader judgement, or design-review approval is **operator work**, not loop work. + +Operator work MUST NOT live as `- [ ]` checkboxes interleaved with implementer tasks. Three reasons: + +1. **Every interleaved manual step is a guaranteed `BLOCKED_HANDOFF`.** The agent reads the task, sees a stop-condition it cannot satisfy, and emits a hand-off. Iteration cost: 1 prompt + 1 supervisor attempt + 1 operator round-trip, for zero implementation progress. +2. **Loop momentum dies on context switches.** Each hand-off forces the operator to leave whatever they were doing, drive Chrome themselves, capture evidence files at exact paths, and resume the loop. A single change with five `[manual]` checkboxes burns five context switches the operator did not budget for. +3. **Manual evidence is rarely on the critical path of the implementation.** It almost always belongs *after* the code, types, tests, and build are green — i.e. as a post-loop acceptance pass the operator runs once, against a finished change, with a single batched checklist. + +### Placement rules + +- **Implementer tasks** (`## 1. …`, `## 2. …`, etc.) contain ONLY checkboxes whose verifiers an agent can run from `bash` / `pnpm` / `npx` / `rg` / a scripted browser. No `[manual]` markers. No "operator opens Chrome" stop conditions. +- **Operator verification** lives in a dedicated, post-loop section titled exactly `## Manual verification (operator-only, post-loop)`. Items in this section are NOT `- [ ]` checkboxes — use `- [op]` markers, plain bullets, or a numbered list. `ralph-run` does not iterate on this section. The final integration gate (last implementer task) references this section as a downstream operator step, not as a loop-step dependency. +- **Mixed tasks are forbidden.** A task that does code + manual verification in one checkbox is split: the code half stays in the implementer list with an automated verifier; the manual half moves to the operator-only section. + +### When you think you need a manual step, try these first + +Before adding ANY operator-only item, exhaust automation: + +| Manual step you are tempted to write | Automate it as | +|---|---| +| "Operator opens `/some-route` in Chrome and reads computed style of `

`" | Boot the prod/dev server in the loop; drive it with Playwright, Puppeteer, or the Chrome DevTools MCP; assert `getComputedStyle()` programmatically; emit a JSON evidence file and a screenshot to a fixed path. | +| "Operator confirms there are no console errors on `/foo`" | Same headless probe; assert `page.on('console')` collected zero `error`-level messages. | +| "Operator visually confirms the layout looks correct" | Snapshot test (`vitest --run --update` for first-run baseline; subsequent diffs are automated). | +| "Operator deploys to staging and clicks through the flow" | Move to the post-loop operator section. Staging deploys are a release pipeline concern, not an implementer-loop concern. | +| "Operator checks accessibility with screen reader / contrast tooling" | Run `axe-core` / `pa11y` / `lighthouse` headless; assert score thresholds. Move qualitative judgement (e.g. "is this announcement clear?") to the post-loop section. | +| "Operator approves design fidelity against Figma" | Move to post-loop. Design-review approval is operator work by definition. | + +If — after honest effort — automation is genuinely impossible for an item, that item belongs in the post-loop operator section. It does not get promoted back into the checkbox list with a `[manual]` tag. + +### Section template + +```markdown +## 7. Final integration gate + +- [ ] **7.1 Final integration gate: all pre-flight gates pass or match baseline** + - Scope: full repository; no code edits in this task + - Done when: `` exit 0 or match the pre-flight baseline. + - Stop and hand off if: any new failure identifier appears. + +## Manual verification (operator-only, post-loop) + +> Run after the loop completes and all implementer checkboxes are checked. +> `ralph-run` does NOT iterate on this section. These items exist to give the +> operator a single batched acceptance pass against the finished change. + +- [op] **Production-served `/docs` H1 paints at the Nextra default scale** + - URL: `http://localhost:/docs/atmosphere/components/accordion` (after `pnpm start`) + - DevTools snippet: + ```js + (() => { + const h1 = document.querySelector('article h1'); + const cs = getComputedStyle(h1); + return { fontSize: cs.fontSize, fontWeight: cs.fontWeight, marginTop: cs.marginTop }; + })() + ``` + - Expected: `{ fontSize: "36px", fontWeight: "700", marginTop: "8px" }` + - Save evidence: `/.ralph/baselines/-prod-h1.txt` and `.png` +- [op] **Visual regression sweep on representative non-`/docs` routes** + - Routes: `/`, `/solutions/cdsi`, `/solutions/atmosphere`, … + - Method: `pnpm start`, viewport screenshot per route, side-by-side compare against `/.ralph/baselines/-sweep-.png`. +``` + +### Refactor example: BAD vs. GOOD + +**BAD — manual step interleaved in implementer list (every loop pass on this task hands off):** + +```markdown +## 4. Implementation — production-build cascade verification + +- [ ] **4.1 Verify cascade-layer wrapper survives `next build`** + - Scope: read-only inspection of `.next/static/css/**` + - Done when: `rg '@layer\s+atm-base' .next/static/css/` reports ≥1 match + - Stop and hand off if: pattern not found + +- [ ] **4.2 [manual] Verify `/docs` H1 computed style against `next start`** + - Scope: manual verification against `pnpm start` + - Done when: operator captures `-prod-h1.txt` and `.png` showing fontSize=36px + - Stop and hand off if: manual verification required — emit BLOCKED_HANDOFF +``` + +**GOOD — automate it, or move it to the post-loop operator section:** + +```markdown +## 4. Implementation — production-build cascade verification + +- [ ] **4.1 Verify cascade-layer wrapper survives `next build`** + - Scope: read-only inspection of `.next/static/css/**` + - Done when: `rg '@layer\s+atm-base' .next/static/css/` reports ≥1 match + - Stop and hand off if: pattern not found + +- [ ] **4.2 Verify `/docs` H1 computed style via headless probe of `next start`** + - Scope: `scripts/probe-prod-h1.mjs` (new); writes evidence under `/.ralph/baselines/` + - Change: A node script boots `next start` on a free port, drives Playwright/Puppeteer + to load `/docs/atmosphere/components/accordion`, reads `getComputedStyle()` on + `article h1`, asserts the three values, writes a labeled `.txt` and a viewport + screenshot, and tears the server down. + - Done when: + - `node scripts/probe-prod-h1.mjs` exits 0 + - `/.ralph/baselines/-prod-h1.txt` exists with labeled lines + `fontSize: "36px"`, `fontWeight: "700"`, `marginTop: "8px"` + - `/.ralph/baselines/-prod-h1.png` exists + - Stop and hand off if: + - `next start` cannot bind a free port in 30s + - the headless browser cannot resolve `nextra-theme-docs/style-prefixed.css` +``` + +Or — when automation is genuinely not feasible (e.g. SSO-gated environments, real device testing): + +```markdown +## Manual verification (operator-only, post-loop) + +- [op] **`/docs` H1 paints at 36px / 700 / 8px on a real Chrome window** + - Steps, snippet, expected values, evidence paths — see template above. +``` + +### Pre-existing `[manual]` checkboxes + +If a `tasks.md` predating this rule still has `[manual]` checkboxes, the pre-scan (rule #6 below) will flag them. Remediate by either (a) automating the step into a real implementer task with a programmatic verifier, or (b) moving it to the post-loop operator section. Do not start the loop until every implementer checkbox is loop-executable. + ## Pre-loop scope-handoff pre-scan Before handing `tasks.md` to `ralph-run`, audit every pending `- [ ]` checkbox for the seven failure modes that most commonly cause `BLOCKED_HANDOFF` mid-loop. Each one is cheap to spot statically and expensive to discover after the loop has burned an iteration plus an auto-resolve attempt on it. @@ -133,7 +255,14 @@ For every pending task, verify in this order: 3. **Verifier scope matches scope statement.** If `Scope:` names one file but `Done when:` runs a command that touches more (`pnpm test:update-snapshots` regenerates snapshots for *all* test files, not just one), either broaden `Scope:` to match the verifier's reach or replace the verifier with a narrower one (`vitest --run `). 4. **Pre-existing failures are classified.** If the verifier is a multi-file gate and the repo has known unrelated failures, they must be enumerated in a "Pre-existing unrelated failures" sub-section with file:line references and an explicit "do not stop on these" clause. See [Quality gates](#quality-gates). 5. **Stop-conditions are objective.** Phrases like "diffs that cannot be explained" or "behavior looks wrong" are subjective and the agent will either over-trigger or under-trigger. Replace with grep-able evidence: "snapshot diff contains `atm-*` class on a docs MDX content element," "`hbr-tab-panel` appears in the rendered DOM." -6. **Manual-only tasks are flagged.** Any task whose `Done when:` requires a human in a browser, eyes on a deployed URL, or visual judgement must be tagged with a `[manual]` marker in its title and have an explicit `Stop and hand off if:` line that says "manual verification required — emit BLOCKED_HANDOFF with verification template." This makes the handoff intentional, not a mid-loop surprise. +6. **Manual-only steps are NOT in the implementer checkbox list.** Per [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop), every `- [ ]` checkbox must be loop-executable: a verifier the agent can run from `bash` / `pnpm` / `npx` / `rg` / a scripted browser, with no human required. Concretely, fail this check if any pending checkbox: + - has a `[manual]` (or equivalent) marker in its title, OR + - has a `Stop and hand off if:` clause containing "manual verification required", "operator captures", "human-in-browser", "eyes on", or any equivalent phrasing whose literal effect is a guaranteed `BLOCKED_HANDOFF` on first encounter, OR + - has a `Done when:` clause whose evidence files can only come from a real human's browser session (DevTools paste, screen-reader transcript, deployed-staging URL screenshot, design-review sign-off). + + Remediate by EITHER (a) replacing the task with a programmatic verifier — boot a server, drive Playwright/Puppeteer/Chrome DevTools MCP, assert `getComputedStyle()` / DOM state / network state in code; OR (b) moving the item out of the numbered implementer sections into a `## Manual verification (operator-only, post-loop)` section using `- [op]` markers (or plain bullets) instead of `- [ ]`. `ralph-run` does not iterate on the operator-only section. See the [Refactor example](#refactor-example-bad-vs-good) for the exact transformation. + + This rule is stricter than the previous "tag manual tasks with `[manual]`" guidance: tagging a task `[manual]` does not authorize keeping it in the loop. The marker exists only for static analysis to catch leftover items that escaped placement; the placement itself is the real fix. 7. **Cross-task scope conflicts are absent.** If task N writes a file that task N-1 already finished, or task N's `Stop and hand off if:` would trigger on the normal completion of task N+1, reorder or merge them. Read tasks in execution order and confirm no two tasks claim ownership of the same file/route/symbol. If any check fails, edit `tasks.md` before starting the loop. The cost of a static edit is a few seconds; the cost of discovering the same issue at iteration 21 is a `BLOCKED_HANDOFF`, a dirty worktree, and a context-poisoned restart. @@ -205,7 +334,7 @@ Before writing tasks, confirm: - `proposal.md` has scope, non-goals, rollout boundaries - `design.md` resolves all policy (no "may be X or Y") - Specs are deterministic (two implementers would make the same choices) -- Human/operator work is outside the `- [ ]` checkbox path +- Human/operator work is outside the `- [ ]` checkbox path — see [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop). Operator items go under `## Manual verification (operator-only, post-loop)` with `- [op]` markers, NOT in numbered implementer sections. If any of these are unresolved, stop and fix the artifact before writing tasks. diff --git a/package.json b/package.json index 2c0ffb7..954dca7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "spec-and-loop", - "version": "3.3.5", + "version": "3.3.6", "description": "OpenSpec + Ralph Loop integration for iterative development with opencode", "main": "index.js", "bin": {