Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Before handing `tasks.md` to `ralph-run` (whether you just authored it or just e
3. The verifier's actual reach matches the `Scope:` statement; broaden scope or narrow the verifier when they disagree.
4. Multi-file gates that may hit pre-existing failures enumerate them in a "Pre-existing unrelated failures" sub-section with file:line references and a "do not stop on these" clause.
5. `Stop and hand off if:` conditions are objective (grep-able evidence, exact class/selector names) — never subjective ("looks wrong", "cannot be explained").
6. Any task requiring human-in-browser verification, deployed-URL checks, or visual judgment is tagged `[manual]` in its title with an explicit "manual verification required — emit BLOCKED_HANDOFF with verification template" stop condition.
6. No `- [ ]` checkbox in a numbered implementer section requires human-in-browser verification, a deployed-URL check, screen-reader judgement, design-review approval, or any other operator-only evidence. Such items either (a) get an automated verifier (Playwright / Puppeteer / Chrome DevTools MCP / `axe-core` / `lighthouse` headless), or (b) move to a dedicated `## Manual verification (operator-only, post-loop)` section that uses `- [op]` markers instead of `- [ ]`. `ralph-run` does not iterate on the operator-only section. Tagging a task `[manual]` and keeping it in the loop is forbidden — every `[manual]` checkbox is, by construction, a guaranteed mid-loop `BLOCKED_HANDOFF`. See the **Human-loop boundary** section in `OPENSPEC-RALPH-BP.md` for the full rule and refactor examples.
7. No two pending tasks claim ownership of the same file/route/symbol; no task's `Stop and hand off if:` would trigger on the normal completion of a later task.

Remediate every finding by editing `tasks.md` directly, then re-run `openspec validate <change>` before starting the loop. If you cannot remediate a finding (it requires a product/policy decision), surface it to the user instead of starting the loop.
133 changes: 131 additions & 2 deletions OPENSPEC-RALPH-BP.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ Baseline artifact compatibility repair template:
- Soft verbs without observables (`ensure X`, `support Y`, `validate Z`)
- Unresolved policy as tasks ("decide whether X or Y")
- Mixing implementation + rollout + manual validation in one checkbox
- **Manual / human-in-the-loop steps as `- [ ]` checkboxes inside the implementer task list.** See [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop). Every `[manual]`-tagged checkbox is, by construction, a guaranteed mid-loop `BLOCKED_HANDOFF`. The runner cannot drive Chrome/DevTools/eyeballs, so the iteration is wasted: the agent reads the task, immediately stops, and the operator now has to context-switch into the loop instead of receiving a single batched verification list at the end.
- File chores (separate tasks for imports, renames, tiny follow-through)
- Tasks whose only proof is "the next task worked"
- `Done when` that only checks unit tests when real behavior is end-to-end
Expand All @@ -122,6 +123,127 @@ Baseline artifact compatibility repair template:
- Repo-wide or slow validators for a narrow task when a focused verifier exists (`npm test`, `make all`, full browser/e2e suites)
- Ambiguous package-manager forwarding such as `npm test -- event-schema` unless confirmed to execute only the intended test scope

## Human-loop boundary (no manual steps mid-loop)

`ralph-run` drives **only** what an agent in a fresh session can execute non-interactively: shell commands, file edits, test runners, static analyzers, scripted browsers (e.g. Playwright, Puppeteer, Chrome DevTools MCP). Anything that needs a human in front of a real browser, eyes on a deployed URL, a touch device, an authenticated SSO session, screen-reader judgement, or design-review approval is **operator work**, not loop work.

Operator work MUST NOT live as `- [ ]` checkboxes interleaved with implementer tasks. Three reasons:

1. **Every interleaved manual step is a guaranteed `BLOCKED_HANDOFF`.** The agent reads the task, sees a stop-condition it cannot satisfy, and emits a hand-off. Iteration cost: 1 prompt + 1 supervisor attempt + 1 operator round-trip, for zero implementation progress.
2. **Loop momentum dies on context switches.** Each hand-off forces the operator to leave whatever they were doing, drive Chrome themselves, capture evidence files at exact paths, and resume the loop. A single change with five `[manual]` checkboxes burns five context switches the operator did not budget for.
3. **Manual evidence is rarely on the critical path of the implementation.** It almost always belongs *after* the code, types, tests, and build are green — i.e. as a post-loop acceptance pass the operator runs once, against a finished change, with a single batched checklist.

### Placement rules

- **Implementer tasks** (`## 1. …`, `## 2. …`, etc.) contain ONLY checkboxes whose verifiers an agent can run from `bash` / `pnpm` / `npx` / `rg` / a scripted browser. No `[manual]` markers. No "operator opens Chrome" stop conditions.
- **Operator verification** lives in a dedicated, post-loop section titled exactly `## Manual verification (operator-only, post-loop)`. Items in this section are NOT `- [ ]` checkboxes — use `- [op]` markers, plain bullets, or a numbered list. `ralph-run` does not iterate on this section. The final integration gate (last implementer task) references this section as a downstream operator step, not as a loop-step dependency.
- **Mixed tasks are forbidden.** A task that does code + manual verification in one checkbox is split: the code half stays in the implementer list with an automated verifier; the manual half moves to the operator-only section.

### When you think you need a manual step, try these first

Before adding ANY operator-only item, exhaust automation:

| Manual step you are tempted to write | Automate it as |
|---|---|
| "Operator opens `/some-route` in Chrome and reads computed style of `<h1>`" | Boot the prod/dev server in the loop; drive it with Playwright, Puppeteer, or the Chrome DevTools MCP; assert `getComputedStyle()` programmatically; emit a JSON evidence file and a screenshot to a fixed path. |
| "Operator confirms there are no console errors on `/foo`" | Same headless probe; assert `page.on('console')` collected zero `error`-level messages. |
| "Operator visually confirms the layout looks correct" | Snapshot test (`vitest --run --update` for first-run baseline; subsequent diffs are automated). |
| "Operator deploys to staging and clicks through the flow" | Move to the post-loop operator section. Staging deploys are a release pipeline concern, not an implementer-loop concern. |
| "Operator checks accessibility with screen reader / contrast tooling" | Run `axe-core` / `pa11y` / `lighthouse` headless; assert score thresholds. Move qualitative judgement (e.g. "is this announcement clear?") to the post-loop section. |
| "Operator approves design fidelity against Figma" | Move to post-loop. Design-review approval is operator work by definition. |

If — after honest effort — automation is genuinely impossible for an item, that item belongs in the post-loop operator section. It does not get promoted back into the checkbox list with a `[manual]` tag.

### Section template

```markdown
## 7. Final integration gate

- [ ] **7.1 Final integration gate: all pre-flight gates pass or match baseline**
- Scope: full repository; no code edits in this task
- Done when: `<gate commands>` exit 0 or match the pre-flight baseline.
- Stop and hand off if: any new failure identifier appears.

## Manual verification (operator-only, post-loop)

> Run after the loop completes and all implementer checkboxes are checked.
> `ralph-run` does NOT iterate on this section. These items exist to give the
> operator a single batched acceptance pass against the finished change.

- [op] **Production-served `/docs` H1 paints at the Nextra default scale**
- URL: `http://localhost:<port>/docs/atmosphere/components/accordion` (after `pnpm start`)
- DevTools snippet:
```js
(() => {
const h1 = document.querySelector('article h1');
const cs = getComputedStyle(h1);
return { fontSize: cs.fontSize, fontWeight: cs.fontWeight, marginTop: cs.marginTop };
})()
```
- Expected: `{ fontSize: "36px", fontWeight: "700", marginTop: "8px" }`
- Save evidence: `<change>/.ralph/baselines/<change>-prod-h1.txt` and `.png`
- [op] **Visual regression sweep on representative non-`/docs` routes**
- Routes: `/`, `/solutions/cdsi`, `/solutions/atmosphere`, …
- Method: `pnpm start`, viewport screenshot per route, side-by-side compare against `<change>/.ralph/baselines/<change>-sweep-<slug>.png`.
```

### Refactor example: BAD vs. GOOD

**BAD — manual step interleaved in implementer list (every loop pass on this task hands off):**

```markdown
## 4. Implementation — production-build cascade verification

- [ ] **4.1 Verify cascade-layer wrapper survives `next build`**
- Scope: read-only inspection of `.next/static/css/**`
- Done when: `rg '@layer\s+atm-base' .next/static/css/` reports ≥1 match
- Stop and hand off if: pattern not found

- [ ] **4.2 [manual] Verify `/docs` H1 computed style against `next start`**
- Scope: manual verification against `pnpm start`
- Done when: operator captures `<change>-prod-h1.txt` and `.png` showing fontSize=36px
- Stop and hand off if: manual verification required — emit BLOCKED_HANDOFF
```

**GOOD — automate it, or move it to the post-loop operator section:**

```markdown
## 4. Implementation — production-build cascade verification

- [ ] **4.1 Verify cascade-layer wrapper survives `next build`**
- Scope: read-only inspection of `.next/static/css/**`
- Done when: `rg '@layer\s+atm-base' .next/static/css/` reports ≥1 match
- Stop and hand off if: pattern not found

- [ ] **4.2 Verify `/docs` H1 computed style via headless probe of `next start`**
- Scope: `scripts/probe-prod-h1.mjs` (new); writes evidence under `<change>/.ralph/baselines/`
- Change: A node script boots `next start` on a free port, drives Playwright/Puppeteer
to load `/docs/atmosphere/components/accordion`, reads `getComputedStyle()` on
`article h1`, asserts the three values, writes a labeled `.txt` and a viewport
screenshot, and tears the server down.
- Done when:
- `node scripts/probe-prod-h1.mjs` exits 0
- `<change>/.ralph/baselines/<change>-prod-h1.txt` exists with labeled lines
`fontSize: "36px"`, `fontWeight: "700"`, `marginTop: "8px"`
- `<change>/.ralph/baselines/<change>-prod-h1.png` exists
- Stop and hand off if:
- `next start` cannot bind a free port in 30s
- the headless browser cannot resolve `nextra-theme-docs/style-prefixed.css`
```

Or — when automation is genuinely not feasible (e.g. SSO-gated environments, real device testing):

```markdown
## Manual verification (operator-only, post-loop)

- [op] **`/docs` H1 paints at 36px / 700 / 8px on a real Chrome window**
- Steps, snippet, expected values, evidence paths — see template above.
```

### Pre-existing `[manual]` checkboxes

If a `tasks.md` predating this rule still has `[manual]` checkboxes, the pre-scan (rule #6 below) will flag them. Remediate by either (a) automating the step into a real implementer task with a programmatic verifier, or (b) moving it to the post-loop operator section. Do not start the loop until every implementer checkbox is loop-executable.

## Pre-loop scope-handoff pre-scan

Before handing `tasks.md` to `ralph-run`, audit every pending `- [ ]` checkbox for the seven failure modes that most commonly cause `BLOCKED_HANDOFF` mid-loop. Each one is cheap to spot statically and expensive to discover after the loop has burned an iteration plus an auto-resolve attempt on it.
Expand All @@ -133,7 +255,14 @@ For every pending task, verify in this order:
3. **Verifier scope matches scope statement.** If `Scope:` names one file but `Done when:` runs a command that touches more (`pnpm test:update-snapshots` regenerates snapshots for *all* test files, not just one), either broaden `Scope:` to match the verifier's reach or replace the verifier with a narrower one (`vitest --run <single-test-file>`).
4. **Pre-existing failures are classified.** If the verifier is a multi-file gate and the repo has known unrelated failures, they must be enumerated in a "Pre-existing unrelated failures" sub-section with file:line references and an explicit "do not stop on these" clause. See [Quality gates](#quality-gates).
5. **Stop-conditions are objective.** Phrases like "diffs that cannot be explained" or "behavior looks wrong" are subjective and the agent will either over-trigger or under-trigger. Replace with grep-able evidence: "snapshot diff contains `atm-*` class on a docs MDX content element," "`hbr-tab-panel` appears in the rendered DOM."
6. **Manual-only tasks are flagged.** Any task whose `Done when:` requires a human in a browser, eyes on a deployed URL, or visual judgement must be tagged with a `[manual]` marker in its title and have an explicit `Stop and hand off if:` line that says "manual verification required — emit BLOCKED_HANDOFF with verification template." This makes the handoff intentional, not a mid-loop surprise.
6. **Manual-only steps are NOT in the implementer checkbox list.** Per [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop), every `- [ ]` checkbox must be loop-executable: a verifier the agent can run from `bash` / `pnpm` / `npx` / `rg` / a scripted browser, with no human required. Concretely, fail this check if any pending checkbox:
- has a `[manual]` (or equivalent) marker in its title, OR
- has a `Stop and hand off if:` clause containing "manual verification required", "operator captures", "human-in-browser", "eyes on", or any equivalent phrasing whose literal effect is a guaranteed `BLOCKED_HANDOFF` on first encounter, OR
- has a `Done when:` clause whose evidence files can only come from a real human's browser session (DevTools paste, screen-reader transcript, deployed-staging URL screenshot, design-review sign-off).

Remediate by EITHER (a) replacing the task with a programmatic verifier — boot a server, drive Playwright/Puppeteer/Chrome DevTools MCP, assert `getComputedStyle()` / DOM state / network state in code; OR (b) moving the item out of the numbered implementer sections into a `## Manual verification (operator-only, post-loop)` section using `- [op]` markers (or plain bullets) instead of `- [ ]`. `ralph-run` does not iterate on the operator-only section. See the [Refactor example](#refactor-example-bad-vs-good) for the exact transformation.

This rule is stricter than the previous "tag manual tasks with `[manual]`" guidance: tagging a task `[manual]` does not authorize keeping it in the loop. The marker exists only for static analysis to catch leftover items that escaped placement; the placement itself is the real fix.
7. **Cross-task scope conflicts are absent.** If task N writes a file that task N-1 already finished, or task N's `Stop and hand off if:` would trigger on the normal completion of task N+1, reorder or merge them. Read tasks in execution order and confirm no two tasks claim ownership of the same file/route/symbol.

If any check fails, edit `tasks.md` before starting the loop. The cost of a static edit is a few seconds; the cost of discovering the same issue at iteration 21 is a `BLOCKED_HANDOFF`, a dirty worktree, and a context-poisoned restart.
Expand Down Expand Up @@ -205,7 +334,7 @@ Before writing tasks, confirm:
- `proposal.md` has scope, non-goals, rollout boundaries
- `design.md` resolves all policy (no "may be X or Y")
- Specs are deterministic (two implementers would make the same choices)
- Human/operator work is outside the `- [ ]` checkbox path
- Human/operator work is outside the `- [ ]` checkbox path — see [Human-loop boundary](#human-loop-boundary-no-manual-steps-mid-loop). Operator items go under `## Manual verification (operator-only, post-loop)` with `- [op]` markers, NOT in numbered implementer sections.

If any of these are unresolved, stop and fix the artifact before writing tasks.

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "spec-and-loop",
"version": "3.3.5",
"version": "3.3.6",
"description": "OpenSpec + Ralph Loop integration for iterative development with opencode",
"main": "index.js",
"bin": {
Expand Down
Loading