|
| 1 | +# examples/ag-ui e2e (aimock replay) Implementation Plan (Part 2b) |
| 2 | + |
| 3 | +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | +
|
| 5 | +**Goal:** Port the relevant `examples/chat` Playwright e2e suite to `examples/ag-ui`, driving the uvicorn `ag-ui-langgraph` backend with aimock-replayed LLM fixtures — proving transport + a2ui parity deterministically (no live LLM). |
| 6 | + |
| 7 | +**Architecture:** aimock mocks the OpenAI provider (below the transport), so the *same* graph over AG-UI replaying the *same* recorded fixtures yields the *same* a2ui surfaces — the suite ports transport-agnostically. The only harness change is `global-setup` launching the uvicorn ag-ui server (`:8000`) instead of `langgraph dev` (`:2024`), and the app on `:4201`. Shell-specific specs (control-palette, mode-routing, model-picker, url-routing, debug-devtools, lifecycle, keyboard-accessibility, visual-polish, regenerate) are NOT ported — the ag-ui example is a simplified single-conversation app. |
| 8 | + |
| 9 | +**Tech Stack:** Playwright, `@copilotkit/aimock` (repo-standard e2e LLM mock — confirmed in scope), Nx; the existing `examples/ag-ui` app + uvicorn backend (Part 2a). |
| 10 | + |
| 11 | +**Spec:** `docs/superpowers/specs/2026-06-06-examples-ag-ui-standalone-design.md` (Part 2, e2e). Branches off `main` (Part 2a merged). CI e2e job + deploy = Part 3. |
| 12 | + |
| 13 | +--- |
| 14 | + |
| 15 | +## Scope: which specs port |
| 16 | + |
| 17 | +**Port (transport/message/a2ui — app-agnostic DOM assertions):** |
| 18 | +- `initial-render.spec.ts`, `send-receive.spec.ts`, `a2ui-single-bubble.spec.ts` (critical a2ui parity + `customEvents` regression guard), `markdown-surfaces.spec.ts`, `color-scheme.spec.ts`, `error-handling.spec.ts`, `browser-hygiene.spec.ts`. |
| 19 | + |
| 20 | +**Do NOT port (test the dropped demo shell):** |
| 21 | +- `control-palette`, `mode-routing`, `model-picker`, `url-routing`, `debug-devtools`, `lifecycle`, `keyboard-accessibility`, `visual-polish`, `regenerate`, `interrupt-approval` (only if the graph exposes interrupts AND the simplified app surfaces the approval card — defer unless trivial), `research-subagent` (subagent surface — defer), `send-receive` keep, `aimock-runner.spec.ts` (a self-test of the runner — port only if it passes unchanged). |
| 22 | + |
| 23 | +Each ported spec MUST be opened and checked: if it asserts a shell feature absent from the ag-ui app (mode tabs, palette, model picker, thread list), drop that spec or trim the shell-coupled assertions. Report any dropped/trimmed. |
| 24 | + |
| 25 | +--- |
| 26 | + |
| 27 | +## File Structure |
| 28 | + |
| 29 | +**New — `examples/ag-ui/angular/e2e/`** (ported from `examples/chat/angular/e2e/`): |
| 30 | +- `aimock-runner.ts`, `test-helpers.ts` — verbatim copies (mock server + DOM helpers; transport-agnostic). |
| 31 | +- `global-setup.ts` — adapted: uvicorn ag-ui backend on `:8000`, app on `:4201`. |
| 32 | +- `global-teardown.ts` — adapted: kill uvicorn + angular. |
| 33 | +- `playwright.config.ts` — `baseURL: http://localhost:4201`. |
| 34 | +- `tsconfig.json`, `.gitignore`, `README.md`. |
| 35 | +- `fixtures/` — copy the fixtures the ported specs use. |
| 36 | +- `<spec subset>.spec.ts` — the ported specs above. |
| 37 | + |
| 38 | +**Modified:** |
| 39 | +- `examples/ag-ui/angular/project.json` — add `e2e` (+ optional `record`/`drift`) targets. |
| 40 | + |
| 41 | +--- |
| 42 | + |
| 43 | +## Task 1: Port the e2e harness |
| 44 | + |
| 45 | +**Files:** create `examples/ag-ui/angular/e2e/{aimock-runner.ts,test-helpers.ts,global-setup.ts,global-teardown.ts,playwright.config.ts,tsconfig.json,.gitignore,README.md,fixtures/**}` |
| 46 | + |
| 47 | +- [ ] **Step 1: Copy the transport-agnostic harness files verbatim** |
| 48 | + |
| 49 | +```bash |
| 50 | +mkdir -p examples/ag-ui/angular/e2e/fixtures |
| 51 | +cp examples/chat/angular/e2e/aimock-runner.ts examples/ag-ui/angular/e2e/aimock-runner.ts |
| 52 | +cp examples/chat/angular/e2e/test-helpers.ts examples/ag-ui/angular/e2e/test-helpers.ts |
| 53 | +cp examples/chat/angular/e2e/tsconfig.json examples/ag-ui/angular/e2e/tsconfig.json |
| 54 | +cp examples/chat/angular/e2e/.gitignore examples/ag-ui/angular/e2e/.gitignore |
| 55 | +cp examples/chat/angular/e2e/global-teardown.ts examples/ag-ui/angular/e2e/global-teardown.ts |
| 56 | +``` |
| 57 | +Then READ `test-helpers.ts` and `global-teardown.ts`: if they reference `examples-chat` project names, `:4200`, `:2024`, or the `langgraph` child in `__AIMOCK_E2E_STATE__`, note them — `global-teardown.ts` likely kills `state.langgraph`; you will rename that to the uvicorn child in Step 3. `test-helpers.ts` should be pure DOM (sendPromptAndWait etc.) — if it has no backend coupling, leave verbatim. |
| 58 | + |
| 59 | +- [ ] **Step 2: Write `examples/ag-ui/angular/e2e/playwright.config.ts`** |
| 60 | + |
| 61 | +```ts |
| 62 | +// SPDX-License-Identifier: MIT |
| 63 | +import { defineConfig } from '@playwright/test'; |
| 64 | + |
| 65 | +export default defineConfig({ |
| 66 | + testDir: '.', |
| 67 | + testMatch: '**/*.spec.ts', |
| 68 | + testIgnore: ['aimock-runner.spec.ts'], |
| 69 | + fullyParallel: false, |
| 70 | + workers: 1, |
| 71 | + retries: process.env.CI ? 2 : 0, |
| 72 | + reporter: process.env.CI ? [['list'], ['html', { open: 'never' }]] : 'list', |
| 73 | + use: { |
| 74 | + baseURL: 'http://localhost:4201', |
| 75 | + trace: 'retain-on-failure', |
| 76 | + }, |
| 77 | + globalSetup: './global-setup.ts', |
| 78 | + globalTeardown: './global-teardown.ts', |
| 79 | +}); |
| 80 | +``` |
| 81 | + |
| 82 | +- [ ] **Step 3: Write `examples/ag-ui/angular/e2e/global-setup.ts`** (uvicorn instead of langgraph dev) |
| 83 | + |
| 84 | +```ts |
| 85 | +// SPDX-License-Identifier: MIT |
| 86 | +import { spawn, type ChildProcess } from 'node:child_process'; |
| 87 | +import { setTimeout as delay } from 'node:timers/promises'; |
| 88 | +import { resolve } from 'node:path'; |
| 89 | +import { startAimock, type AimockHandle } from './aimock-runner'; |
| 90 | + |
| 91 | +interface SharedState { |
| 92 | + aimock: AimockHandle; |
| 93 | + backend: ChildProcess; |
| 94 | + angular: ChildProcess; |
| 95 | +} |
| 96 | + |
| 97 | +declare global { |
| 98 | + // eslint-disable-next-line no-var |
| 99 | + var __AIMOCK_E2E_STATE__: SharedState | undefined; |
| 100 | +} |
| 101 | + |
| 102 | +const REPO_ROOT = resolve(__dirname, '../../../..'); |
| 103 | +const FIXTURE_PATH = process.env.AIMOCK_FIXTURE |
| 104 | + ? resolve(__dirname, process.env.AIMOCK_FIXTURE) |
| 105 | + : resolve(__dirname, 'fixtures'); |
| 106 | + |
| 107 | +async function waitForPort(url: string, timeoutMs: number): Promise<void> { |
| 108 | + const start = Date.now(); |
| 109 | + while (Date.now() - start < timeoutMs) { |
| 110 | + try { |
| 111 | + const res = await fetch(url); |
| 112 | + if (res.ok || res.status === 404) return; |
| 113 | + } catch { |
| 114 | + // not up yet |
| 115 | + } |
| 116 | + await delay(500); |
| 117 | + } |
| 118 | + throw new Error(`Server at ${url} did not become ready within ${timeoutMs}ms`); |
| 119 | +} |
| 120 | + |
| 121 | +export default async function globalSetup(): Promise<void> { |
| 122 | + const aimock = await startAimock({ mode: 'replay', fixturePath: FIXTURE_PATH }); |
| 123 | + console.log(`[aimock-e2e] aimock listening at ${aimock.baseUrl}`); |
| 124 | + |
| 125 | + const backend = spawn( |
| 126 | + 'uv', |
| 127 | + ['run', 'uvicorn', 'src.server:app', '--port', '8000'], |
| 128 | + { |
| 129 | + cwd: resolve(REPO_ROOT, 'examples/ag-ui/python'), |
| 130 | + env: { |
| 131 | + ...process.env, |
| 132 | + OPENAI_BASE_URL: aimock.baseUrl, |
| 133 | + OPENAI_API_KEY: 'test-not-used', |
| 134 | + }, |
| 135 | + stdio: 'pipe', |
| 136 | + }, |
| 137 | + ); |
| 138 | + backend.stdout?.on('data', (b) => process.stdout.write(`[uvicorn] ${b}`)); |
| 139 | + backend.stderr?.on('data', (b) => process.stderr.write(`[uvicorn] ${b}`)); |
| 140 | + |
| 141 | + await waitForPort('http://localhost:8000/ok', 60_000); |
| 142 | + console.log('[aimock-e2e] ag-ui backend ready on :8000'); |
| 143 | + |
| 144 | + const angular = spawn( |
| 145 | + 'npx', |
| 146 | + ['nx', 'serve', 'examples-ag-ui-angular', '--port', '4201'], |
| 147 | + { cwd: REPO_ROOT, env: { ...process.env }, stdio: 'pipe' }, |
| 148 | + ); |
| 149 | + angular.stdout?.on('data', (b) => process.stdout.write(`[angular] ${b}`)); |
| 150 | + angular.stderr?.on('data', (b) => process.stderr.write(`[angular] ${b}`)); |
| 151 | + |
| 152 | + await waitForPort('http://localhost:4201/', 120_000); |
| 153 | + console.log('[aimock-e2e] angular ready on :4201'); |
| 154 | + |
| 155 | + globalThis.__AIMOCK_E2E_STATE__ = { aimock, backend, angular }; |
| 156 | +} |
| 157 | +``` |
| 158 | + |
| 159 | +- [ ] **Step 4: Update `global-teardown.ts` to the renamed `backend` child** |
| 160 | + |
| 161 | +Read the copied `global-teardown.ts`. It references `state.langgraph` (and/or `state.angular`, `state.aimock`). Rename `langgraph` → `backend` to match the new `SharedState`. Keep the kill logic identical (e.g. `state.backend?.kill('SIGTERM')`, `state.angular?.kill('SIGTERM')`, `await state.aimock.stop()`). If the teardown imports a `SharedState` type, ensure it matches global-setup's (or is duplicated identically). |
| 162 | + |
| 163 | +- [ ] **Step 5: Copy the fixtures the ported specs need** |
| 164 | + |
| 165 | +Determine which fixtures the to-be-ported specs reference (read each spec for `AIMOCK_FIXTURE` or prompt strings; the runner loads the whole `fixtures/` dir by default). Simplest + safe: copy the whole fixtures dir (same graph → all fixtures valid): |
| 166 | +```bash |
| 167 | +cp examples/chat/angular/e2e/fixtures/*.json examples/ag-ui/angular/e2e/fixtures/ |
| 168 | +``` |
| 169 | +Copy `examples/chat/angular/e2e/README.md` → adapt the ports (2024→8000, 4200→4201) and project name, or write a short fresh README. |
| 170 | + |
| 171 | +- [ ] **Step 6: Add e2e target to `examples/ag-ui/angular/project.json`** |
| 172 | + |
| 173 | +Read `examples/chat/angular/project.json`'s `e2e` target and mirror it with paths swapped: |
| 174 | +```json |
| 175 | +"e2e": { |
| 176 | + "executor": "@nx/playwright:playwright", |
| 177 | + "options": { "config": "examples/ag-ui/angular/e2e/playwright.config.ts" } |
| 178 | +} |
| 179 | +``` |
| 180 | +(Add `record`/`drift` targets only if you also port `scripts/record.ts`/`drift.ts` — OPTIONAL; defer if not needed for the e2e run.) |
| 181 | + |
| 182 | +- [ ] **Step 7: Commit** |
| 183 | + |
| 184 | +```bash |
| 185 | +git add examples/ag-ui/angular/e2e examples/ag-ui/angular/project.json |
| 186 | +git commit -m "test(examples/ag-ui): port aimock e2e harness (uvicorn backend, :4201)" |
| 187 | +``` |
| 188 | + |
| 189 | +--- |
| 190 | + |
| 191 | +## Task 2: Port the spec subset |
| 192 | + |
| 193 | +**Files:** create `examples/ag-ui/angular/e2e/<subset>.spec.ts` |
| 194 | + |
| 195 | +- [ ] **Step 1: Copy the transport/a2ui spec subset** |
| 196 | + |
| 197 | +```bash |
| 198 | +cd examples/chat/angular/e2e |
| 199 | +for s in initial-render send-receive a2ui-single-bubble markdown-surfaces color-scheme error-handling browser-hygiene; do |
| 200 | + cp "$s.spec.ts" ../../../ag-ui/angular/e2e/"$s.spec.ts" |
| 201 | +done |
| 202 | +cd ../../../.. |
| 203 | +``` |
| 204 | + |
| 205 | +- [ ] **Step 2: Audit each ported spec for shell coupling** |
| 206 | + |
| 207 | +Open each copied spec. For each, check whether it: |
| 208 | +- navigates to a shell route (e.g. `/embed`, `/popup`, mode tabs) that the ag-ui app doesn't have — change to `/` (the ag-ui app is single-route), |
| 209 | +- asserts a shell element absent in the ag-ui app (mode tabs, palette, model picker, project/thread list) — remove that assertion or drop the spec, |
| 210 | +- relies on `examples-chat`-specific selectors — the `<chat>` composition selectors (`chat-message`, `a2ui-surface`, `chat-streaming-md`, etc.) are shared and fine. |
| 211 | + |
| 212 | +The ag-ui `app.html` renders `<chat main [agent]="agent" />` at route `/`. Most message/a2ui assertions work unchanged. Report exactly what you trimmed/dropped per spec. |
| 213 | + |
| 214 | +- [ ] **Step 3: Run the suite locally (aimock replay — deterministic, no live LLM)** |
| 215 | + |
| 216 | +```bash |
| 217 | +npx nx e2e examples-ag-ui-angular 2>&1 | tail -40 |
| 218 | +``` |
| 219 | +Expected: the ported specs pass. The harness auto-starts aimock + uvicorn + angular. If a spec fails: |
| 220 | +- If it's a shell-coupled assertion you missed, trim it (Step 2). |
| 221 | +- If `a2ui-single-bubble` fails on the surface not rendering, that's a real signal — check the uvicorn log in the Playwright output; the a2ui surface depends on the `customEvents` signal (now on main) + the `render_a2ui_surface` tool call. Report BLOCKED with the failure if it's a genuine parity gap, not a shell-coupling issue. |
| 222 | + |
| 223 | +> If `uv` / `nx serve` aren't available in the runner, that's an environment problem — report it; do not fake the run. |
| 224 | +
|
| 225 | +- [ ] **Step 4: Commit** |
| 226 | + |
| 227 | +```bash |
| 228 | +git add examples/ag-ui/angular/e2e/*.spec.ts |
| 229 | +git commit -m "test(examples/ag-ui): port transport + a2ui e2e specs (aimock replay)" |
| 230 | +``` |
| 231 | + |
| 232 | +--- |
| 233 | + |
| 234 | +## Task 3: PR |
| 235 | + |
| 236 | +- [ ] **Step 1: Push + open PR** |
| 237 | + |
| 238 | +```bash |
| 239 | +git push -u origin claude/examples-ag-ui-e2e |
| 240 | +gh pr create --title "test(examples/ag-ui): aimock e2e — transport + a2ui parity (Part 2b)" --body "$(cat <<'EOF' |
| 241 | +## Summary |
| 242 | +
|
| 243 | +Ports the relevant `examples/chat` Playwright e2e to `examples/ag-ui`, driving the uvicorn ag-ui backend with aimock-replayed OpenAI fixtures. aimock mocks the LLM provider (below the transport), so the same graph over AG-UI + same fixtures = same a2ui surfaces — deterministic parity proof. |
| 244 | +
|
| 245 | +- Harness ported: `aimock-runner`/`test-helpers` verbatim; `global-setup` launches `uvicorn src.server:app` on `:8000` (was `langgraph dev :2024`); app on `:4201`. |
| 246 | +- Specs ported (transport/message/a2ui): initial-render, send-receive, a2ui-single-bubble (the parity + `customEvents` regression guard), markdown-surfaces, color-scheme, error-handling, browser-hygiene. |
| 247 | +- Shell-specific specs (palette/modes/model-picker/url-routing/etc.) intentionally NOT ported — the ag-ui example is a simplified single-conversation app. |
| 248 | +
|
| 249 | +aimock is the repo-standard e2e LLM mock (already used by examples/chat, cockpit, website) — in scope per prior decision. |
| 250 | +
|
| 251 | +Part 2b of `docs/superpowers/specs/2026-06-06-examples-ag-ui-standalone-design.md`. CI e2e job + deploy = Part 3. |
| 252 | +
|
| 253 | +## Test plan |
| 254 | +- [x] `nx e2e examples-ag-ui-angular` green locally (aimock replay). |
| 255 | +- [ ] CI green (the e2e CI job lands with Part 3). |
| 256 | +
|
| 257 | +🤖 Generated with [Claude Code](https://claude.com/claude-code) |
| 258 | +EOF |
| 259 | +)" |
| 260 | +``` |
| 261 | + |
| 262 | +- [ ] **Step 2: Arm auto-merge** |
| 263 | + |
| 264 | +```bash |
| 265 | +gh pr merge <PR_NUMBER> --squash --auto --delete-branch |
| 266 | +``` |
| 267 | + |
| 268 | +- [ ] **Step 3: Wait for green + merge.** If `Vercel – threadplane` preview fails on the known transient npm-registry 404, redeploy that preview via the Vercel API. If `chat:lint`-style broken-main blocks it, investigate whether main regressed (as in #607). |
| 269 | + |
| 270 | +--- |
| 271 | + |
| 272 | +## Self-Review |
| 273 | + |
| 274 | +- [ ] **Spec coverage (Part 2 e2e):** harness port + uvicorn adaptation → Task 1. Transport/a2ui spec subset → Task 2. Local green → Task 2 Step 3. Create-app smoke + CI e2e job deferred (smoke → later; CI → Part 3) — noted. |
| 275 | +- [ ] **No placeholders:** harness files have full content or verbatim-copy instructions; specs are copy+audit with explicit drop/trim guidance. |
| 276 | +- [ ] **Consistency:** `SharedState.backend` (renamed from `langgraph`) consistent between global-setup (Task 1 Step 3) and global-teardown (Task 1 Step 4); ports `:8000`/`:4201` consistent across global-setup + playwright.config + project.json; project name `examples-ag-ui-angular` consistent. |
| 277 | +- [ ] **Honesty:** the spec subset is a deliberate reduction from examples/chat's full suite (shell specs dropped) — documented in the PR body and Scope section, not silently truncated. |
0 commit comments