Replace frontend source greps with runtime behavior coverage

[alexeygrigorev]

Replace frontend source greps with runtime behavior coverage

Status: blocked — groomed; implement after #180 fixes and splits the overlapping browser journeys
Tags: bug, frontend, backend, testing, P1
Depends on: #180
Blocks: None

Scope

Finish the source-text-test cleanup identified by the 2026-08-13 suite review without weakening the canonical frontend or deployment gates.

Delete the remaining Home, Knowledge, Operations/Inbox/Assistants, Artifacts, and runtime-template marker greps from backend/tests/frontend-convergence.test.ts. Keep and make data-driven the real backend/frontend boundary that belongs there: the TypeScript handler serves the canonical root and every manifest-declared same-origin asset, rejects traversal and the retired /public/* namespace, and fails explicitly when the canonical artifact is absent. Do not mirror the frontend module inventory, UI copy, private function names, CSS selectors, asset count, or file bytes in a second hard-coded list. Product behavior already exercised by frontend unit tests and normal browser journeys is not replaced one-for-one with another static assertion.

Replace backend/scripts/frontend-capabilities.test.mjs test-title/file/source parsing with stable coverage evidence emitted by the normal Playwright run. The capability catalog remains the durable definition of retained routes, API seams, roles, and observable states, but a human-readable test title or file path is no longer its identity. A passing browser journey records structured capability-state evidence through a shared Playwright helper/annotation contract, and a reporter or equivalent same-run collector validates the completed results. It must not read spec source, parse test declarations, regex-match test bodies, run Playwright discovery a second time, or infer coverage from screenshots.

The runtime evidence contract must record the stable capability/state ID, the allowed role actually exercised, and the canonical served route. Evidence counts only when the browser test completes successfully. Failed, skipped, interrupted, or did-not-run tests provide no coverage. Role-specific denial/mutation states must retain their exact role evidence; ordinary operator/admin states need at least one allowed authenticated role unless the catalog explicitly requires both. Unknown states, wrong roles, duplicate/conflicting declarations, wrong routes, and uncovered required states fail with the affected stable IDs in the diagnostic.

Keep static catalog validation and deployment-placement validation in the fast backend/Node suite. Runtime browser coverage completeness belongs only to the existing independent Playwright workflow. Parse workflow structure and/or an observable dry-run command graph rather than matching unrelated source text. The deploy workflow must remain independent of Playwright.

Coordination and implementation order

1. Wait for Restore behavior-based browser coverage and fix canonical UI accessibility races #180 to finish its fixes and split the two large browser specs. Restore behavior-based browser coverage and fix canonical UI accessibility races #180 owns product/accessibility behavior, deterministic fixtures, browser diagnostics, serial-fallout removal, and its screenshots. Replace frontend source greps with runtime behavior coverage #184 then adds stable capability evidence to the resulting journey-sized specs instead of editing the same files in parallel.
2. Remove the redundant lower marker suites and their now-unused source reads from backend/tests/frontend-convergence.test.ts; retain the served-route, canonical-asset, retired-namespace/traversal, and missing-artifact behavior.
3. Replace file plus title coverage pointers in backend/e2e/frontend-capabilities.json with the runtime evidence schema. Do not add file paths, titles, line numbers, selectors, copy, or asset hashes as identity.
4. Add the small shared Playwright evidence helper and same-run coverage collector/reporter, then instrument the final Restore behavior-based browser coverage and fix canonical UI accessibility races #180 browser journeys plus runtime-template-readonly.spec.js and card-template-updates.spec.js as needed. The helper records evidence only after the journey has reached and asserted the relevant route/state.
5. Keep the catalog-schema and workflow-placement checks deterministic and browser-free in the backend suite. Add focused collector tests using synthetic Playwright result objects; those tests must not launch Chromium, a server, SAM, npm install, or a second E2E run.

Acceptance Criteria

• backend/tests/frontend-convergence.test.ts contains no reads or assertions for frontend private function names, incidental UI sentences, CSS selectors/classes, or source marker lists. The obsolete lower Home/Knowledge/Operations/Artifacts/runtime-template marker tests are deleted.
• The retained convergence contract drives the real handler and proves /, every path declared by the canonical frontend asset manifest, content-type behavior, retired /public/* rejection, traversal rejection, and the explicit missing-canonical-artifact failure without a duplicated hard-coded asset inventory or count.
• The capability catalog still classifies every retained route exactly once and validates its API boundaries, allowed roles, state IDs/kinds, descriptions, and public-safe content.
• Capability evidence uses stable structured state IDs emitted by the running Playwright journey. Renaming a human test title or moving a spec file leaves coverage valid; changing/removing the stable evidence or failing to exercise its route/role/state makes coverage fail.
• A state is covered only by a passed real-browser result. Failed, skipped, interrupted, and did-not-run cases are excluded, and the failure report lists every uncovered state ID plus the non-passing journeys that had claimed it.
• Evidence validates the exercised canonical route and allowed role. Narrow role states such as signed-out/expired/disabled denial, operator denial/read-only behavior, and admin mutation cannot be satisfied by a different role.
• No capability check scans Playwright source, extracts literal test declarations, matches test titles/files, searches test bodies for page/expect calls, or uses Playwright --list/discovery as a substitute for an executed passing journey.
• Restore behavior-based browser coverage and fix canonical UI accessibility races #180's final independent journeys remain independently runnable; the coverage collector consumes that one normal run and does not introduce serial ordering, shared outcome dependence, a second browser invocation, fixed waits, retries, request interception, or new product behavior.
• Runtime completeness runs only in .github/workflows/validate-backend-e2e.yml through the existing full Playwright command. The workflow remains non-deploy-blocking and continues to upload backend/playwright-report/ and backend/test-results/ on failure.
• .github/workflows/deploy-dataops-v1.yml does not run, discover, or depend on Playwright. Its checks remain browser-free.
• The accepted Run the packaged frontend artifact isolation contract in CI #178/Isolate SAM dependency installation from the shared developer workspace #185 artifact path is preserved: the deploy job performs exactly one make sam-build, then verifies the canonical frontend, runtime boundary, and isolated packaged handler against that same artifact before sam deploy; the checks job adds no SAM build, and local make ci retains one build followed by the same three gates.
• No gate rebuilds, reinstalls, copies a second artifact, or pins a digest, byte sequence, asset/test count, copied inventory, source line, or screenshot pixels. Dynamic comparison of the built artifact with its current manifest/source and the explicitly required single SAM build remain valid behavior contracts.
• Focused schema/collector/workflow tests, frontend unit coverage, backend tests/typecheck/build, the complete independent E2E suite, and one local make ci pass. Tester reports observed counts and the capability-state result inventory as evidence; no expected test/asset count is committed.
• No import, migration, restore, backfill, export-validation, raw-source data suite, AWS request, deployment, or production/private data operation is added or run for this issue.

Test Scenarios

Scenario: harmless browser test rename

Given: a passing browser journey records stable evidence for a catalog state
When: its human-readable title changes or the spec is moved during #180's split
Then: the state remains covered because the executed evidence ID, route, and role are unchanged

Scenario: claimed journey does not pass

Given: a required state is recorded by one journey
When: that journey fails, is skipped, is interrupted, or does not run
Then: the collector fails the independent browser run and names the uncovered state and non-passing journey

Scenario: role or route mismatch

Given: a state is restricted to an operator, admin, signed-out, expired, or disabled role and belongs to one canonical route
When: evidence reports another role or route
Then: the runtime contract rejects it even if the browser test itself otherwise passes

Scenario: behavior unchanged after implementation rename

Given: the frontend still serves and performs the same observable behavior
When: a private renderer/helper, CSS class, or incidental sentence is renamed
Then: convergence and capability coverage remain green because no source marker is a contract

Scenario: canonical asset serving regresses

Given: the canonical frontend manifest declares an asset
When: the real handler omits it, serves the wrong content type, exposes traversal/retired namespace fallback, or lacks its configured artifact
Then: the retained convergence behavior test fails with the requested route; no marker grep is needed

Scenario: one independent browser run supplies coverage

Given: #180's complete independently runnable Playwright suite
When: npm --prefix backend run test:e2e finishes
Then: the same run proves all required capability states or fails with the missing stable IDs, and no second discovery/browser run occurs

Scenario: deployment gates remain isolated

Given: the browser coverage machinery is installed
When: the workflow and local command graph are inspected and make ci runs once
Then: Playwright remains only in the independent E2E workflow and one SAM artifact flows through the three existing package/runtime gates without rebuild

Verification

Engineer may use focused Node/frontend/browser cases while iterating. Before Tester acceptance, record exact commands, exit codes, observed test counts, and the final stable state-evidence inventory:

node --test backend/scripts/frontend-capabilities.test.mjs
npm run test:frontend:coverage
npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
npm --prefix backend run test:e2e
make ci
git diff --check

make ci is the one SAM build for this verification and must reuse that exact artifact for all three gates. Do not run a separate make sam-build, a second make ci, or a Playwright command from the deploy workflow. UI screenshots are not required for #184 because it changes test plumbing only; #180 owns browser screenshots for its product/accessibility changes.

Out of Scope

• Product, copy, visual, route, role, API, Card/Task, assistant-lifecycle, or accessibility behavior changes. A real behavior defect discovered here is filed separately or returned to Restore behavior-based browser coverage and fix canonical UI accessibility races #180/Make assistant output review safe and lifecycle-complete #158 as applicable.
• Reworking Run the packaged frontend artifact isolation contract in CI #178's dynamic canonical-frontend verifier, Isolate SAM dependency installation from the shared developer workspace #185's isolated/fingerprinted SAM builder, the runtime-boundary verifier, or the packaged-handler isolation behavior.
• Reintroducing exact allowlists, hashes, copied golden artifacts, source/test-title pointers, test-count pins, screenshot goldens, or source-code coverage by grep.
• Adding Playwright to deploy checks, making Playwright deploy-blocking, or running a second E2E/discovery pass only to calculate coverage.
• Import, migration, restore, backfill, export-validation, raw-source data tests, AWS/deployment activity, production writes, private-repository reads, external provider calls, or source-repository changes.

Dependencies

• Restore behavior-based browser coverage and fix canonical UI accessibility races #180 owns the overlapping canonical-capability-behavior.spec.js and canonical-route-parity.spec.js restructuring and must complete local implementation/review first. Replace frontend source greps with runtime behavior coverage #184 consumes those final journey boundaries and must not recreate serial mega-specs or overwrite its UI/accessibility fixes.
• Run the packaged frontend artifact isolation contract in CI #178 is closed and deployed. Isolate SAM dependency installation from the shared developer workspace #185 is Tester/PM accepted and locally committed at 247ea7c. Their exact-artifact, isolated-runtime, one-build, and immutable-workspace contracts are prerequisites to preserve, not scope to redesign.
• Make assistant output review safe and lifecycle-complete #158 remains the owner of assistant lifecycle mutations. Deleting the old assistant source-marker grep does not claim that lifecycle is covered by Replace frontend source greps with runtime behavior coverage #184.

[alexeygrigorev]

PM GROOMED — BLOCKED ON #180

#184 is now an implementation-ready cleanup of the remaining false coverage seams.

The contract now requires:

• deleting the lower Home/Knowledge/Operations/Artifacts/runtime-template marker greps while retaining data-driven real-handler serving, traversal, retired-namespace, and missing-artifact behavior;
• replacing Playwright file/title/body parsing with stable route/role/state evidence emitted only by passed tests in the existing normal browser run;
• keeping static catalog and workflow-placement checks browser-free, while runtime coverage completeness runs only in the independent E2E workflow;
• preserving the closed/deployed Run the packaged frontend artifact isolation contract in CI #178 package gates and the accepted/local Isolate SAM dependency installation from the shared developer workspace #185 isolated one-build contract: one SAM artifact, three gates, no rebuild/install;
• no byte/digest/asset/test-count/copy pins and no import/migration/restore/backfill/export-validation or AWS/data operations.

Status is blocked because #180 owns and is actively restructuring the exact two large browser specs that carry most current capability evidence. #184 should instrument the final #180 journey boundaries after that work passes review, not create merge conflicts or restore serial mega-specs. #178 is already deployed; #185 is accepted and committed locally at 247ea7c and is a preservation gate, not redesign scope.

Labels are now bug, frontend, backend, testing, P1; needs grooming was removed. Next owner after #180: Software Engineer.

[alexeygrigorev]

SWE handoff — implementation ready for independent Tester

Implemented the groomed runtime-evidence contract without file/title/source identity and without changing deploy or infrastructure workflows.

Observable behavior now in the frozen 13-path diff:

• frontend-capabilities.json is the durable route/role/state catalog. All 138 states use stable IDs and runtime evidence; it carries no spec filename, journey title, selector, or source-location mapping.
• Passing Playwright journeys emit explicit { stateId, route, roleId } annotations only after their browser assertions. Renaming/moving a journey cannot change the state identity.
• The same Playwright invocation collects final test results. Failed, timed-out, skipped, interrupted, or absent journeys cannot satisfy a state.
• State-local roleIds are all-required. A state inheriting capability roles needs one passing allowed role. Focused synthetic tests cover both policies and missing-role diagnostics.
• The independent E2E command attaches the reporter once. There is no discovery/list/source-scan or second Playwright run. Deploy remains Playwright-free; the existing exactly-one-SAM-build placement is statically preserved.
• The old frontend source-text greps and copied asset inventory are gone. Backend convergence now drives the real handler over the current manifest, MIME boundaries, retired namespace, traversal/undeclared paths, and explicit missing-artifact failure.

Owned paths:

• backend/e2e/canonical-capability-behavior.spec.js
• backend/e2e/canonical-route-parity.spec.js
• backend/e2e/card-template-updates.spec.js
• backend/e2e/runtime-template-readonly.spec.js
• backend/e2e/frontend-capabilities.json
• backend/e2e/helpers/capability-evidence.js
• backend/e2e/capability-evidence.js
• backend/e2e/capability-evidence-reporter.js
• backend/scripts/frontend-capabilities.test.mjs
• backend/scripts/frontend-capability-evidence.test.mjs
• backend/tests/frontend-convergence.test.ts
• backend/playwright.config.js
• backend/package.json

Verification:

• cd backend && node --test scripts/frontend-capabilities.test.mjs scripts/frontend-capability-evidence.test.mjs — 15/15 pass.
• Focused tests/frontend-convergence.test.ts with the normal test environment — 3/3 pass.
• cd backend && npx playwright test e2e/runtime-template-readonly.spec.js e2e/card-template-updates.spec.js --workers=1 — 4/4 pass.
• cd backend && npm run test:e2e — one Playwright invocation, 63/63 pass; attached reporter: 138 stable states, 143 passing role claims, 0 uncovered, 0 errors.
• cd backend && npm run typecheck — pass.
• cd backend && npm run build — pass, including runtime boundary and canonical frontend artifact verification.
• npm run test:frontend:coverage from repository root — 192/192 pass.
• node --check for the changed/new JavaScript collector, reporter, helper, config, and instrumented specs — pass.
• git diff --check — pass.

No import, migration, restore, raw-data movement, AWS, SAM, deploy, or second Playwright run was executed. No workflow or aws-infra path changed. There is no full-E2E readiness blocker. The diff is uncommitted pending independent Tester and PM acceptance.