Restore render-impact screenshot evidence in normal CI

[alexeygrigorev]

Parent epic: #11
Related accepted workflow baselines: #104, #113, #143
Observed on exact main SHA: 4b8c852681c0ace77f0da94b7f7c38e807c42984

Outcome

Normal push CI must produce the required screenshot evidence whenever the verification planner marks a change as render-impacting, so the aggregate gate can validate a release instead of rejecting an otherwise green run and skipping publish/deploy.

Confirmed failure

Run: https://github.com/DataTalksClub/website/actions/runs/31691360510

• resolve-release, classification, quality, Django, Playwright, and container all passed.
• ci-gate.json is correctly fail-closed with verdict=failure.
• The plan marked render.impact=true because Implement confirmed design-audit Phase 1 quality fixes #144 changed courses/templates/homework/stats.html and course CSS.
• The verification report has screenshots in the skipped bucket with reason=required_result_missing.
• .github/workflows/ci.yml has no screenshot component/result for render-impact pushes.
• Publish/deploy were therefore skipped; this is a deterministic workflow/evidence defect, not a product test failure.

Scope

• Add the normal-CI screenshot component/job or otherwise wire the existing screenshot capture/evidence contract into the push workflow for render-impact selections.
• Use the planner's exact screenshot requirements and record digest-bound evidence through the existing ci.verification contract.
• Keep screenshots synthetic and free of credentials/PII; preserve the fail-closed aggregate gate.
• Add workflow/contract tests proving render-impact changes cannot finish without a valid screenshot result, while non-render changes remain allowed to use not_applicable.
• Verify the accepted Fix deterministic Scheduled full regression workflow failures #143 scheduled workflow contract remains unchanged.

Non-goals

• Do not weaken ci.gate.normal, allow pending screenshot evidence, or mark render-impact screenshots not applicable.
• Do not change application behavior, templates, CSS, deployment, or product routes.
• Do not delete screenshot assertions or bypass evidence validation.

Acceptance criteria

• A normal push with render-impacting template/static changes runs the screenshot component using the planner-authorized command/profile and records a valid digest-bound result.
• The aggregate gate accepts a complete render-impact run only when screenshot evidence is present, valid, and tied to the exact source/plan/environment; missing or malformed evidence still fails closed.
• A non-render push retains the existing screenshots: not_applicable behavior.
• Workflow tests cover job dependencies, exact command/profile binding, artifact upload, evidence recording, and fail-closed missing-result behavior.
• Synthetic desktop/mobile captures are inspected for the changed surface; no credentials, PII, debug page, overflow, or minified template output appears.
• make test-ci, focused Django/Playwright checks, git diff --check, and the exact normal push workflow complete successfully on the candidate.
• A post-merge normal CI run at the exact merged SHA is terminal green and reaches publish/deploy gates.

Required process

Follow _docs/PROCESS.md: groomed issue, engineer frozen uncommitted handoff, independent tester PASS, PM acceptance, local merge/push, then on-call observes the exact push run. Do not manually rerun the failed SHA as proof.

[alexeygrigorev]

Grooming Complete

Verdict: READY FOR ENGINEERING — deterministic normal-CI evidence gap confirmed.

The exact 4b8c852 push run has six successful component results but no screenshot result even though the planner classified the change as render-impacting. The aggregate gate correctly rejected the incomplete report. This issue owns only the normal push workflow/evidence wiring and its contract tests; it must not weaken the gate or alter product files. The accepted scheduled workflow (#143) is a separate contract and remains unchanged.

[alexeygrigorev]

Engineer handoff available

The #146 engineer reports an uncommitted candidate in .tmp/website-146-engineer with normal-CI screenshot capture/evidence wiring, workflow/gate contract coverage, and no scheduled-workflow changes. Reported checks: make test-ci 505 passed, targeted CI tests 77 passed, lint/typecheck passed, core Playwright 154 passed / 27 deselected, and git diff --check passed. make test-ci-focused correctly requires a planner artifact and remains a controller-level check.

This remains pending independent tester and PM gates; no commit or push has been made.

[alexeygrigorev]

Software Engineer Remediation Report

Issue: #146 — Restore render-impact screenshot evidence in normal CI

Frozen candidate

• Worktree: /home/alexey/git/dtc-website/.tmp/website-146-engineer
• Branch: issue-146-engineer
• Candidate HEAD/base: 4b8c852681c0ace77f0da94b7f7c38e807c42984
• Current origin/main: 31c5447f4686c2ff87707b137b97ef85a332931a (the candidate is intentionally based before the Add controlled upstream synchronization from CMP main #145 merge; the tracked candidate patch applies cleanly to current main in a scratch checkout)
• No commit or push was made.

Remediation

The independent tester found one stale contract assertion after the screenshot job was added to normal-CI gate dependencies:

make test-core  # before remediation: exit 2
core.tests.test_deployment_workflow.DeploymentWorkflowContractTests.test_one_image_is_built_tested_and_published_only_after_all_gates
AssertionError: 'needs: [resolve-release, quality, django, playwright, container]' not found

I updated only the exact deployment contract assertion in core/tests/test_deployment_workflow.py to require screenshots in the ci-gate needs list. Because that file is hash-bound by the existing terminology compatibility policy, I updated its single recorded SHA-256 in _docs/compatibility/development-terminology-allowlist.json to the resulting digest 3a6dc74288810a949cede0f59d0e8fda9326bec67d4e7642b6fbef0c0be3f720. No application, template, CSS, route, or deployment behavior was changed.

Verification after remediation

• make test-core — 790 tests passed (exit 0).
• uv run --frozen pytest core/tests/test_deployment_workflow.py -q — 119 passed, 303 subtests passed.
• make test-ci — 505 passed.
• uv run --frozen pytest ci/tests/test_workflows.py ci/tests/test_gate.py tests_ci/test_verification.py tests_ci/test_evidence.py tests_ci/test_runner.py -q — 82 passed.
• make format-check — 489 files already formatted.
• make lint — passed.
• make typecheck — no issues in 365 source files.
• make migrations-check — no changes detected.
• git diff --check — passed.

The independent tester's prior make test-playwright-core also completed 154 passed, 27 deselected. Focused screenshot/evidence tests completed 7 passed. The candidate's fresh tester plan remains full profile with render.impact=false, no required captures, and screenshots=not_applicable (render_inputs_unchanged_without_reusable_visual); the change is CI/evidence/contract tooling only. Render-impact plans still require screenshot rerun evidence, and the missing-evidence test remains fail-closed.

The candidate is still uncommitted for the tester/PM lifecycle gates.

[alexeygrigorev]

TEST Review — remediation

Verdict: PASS — remediation is independently verified.

The initial #146 candidate did not pass the tester gate: make test-core failed because the deployment contract test still expected the pre-#146 ci-gate dependency list without screenshots. This re-review covers the engineer's uncommitted remediation; I did not commit or push.

Frozen candidate and scope

• Worktree: /home/alexey/git/dtc-website/.tmp/website-146-engineer
• Branch/HEAD: issue-146-engineer / 4b8c852681c0ace77f0da94b7f7c38e807c42984
• Tester plan: .tmp/verification-tester-146/verification-plan.json
• Plan: full, reason test_infrastructure, browser profile core
• Render decision: impact=false, no required captures; screenshots=not_applicable with reason render_inputs_unchanged_without_reusable_visual
• The remediation adds only the exact expected screenshots dependency assertion in core/tests/test_deployment_workflow.py and its required hash-bound metadata update in _docs/compatibility/development-terminology-allowlist.json. No application, template, CSS, route, scheduled-workflow, or deployment behavior files were added/changed beyond the existing Restore render-impact screenshot evidence in normal CI #146 candidate scope.
• Candidate status remains the expected Restore render-impact screenshot evidence in normal CI #146 uncommitted change set; git diff --check passed.

Re-run evidence

• make test-core — 790 tests passed, exit 0.
• uv run --frozen pytest core/tests/test_deployment_workflow.py -q — 119 passed, 303 subtests passed.
• make test-ci — 505 passed, exit 0.
• uv run --frozen pytest ci/tests/test_workflows.py ci/tests/test_gate.py tests_ci/test_verification.py tests_ci/test_evidence.py tests_ci/test_runner.py -q — 82 passed.
• Screenshot/evidence-focused tests: uv run --frozen pytest tests_ci -k 'screenshot' -q — 7 passed, 92 deselected.
• make format-check — 489 files already formatted.
• make lint — passed.
• make typecheck — no issues in 365 source files.
• make migrations-check — no changes detected.
• git diff --check — passed.
• Existing make test-playwright-core from the first independent review — 154 passed, 27 deselected.

The fresh tester plan remains non-rendering for this CI/evidence-only change, so screenshots are correctly not applicable. I also invoked ci.screenshot_capture against that plan and it failed closed with RuntimeError: screenshot capture requires a rerun disposition; render-impact plans still require a rerun disposition and the focused missing-result test remains fail-closed. The terminology policy SHA now exactly matches the changed deployment contract test (3a6dc74288810a949cede0f59d0e8fda9326bec67d4e7642b6fbef0c0be3f720).

Independent tester gate now passes. Candidate remains uncommitted for PM acceptance.

[alexeygrigorev]

Product Manager Acceptance

Verdict: ACCEPT — the #146 candidate satisfies the pre-merge contract and is ready for the focused commit gate (Closes #146).

I reviewed the groomed issue and its linked verification contract, the frozen candidate at /home/alexey/git/dtc-website/.tmp/website-146-engineer (issue-146-engineer, base/head 4b8c852681c0ace77f0da94b7f7c38e807c42984), the complete uncommitted diff, and the independent tester PASS (remediation review). I did not edit, commit, push, dispatch, or manually rerun the failed historical SHA.

Acceptance mapping

• Normal push screenshot component: .github/workflows/ci.yml now exports the planner's screenshots_mode, runs a dedicated screenshots job for rerun plans, restores the exact reviewed plan, provisions locked release/controller environments, installs the pinned Chromium browser, migrates the synthetic SQLite database, starts the selected release locally, and invokes ci.screenshot_capture for every planner-authorized route/viewport. The job records the actual environment and digest-bound result through ci.verification record, uploads the PNG/manifest/result artifacts, and is trusted as the screenshots CI component.
• Fail-closed aggregate gate: screenshots is included in ci.gate.NORMAL_REQUIRED_JOBS and in the ci-gate dependency list/result inputs. The gate still validates the complete plan-bound report and requires every normalized job result to be success; no pending or missing screenshot result can pass. The new test_ci_render_plan_fails_closed_without_screenshot_evidence regression proves a render-impact plan reports required_result_missing, and the tester also verified the capture command rejects a non-rerun plan.
• Non-render behavior: The shared planner continues to mark screenshots not_applicable with render_inputs_unchanged_without_reusable_visual; the workflow's explicit no-op branch succeeds for that disposition. The fresh tester plan for this CI-only candidate has render.impact=false, no required captures, and screenshots=not_applicable.
• Workflow/evidence contracts: Focused tests assert classifier output wiring, screenshot job dependencies, exact planner command binding, Chromium/install/migration/capture invocation, artifact naming/upload, evidence recording, aggregate dependencies, and manual probe exclusion. The scheduled workflow remains untouched and its existing contract is still covered.
• Visual/synthetic safety: This candidate changes no templates, static assets, routes, or product behavior, so the planner correctly makes changed-surface screenshots not applicable. The new capture implementation nevertheless enforces planner route/viewport identity, expected HTTP status (including the synthetic 404), non-empty body, debug-marker rejection, horizontal-overflow rejection, independent inspection, and render/image SHA-256 binding for render-impact runs. No credentials, PII, or production data are introduced.
• Verification: The independent tester reported make test-core 790 passed; make test-ci 505 passed; targeted CI/evidence tests 82 passed; screenshot-focused tests 7 passed; core Playwright 154 passed / 27 deselected; format, lint, typecheck, migrations, and git diff --check passed. My read-only rerun independently passed the focused workflow/verification suite (34 passed) and make test-core (790 tests, 0 failures). The remediation's single deployment-contract assertion and its required terminology allowlist SHA agree exactly (3a6dc74288810a949cede0f59d0e8fda9326bec67d4e7642b6fbef0c0be3f720).

The candidate is limited to the normal CI/evidence contract (.github/workflows/ci.yml, ci/evidence.py, ci/gate.py, ci/verification.py, ci/screenshot_capture.py, focused tests, and the hash-bound deployment assertion/metadata). No application, template, CSS, route, scheduled-workflow, or deployment behavior change is included.

The required post-merge normal CI run at the exact merged SHA, including terminal publish/deploy gates, remains pending by design. The engineer may now create the focused commit; the orchestrator owns the local no-PR merge/push, and on-call owns observing that exact pushed run. Keep #146 open until that operational gate is terminal green; this acceptance does not substitute local evidence for the post-merge run.

[alexeygrigorev]

On-call report — post-merge normal CI

Verdict: GREEN — the exact pushed SHA completed CI, aggregate validation, publish, and deploy successfully.

• Commit: bce6ec9e399262a1bdc3741cc1758697a97c2a36
• Event/attempt: push, attempt 1
• Run: CI #31718697834
• Started: 2026-08-13 16:03:18Z
• Terminal success: 2026-08-13 16:29:37Z
• No rerun, manual dispatch, failure dismissal, or on-call code change was made.

Component evidence

All required non-probe components passed on the exact SHA:

• resolve-release — success (20s)
• classification — success (28s)
• quality — success (1m34s)
• django — success (6m11s)
• playwright — success (10m58s)
• container — success (1m09s)
• screenshots — success (21s)

The screenshot component restored the exact same-run selection artifact and executed its planner disposition. For this SHA the planner reported no render-impact changes, so the job explicitly ran No render-impact changes; screenshot evidence is not applicable. It preserved the successful screenshot-component result; no capture files were required for this non-render disposition. The screenshot job therefore did not silently skip a required render-impact capture.

The five manual-probe jobs (probe-contract, probe-deployer, probe-wrong-environment-claim, probe-wrong-main-claims, probe-publisher) were intentionally skipped by the normal push path; none was a required release check.

Aggregate gate

• ci-gate — success (26s)
• It restored fresh component evidence, built the exhaustive verification report, and required every non-probe release check.
• The gate received resolve-release=success, classification=success, quality=success, django=success, playwright=success, container=success, and screenshots=success, bound to release SHA bce6ec9e399262a1bdc3741cc1758697a97c2a36, run 31718697834, attempt 1, event push, and selection digest 0891201925dfbb602eeaed9fe92115d00865234a09727ea47487099f18b1ea0c.
• Aggregate evidence was preserved as artifact verification-evidence-31718697834-attempt-1 (artifact 9188769206).

Release gates

• auto-capture-prior — success (33s); exact managed active web/worker pair captured.
• publish — success (1m39s); publisher configuration/OIDC checks passed, the image was pushed once, its immutable digest was verified, and the published-image record was preserved.
• deploy — success (11m24s); deployer configuration/OIDC and pre-mutation checkpoint checks passed, the atomic web-and-worker promotion completed, and redacted deployment plus read-only browser/HTTP evidence were preserved.

The workflow is terminal green and reached both publish and deploy as required by this issue. GitHub emitted non-blocking Node.js 20 deprecation annotations for third-party actions forced onto Node.js 24; no job failed or was cancelled.