Fix deterministic Scheduled full regression workflow failures

[alexeygrigorev]

Outcome

Restore the dedicated Scheduled full regression workflow so a selected run can complete every required component, retain valid digest-bound evidence, and become a trustworthy coverage anchor again. The correction is workflow/test-infrastructure only; product behavior and the deployed runtime remain unchanged.

Authority and exact baseline

• _docs/PROCESS.md: role-separated engineering, independent QA, product acceptance, merge, and on-call gates.
• _docs/specs/10-verification-strategy.md: ordinary-CI SQLite, core Playwright, complete scheduled coverage, and fail-closed release evidence.
• _docs/ci/change-selective-ci.md: shared planner/evidence contract, concrete runner identity, four-bucket reports, scheduled anchors, and least-privilege scheduled authority.
• Accepted source-selection/scheduler baseline: Run change-selective CI with four-hour full regression #104.
• Accepted ownership/evidence baseline: Reuse digest-bound QA evidence without weakening coverage #113.
• Current exact main and issue base: c86120e673b80068df541e95a8965531915bacea (Allow same-workflow hosted runner image drift).

The accepted planner is authoritative. For an empty scheduled diff at this base it selects profile=full, browser_profile=core, Django command make test, and Playwright command make test-playwright-core. The current drift API intentionally permits a fresh GitHub Actions component to record a newer patch revision of the same hosted-runner image family while retaining the component's actual concrete image in its envelope; different families, local runners, or any other environment change still fail closed.

Confirmed failing boundary

• Latest scheduled run: https://github.com/DataTalksClub/website/actions/runs/31662645190 — terminal failure at exact SHA c86120e673b80068df541e95a8965531915bacea.
• Same-SHA push run: https://github.com/DataTalksClub/website/actions/runs/31655624461 — terminal success, including quality, Django, Playwright, container, aggregate gate, publish, and deploy.
• The scheduled selector chose run_full with reason=retry_after_failure; its retained plan and selector envelope are valid and record one assertion.
• Successful scheduled components: container (12 retained assertions), migrations (16 tests), and factories (139 tests).
• full-regression and scheduled-gate failed only because required upstream components were non-successful. Their fail-closed behavior is correct and must be preserved.

Three independent deterministic workflow defects are confirmed:

1. Runner drift was not propagated to the scheduled workflow. The selector planned ubuntu24@20260720.247.2; the quality job ran on ubuntu24@20260810.271.1. The complete quality command passed, including 502 CI-policy tests, but environment capture failed with VerificationError: executing component environment does not match the authorized plan. The same-workflow drift flags added to .github/workflows/ci.yml at c86120e are absent from the corresponding scheduled environment-capture and evidence-recording calls.
2. Scheduled component checkouts are shallow. Compatibility passed 304 tests and Django discovered 1,759 tests, then content.tests.test_event_description_bridge.PublicEventDescriptionTests.test_projection_preserves_baseline_identity_and_empty_rollback_is_luma_free failed because scripts.build_event_description_bridge._load_projection_events() could not read 4cad269d576217679ac6c9ce02286e7939d8b043:content/public_projection/events.json. The selector checkout has full history, but the Django checkout defaults to depth 1. Parallel-test teardown then emitted the secondary TypeError: cannot pickle 'traceback' object; it is not a separate root cause.
3. Scheduled Playwright bypasses the plan's exact command binding. The workflow ran make test-playwright; 177 tests passed with 3 deselected. Recording then failed with ci.evidence.EvidenceError: evidence command does not match the verification plan because the retained plan authorizes make test-playwright-core.

The green same-SHA push run and successful raw component commands rule out a product/runtime regression, production-data problem, GitHub outage, or flaky-test explanation. Rerunning this unchanged workflow is not a correction: the shallow history and Playwright command mismatch are deterministic, and runner allocation could only mask the first defect.

Scope

Expected change surface is deliberately limited to:

• .github/workflows/scheduled-full-regression.yml;
• ci/tests/test_workflows.py.

Implement all of the following:

• Pass --allow-hosted-runner-drift symmetrically to every scheduled fresh evidence-producing component's ci.verification environment and ci.verification record invocation: selector, quality, evidence validation, Django, Playwright, and container. Keep the actual concrete component environment in each envelope.
• Make every actions/checkout@v4 step in the scheduled workflow check out exact ${{ github.sha }} with fetch-depth: 0. This includes component and aggregate/gate jobs, so history-dependent verification cannot vary by job checkout depth.
• Run and record exactly make test-playwright-core, matching the current shared plan. Do not change planner policy to authorize the broader suite for this repair.
• Extend workflow-contract coverage so these three properties cannot drift independently again, while retaining the existing fail-closed gate, evidence, permissions, concurrency, and no-deployment assertions.

If implementation requires a change to ci/evidence.py, ci/verification.py, ci/ownership.json, planner policy, application code, dependencies, or the Make target contracts, return to PM before expanding scope: current main already supplies the required APIs and commands.

Non-goals

• No Django application, model, migration, template, static asset, public page, Studio/admin API, or deployed-runtime behavior change.
• No change to the ownership graph, evidence schema, environment-matching semantics, validity/reuse policy, four-bucket report, or accepted Run change-selective CI with four-hour full regression #104/Reuse digest-bound QA evidence without weakening coverage #113 selection policy.
• No test deletion, quarantine, expectation weakening, or change to the Ship reviewed event descriptions without public Luma CTAs #131 accepted-baseline/rollback contract.
• No change to cron, concurrency, history-anchor selection, artifact retention, normal push CI, publish/deploy/probe behavior, AWS authority, secrets, or production data.
• No new scheduled workflow_dispatch trigger and no rerun of the known-broken SHA as evidence of a fix.
• No product screenshots: this internal workflow/test-infrastructure change has render_impact=false; screenshots are not_applicable.

Dependencies

No open dependency blocks implementation. #104 and #113 are accepted baselines already present on main; the same-family hosted-runner drift API is present at c86120e. #131 supplies the existing history-dependent regression test that exposed the shallow checkout, but no #131 behavior changes in this issue.

Acceptance criteria

• Every scheduled ci.verification environment and matching ci.verification record call for selector, quality, evidence validation, Django, Playwright, and container explicitly opts into same-workflow hosted-runner drift; contract tests fail if either half is missing.
• Fresh same-family hosted-runner patch drift can produce a valid envelope containing the component's actual concrete runner image, while a different runner family, local runner, or any other environment difference remains rejected by the existing evidence contract.
• Every scheduled checkout is pinned to exact ${{ github.sha }} with fetch-depth: 0; the Django job can resolve commit 4cad269d576217679ac6c9ce02286e7939d8b043 and content/public_projection/events.json from that commit.
• Scheduled Playwright executes and records byte-for-byte make test-playwright-core, equal to the shared planner-authorized command for browser_profile=core; the workflow does not authorize itself by changing planner policy.
• A selected scheduled run still requires successful selector, quality, factories, migrations, Django, Playwright, container, full-regression, and scheduled-gate outcomes. Missing, malformed, mismatched, or failed component evidence cannot create an aggregate envelope or coverage anchor.
• The scheduled workflow retains contents: read plus actions: read only, exact cron/queued concurrency, and no environment, service, id-token, AWS, publish, deploy, probe, production-data, or repository-write authority.
• Focused workflow/evidence contract tests and the graph-selected full local verification pass. The candidate changes only the scheduled workflow and CI contract tests, with screenshots explicitly not_applicable.
• After merge, normal push CI is terminal green and one fresh naturally scheduled run at that exact merged main SHA is terminal green with valid retained component and aggregate evidence. A manual-equivalent check may supplement local verification but does not replace this exact scheduled result or become an anchor.

Required BDD and contract scenarios

1. Hosted runner patch drift: Given a plan produced on ubuntu24@A and a fresh scheduled component assigned ubuntu24@B, when that component captures and records its environment, then the scheduled invocation explicitly allows same-family patch drift and retains ubuntu24@B; when family/locality or another dimension differs, the existing validator rejects it.
2. Symmetric invocation coverage: Given each scheduled evidence-producing job, when workflow YAML is parsed, then every environment-capture call and every matching record call contains the explicit drift flag. Omitting the flag from either side fails make test-ci.
3. History-dependent Django: Given exact scheduled SHA c86120e... and the Ship reviewed event descriptions without public Luma CTAs #131 baseline revision, when the scheduled Django checkout is established, then both the exact head and 4cad269d...:content/public_projection/events.json are resolvable before make test; all other scheduled checkouts use the same full-history exact-SHA contract.
4. Exact browser command: Given the retained plan has browser_profile=core, when the scheduled Playwright job runs and records evidence, then both command strings are exactly make test-playwright-core. A workflow/plan or execution/recording mismatch fails the contract test and evidence recording.
5. Fail-closed aggregation: Given any required job failure, missing artifact, malformed envelope, command/environment mismatch, or incomplete component set, when full-regression and scheduled-gate evaluate the run, then both fail and no scheduled state anchor is retained. Given all valid components, both may succeed.
6. Least privilege and product preservation: Given the corrected workflow, when its contract is inspected, then cron, queueing, read-only permissions, absence of deployment/cloud authority, and exact component dependencies are unchanged; the git diff contains no product/runtime/template/model/migration/dependency change.

Verification and handoff

Engineering must follow the versioned plan/report workflow from _docs/PROCESS.md, including exact base/head, graph and plan digests, four buckets, commands/counts, and frozen uncommitted handoff. At minimum, run the focused workflow/evidence tests through make test-ci; the graph-derived full verification remains authoritative for this .github/**/test-infrastructure change.

Independent QA must recompute the plan from the frozen base/head, inspect the complete diff, run every planned rerun, and directly verify all acceptance criteria. Screenshots must be reported as not_applicable, not pending or silently omitted.

After tester PASS and product acceptance, commit with Refs #143 and deliver through the repository's local merge process. On-call first observes the exact push CI run. Keep #143 open until a later naturally scheduled run executes at the exact merged SHA and the selector, quality, factories, migrations, Django, Playwright, container, full-regression, and scheduled-gate jobs are all successful. Inspect the retained envelopes/report for exact source SHA, commands, concrete environment identities, counts, artifact digests, and a successful scheduled state. The scheduled workflow must not publish or deploy. Close #143 only after that automated operational gate passes; no human label is required.

[alexeygrigorev]

Grooming Complete

Verdict: READY FOR ENGINEERING — three deterministic scheduled-workflow contract gaps, no product/runtime defect.

I groomed this from isolated detached worktree /home/alexey/git/dtc-website/.tmp/pm-issue-143 at exact current origin/main c86120e673b80068df541e95a8965531915bacea; the existing dirty /home/alexey/git/dtc-website branch was not touched.

The latest scheduled run 31662645190, retained artifacts, exact job logs, same-SHA green push run 31655624461, accepted #104/#113 contracts, current planner/recorder, #131 baseline loader, workflow tests, and repository history confirm three independent causes: missing scheduled drift flags, shallow scheduled component checkouts, and make test-playwright versus the plan-authorized make test-playwright-core. The aggregate failures are correct downstream denials, not fourth defects.

Scope is limited to .github/workflows/scheduled-full-regression.yml and ci/tests/test_workflows.py: apply drift opt-in symmetrically to capture/record calls, make every scheduled exact-SHA checkout full-history, align Playwright execution/recording with the planner, and lock all three plus fail-closed/least-privilege behavior in contract tests. No open dependency blocks work; planner/evidence/product/runtime changes require renewed grooming. Screenshots are not_applicable.

needs grooming is removed; bug, P0, infra, operations, and testing remain. After independent QA and PM acceptance, commit with Refs #143; the issue remains open until normal push CI and a fresh natural scheduled run at the exact merged SHA both pass with valid retained evidence. No implementation, tests, commit, push, dispatch, or rerun was performed during grooming.

[alexeygrigorev]

Software Engineer Report

Verdict: PASS — implementation complete and frozen uncommitted for independent tester review.

Identity and scope

• Worktree: /home/alexey/git/dtc-website/.tmp/website-143-engineer
• Branch: issue-143-engineer
• Reviewed base / committed HEAD: c86120e673b80068df541e95a8965531915bacea
• Current origin/main: c86120e673b80068df541e95a8965531915bacea
• Candidate source mode: committed HEAD plus the uncommitted two-file worktree diff
• Changed files only:
  • .github/workflows/scheduled-full-regression.yml
  • ci/tests/test_workflows.py
• No commit, push, dispatch, rerun, product/runtime/planner/recorder/graph/dependency change, or production-data access was performed.

Implemented behavior

• Every scheduled fresh ci.verification environment and matching record block now passes --allow-hosted-runner-drift: selector; the quality/evidence-validation loop; Django; Playwright; and container.
• Every one of the nine scheduled actions/checkout@v4 steps now pins ref: ${{ github.sha }} and fetch-depth: 0, including selector, all components, aggregate, and gate.
• Scheduled Playwright now executes and records the planner-owned exact command make test-playwright-core.
• Contract tests parse logical CLI invocations so a missing drift flag on either capture or record fails independently; require exact full-history checkout settings for every job; and compare both Playwright command sites to ci/ownership.json without accepting the broader command as a substring.
• Existing cron, queued concurrency, read-only permissions, component dependencies, fail-closed aggregate/gate, artifact handling, and absence of deployment/AWS authority remain unchanged.
• The historical object and file resolve locally: 4cad269d576217679ac6c9ce02286e7939d8b043^{commit} and 4cad269d576217679ac6c9ce02286e7939d8b043:content/public_projection/events.json.

Frozen verification plan

• Profile: full; browser profile: core
• Direct nodes: surface.ci, surface.github; downstream nodes: none
• Risk flags: deployment_runtime, test_infrastructure
• Render impact: false; screenshots: not_applicable
• Policy version: 4
• Graph SHA-256: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19
• Report-bound plan SHA-256: ed236ffe9abb9e2755b0c82a2d3e1aa8d40d90f519525419bf6f5429ebadbb24
• Retained plan-file SHA-256: e71b49081be20f1499cb800b7e60ae21db5d605206f07fb10ea9ec83085c8b8a
• Verification-state SHA-256: dd1def293ecaf4483725427e2231b0ca55b63686af00f2cd30959a4f49cee799
• Source-manifest SHA-256: b16c27b4874ddab670e035adc5fdaaedc909b238e69f49c88d04c531e712f997
• Report-file SHA-256: 90ec34fc3debe0e1e758d0f4970a4ec002d9591456a87b3c51f1cefaf42fa1fb

Four buckets and evidence

• rerun:
  • selector — 1 assertion; evidence ec02d887d4e192fbeacb341473abcfa5ec7640d9bf649bb9cf587fd110f9d56a
  • quality (quality-contract-v1) — terminology, portability, Ruff lint/format, mypy (361 files), migration drift, Django/deploy checks, and 504 CI tests; evidence 166b57c4267b41cc13d8bc0f170c77ddc2e991b23527b900200fedbec6b78120
  • evidence validation (make test-ci) — 504 passed; evidence c356e4619482132ff11f5ab8b904f7526682d00eaf80bc027f6bdb606472acdb
  • Django (make test) — 2,063 parsed tests/assertions: 2,055 passed, 8 skipped (304 compatibility plus 1,759 Django); evidence 2f041af67ab900475da202bd7f9550fcce4632865d117f6d66a8f5625d2dd3a0
  • Playwright (make test-playwright-core) — 153 passed, 27 collection deselections; evidence 3ee16b89434a7fe67f47ea61e3f440f034e39a2c486d77f778b35f8254b4c001
  • container (make verification-container) — 12 assertions; evidence f5407cc9a2da4de3218b2e03eb5cc480a8616d93160ac00fb70b3a85ea4bf6c8
• reused: none
• skipped: none
• not_applicable:
  • compatibility component — outside impact closure (the Django command still ran the compatibility prerequisite)
  • content invariants — outside large-content closure
  • screenshots — render inputs unchanged; no product-page impact

Exact commands and retained artifacts

• uv run --frozen pytest ci/tests/test_workflows.py -q → 12 passed
• make test-ci → 504 passed
• git cat-file -e 4cad269d576217679ac6c9ce02286e7939d8b043^{commit} → pass
• git show 4cad269d576217679ac6c9ce02286e7939d8b043:content/public_projection/events.json → pass
• make verification-plan VERIFY_ISSUE=143 VERIFY_WORKTREE=issue-143-engineer → pass
• make verification-run VERIFY_ISSUE=143 VERIFY_WORKTREE=issue-143-engineer → pass, including report/evidence validation
• Plan: .tmp/verification/verification-plan.json
• Report: .tmp/verification/verification-report.json
• Component environments, outputs, result records, and envelopes: .tmp/verification/evidence/

Remaining gates and limitations

• Independent tester recomputation/verification and PM acceptance are pending.
• The worktree is intentionally uncommitted and frozen; after both gates pass the required commit body is Refs #143.
• Normal push CI and a fresh natural scheduled run at the exact merged SHA are operational post-merge gates and remain unchecked. Only that later natural run can prove the hosted scheduled runner and retained coverage anchor end to end; no manual dispatch or broken-SHA rerun was used as substitute evidence.

[alexeygrigorev]

TESTER PASS — issue #143 (current frozen candidate, independently verified 2026-08-13)

The workflow-only candidate passes the independent tester gate. The diff remains limited exactly to .github/workflows/scheduled-full-regression.yml and ci/tests/test_workflows.py; no product/runtime/planner/recorder/dependency files changed, no commit/push/dispatch was performed, and screenshots are not_applicable because render impact is false.

Evidence:

• Recomputed tester plan: profile full, browser profile core, base/head c86120e673b80068df541e95a8965531915bacea, graph SHA-256 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19; report .tmp/verification-tester-143/verification-report.json has phase=tester, verdict=success, and invalid_evidence=[].
• Graph-selected reruns all passed: selector (1 assertion), quality (504), evidence validation / make test-ci (504), Django / make test (2,063: 2,055 passed, 8 skipped), Playwright / make test-playwright-core (153 passed, 27 deselected), and container (12 assertions).
• Focused workflow contract tests: ci/tests/test_workflows.py 12 passed. The quality contract also reran the complete CI test set (504 passed).
• Workflow inspection confirms all 9 scheduled checkouts pin exact ${{ github.sha }} with fetch-depth: 0; every scheduled verification environment/record pair carries --allow-hosted-runner-drift; Playwright execution and recording both use the planner-owned exact make test-playwright-core command.
• git diff --check passed; changed-file credential/private-key scan was clear.

The required post-merge operational gates (normal push CI and a later naturally scheduled run at the merged SHA) remain pending by design and are not substituted by local evidence.

[alexeygrigorev]

Product Acceptance

Verdict: ACCEPT — the scoped scheduled-workflow correction satisfies the groomed product contract.

I reviewed the groomed issue, the frozen uncommitted candidate at base/head c86120e673b80068df541e95a8965531915bacea, the exact two-file diff, and the current independent tester PASS: #143 (comment)

Acceptance mapping:

• Scheduled selector, quality/evidence-validation, Django, Playwright, and container evidence capture/recording now carry the hosted-runner drift opt-in symmetrically; the existing environment/evidence contract and fail-closed semantics remain in force.
• All nine scheduled checkout steps pin ${{ github.sha }} with fetch-depth: 0, preserving history-dependent verification; the candidate does not alter planner policy or application/runtime behavior.
• Playwright execution and recording both use the planner-owned exact make test-playwright-core command.
• The workflow retains its exact cron, queued concurrency, read-only contents/actions permissions, component dependencies, aggregate/gate checks, and absence of deployment, cloud, publish, probe, or repository-write authority.
• Independent QA recomputed the full/core plan and reported phase=tester, verdict=success, and invalid_evidence=[]: selector 1 assertion; quality 504; evidence validation / make test-ci 504; Django make test 2,063 (2,055 passed, 8 skipped); Playwright core 153 passed / 27 deselected; container 12 assertions; focused workflow tests 12 passed. Scope, diff-check, and credential/private-key scans were clear. Screenshots are correctly not_applicable for this render-impact-free infrastructure change.

The candidate changes only .github/workflows/scheduled-full-regression.yml and ci/tests/test_workflows.py; no product/runtime, planner, recorder, dependency, or production-data change is included. I authorize the engineer to create the focused uncommitted commit with Refs #143. The orchestrator owns local merge, push, and CI observation.

Normal push CI and one fresh naturally scheduled run at the exact merged main SHA remain mandatory post-merge operational gates. This acceptance does not authorize closing #143; keep it open until those retained scheduled envelopes and aggregate/gate outcomes are green.

[alexeygrigorev]

On-call — terminal PASS

Exact merged head 2a84fd860335bc2ef04a0559f284fe138ad3f7bd workflow 31671780961 completed success. Required quality, container, Django, Playwright, CI gate, publish, and deploy stages are green. The scheduled-workflow repair is verified on the exact merged SHA; no further source changes were made by on-call.