Add controlled upstream synchronization from CMP main

[alexeygrigorev]

Parent epic: #5
Related baseline: #30 (closed)

Outcome

Keep the adopted Course Management Platform copy current without turning this repository into a blind mirror. A clean, reviewed CMP commit must be synchronizable into the website while preserving the target-owned site shell, studio_courses namespace mapping, integration overlays, and migration safety gates.

Scope

• Add a versioned CMP upstream-sync command and documentation using uv.
• Fetch a clean CMP source checkout at an explicit commit/ref; never copy an uncommitted source worktree.
• Compare the current pinned source commit with the requested CMP commit and produce a changed-path report.
• Reuse the adoption allowlist and literal source-to-target mapping from Copy the existing course platform verbatim and establish its characterization baseline #30, including cadmin/* → studio_courses/* and source templates → course_platform_templates/*.
• Copy new/changed non-overlaid allowlisted files mechanically and regenerate copied-files.tsv plus the pinned-source metadata.
• Preserve files listed in integration-patched-files.tsv; generate a deterministic conflict report when the upstream source side of an overlay changes.
• Fail closed for source deletions/renames, migration removals/replacements, target-owned collisions, unknown allowlisted roots, or a dirty source checkout.
• Add a dry-run mode that makes no target changes and reports the exact source commit, changed files, conflicts, and required follow-up.
• Add focused tests for clean sync, dry-run, path mapping, overlay conflict detection, and destructive-change rejection.
• Add a manual CI workflow (and a non-mutating scheduled drift check if practical) that verifies CMP main drift but never deploys or merges it automatically.
• Keep the existing adoption verifier authoritative after a sync.

Non-goals

• Do not automatically deploy or merge CMP changes.
• Do not copy source secrets, databases, caches, deployment files, editor metadata, or untracked worktree changes.
• Do not overwrite target-owned integration files or silently resolve overlay conflicts.
• Do not squash, rewrite, or remove existing Django migrations.
• Do not redesign Course → Cohort models, routes, Studio, or the website shell.
• Do not replace the tester/PM lifecycle with a green workflow check.

Acceptance criteria

• A documented uv run command can dry-run a clean CMP commit and reports the exact pinned/current SHA and changed source paths without changing tracked files.
• A successful sync uses the recorded allowlist/path mapping and updates the source pin and copied-file checksums deterministically.
• cadmin/* maps to studio_courses/*, source templates map to course_platform_templates/*, and target-owned cadmin compatibility adapters remain untouched.
• Any changed destination listed in integration-patched-files.tsv is preserved and reported as a conflict with old source, new source, and target-overlay evidence.
• Dirty source checkouts, deleted/renamed allowlisted files, migration removals/replacements, unknown roots, and target-owned collisions fail before mutation.
• Sync never removes an existing target migration or creates a squash/replacement migration.
• Unit tests cover the dry-run, successful non-overlap, mapping, overlay conflict, and fail-closed paths.
• The existing adoption verifier, migration check, lint/format checks, and focused copied-course tests pass after a no-op sync.
• CI provides manual sync/drift invocation and a non-mutating report; no workflow can merge or deploy upstream changes by itself.
• Documentation explains the clean-commit requirement, conflict resolution, review gates, rollback to the previous pin, and the current uncommitted CMP-worktree limitation.

Test scenarios

1. Dry-run against the current pinned commit: no file changes and a zero-change report.
2. Dry-run against a clean CMP commit containing only an excluded .claude/metadata change: report it as excluded and make no target change.
3. Sync a fixture source with one changed non-overlaid allowlisted file: copy it, update the manifest, and verify the adoption checker.
4. Change a source file whose destination is in integration-patched-files.tsv: preserve the overlay and emit a stable conflict report.
5. Delete or rename an allowlisted file, remove a migration, dirty the source checkout, or collide with a target-owned file: fail before writing.
6. Run the no-op sync and then uv run python scripts/verify_course_platform_adoption.py, make migrations-check, and the focused course test set.
7. Run the CI drift check: it reports differences but does not modify the branch, issue state, deployment, or source checkout.

Dependencies

• Copy the existing course platform verbatim and establish its characterization baseline #30 adoption provenance and verifier (complete).
• CMP source changes must be committed and pushed before they are eligible for synchronization.
• Tester and PM gates remain mandatory under _docs/PROCESS.md.

[alexeygrigorev]

Grooming Complete

Verdict: READY FOR ENGINEERING — controlled CMP synchronization is bounded and preserves the #30 adoption contract.

The current adoption is pinned to 98a235283904b4ef9ad29e196298540756cf1bcc, with 768 copied-file checksums, explicit integration overlays, a cadmin → studio_courses destination mapping, and target-owned compatibility shims. CMP source synchronization must therefore consume only a clean, committed source revision and must never be a blind repository merge.

The implementation is limited to a uv-invokable sync/dry-run command, deterministic changed-path/conflict reporting, manifest/pin updates, focused tests, and a non-mutating CI drift check. It does not change course business behavior, Course → Cohort models, routes, migrations, Studio, or the site shell. Existing tester/PM gates remain required before any pin update is merged or deployed.

The current local CMP checkout contains uncommitted application changes. Those are intentionally ineligible until committed and pushed in CMP; this issue must not copy a dirty worktree.

[alexeygrigorev]

Current CMP drift snapshot

At 2026-08-13, the clean CMP main commit is 9bed3a67a510349446fbce1212c19255dbff9af7. Comparing it to the website pin 98a235283904b4ef9ad29e196298540756cf1bcc across the adopted allowlist (accounts, api, cadmin, course_management, courses, data, e2e, scripts, templates) yields no eligible source-path changes; the clean delta is limited to excluded CMP incident-response metadata.

The local CMP checkout also has uncommitted eligible changes in courses/views/course_homepage.py, courses/views/course_list.py, courses/views/course_page_context.py, and a new test. These remain intentionally ineligible until committed/pushed in CMP and must be rejected by the sync command.

[alexeygrigorev]

Software Engineer Report

Issue: #145 — Add controlled upstream synchronization from CMP main

Frozen source

• Worktree: /home/alexey/git/dtc-website/.tmp/website-145-engineer
• Branch: issue-145-engineer
• Base/head: 4b8c852681c0ace77f0da94b7f7c38e807c42984 (uncommitted worktree changes included)
• Graph digest: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19
• Verification plan: .tmp/verification-145/verification-plan.json
• Plan SHA-256: d9399d49ddb454cfd0c78c93d58add4b73752268d3f7979750f1f818cb3a1ad0
• Source mode: worktree; render impact: not applicable (internal sync tooling/CI only)
• Screenshots: not_applicable (no product-page render inputs changed)
• Commits/pushes: none; worktree is intentionally uncommitted for tester review

Implementation

• Added scripts/sync_course_platform.py: clean detached source provisioning, exact SHA diff/report, allowlist mapping, cadmin/* → studio_courses/*, template mapping, overlay conflict evidence, fail-closed deletion/rename/migration/collision/dirty checks, atomic staged apply, and deterministic source pin/manifest updates.
• Added scripts/prepare_course_platform_source.py and source-pin.json so a fresh worktree can provision and verify the exact pinned source checkout.
• Updated scripts/verify_course_platform_adoption.py to read the versioned source pin while retaining the baseline fallback.
• Added scripts/tests/test_sync_course_platform.py (8 focused tests).
• Added _docs/adoption/course-platform/upstream-sync.md and linked it from README.md, including clean-commit prerequisite, commands, mapping, conflicts, verification, review, rollback, and dirty-worktree limitation.
• Added Makefile targets: course-platform-source-checkout, course-platform-sync-dry-run, course-platform-sync, and test-course-platform-sync.
• Added .github/workflows/cmp-upstream-sync.yml: scheduled/manual, read-only drift report with artifact upload; no apply, merge, push, deployment, or issue mutation.
• Added workflow contract coverage in ci/tests/test_workflows.py and recorded new cadmin references in the adoption allowlist.

Evidence

Exact commands and results:

uv run --frozen pytest scripts/tests/test_sync_course_platform.py -q
8 passed

uv run --frozen pytest core/tests/test_course_platform_adoption.py -q
9 passed, 909 subtests passed

uv run --frozen pytest ci/tests/test_workflows.py -q
13 passed

make test-ci
505 passed

uv run --frozen python scripts/prepare_course_platform_source.py
provisioned clean CMP checkout at 98a235283904b4ef9ad29e196298540756cf1bcc

uv run --frozen python scripts/verify_course_platform_adoption.py
verified 768 copied files ... and 4 target-owned compatibility shims

make migrations-check
No changes detected

make format-check && make lint && make typecheck
491 files formatted; ruff passed; mypy passed (366 source files)

Fresh remote no-op dry-run (using a clean checkout path) passed:

make course-platform-sync-dry-run \
  CMP_SOURCE_REF=98a235283904b4ef9ad29e196298540756cf1bcc \
  CMP_SOURCE_REPOSITORY=https://github.com/DataTalksClub/course-management-platform.git \
  CMP_SOURCE_CHECKOUT=.tmp/cmp-source-sync-fresh2

Report .tmp/course-platform-sync-report.json: status=no_change,
current_pinned_commit=requested_commit=98a235283904b4ef9ad29e196298540756cf1bcc,
zero changed/copy/conflict/error paths. A dirty local CMP worktree correctly exits
with status 2 and a blocked report; it is intentionally never copied.

The generated verification plan selects a fresh full regression because the
change touches dependency/toolchain, deployment/runtime, and test-infrastructure
surfaces. Selector, evidence-validation, and quality envelopes are present under
.tmp/verification-145/evidence; Django, Playwright, and container components
remain fresh-run work for the verification controller/tester. Compatibility,
content-invariants, and screenshots are explicitly not_applicable per the plan.

[alexeygrigorev]

Sync guard evidence

The implementation was also run against the current clean remote CMP main snapshot during engineering. It resolved the exact commit, reported 15 changed paths (10 allowlisted and 5 excluded metadata), and returned status=blocked with 10 explicit overlay conflicts and zero fatal parser errors. No target files were changed. This is the intended behavior while CMP changes overlap the website’s recorded integration overlays: the source is visible for review, but cannot be copied silently.

The pinned commit was run through a fresh remote checkout and returned status=no_change with zero changed, copy, conflict, or fatal paths.

[alexeygrigorev]

TEST Review

Verdict: PASS — issue #145 candidate is independently verified.

I reviewed the frozen, intentionally uncommitted engineer worktree without editing, committing, or pushing it.

Frozen candidate and verification envelope

• Worktree: /home/alexey/git/dtc-website/.tmp/website-145-engineer
• Branch: issue-145-engineer
• Base/head: 4b8c852681c0ace77f0da94b7f7c38e807c42984 / 4b8c852681c0ace77f0da94b7f7c38e807c42984
• Fresh tester plan: .tmp/verification-tester-145/verification-plan.json
• Report: .tmp/verification-tester-145/verification-report.json
• Graph SHA-256: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19
• Plan SHA-256: 4f4c56f6602526f612f8f14416b486b8bd9efef9aba600989fe2cbdcfb57535c
• Verification report: verdict=success, phase=tester, profile=full, invalid_evidence=[]
• Candidate status remained the expected 11 changed paths throughout QA; git diff --check passed.

The plan records render impact as false for this internal synchronization/CI change. Screenshots are therefore not_applicable (render_inputs_unchanged_without_reusable_visual); no product-page captures were required or taken.

Fresh verification controller results

• Django: make test — 2084 passed, 8 skipped (2092 tests; 0 failed).
• Browser: make test-playwright-core — 154 passed, 27 deselected (0 failed).
• Evidence/control plane: make test-ci — 505 passed.
• Quality contract: 505 passed (lint, format, mypy, migrations, security, parity, deployment/OpenAPI checks).
• Container verification: 12 assertions passed (amd64 image, revision label, non-root UID/GID, static manifest guards, health, unified route).
• Selector and evidence validation: passed; no components skipped.

Issue-specific checks

• uv run --frozen pytest scripts/tests/test_sync_course_platform.py -q — 8 passed (no-op, excluded changes, deletion, dirty source, copy/manifest, mapping, overlay, migration guards).
• uv run --frozen pytest core/tests/test_course_platform_adoption.py -q — 9 passed, 909 subtests passed.
• uv run --frozen pytest ci/tests/test_workflows.py -q — 13 passed.
• make migrations-check — no changes detected.
• make format-check — 491 files already formatted; make lint passed; make typecheck passed with no issues in 366 source files.
• A fresh clean detached checkout at the pinned CMP commit 98a235283904b4ef9ad29e196298540756cf1bcc was provisioned and verified: 768 copied files and 4 target-owned compatibility shims.

Fail-closed sync safety

• Fresh no-op dry-run report .tmp/qa-noop-145.json: status=no_change, current/requested pin both 98a235283904b4ef9ad29e196298540756cf1bcc, zero changed/copy/conflict/fatal paths. Target status and binary diff were unchanged.
• Dirty-source dry-run report .tmp/qa-dirty-145.json: exit code 2, status=blocked, explicit dirty-source fatal; the intentionally dirty local CMP checkout was not copied and target status/binary diff were unchanged.

The allowlist/mapping (cadmin/template destinations), overlay conflict handling, deletion/rename/copy rejection, migration rewrite guard, collision checks, staged apply/manifest/pin/README behavior, and read-only workflow contract are covered by the focused tests and all passed. The scheduled/manual workflow requests read-only contents access, performs dry-run reporting/artifact upload only, and contains no apply, push, deploy, or issue mutation path.

No secrets, tokens, registration data, or production data were exposed. Candidate remains uncommitted for PM acceptance.

[alexeygrigorev]

Product Acceptance

Verdict: ACCEPT — issue #145 is ready for the engineer commit gate.

I reviewed the groomed issue, the #30 course-platform adoption provenance, the frozen uncommitted candidate, the engineer report, and the independent tester PASS. I did not edit the candidate, commit, push, or create a pull request.

Review envelope

• Candidate worktree: /home/alexey/git/dtc-website/.tmp/website-145-engineer
• Branch: issue-145-engineer
• Frozen base/head: 4b8c852681c0ace77f0da94b7f7c38e807c42984 / 4b8c852681c0ace77f0da94b7f7c38e807c42984
• Tester review: Add controlled upstream synchronization from CMP main #145 (comment)
• Fresh tester report: .tmp/verification-tester-145/verification-report.json
• Fresh report verdict: success, phase=tester, profile=full, invalid_evidence=[]
• Graph SHA-256: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19
• Fresh tester plan SHA-256: 4f4c56f6602526f612f8f14416b486b8bd9efef9aba600989fe2cbdcfb57535c

Acceptance review

• Controlled dry-run and exact provenance — PASS. The runbook documents the uv run dry-run command and exact immutable CMP SHA flow. The fresh no-op report .tmp/qa-noop-145.json records status=no_change, the pinned/requested SHA 98a235283904b4ef9ad29e196298540756cf1bcc, zero changed/copy/conflict/fatal paths, and no target status or binary-diff change.
• Deterministic sync and adoption manifest — PASS. The implementation stages byte/checksum-validated copies, regenerates copied-files.tsv, updates source-pin.json and the single README pin, and leaves target files untouched until preflight succeeds. A clean detached pinned source verified 768 copied files and 4 target-owned compatibility shims.
• Namespace and overlay ownership — PASS. The documented mapping preserves cadmin/* → studio_courses/*, maps cadmin/templates/cadmin/* to studio_courses/templates/studio_courses/*, keeps the target-owned compatibility adapter out of copied source, and reports/preserves changed integration overlays with source/target checksum evidence and rationale.
• Fail-closed safety — PASS. Dirty source, allowlisted deletion/rename/copy, migration replacement/removal, unknown mapping, stale manifests/overlays, and target-owned collisions are preflight guards. The dirty-source report .tmp/qa-dirty-145.json returned exit 2 with status=blocked and no target mutation; focused tests cover no-op, excluded changes, non-overlap copy, mapping, overlay, deletion, dirty-source, and migration safety.
• Migration and existing adoption gates — PASS. make migrations-check reported no drift; adoption verification and the copied-course checks passed.
• Verification and quality — PASS. Fresh full verification completed with Django 2084 passed, 8 skipped, Playwright core 154 passed (27 deselected), make test-ci 505 passed, quality contract 505 passed, and 12 container assertions passed. No verification components were skipped.
• Read-only automation — PASS. .github/workflows/cmp-upstream-sync.yml supports schedule and manual source-ref invocation, requests only contents: read, runs dry-run/reporting, uploads an artifact, and has no apply, merge, push, deployment, or issue-mutation path.
• Documentation and rollback — PASS. The README links the machine-readable source pin and runbook. The runbook explains clean pushed commits, current dirty CMP-worktree rejection, mappings, conflict resolution, review gates, rollback by reverting the approved website sync commit, and the migration-history restriction.

Product/user impact

This issue adds bounded operator tooling and CI reporting only. It does not change public or Studio routes, authentication/denial behavior, course business behavior, migrations, content, or the website shell. Render impact is false in the fresh plan, so screenshots are correctly not_applicable under the tester policy; no browser-page acceptance was required. The documented dirty local CMP checkout remains ineligible and is never copied.

No secrets, tokens, registration data, or production data were exposed. The candidate remains intentionally uncommitted. The next lifecycle step is the engineer's focused commit with Closes #145, followed by the orchestrator's local no-PR merge/push and on-call CI observation.

[alexeygrigorev]

On-call CI report — PASS

Observed the exact pushed merge for #145:

• Main SHA: 31c5447f4686c2ff87707b137b97ef85a332931a
• Run: https://github.com/DataTalksClub/website/actions/runs/31712254201
• Required jobs: resolve-release, classification, quality, django, playwright, container, and ci-gate — all successful.
• Release jobs: publish and deploy — successful.
• Probe jobs were correctly skipped because this change did not require them.

No rerun, source mutation, or manual deployment was performed. The controlled CMP synchronization documentation and tooling are now deployed through the normal release path.