From f2abda3fc6ac03d7f26f94dad3210b7cf14cb7a3 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 4 Aug 2026 18:40:53 -0700 Subject: [PATCH] ci: add reviewed Codex unstable integration lane codex-unstable needs to carry preview topics above the generated production output, but the earlier prototype discovered every matching *-unstable ref. A branch push could therefore enroll unreviewed code, and replaying that prototype would also drop the reviewed-topic controls already protecting codex. Add a lane-aware preview path that keeps published config authoritative. An explicit local enable creates a bot-authored, tree-identical child of the current Codex output so the first preview pull request has a safe target without relaxing strict ancestry. Preview topics enter only through an authenticated reviewed merge into that branch; bare suffix refs stay inert. Prepare stable and preview candidates from one snapshot, run independent exact-SHA staging CI, and publish both lanes, meta, topics, and staging cleanup with one leased atomic push. Keep releases restricted to codex, preserve stable v1 no-op behavior, and fail closed on cross-lane ancestry, forged sentinels, stale staging, and unsafe workflow changes. t/t9905-codex-branch.sh now covers all 66 existing production cases and 29 preview admission, lifecycle, graph, recovery, and publication cases. --- .github/CODEX.md | 357 ++-- .github/rulesets/codex-unstable-branch.json | 71 + .github/workflows/codex-admission.yml | 111 +- .github/workflows/codex-branch.sh | 1403 ++++++++++++++-- .github/workflows/codex.yml | 28 +- t/t9905-codex-branch.sh | 1667 ++++++++++++++++++- 6 files changed, 3364 insertions(+), 273 deletions(-) create mode 100644 .github/rulesets/codex-unstable-branch.json diff --git a/.github/CODEX.md b/.github/CODEX.md index caf05e90f4aa16..b4bf905684acf0 100644 --- a/.github/CODEX.md +++ b/.github/CODEX.md @@ -1,21 +1,26 @@ -# Maintaining `codex` +# Maintaining `codex` and `codex-unstable` -`master` stays equivalent to upstream. The orphan `meta` branch contains the -controller, reusable workflows, tests, documentation, and ruleset recipes. -Never merge `meta` into `master` or `codex`. +`master` stays equivalent to upstream. `codex` is the production integration +branch distributed with Codex. When enabled, `codex-unstable` starts at that +exact production commit and adds reviewed preview topics; it never publishes a +production release. The orphan `meta` branch contains the controller, reusable +workflows, tests, documentation, and ruleset recipes. Never merge `meta` into +`master`, `codex`, or `codex-unstable`. Topics use branches named `??/codex/*`, where `??` is a two-character owner -name. The existing rows in `meta:codex.config` stay enrolled. After this -migration, a new branch becomes an active input only after its reviewed pull -request has merged into `codex` and the controller has enrolled it there. -Merely pushing a matching branch never includes it in a build. The `-wip`, -`-stale`, and reserved `-unstable` suffixes cannot enter the production -branch. +name. A topic ending in `-unstable` belongs only in `codex-unstable`; other +active topics belong in `codex`. The existing rows in `meta:codex.config` stay +enrolled. A new topic enters either lane only after its reviewed pull request +merges into that lane and the controller enrolls its retained head. Merely +pushing a matching branch, including an `-unstable` branch, never includes it +in a build. The `-wip` and `-stale` suffixes are inactive. The generated `codex.config` records each enrolled topic's prerequisite and -last published tip. Those prerequisite edges form a partial order; there is -no global topic order. Rows are sorted only so the generated file has a -stable representation. +last published tip. When the preview lane is enabled, it also records the exact +production commit underlying `codex-unstable` and its published preview tip. +Prerequisite edges form a partial order within each lane; there is no global +topic order. Rows are sorted only to give the generated file a stable +representation. The saved tips are rebase boundaries, not another copy of the patches. They let a fresh runner distinguish “the prerequisite was rewritten” from “these @@ -24,15 +29,16 @@ is what makes amending, dropping, or replacing an already-published commit safe without relying on a runner's local reflogs. The generated `codex` history contains one explicit two-parent integration -commit for every active topic, including prerequisites, fast-forwardable -topics, and empty topics. Prerequisites are integrated before their dependents; -lexical order only breaks ties among topics that are ready at the same time. -This order makes the history deterministic but does not add semantic edges to -the topic graph. When the first topic is empty because its tip equals the base, -the controller first creates a tree-identical `Begin codex integration` commit -so that topic can still have two distinct parents. +commit for every production topic, including prerequisites, fast-forwardable +topics, and empty topics. `codex-unstable` starts at the exact generated +`codex` commit and adds one explicit integration merge per enrolled preview +topic. It remains strictly ahead even before its first topic because enabling +it creates a tree-identical bootstrap commit. Prerequisites are integrated +before their dependents; lexical order only breaks ties among topics that are +ready at the same time. This makes the history deterministic without inventing +semantic dependencies. -## Topic branches and pull requests +## Add a production topic Create a topic from `master`, or from the topic it depends on: @@ -58,43 +64,111 @@ skipped, and another topic cannot pass the merge queue until the controller atomically publishes the rebuilt `codex` and its matching `meta` state. The pending commit remains visible to anyone fetching `codex` directly. +## Enable the preview lane + +After deploying the updated controller, automation trampoline, and unstable +branch ruleset, initialize `codex-unstable` explicitly: + +```sh +Meta/rebuild --local --enable-unstable +``` + +The controller creates a tree-identical `Initialize codex-unstable` commit +directly on top of `codex`, records the enabled lane in `meta:codex.config`, +and publishes both changes together. The production tip does not change, and +no existing `*-unstable` branch is enrolled. The preview branch now exists as +a pull-request target, is strictly ahead of production, and initially has +exactly the same contents. + +Enabling and disabling are explicit, local-only operations; dispatching the +ordinary GitHub Action cannot change whether the lane exists. Once enabled, +both `Meta/rebuild` and `Meta/rebuild --local` maintain the preview lane. + +## Add a preview topic + +Create the topic from the published `codex`, or from another enrolled preview +topic it depends on: + +```sh +git switch -c tb/codex/my-topic-unstable origin/codex +# Edit, test, and commit. +git push -u origin HEAD +``` + +Pushing this branch does not change `codex-unstable`. Open a pull request from +the topic to `codex-unstable`, obtain the required approval, and select +**Merge when ready**. Its separate one-at-a-time merge queue accepts only a +same-repository `??/codex/*-unstable` topic. Run `Meta/rebuild` or +`Meta/rebuild --local` to authenticate the reviewed merge, enroll the exact +topic head, run staging CI, and publish the rebuilt generation. + +While a preview merge is pending, another preview topic cannot pass its merge +queue. Production and preview admission are tracked independently; a single +controller run can consume one pending merge from each lane. An unreviewed +preview branch, including a whole preexisting stack of matching branches, +remains excluded until its own pull request is merged. + +Production topics may never depend on preview topics. Root preview topics are +based on `codex`; their prerequisites, when present, must be other enrolled +preview topics. Update, rebase, replace, or reorder a preview topic using the +same ancestry rules as a production topic, substituting `codex` for `master`. +Before retiring a preview prerequisite, restack its children onto a surviving +preview topic or `codex`. + +When the lane is no longer needed, first retire every enrolled preview topic, +then explicitly remove the generated branch and its recorded state: + +```sh +Meta/rebuild --local --disable-unstable +``` + +Topic-branch deletion still requires an authorized bypass of the topic ruleset. +Creating a separate inactive copy without removing the enrolled ref does not +retire that topic. + +## Update, reorder, or remove a topic + Already enrolled topics remain active across rebuilds. You may push an update and run the controller directly, or use another reviewed topic pull request before rebuilding. Keep topic history linear. Rebase a dependent topic onto its prerequisite; -never merge `codex` into a topic or use GitHub's **Update branch** button on -these pull requests. +never merge either generated branch into a topic or use GitHub's **Update +branch** button on these pull requests. -For a new topic, the controller infers one prerequisite: its nearest active -topic-tip ancestor, or `master` for a root topic. After publication it keeps -that recorded edge across prerequisite rewrites. If sibling topics share -private commits, create an active topic at that shared prefix and base both -siblings on it. Otherwise the controller rejects the ambiguous overlap. +For a new topic, the controller infers one prerequisite: its nearest enrolled +same-lane topic-tip ancestor, or that lane's root. The production root is +`master`; the preview root is the exact generated `codex`. After publication +it keeps that recorded edge across prerequisite rewrites. If sibling topics +share private commits, create an enrolled topic at that shared prefix and base +both siblings on it. Otherwise the controller rejects the ambiguous overlap. You may append, amend, reorder, or drop commits on an existing topic. If it has dependents, you may leave them at the last published prerequisite tip; the controller uses the recorded boundary to restack them. To change a topic's prerequisite, rebase it so the new topic's exact current tip is in its history. -To make it a root topic, rebase it onto current `master` and remove the old -private prerequisite from its history. Those exact ancestry changes are the -reordering signal; patch similarity and lexical branch order are never used. +To make it a root topic, rebase it onto the current `master` or `codex`, as +appropriate, and remove the old private prerequisite from its history. Those +exact ancestry changes are the reordering signal; patch similarity and +lexical branch order are never used. Before making a prerequisite inactive, first restack every child onto a -surviving topic or current `master`. The controller refuses to guess whether -the retired topic's commits should be discarded or transferred to a child. +surviving topic in the same lane or that lane's root. The controller refuses +to guess whether the retired topic's commits should be discarded or +transferred to a child. Deleting or renaming an enrolled active ref explicitly retires it on the next rebuild; because topic refs are protected against deletion, this requires an authorized ruleset bypass. Creating a separate `-stale` copy without removing the enrolled active ref does not retire it. -Every admission pull request must have a same-repository `??/codex/*` head -and target `codex`. The topic ruleset retains that head after the merge. Do +Every admission pull request must have a same-repository `??/codex/*` head and +target the matching lane: ordinary topics go to `codex`; `-unstable` topics go +to `codex-unstable`. The topic ruleset retains that head after the merge. Do not delete, force-rewrite, or otherwise change the reviewed head before the -controller rebuilds it. The controller rejects squash commits, unrelated -direct commits, multiple pending merges, octopus merges, and merge-only edits. +controller rebuilds it. Each lane rejects squash commits, unrelated direct +commits, multiple pending merges, octopus merges, and merge-only edits. -## Keep the dispatch and admission workflows on `codex` +## Keep dispatch and admission on the generated branches GitHub shows **Run workflow** only for a workflow present on the default branch. Exactly one active `??/codex/automation` topic must therefore be based @@ -109,6 +183,7 @@ on: pull_request: branches: - codex + - codex-unstable types: - opened - reopened @@ -129,20 +204,35 @@ jobs: uses: openai/git/.github/workflows/codex.yml@meta admission: name: Codex admission - if: github.event_name == 'pull_request' || github.event_name == 'merge_group' + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex') + permissions: + contents: read + pull-requests: read + uses: openai/git/.github/workflows/codex-admission.yml@meta + unstable_admission: + name: Codex unstable admission + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex-unstable') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex-unstable') permissions: contents: read pull-requests: read uses: openai/git/.github/workflows/codex-admission.yml@meta ``` -The admission job handles both ordinary pull-request checks and merge-group -checks. It accepts a queued merge only while the published output and `meta` -agree, verifies that the group introduces exactly one eligible reviewed topic, -and rejects changes to the trusted dispatch or release workflows. The -controller accepts the old dispatch-only trampoline during rollout; once the -updated automation topic is published, this exact file remains in every -generated `codex` tree. Both implementations stay on orphan `meta`. No +Each lane's admission job handles both ordinary pull-request checks and +merge-group checks. It accepts a queued merge only while that lane's published +output and `meta` agree, verifies that the group introduces exactly one +eligible reviewed topic, and rejects changes to every GitHub Actions workflow. +The production and preview jobs have separate required-check names. Once the +updated automation topic is published, this exact file remains in both +generated branch trees. The implementations stay on orphan `meta`; no controller files or custom patches belong on `master`. ## Initialize the published topology once @@ -185,7 +275,7 @@ inspection and conflict recovery. Normal publication uses `Meta/rebuild` or `Meta/rebuild --local`; use `Meta/publish ` only to finish a preparation started through GitHub. -## Refresh `codex` +## Refresh the enabled branches From a clean, complete clone with a linked `Meta` worktree, run: @@ -195,10 +285,11 @@ Meta/rebuild `Meta/rebuild` updates the clean `Meta` worktree to current `origin/meta`, dispatches **Refresh codex**, and prints the exact run ID and URL returned by -GitHub. It reports preparation status, validates the successful artifact, -pushes the candidate to `codex-staging`, reports staging-CI job progress, and -performs the atomic promotion only after that exact candidate passes. Leave -the command running until it reports the published candidate. +GitHub. It reports preparation status, validates the successful artifact, and +stages each generated candidate separately: `codex-staging` for production +and, when enabled, `codex-unstable-staging` for preview. It reports staging-CI +job progress and performs one atomic promotion only after every exact +candidate passes. Leave the command running until it reports publication. To do the rebase and assembly on your machine instead of in the preparation Action, run: @@ -208,19 +299,19 @@ Meta/rebuild --local ``` This skips only the preparation Action. It fetches the current heads from -GitHub, prepares the topics, integration commits, `codex`, and the next -`codex.config` state in an isolated temporary repository, then imports and -verifies the resulting bundle in the publisher clone. The temporary -repository has its own rerere cache and Git configuration, so an old local -resolution, hook, or concurrent local preparation cannot leak through shared -repository state. The command prints the path of a persistent local session -containing the input snapshot, update manifest, candidate OID, bundle, and any -conflict report. +GitHub, prepares the topics, integration commits, enabled output branches, +and the next `codex.config` state in an isolated temporary repository, then +imports and verifies the resulting bundle in the publisher clone. The +temporary repository has its own rerere cache and Git configuration, so an +old local resolution, hook, or concurrent local preparation cannot leak +through shared repository state. The command prints the path of a persistent +local session containing the input snapshot, update manifest, candidate OID, +bundle, and any conflict report. `--local` does not skip CI or make a private local-only publication. After -preparation, it uses the same user-authenticated `codex-staging` push, waits -for the same fresh `main.yml` run for the exact candidate SHA, and performs -the same atomic exact-lease promotion to GitHub. Use `Meta/codex refresh +preparation, it uses the same user-authenticated staging pushes, waits for a +fresh `main.yml` run for each exact candidate SHA, and performs the same +atomic exact-lease promotion to GitHub. Use `Meta/codex refresh --require-automation` when you want only a local preview with no server-side ref update. @@ -249,8 +340,8 @@ but rejects every other change. To start preparation in the GitHub UI instead, open **Actions > Refresh codex**, select `codex`, and choose **Run workflow**. After **Rebase topics and -assemble codex** succeeds, finish that exact run from the same kind of clean -clone: +assemble Codex branches** succeeds, finish that exact run from the same kind +of clean clone: ```sh Meta/publish @@ -268,14 +359,16 @@ snapshot the current refs and stage a fresh candidate. The Action: -1. reads the published enrollment from `meta/codex.config`, verifies any one - pending reviewed pull-request merge, and snapshots `meta`, `master`, - `codex`, the enrolled topics, and the newly admitted topic if present; +1. reads the published enrollment from `meta/codex.config`, verifies at most + one pending reviewed pull-request merge per enabled lane, and snapshots + `meta`, `master`, both outputs, their enrolled topics, and any newly + admitted heads; 2. reads the published boundaries, infers only a newly admitted topic or explicitly restacked edges, and rebases topics sequentially in dependency order, using rerere resolutions learned from the old `codex`; -3. integrates every active topic at its rewritten tip with an explicit merge - commit and freezes the result without building or executing candidate code, +3. integrates each enrolled production topic with an explicit merge, then + integrates enrolled preview topics on top of the exact generated `codex` + commit; freezes both results without building or executing candidate code; and, when state changed, creates a direct child of the pinned `meta` tip that changes only `codex.config`; 4. uploads the bundle, pinned input snapshot, update manifest, and canonical @@ -299,16 +392,20 @@ recorded by GitHub for `meta`. It checks the caller SHA against the snapshotted `codex`, requires the artifact controller to equal `Meta/HEAD`, validates the ZIP allowlist and bundle heads, and revalidates the complete input snapshot. -Only then does it record the existing CI run ID and push the exact candidate -to the fixed `codex-staging` branch. That user-authenticated push starts -ordinary push CI, including when the candidate changes a workflow file. The -helper binds to a newer `main.yml` push run whose branch and SHA exactly match -the staged candidate. While it waits, it prints the run URL and reports changes +Only then does it record the existing CI run IDs and push the exact production +candidate to `codex-staging`. When preview is enabled, it also stages the +exact preview candidate at `codex-unstable-staging`. Each user-authenticated +push starts ordinary CI, including when its candidate changes a workflow file. +The helper binds each candidate to a newer `main.yml` push run whose branch and +SHA match exactly. While waiting, it prints each run URL and reports changes in status, completed-job count, and failure count, plus a heartbeat every five -minutes when those values do not change. After the full run and its unique -`config` job succeed, it revalidates the snapshot and atomically updates -`meta`, every topic, and `codex` with exact leases while deleting -`codex-staging`. +minutes when those values do not change. + +After every required run and its unique `config` job succeeds, the helper +revalidates the snapshot and atomically updates `meta`, every enrolled topic, +`codex`, and the enabled `codex-unstable` branch with exact leases while +deleting both staging refs. A failed preview build cannot publish production; +a failed production build cannot publish preview. Every push to `codex` starts the inexpensive release-publication check. The expensive build and release jobs proceed only when the triggering SHA is the @@ -317,20 +414,23 @@ therefore publishes no release; the controller's atomic `codex`/`meta` promotion does. The check does not require that SHA to remain the live `codex` tip, so a later pending merge cannot suppress a release that the controller already triggered. A canonical no-op reuses the existing tips, and -staging never releases. Ordinary CI may reuse its own earlier successful -result when the commit or tree is identical. +staging never releases. `codex-unstable` also never releases: the release +workflow listens only for pushes to `codex`. Ordinary CI may reuse its own +earlier successful result when the commit or tree is identical. -Until the final push, `meta`, `codex`, and all topic refs remain unchanged. A -CI, validation, lease, or promotion failure after staging leaves -`codex-staging` at the candidate for inspection. Successful promotion updates -all primary refs and deletes staging as one atomic transaction. +Until the final push, `meta`, both generated outputs, and all topic refs remain +unchanged. A CI, validation, lease, or promotion failure after staging leaves +the affected staging refs available for inspection. Successful promotion +updates all primary refs and deletes every staging ref as one atomic +transaction. Git can place server-side leases only on refs included in the push. The -controller therefore refetches `meta`, `master`, `codex`, and the complete -topic namespace immediately before promotion, then places exact leases on -every ref it mutates or deletes in the atomic transaction. The generated -state can therefore never describe a different published generation. A new -input created in the final fetch-to-push window is handled by the next run. +controller therefore refetches `meta`, `master`, both enabled outputs, and the +complete topic namespace immediately before promotion, then places exact +leases on every ref it mutates or deletes in the atomic transaction. The +generated state can therefore never describe a different published generation. +A newly created but unadmitted topic remains excluded; a newly merged topic is +handled by the next verified snapshot. Both preparation modes use `refs/remotes/origin/master` after fetching the current `openai/git` heads. They do not use the caller's local `master` @@ -346,8 +446,8 @@ unchanged ref would require server-side transaction support. ## Resolve a rebase conflict -A failed Action summary says **No refs were updated** and prints a pinned -`Meta/codex resolve` command. A failed `Meta/rebuild --local` prints the +A production-topic conflict leaves every published ref unchanged and prints a +pinned `Meta/codex resolve` command. A failed `Meta/rebuild --local` prints the persistent session path and leaves the same conflict report there. In either case, follow the exact commands in that report. For Action preparation, start from a clean clone that does not already have a `Meta` worktree; the printed @@ -391,6 +491,13 @@ practice. Rebase the conflicting topic and its descendants onto the real prerequisite, push that coherent graph, and start a fresh dispatch. Do not fix this by merging `codex` into a topic or by maintaining a manual order. +A preview-topic conflict also leaves every published ref unchanged, but the +production-only `resolve`, `continue`, and `publish-topics` commands do not +reconstruct the nested preview lane. Manually restack the affected +`*-unstable` topic and its descendants onto their current prerequisite, push +that coherent topic graph atomically with exact leases, and run +`Meta/rebuild` or `Meta/rebuild --local` again. + ## Configure publishing Publishing needs no repository secret, deploy key, GitHub App, or protected @@ -409,18 +516,19 @@ environment. It uses the publisher's ordinary credentials: `Meta/rebuild`, `Meta/rebuild --local`, and `Meta/publish` accept only the standard SSH or HTTPS URL for `openai/git`. None reads a token from the repository, local session, or artifact. -3. In the existing **Protect generated Codex branch** ruleset and the - **Protect Codex controller branch** ruleset, add an **always** bypass for - each exact human publisher. The checked-in recipes authorize only the +3. In **Protect generated Codex branch**, **Protect generated unstable Codex + branch**, and **Protect Codex controller branch**, add an **always** bypass + for each exact human publisher. The checked-in recipes authorize only the `ttaylorr-oai` user (`301000140`) plus the existing organization-admin - break-glass actor. Import the `meta` recipe only if that ruleset is absent. + break-glass actor. Update existing rulesets; import a recipe only when its + matching ruleset is absent. The bypass is intentionally personal and visible. The Action cannot publish; it only prepares an immutable artifact. Running `Meta/rebuild`, `Meta/rebuild --local`, or `Meta/publish` is the approval, and Git records the configured user as the pusher. To add or remove a publisher, change the exact -`User` actor in both rulesets. Do not replace it with a broad repository role -or a shared long-lived credential. +`User` actor in each generated-output and controller ruleset. Do not replace +it with a broad repository role or a shared long-lived credential. The final push runs from the operator's machine. Preparation runs either in Actions or in the isolated local repository; staging CI still runs in Actions. @@ -439,8 +547,8 @@ For an Actions preparation, the local helper downloads the artifact through `gh`, but pushes through `origin`. Those credentials can theoretically identify different users, so every publishing path prints the authenticated `gh` login before staging. Verify the configured Git identity when changing -machines or credentials. A failed CI run leaves `codex-staging` for -inspection; it exposes no publishing secret. +machines or credentials. A failed CI run leaves its production or preview +staging branch for inspection; neither exposes a publishing secret. The automation topic is the only topic allowed to change the dispatch and admission trampoline. A directly merged topic pull request may change neither @@ -458,19 +566,19 @@ rebase committers use Generated merge and `meta` state commits use that identity as both author and committer. These commits are deliberately unsigned and do not claim verified GitHub App authentication. GitHub records the local credential owner as the -pusher. Integration subjects remain `Merge into codex`. +pusher. Integration subjects are `Merge into codex` or +`Merge into codex-unstable`. ## Repository rulesets -Create or update the repository rulesets to match the three JSON files. Do not -layer a duplicate over an existing matching ruleset: a bypass in the new +Create or update the repository rulesets to match the checked-in JSON files. +Do not layer a duplicate over an existing matching ruleset: a bypass in one ruleset does not bypass another applicable ruleset. For a missing ruleset, use **Settings > Rules > Rulesets > New ruleset > Import a ruleset**. -In `openai/git`, edit the existing **Protect generated Codex branch** ruleset -to add the exact-user publisher bypass, keep the existing topic ruleset aligned -with its recipe, and import only the missing `meta` ruleset. Verify that exactly -one active ruleset covers each of `codex`, `??/codex/*`, and `meta`. +In `openai/git`, update the existing production, preview, topic, and controller +rulesets in place. Verify that exactly one active policy ruleset covers each +of `codex`, `codex-unstable`, `??/codex/*`, and `meta`. - `.github/rulesets/codex-topics.json` matches `??/codex/*` and blocks deletion, preserving topic heads after pull-request merges. @@ -478,17 +586,22 @@ one active ruleset covers each of `codex`, `??/codex/*`, and `meta`. review, deletion, force-push, one-at-a-time merge-queue, and trusted admission-check rules. Its exact `ttaylorr-oai` user bypass permits local publication; the organization-admin entry remains for break-glass access. +- `.github/rulesets/codex-unstable-branch.json` gives `codex-unstable` the + same review and one-at-a-time merge-queue protections, but requires the + distinct **Codex unstable admission / Verify reviewed topic** check. It + retains the same narrowly scoped publication bypasses. - `.github/rulesets/codex-meta.json` protects the `meta` controller with pull-request, review, deletion, and force-push rules. Its matching exact-user bypass lets the same atomic push advance the generated state; the organization-admin entry remains for break-glass access. -The trusted admission check verifies that the pull-request head matches the -eligible `??/codex/*` namespace. Do not require topic heads to be up to date -with `codex`: the merge queue checks the proposed merge against the current -base without copying the generated aggregate into a topic. An actor with an -explicit ruleset bypass can still override the queue; keep publisher bypasses -narrow and reserve organization-admin access for emergencies. +Each trusted admission check verifies that the pull-request head matches the +eligible `??/codex/*` namespace and belongs to its target lane. Do not require +topic heads to be up to date with either generated branch: the merge queue +checks the proposed merge against the current base without copying the +aggregate into a topic. An actor with an explicit ruleset bypass can still +override the queue; keep publisher bypasses narrow and reserve +organization-admin access for emergencies. The required check is pinned to the GitHub Actions App, but a repository ruleset cannot pin the source workflow file. SCM review is therefore still the @@ -498,10 +611,16 @@ workflow rule or use a dedicated admission App before enabling this flow. Deploy these changes in order: -1. Merge the new controller and admission workflow into `meta`. -2. Update the enrolled automation and release topics, then run - `Meta/rebuild --local` once to publish the new trampoline and release guard. -3. Update the existing `codex` ruleset from its checked-in recipe to require - the one-at-a-time merge queue and trusted admission check. - -Do not merge a new topic pull request before all three steps are complete. +1. Merge the controller, lane-aware trusted admission workflow, and ruleset + recipe into `meta` without creating a merge commit there. +2. Update the already-enrolled automation topic, then run + `Meta/rebuild --local` once to publish the exact two-lane trampoline while + preserving the production release guard and stable merge-queue settings. +3. Update the existing **Protect generated unstable Codex branch** ruleset + from its recipe, including its separate required check and serial queue. +4. Run `Meta/rebuild --local --enable-unstable` to create the protected, + strictly-ahead preview target without enrolling any preexisting topic. +5. Open reviewed `*-unstable` pull requests against `codex-unstable`, merge + them through its queue, and run the controller to publish each generation. + +Do not merge a preview pull request until all four setup steps are complete. diff --git a/.github/rulesets/codex-unstable-branch.json b/.github/rulesets/codex-unstable-branch.json new file mode 100644 index 00000000000000..52c402ffb9b2d2 --- /dev/null +++ b/.github/rulesets/codex-unstable-branch.json @@ -0,0 +1,71 @@ +{ + "name": "Protect generated unstable Codex branch", + "target": "branch", + "source_type": "Repository", + "enforcement": "active", + "conditions": { + "ref_name": { + "exclude": [], + "include": [ + "refs/heads/codex-unstable" + ] + } + }, + "rules": [ + { + "type": "deletion" + }, + { + "type": "non_fast_forward" + }, + { + "type": "pull_request", + "parameters": { + "allowed_merge_methods": [ + "merge" + ], + "require_code_owner_review": false, + "require_last_push_approval": true, + "dismiss_stale_reviews_on_push": false, + "required_approving_review_count": 1, + "required_review_thread_resolution": false + } + }, + { + "type": "merge_queue", + "parameters": { + "check_response_timeout_minutes": 60, + "grouping_strategy": "ALLGREEN", + "max_entries_to_build": 1, + "max_entries_to_merge": 1, + "merge_method": "MERGE", + "min_entries_to_merge": 1, + "min_entries_to_merge_wait_minutes": 0 + } + }, + { + "type": "required_status_checks", + "parameters": { + "required_status_checks": [ + { + "context": "Codex unstable admission / Verify reviewed topic", + "integration_id": 15368 + } + ], + "strict_required_status_checks_policy": false + } + } + ], + "bypass_actors": [ + { + "actor_id": 301000140, + "actor_type": "User", + "bypass_mode": "always" + }, + { + "actor_id": 1, + "actor_type": "OrganizationAdmin", + "bypass_mode": "always" + } + ] +} diff --git a/.github/workflows/codex-admission.yml b/.github/workflows/codex-admission.yml index 308510be724ce9..eef94a033f6141 100644 --- a/.github/workflows/codex-admission.yml +++ b/.github/workflows/codex-admission.yml @@ -58,17 +58,60 @@ jobs: "repos/$GITHUB_REPOSITORY/contents/codex.config?ref=$meta_oid" \ -H 'Accept: application/vnd.github.raw+json') || die 'could not read published codex.config' - published_oid=$(printf '%s\n' "$config" | + published_codex_oid=$(printf '%s\n' "$config" | git config --no-includes --file /dev/stdin \ --get codex.output-tip) || die 'published codex.config has no codex.output-tip' case "$GITHUB_EVENT_NAME" in pull_request) - test "$PULL_BASE_REF" = codex || - die "pull request targets '$PULL_BASE_REF', not codex" - test "$PULL_BASE_SHA" = "$codex_oid" || - die 'the pull request base no longer matches published codex' + lane=$PULL_BASE_REF + ;; + merge_group) + test "$EVENT_ACTION" = checks_requested || + die "unexpected merge-group action '$EVENT_ACTION'" + case "$GROUP_BASE_REF" in + refs/heads/codex|refs/heads/codex-unstable) + lane=${GROUP_BASE_REF#refs/heads/} + ;; + *) die "unexpected merge-group target '$GROUP_BASE_REF'" ;; + esac + ;; + *) + die "unexpected event '$GITHUB_EVENT_NAME'" + ;; + esac + + case "$lane" in + codex) + lane_oid=$codex_oid + published_oid=$published_codex_oid + ;; + codex-unstable) + lane_oid=$(api \ + "repos/$GITHUB_REPOSITORY/git/ref/heads/codex-unstable" \ + --jq '.object.sha') || + die 'could not resolve the codex-unstable branch' + published_oid=$(printf '%s\n' "$config" | + git config --no-includes --file /dev/stdin \ + --get codex-unstable.output-tip) || + die 'published codex.config has no codex-unstable.output-tip' + unstable_base=$(printf '%s\n' "$config" | + git config --no-includes --file /dev/stdin \ + --get codex-unstable.base-tip) || + die 'published codex.config has no codex-unstable.base-tip' + test "$unstable_base" = "$published_codex_oid" || + die 'codex-unstable is not based on the published codex output' + test "$published_oid" != "$published_codex_oid" || + die 'codex-unstable is not strictly ahead of codex' + ;; + *) die "unexpected pull-request target '$lane'" ;; + esac + + case "$GITHUB_EVENT_NAME" in + pull_request) + test "$PULL_BASE_SHA" = "$lane_oid" || + die "the pull request base no longer matches published $lane" test "$PULL_HEAD_REPOSITORY" = "$GITHUB_REPOSITORY" || die 'the pull request topic must belong to openai/git' test "$PULL_DRAFT" = false || @@ -81,24 +124,17 @@ jobs: pull_number=$PULL_NUMBER ;; merge_group) - test "$EVENT_ACTION" = checks_requested || - die "unexpected merge-group action '$EVENT_ACTION'" - test "$GROUP_BASE_REF" = refs/heads/codex || - die "merge group targets '$GROUP_BASE_REF', not codex" - test "$codex_oid" = "$published_oid" || - die 'codex already contains a pending topic; run Meta/rebuild first' - test "$GROUP_BASE_SHA" = "$codex_oid" || - die 'merge group does not start at the published codex tip' + test "$lane_oid" = "$published_oid" || + die "$lane already contains a pending topic; run Meta/rebuild first" + test "$GROUP_BASE_SHA" = "$lane_oid" || + die "merge group does not start at the published $lane tip" test "$GROUP_HEAD_SHA" = "$GITHUB_SHA" || die 'merge-group event does not describe its checked commit' case "$GROUP_HEAD_REF" in - refs/heads/gh-readonly-queue/codex/*) ;; + "refs/heads/gh-readonly-queue/$lane/"*) ;; *) die "unexpected merge-group ref '$GROUP_HEAD_REF'" ;; esac ;; - *) - die "unexpected event '$GITHUB_EVENT_NAME'" - ;; esac read -r candidate_oid parent_count first_parent second_parent <<<"$( @@ -111,15 +147,15 @@ jobs: die 'the inspected candidate does not match the workflow commit' test "$parent_count" = 2 || die 'Codex admission requires exactly one explicit topic merge' - test "$first_parent" = "$codex_oid" || - die 'the integration does not start at the published codex tip' + test "$first_parent" = "$lane_oid" || + die "the integration does not start at the published $lane tip" if test "$GITHUB_EVENT_NAME" = merge_group then associated=$(api --paginate \ "repos/$GITHUB_REPOSITORY/commits/$second_parent/pulls?per_page=100" \ --jq '.[] | select(.state == "open" and - .base.ref == "codex" and + .base.ref == "'"$lane"'" and .head.sha == "'"$second_parent"'" and .head.repo.full_name == "'"$GITHUB_REPOSITORY"'" and .draft == false) | @@ -154,10 +190,15 @@ jobs: case "${topic_name#??/codex/}" in */*) die "topic '$topic_name' contains another namespace component" ;; esac - case "$topic_name" in - *-unstable|*-wip|*-stale) + case "$lane:$topic_name" in + codex:*-unstable|codex:*-wip|codex:*-stale) die "topic '$topic_name' is not eligible for production codex" ;; + codex:*) ;; + codex-unstable:*-unstable) ;; + codex-unstable:*) + die "topic '$topic_name' is not eligible for codex-unstable" + ;; esac current_topic_oid=$(api \ @@ -172,9 +213,9 @@ jobs: .github/workflows/codex-release.yml do published_blob=$(api \ - "repos/$GITHUB_REPOSITORY/contents/$protected_path?ref=$codex_oid" \ + "repos/$GITHUB_REPOSITORY/contents/$protected_path?ref=$lane_oid" \ --jq '.sha') || - die "published codex is missing '$protected_path'" + die "published $lane is missing '$protected_path'" candidate_blob=$(api \ "repos/$GITHUB_REPOSITORY/contents/$protected_path?ref=$GITHUB_SHA" \ --jq '.sha') || @@ -184,7 +225,7 @@ jobs: done published_workflows=$(api \ - "repos/$GITHUB_REPOSITORY/contents/.github/workflows?ref=$codex_oid" \ + "repos/$GITHUB_REPOSITORY/contents/.github/workflows?ref=$lane_oid" \ --jq '[.[] | select(.type == "file" and (.name | test("\\.ya?ml$"))) | [.name, .sha]] | sort | @json') || @@ -199,7 +240,7 @@ jobs: die 'the candidate adds, removes, or modifies a GitHub Actions workflow' automation=$(api \ - "repos/$GITHUB_REPOSITORY/contents/.github/workflows/codex.yml?ref=$codex_oid" \ + "repos/$GITHUB_REPOSITORY/contents/.github/workflows/codex.yml?ref=$lane_oid" \ -H 'Accept: application/vnd.github.raw+json') || die 'could not read the published automation trampoline' grep -Fxq ' pull_request:' <<<"$automation" && @@ -213,7 +254,7 @@ jobs: die 'published codex does not contain the canonical admission trampoline' release=$(api \ - "repos/$GITHUB_REPOSITORY/contents/.github/workflows/codex-release.yml?ref=$codex_oid" \ + "repos/$GITHUB_REPOSITORY/contents/.github/workflows/codex-release.yml?ref=$lane_oid" \ -H 'Accept: application/vnd.github.raw+json') || die 'could not read the published release workflow' grep -Fxq ' publication:' <<<"$release" && @@ -242,7 +283,10 @@ jobs: git config --no-includes --file /dev/stdin \ --get "branch.$other_name.codex-tip" >/dev/null then - continue + case "$lane:$other_name" in + codex:*-unstable) ;; + *) continue ;; + esac fi shared_base=$(api \ @@ -251,6 +295,7 @@ jobs: die "could not inspect prerequisite '$other_name'" test -n "$shared_base" || die "prerequisite '$other_name' has no common base" + test "$shared_base" != "$topic_oid" || continue published_relationship=$(api \ "repos/$GITHUB_REPOSITORY/compare/$published_oid...$shared_base" \ @@ -273,12 +318,12 @@ jobs: esac done <<<"$topics" - current_codex_oid=$(api \ - "repos/$GITHUB_REPOSITORY/git/ref/heads/codex" \ + current_lane_oid=$(api \ + "repos/$GITHUB_REPOSITORY/git/ref/heads/$lane" \ --jq '.object.sha') || - die 'could not recheck the codex branch' - test "$current_codex_oid" = "$codex_oid" || - die 'codex changed while its topic admission was being verified' + die "could not recheck the $lane branch" + test "$current_lane_oid" = "$lane_oid" || + die "$lane changed while its topic admission was being verified" printf 'Approved pull request #%s: %s at %s\n' \ "$pull_number" "$topic_name" "$topic_oid" diff --git a/.github/workflows/codex-branch.sh b/.github/workflows/codex-branch.sh index 795a01f39b6af8..4ac68c242fe26f 100755 --- a/.github/workflows/codex-branch.sh +++ b/.github/workflows/codex-branch.sh @@ -26,21 +26,25 @@ usage () { cat <<-\EOF usage: codex-branch check-topic or: codex-branch rebuild [--local] + [--enable-unstable | --disable-unstable] or: codex-branch publish or: codex-branch initialize [--remote ] [--base ] [--codex ] [--output ] [--require-automation] or: codex-branch refresh [--session ] [--remote ] [--base ] [--codex ] [--rerere-from ] [--require-automation] + [--enable-unstable | --disable-unstable] or: codex-branch rewrite [--remote ] [--base ] [--codex ] [--rerere-from ] [--result ] [--updates ] [--inputs ] [--bundle ] [--failure ] [--worktree ] [--require-automation] + [--enable-unstable | --disable-unstable] or: codex-branch verify-inputs [--remote ] [--base ] [--codex ] or: codex-branch verify-output --inputs --updates --result [--require-automation] + [--stable-recovery] or: codex-branch stage [--remote ] [--staging ] --inputs --updates [--require-automation] or: codex-branch promote [--remote ] [--staging ] @@ -177,6 +181,40 @@ is_stable_topic_name () ( esac ) +is_unstable_topic_name () ( + is_active_topic_name "$1" || return 1 + case "$1" in + *-unstable) ;; + *) return 1 ;; + esac +) + +null_oid () { + git hash-object --stdin &2 @@ -208,6 +246,7 @@ legacy_control_paths_unchanged () ( .github/rulesets/codex-branch.json \ .github/rulesets/codex-meta.json \ .github/rulesets/codex-topics.json \ + .github/rulesets/codex-unstable-branch.json \ .github/workflows/codex-admission.yml \ .github/workflows/codex-topic.yml \ .github/workflows/codex.yml \ @@ -227,6 +266,7 @@ meta_control_paths_unchanged () ( .github/rulesets/codex-branch.json \ .github/rulesets/codex-meta.json \ .github/rulesets/codex-topics.json \ + .github/rulesets/codex-unstable-branch.json \ .github/workflows/codex-admission.yml \ .github/workflows/codex-topic.yml \ .github/workflows/codex-branch.sh \ @@ -247,6 +287,59 @@ write_automation_workflow () { cat <<-'EOF' name: Refresh codex + on: + workflow_dispatch: + pull_request: + branches: + - codex + - codex-unstable + types: + - opened + - reopened + - synchronize + - ready_for_review + merge_group: + types: + - checks_requested + + permissions: + actions: read + contents: read + pull-requests: read + + jobs: + refresh: + if: github.event_name == 'workflow_dispatch' + uses: openai/git/.github/workflows/codex.yml@meta + admission: + name: Codex admission + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex') + permissions: + contents: read + pull-requests: read + uses: openai/git/.github/workflows/codex-admission.yml@meta + unstable_admission: + name: Codex unstable admission + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex-unstable') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex-unstable') + permissions: + contents: read + pull-requests: read + uses: openai/git/.github/workflows/codex-admission.yml@meta + EOF +} + +write_stable_automation_workflow () { + cat <<-'EOF' + name: Refresh codex + on: workflow_dispatch: pull_request: @@ -306,12 +399,23 @@ automation_workflow_is_current () { "$tmp_dir/actual-automation.yml" } -automation_workflow_matches () { +automation_workflow_is_reviewed () { head_oid=$1 if automation_workflow_is_current "$head_oid" then return 0 fi + write_stable_automation_workflow >"$tmp_dir/expected-automation.yml" + cmp -s "$tmp_dir/expected-automation.yml" \ + "$tmp_dir/actual-automation.yml" +} + +automation_workflow_matches () { + head_oid=$1 + if automation_workflow_is_reviewed "$head_oid" + then + return 0 + fi write_legacy_automation_workflow >"$tmp_dir/expected-automation.yml" cmp -s "$tmp_dir/expected-automation.yml" \ "$tmp_dir/actual-automation.yml" @@ -417,6 +521,7 @@ authenticate_pending_codex_merge () ( current=$3 output=$4 snapshot_head=${5:-} + lane=${6:-codex} state=$output.state mkdir -p "$state" || die "could not prepare Codex admission verification" @@ -458,10 +563,10 @@ authenticate_pending_codex_merge () ( then die "could not authenticate the merged Codex pull request" fi - awk -F '\t' -v merge="$merge" -v head="$head" ' + awk -F '\t' -v merge="$merge" -v head="$head" -v base="$lane" ' NF == 11 && $1 ~ /^[0-9]+$/ && $2 == "closed" && $3 != "-" && $4 == merge && $5 == "openai/git" && - $6 == "codex" && $7 == "openai/git" && $9 == head && + $6 == base && $7 == "openai/git" && $9 == head && $10 == "false" && $11 != "-" { print } ' "$state/pull-requests" >"$state/matching-pull-requests" || die "could not authenticate the merged Codex pull request" @@ -471,8 +576,17 @@ authenticate_pending_codex_merge () ( base_repository base_name head_repository name head_oid draft author \ <"$state/matching-pull-requests" || die "could not authenticate the merged Codex pull request" - is_stable_topic_name "$name" || - die "could not authenticate the merged Codex pull request" + case "$lane" in + codex) + is_stable_topic_name "$name" || + die "could not authenticate the merged Codex pull request" + ;; + codex-unstable) + is_unstable_topic_name "$name" || + die "could not authenticate the merged unstable Codex pull request" + ;; + *) die "cannot authenticate unknown Codex output '$lane'" ;; + esac if test "$remote" = - then current_head=$snapshot_head @@ -518,6 +632,7 @@ collect_topics () ( published_state=$3 codex_oid=$4 base_oid=$5 + lane=${6:-codex} root=refs/remotes/$remote/ admission=$output.admission : >"$admission" @@ -538,12 +653,26 @@ collect_topics () ( exit 0 fi - published=$published_state/published-topics - published_codex=$(state_value "$published_state" published-codex-oid) + if test "$lane" = codex-unstable + then + published=$published_state/published-unstable-topics + if test "$(state_value "$published_state" config-version)" = 2 + then + published_codex=$(state_value "$published_state" \ + published-unstable-oid) + else + published_codex=$(null_oid) + fi + else + published=$published_state/published-topics + published_codex=$(state_value "$published_state" published-codex-oid) + fi if test "$published_codex" != "$codex_oid" then + ! is_null_oid "$published_codex" && ! is_null_oid "$codex_oid" || + die "the '$lane' output does not match its published state" authenticate_pending_codex_merge "$remote" "$published_codex" \ - "$codex_oid" "$admission" + "$codex_oid" "$admission" '' "$lane" fi : >"$output.unsorted" @@ -569,11 +698,14 @@ collect_topics () ( fi LC_ALL=C sort "$output.unsorted" >"$output" || die "could not sort enrolled Codex topics" - test -s "$output" || + test "$lane" = codex-unstable || test -s "$output" || die "all enrolled Codex topics were removed" - reject_unadmitted_topic_history "$remote" "$output" \ - "$published_codex" "$base_oid" + if test -s "$output" + then + reject_unadmitted_topic_history "$remote" "$output" \ + "$published_codex" "$base_oid" "$lane" + fi ) reject_unadmitted_topic_history () ( @@ -581,6 +713,7 @@ reject_unadmitted_topic_history () ( topics=$2 published_codex=$3 base_oid=$4 + lane=${5:-codex} root=refs/remotes/$remote/ state=$topics.unadmitted-history mkdir -p "$state" || @@ -603,6 +736,10 @@ reject_unadmitted_topic_history () ( while IFS="$tab" read -r selected selected_oid do test "$oid" = "$selected_oid" && continue + if git merge-base --is-ancestor "$selected_oid" "$oid" + then + continue + fi git merge-base --all "$oid" "$selected_oid" \ >"$state/merge-bases" || die "could not compare unadmitted Codex topic '$name' with '$selected'" @@ -633,11 +770,26 @@ write_input_snapshot () ( topics=$6 output=$7 admission=$8 + unstable_topics=${9:-} + shift 9 || : + unstable_oid=${1:-} + unstable_admission=${2:-} + lane_mode=${3:-} { printf 'controller\trefs/heads/meta\t%s\n' "$controller_oid" printf 'base\trefs/heads/%s\t%s\n' "$base_name" "$base_oid" printf 'codex\trefs/heads/%s\t%s\n' "$codex_name" "$codex_oid" + if test -n "$lane_mode" + then + printf 'lane-mode\trefs/heads/codex-unstable\t%s\n' \ + "$lane_mode" + fi + if test -n "$unstable_oid" + then + printf 'unstable\trefs/heads/codex-unstable\t%s\n' \ + "$unstable_oid" + fi if test -s "$admission" then IFS="$tab" read -r name oid number merge <"$admission" || @@ -645,10 +797,25 @@ write_input_snapshot () ( printf 'admission\trefs/heads/%s\t%s\t%s\t%s\n' \ "$name" "$oid" "$number" "$merge" fi + if test -n "$unstable_admission" && test -s "$unstable_admission" + then + IFS="$tab" read -r name oid number merge <"$unstable_admission" || + die "could not inspect the reviewed unstable Codex admission" + printf 'unstable-admission\trefs/heads/%s\t%s\t%s\t%s\n' \ + "$name" "$oid" "$number" "$merge" + fi while IFS="$tab" read -r name oid do printf 'topic\trefs/heads/%s\t%s\n' "$name" "$oid" done <"$topics" + if test -n "$unstable_topics" + then + while IFS="$tab" read -r name oid + do + printf 'unstable-topic\trefs/heads/%s\t%s\n' \ + "$name" "$oid" + done <"$unstable_topics" + fi } >"$output" ) @@ -659,6 +826,7 @@ snapshot_inputs () { output=$4 topics_output=$5 controller_oid=${6:-${CODEX_CONTROLLER_OID:-}} + lane_mode=${7:-} remote_controller_oid=$(resolve_commit "$(remote_ref "$remote" meta)") if test -n "$controller_oid" @@ -682,9 +850,59 @@ snapshot_inputs () { fi collect_topics "$remote" "$topics_output" "$published_state" \ "$codex_oid" "$base_oid" + unstable_topics= + unstable_oid= + unstable_admission= + if test -n "$published_state" + then + version=$(state_value "$published_state" config-version) + case "$lane_mode:$version" in + enable:1|disable:2|:1|:2) ;; + enable:2) die "codex-unstable is already enabled" ;; + disable:1) die "codex-unstable is not enabled" ;; + *) die "invalid unstable-lane mode '$lane_mode'" ;; + esac + if test "$version" = 1 && test "$lane_mode" != enable && + git rev-parse --verify \ + "$(remote_ref "$remote" codex-unstable)^{commit}" \ + >/dev/null 2>&1 + then + die "$meta_config_path does not describe the existing codex-unstable output" + fi + if test "$version" = 2 || test "$lane_mode" = enable + then + unstable_topics=$topics_output.unstable + unstable_oid=$(git rev-parse --verify \ + "$(remote_ref "$remote" codex-unstable)^{commit}" \ + 2>/dev/null || :) + if test "$version" = 1 + then + test -z "$unstable_oid" || + die "codex-unstable already exists outside its published state" + unstable_oid=$(null_oid) + : >"$unstable_topics" + : >"$unstable_topics.admission" + else + test -n "$unstable_oid" || + die "$meta_config_path records codex-unstable, but its output disappeared" + collect_topics "$remote" "$unstable_topics" \ + "$published_state" "$unstable_oid" "$codex_oid" \ + codex-unstable + fi + unstable_admission=$unstable_topics.admission + if test "$lane_mode" = disable + then + ! test -s "$unstable_topics" || + die "cannot disable codex-unstable while enrolled topics remain" + ! test -s "$unstable_admission" || + die "cannot disable codex-unstable with a pending pull-request merge" + fi + fi + fi write_input_snapshot "$controller_oid" "$base_name" "$base_oid" \ "$codex_name" "$codex_oid" "$topics_output" "$output" \ - "$topics_output.admission" + "$topics_output.admission" "$unstable_topics" "$unstable_oid" \ + "$unstable_admission" "$lane_mode" } input_oid () { @@ -795,14 +1013,34 @@ write_meta_config () ( codex_oid=$4 rows=$5 output=$6 + unstable_rows=${7:-} + unstable_base=${8:-} + unstable_output=${9:-} { printf '[codex]\n' - printf '\tversion = 1\n' + if test -n "$unstable_rows" + then + printf '\tversion = 2\n' + else + printf '\tversion = 1\n' + fi printf '\tbase-ref = refs/heads/%s\n' "$base_name" printf '\tbase-tip = %s\n' "$base_oid" printf '\toutput-ref = refs/heads/%s\n' "$codex_name" printf '\toutput-tip = %s\n' "$codex_oid" + if test -n "$unstable_rows" + then + printf '\n[codex-unstable]\n' + printf '\tbase-ref = refs/heads/%s\n' "$codex_name" + printf '\tbase-tip = %s\n' "$unstable_base" + printf '\toutput-ref = refs/heads/codex-unstable\n' + printf '\toutput-tip = %s\n' "$unstable_output" + cat "$rows" "$unstable_rows" | LC_ALL=C sort \ + >"$output.rows" || + die "could not sort stable and unstable topic state" + rows=$output.rows + fi while IFS="$tab" read -r name tip prerequisite do quoted=$(config_subsection_quote "$name") @@ -812,6 +1050,7 @@ write_meta_config () ( printf '\tcodex-tip = %s\n' "$tip" done <"$rows" } >"$output" || die "could not write $meta_config_path" + test -z "$unstable_rows" || rm -f "$output.rows" ) config_get_one () ( @@ -845,11 +1084,12 @@ read_meta_config () ( state=$4 config=$state/published-config rows=$state/published-topics + unstable_rows=$state/published-unstable-topics git show "$controller_oid:$meta_config_path" >"$config" 2>/dev/null || die "meta has no $meta_config_path; run Meta/codex initialize before refreshing" version=$(config_get_one "$config" codex.version) - test "$version" = 1 || + test "$version" = 1 || test "$version" = 2 || die "$meta_config_path has unsupported version '$version'" base_ref=$(config_get_one "$config" codex.base-ref) base_tip=$(config_get_one "$config" codex.base-tip) @@ -861,37 +1101,84 @@ read_meta_config () ( die "$meta_config_path records output '$output_ref', not refs/heads/$codex_name" require_full_commit_oid "$base_tip" require_full_commit_oid "$output_tip" + if test "$version" = 2 + then + unstable_base_ref=$(config_get_one "$config" \ + codex-unstable.base-ref) + unstable_base_tip=$(config_get_one "$config" \ + codex-unstable.base-tip) + unstable_output_ref=$(config_get_one "$config" \ + codex-unstable.output-ref) + unstable_output_tip=$(config_get_one "$config" \ + codex-unstable.output-tip) + test "$unstable_base_ref" = "refs/heads/$codex_name" || + die "$meta_config_path records unstable base '$unstable_base_ref', not refs/heads/$codex_name" + test "$unstable_output_ref" = refs/heads/codex-unstable || + die "$meta_config_path records invalid unstable output '$unstable_output_ref'" + require_full_commit_oid "$unstable_base_tip" + require_full_commit_oid "$unstable_output_tip" + test "$unstable_base_tip" = "$output_tip" || + die "$meta_config_path unstable base does not match its published codex output" + git merge-base --is-ancestor "$unstable_base_tip" \ + "$unstable_output_tip" || + die "$meta_config_path unstable output is not based on codex" + test "$unstable_base_tip" != "$unstable_output_tip" || + die "$meta_config_path unstable output is not strictly ahead of codex" + fi : >"$rows" + : >"$unstable_rows" git config --no-includes --file "$config" --get-regexp \ '^branch\..*\.codex-tip$' >"$state/published-tip-keys" || : while IFS=' ' read -r key tip do name=${key#branch.} name=${name%.codex-tip} - is_stable_topic_name "$name" || + is_active_topic_name "$name" || die "$meta_config_path records invalid topic '$name'" remote=$(config_get_one "$config" "branch.$name.remote") merge=$(config_get_one "$config" "branch.$name.merge") test "$remote" = . || die "$meta_config_path gives '$name' non-local remote '$remote'" - case "$merge" in - "refs/heads/$base_name") prerequisite=$base_name ;; - refs/heads/*) - prerequisite=${merge#refs/heads/} - is_stable_topic_name "$prerequisite" || - die "$meta_config_path gives '$name' invalid prerequisite '$merge'" - ;; - *) die "$meta_config_path gives '$name' invalid prerequisite '$merge'" ;; - esac + if is_unstable_topic_name "$name" + then + test "$version" = 2 || + die "$meta_config_path version 1 records unstable topic '$name'" + case "$merge" in + "refs/heads/$codex_name") prerequisite=$codex_name ;; + refs/heads/*) + prerequisite=${merge#refs/heads/} + is_unstable_topic_name "$prerequisite" || + die "$meta_config_path gives unstable topic '$name' invalid prerequisite '$merge'" + ;; + *) die "$meta_config_path gives '$name' invalid prerequisite '$merge'" ;; + esac + target_rows=$unstable_rows + else + case "$merge" in + "refs/heads/$base_name") prerequisite=$base_name ;; + refs/heads/*) + prerequisite=${merge#refs/heads/} + is_stable_topic_name "$prerequisite" || + die "$meta_config_path gives '$name' invalid prerequisite '$merge'" + ;; + *) die "$meta_config_path gives '$name' invalid prerequisite '$merge'" ;; + esac + target_rows=$rows + fi require_full_commit_oid "$tip" - printf '%s\t%s\t%s\n' "$name" "$tip" "$prerequisite" >>"$rows" || + printf '%s\t%s\t%s\n' "$name" "$tip" "$prerequisite" \ + >>"$target_rows" || die "could not read '$name' from $meta_config_path" done <"$state/published-tip-keys" LC_ALL=C sort -o "$rows" "$rows" + LC_ALL=C sort -o "$unstable_rows" "$unstable_rows" test "$(cut -f1 "$rows" | sort -u | wc -l | tr -d ' ')" = \ "$(wc -l <"$rows" | tr -d ' ')" || die "$meta_config_path records a topic more than once" + test "$(cut -f1 "$unstable_rows" | sort -u | wc -l | tr -d ' ')" = \ + "$(wc -l <"$unstable_rows" | tr -d ' ')" || + die "$meta_config_path records an unstable topic more than once" while IFS="$tab" read -r name tip prerequisite do @@ -914,15 +1201,62 @@ read_meta_config () ( git merge-base --is-ancestor "$tip" "$output_tip" || die "$meta_config_path output does not contain published topic '$name'" done <"$rows" + if test "$version" = 2 + then + if ! test -s "$unstable_rows" + then + unstable_sentinel_is_canonical "$unstable_base_tip" \ + "$unstable_output_tip" || + die "$meta_config_path empty unstable output is not its canonical sentinel" + fi + while IFS="$tab" read -r name tip prerequisite + do + test "$name" != "$prerequisite" || + die "$meta_config_path makes '$name' its own prerequisite" + if published_depends_on "$unstable_rows" "$prerequisite" \ + "$name" + then + die "$meta_config_path contains an unstable dependency cycle through '$name'" + fi + if test "$prerequisite" = "$codex_name" + then + prerequisite_tip=$unstable_base_tip + else + prerequisite_tip=$(published_tip "$unstable_rows" \ + "$prerequisite") + test -n "$prerequisite_tip" || + die "$meta_config_path records missing prerequisite '$prerequisite' for '$name'" + fi + git merge-base --is-ancestor "$prerequisite_tip" "$tip" || + die "$meta_config_path boundary for unstable topic '$name' is not in its published history" + git merge-base --is-ancestor "$tip" "$unstable_output_tip" || + die "$meta_config_path output does not contain published unstable topic '$name'" + done <"$unstable_rows" + fi git merge-base --is-ancestor "$base_tip" "$output_tip" || die "$meta_config_path output is not based on its recorded base" - write_meta_config "$base_name" "$base_tip" "$codex_name" "$output_tip" \ - "$rows" "$state/canonical-published-config" + if test "$version" = 2 + then + write_meta_config "$base_name" "$base_tip" "$codex_name" \ + "$output_tip" "$rows" "$state/canonical-published-config" \ + "$unstable_rows" "$unstable_base_tip" "$unstable_output_tip" + else + write_meta_config "$base_name" "$base_tip" "$codex_name" \ + "$output_tip" "$rows" "$state/canonical-published-config" + fi cmp -s "$config" "$state/canonical-published-config" || die "$meta_config_path is not in canonical form" + printf '%s\n' "$version" >"$state/config-version" printf '%s\n' "$base_tip" >"$state/published-base-oid" printf '%s\n' "$output_tip" >"$state/published-codex-oid" + if test "$version" = 2 + then + printf '%s\n' "$unstable_base_tip" \ + >"$state/published-unstable-base-oid" + printf '%s\n' "$unstable_output_tip" \ + >"$state/published-unstable-oid" + fi ) topic_contains_commit () ( @@ -1453,13 +1787,16 @@ prepare_stateful_plan () ( { test -n "$published_parent_is_upstream" || ! git merge-base --is-ancestor \ "$published_prerequisite_tip" "$current_tip"; } && - git merge-base --is-ancestor "$base_oid" "$current_tip" + { git merge-base --is-ancestor "$base_oid" "$current_tip" || + git merge-base --is-ancestor "$published_base" \ + "$current_tip"; } then # With the published prerequisite absent, a root history is - # an explicit restack onto master. + # an explicit restack onto the current or published base. prerequisite=$base_name prerequisite_tip=$base_oid - old_base=$base_oid + old_base=$(root_replay_boundary "$name" \ + "$published_base" "$base_oid" "$current_tip" "$state") reparented=t elif test "$old_prerequisite" = "$base_name" || test -n "$old_prerequisite_tip" @@ -1473,9 +1810,17 @@ prepare_stateful_plan () ( if test "$reparented" = t then - git merge-base --is-ancestor "$prerequisite_tip" \ - "$current_tip" || - die "new prerequisite '$prerequisite' is not in '$name'" + if ! git merge-base --is-ancestor "$prerequisite_tip" \ + "$current_tip" + then + test "$prerequisite" = "$base_name" && + test -n "$old_base" && + git merge-base --is-ancestor "$old_base" \ + "$current_tip" && + git merge-base --is-ancestor "$old_base" \ + "$prerequisite_tip" || + die "new prerequisite '$prerequisite' is not in '$name'" + fi test -n "${old_base:-}" || old_base=$prerequisite_tip else old_base=$(choose_replay_boundary "$name" \ @@ -1787,6 +2132,19 @@ write_complete_updates () { printf 'refs/heads/%s\t%s\t%s\n' \ "$codex_name" "$codex_oid" "$candidate" >>"$output.unsorted" || die "could not record the codex update" + if test -f "$state/unstable-output-oid" + then + unstable_old=$(state_value "$state" unstable-oid) + unstable_new=$(state_value "$state" unstable-output-oid) + printf 'refs/heads/codex-unstable\t%s\t%s\n' \ + "$unstable_old" "$unstable_new" >>"$output.unsorted" || + die "could not record the codex-unstable update" + if test -f "$state/unstable/topic-updates" + then + cat "$state/unstable/topic-updates" >>"$output.unsorted" || + die "could not record unstable topic updates" + fi + fi LC_ALL=C sort "$output.unsorted" >"$output" || die "could not sort the complete update manifest" rm -f "$output.unsorted" @@ -1810,10 +2168,38 @@ write_next_meta_config () ( die "could not record next state for '$name'" done <"$state/topics" LC_ALL=C sort -o "$rows" "$rows" - write_meta_config "$(state_value "$state" base-name)" \ - "$(state_value "$state" base-oid)" \ - "$(state_value "$state" codex-name)" "$candidate" \ - "$rows" "$state/next-meta-config" + if test -f "$state/unstable-output-oid" && + ! is_null_oid "$(state_value "$state" unstable-output-oid)" + then + unstable_state=$state/unstable + unstable_rows=$state/next-published-unstable-topics + : >"$unstable_rows" + while IFS="$tab" read -r name old + do + new=$(result_lookup "$unstable_state/results" "$name") + test -n "$new" || + die "unstable topic '$name' has no rewritten tip" + prerequisite=$(awk -F '\t' -v name="$name" \ + '$1 == name { value=$3 } END { if (value != "") print value }' \ + "$unstable_state/plan") + test -n "$prerequisite" || + die "unstable topic '$name' has no recorded prerequisite" + printf '%s\t%s\t%s\n' "$name" "$new" "$prerequisite" \ + >>"$unstable_rows" || + die "could not record next state for unstable topic '$name'" + done <"$unstable_state/topics" + LC_ALL=C sort -o "$unstable_rows" "$unstable_rows" + write_meta_config "$(state_value "$state" base-name)" \ + "$(state_value "$state" base-oid)" \ + "$(state_value "$state" codex-name)" "$candidate" \ + "$rows" "$state/next-meta-config" "$unstable_rows" \ + "$candidate" "$(state_value "$state" unstable-output-oid)" + else + write_meta_config "$(state_value "$state" base-name)" \ + "$(state_value "$state" base-oid)" \ + "$(state_value "$state" codex-name)" "$candidate" \ + "$rows" "$state/next-meta-config" + fi ) create_meta_commit () ( @@ -1981,6 +2367,38 @@ write_failure () { } >"$path" } +write_unstable_failure () { + path=$1 + state=$2 + worktree=$3 + test -n "$path" || return 0 + + failed_owner=$(state_value "$state" failed-owner) + failed_commit=$(state_value "$state" failed-commit) + { + say "## No refs were updated" + say + say "Rebasing unstable topic \`$failed_owner\` stopped while applying \`$failed_commit\`." + say "Neither \`codex\`, \`codex-unstable\`, \`meta\`, nor a topic branch was updated." + say + say "Resolve this experimental conflict manually: restack the affected" + say "unstable topic and its descendants onto their current prerequisite," + say "publish the coherent topic graph in one exact-lease atomic push," + say "and run \`Meta/rebuild\` again." + say "The production-only \`resolve\`/\`continue\` recovery commands do not" + say "reconstruct this nested unstable integration lane." + if test -n "$(git -C "$worktree" -c core.fsmonitor=false \ + diff --name-only --diff-filter=U)" + then + say + say "Conflicted paths:" + git -C "$worktree" -c core.fsmonitor=false \ + diff --name-only --diff-filter=U | + sed 's/^/- `/' | sed 's/$/`/' + fi + } >"$path" +} + write_integration_failure () { path=$1 state=$2 @@ -2024,10 +2442,11 @@ merge_topic () { worktree=$1 name=$2 oid=$3 + output_name=${4:-codex} before=$(git -C "$worktree" rev-parse HEAD) || die "could not resolve the candidate before integrating '$name'" - message=$(printf 'Merge %s into codex\n\nIntegrate the current %s topic into the internally distributed codex branch.\n\nCodex-Integration: %s@%s' \ - "$name" "$name" "$name" "$oid") + message=$(printf 'Merge %s into %s\n\nIntegrate the current %s topic into the internally distributed %s branch.\n\nCodex-Integration: %s@%s' \ + "$name" "$output_name" "$name" "$output_name" "$name" "$oid") if git -C "$worktree" merge-base --is-ancestor "$oid" "$before" then @@ -2035,7 +2454,8 @@ merge_topic () { then tree=$(git -C "$worktree" rev-parse "$before^{tree}") || die "could not resolve the integration base tree" - anchor_message=$(printf 'Begin codex integration\n\nCreate a distinct first parent for explicit topic integration commits.\n') + anchor_message=$(printf 'Begin %s integration\n\nCreate a distinct first parent for explicit topic integration commits.\n' \ + "$output_name") anchor=$(printf '%s' "$anchor_message" | \ GIT_AUTHOR_NAME=$bot_name GIT_AUTHOR_EMAIL=$bot_email \ GIT_COMMITTER_NAME=$bot_name \ @@ -2142,13 +2562,14 @@ codex_has_expected_integrations () ( state=$1 head_oid=$2 base_oid=$(state_value "$state" base-oid) + output_name=$(state_value "$state" codex-name) expected=$state/expected-integrations actual=$state/actual-integrations : >"$expected" while IFS="$tab" read -r name oid do - printf '%s@%s\t%s\tMerge %s into codex\t%s\t%s\t%s\t%s\n' \ - "$name" "$oid" "$oid" "$name" \ + printf '%s@%s\t%s\tMerge %s into %s\t%s\t%s\t%s\t%s\n' \ + "$name" "$oid" "$oid" "$name" "$output_name" \ "$bot_name" "$bot_email" "$bot_name" "$bot_email" \ >>"$expected" || return 1 done <"$state/integration-topics" @@ -2177,6 +2598,7 @@ assemble_candidate () { worktree=$1 state=$2 base_oid=$(state_value "$state" base-oid) + output_name=$(state_value "$state" codex-name) rm -f "$state/integration-failed-name" \ "$state/integration-failed-oid" || die "could not clear old integration state" @@ -2187,10 +2609,10 @@ assemble_candidate () { git -C "$worktree" -c core.fsmonitor=false \ -c advice.detachedHead=false switch --detach "$base_oid" >/dev/null || - die "could not check out master while assembling codex" + die "could not check out the base while assembling $output_name" while IFS="$tab" read -r name oid do - if ! merge_topic "$worktree" "$name" "$oid" + if ! merge_topic "$worktree" "$name" "$oid" "$output_name" then printf '%s\n' "$name" >"$state/integration-failed-name" || die "could not record the conflicting integration topic" @@ -2212,7 +2634,8 @@ assemble_candidate () { codex_has_expected_integrations "$state" "$candidate" || die "candidate does not contain one canonical integration merge per topic" require_automation=$(state_value "$state" require-automation) - if ! test -f "$state/initializing" + if ! test -f "$state/initializing" && + test "$output_name" != codex-unstable then verify_control_paths "$state/inputs" "$state/topic-updates" \ "$candidate" "$state" "$require_automation" @@ -2233,6 +2656,14 @@ create_bundle () { git update-ref refs/codex-output/candidate "$candidate" || die "could not retain the bundle candidate" set -- git bundle create "$bundle" refs/codex-output/candidate + if test -f "$state/unstable-output-oid" && + ! is_null_oid "$(state_value "$state" unstable-output-oid)" + then + git update-ref refs/codex-output/unstable \ + "$(state_value "$state" unstable-output-oid)" || + die "could not retain the unstable bundle candidate" + set -- "$@" refs/codex-output/unstable + fi if test "$meta_oid" != "$controller_oid" then git update-ref refs/codex-output/meta "$meta_oid" || @@ -2264,14 +2695,44 @@ create_bundle () { if ! "$@" then git update-ref -d refs/codex-output/candidate || : + git update-ref -d refs/codex-output/unstable || : git update-ref -d refs/codex-output/meta || : die "could not create candidate bundle" fi git update-ref -d refs/codex-output/candidate || die "could not remove the temporary bundle ref" + git update-ref -d refs/codex-output/unstable || : git update-ref -d refs/codex-output/meta || : } +validate_lane_isolation () ( + state=$1 + stable_topics=$2 + unstable_topics=$3 + codex_oid=$4 + + cp "$unstable_topics" "$state/cross-lane-unstable-tips" || + die "could not prepare current unstable ownership checks" + awk -F '\t' '{ printf "%s\t%s\n", $1, $2 }' \ + "$state/published-unstable-topics" \ + >>"$state/cross-lane-unstable-tips" || + die "could not prepare published unstable ownership checks" + while IFS="$tab" read -r unstable_name unstable_tip + do + while IFS="$tab" read -r stable_name stable_tip + do + git merge-base --all "$unstable_tip" "$stable_tip" \ + >"$state/cross-lane-bases" || + die "could not compare stable topic '$stable_name' with unstable topic '$unstable_name'" + while read -r shared + do + git merge-base --is-ancestor "$shared" "$codex_oid" || + die "stable topic '$stable_name' contains private commits from unstable topic '$unstable_name'" + done <"$state/cross-lane-bases" + done <"$stable_topics" + done <"$state/cross-lane-unstable-tips" +) + initialize_rewrite () { remote=$1 base_name=$2 @@ -2286,22 +2747,40 @@ initialize_rewrite () { mkdir -p "$state" cp "$inputs" "$state/inputs" cp "$topics" "$state/topics" + if test -f "$topics.unstable" + then + cp "$topics.unstable" "$state/unstable-topics" || + die "could not retain unstable topic inputs" + else + : >"$state/unstable-topics" + fi controller_oid=$(awk -F '\t' '$1 == "controller" { print $3 }' "$inputs") base_oid=$(awk -F '\t' '$1 == "base" { print $3 }' "$inputs") base_ref=$(awk -F '\t' '$1 == "base" { print $2 }' "$inputs") base_name=${base_ref#refs/heads/} codex_oid=$(awk -F '\t' '$1 == "codex" { print $3 }' "$inputs") + unstable_oid=$(awk -F '\t' '$1 == "unstable" { print $3 }' "$inputs") + lane_mode=$(awk -F '\t' '$1 == "lane-mode" { print $3 }' "$inputs") printf '%s\n' "$controller_oid" >"$state/controller-oid" printf '%s\n' "$remote" >"$state/remote" printf '%s\n' "$base_name" >"$state/base-name" printf '%s\n' "$base_oid" >"$state/base-oid" printf '%s\n' "$codex_name" >"$state/codex-name" printf '%s\n' "$codex_oid" >"$state/codex-oid" + test -z "$unstable_oid" || + printf '%s\n' "$unstable_oid" >"$state/unstable-oid" + printf '%s\n' "$lane_mode" >"$state/unstable-mode" printf '%s\n' "$rerere_name" >"$state/rerere-name" printf '%s\n' "$require_automation" >"$state/require-automation" printf '%s\n' "$script_path" >"$state/helper" read_meta_config "$controller_oid" "$base_name" "$codex_name" "$state" + if test -s "$state/published-topics" && ! test -s "$state/topics" + then + die "all previously published production topics disappeared" + fi + validate_lane_isolation "$state" "$state/topics" \ + "$state/unstable-topics" "$codex_oid" published_codex_oid=$(state_value "$state" published-codex-oid) validate_live_codex_delta "$published_codex_oid" "$codex_oid" \ "$state/topics" "$state" @@ -2312,6 +2791,155 @@ initialize_rewrite () { fi } +create_unstable_sentinel () ( + base=$1 + tree=$(git rev-parse "$base^{tree}") || + die "could not resolve the codex-unstable sentinel tree" + printf 'Initialize codex-unstable\n' | + GIT_AUTHOR_NAME=$bot_name GIT_AUTHOR_EMAIL=$bot_email \ + GIT_COMMITTER_NAME=$bot_name GIT_COMMITTER_EMAIL=$bot_email \ + git -c commit.gpgSign=false commit-tree "$tree" -p "$base" || + die "could not create the empty codex-unstable sentinel" +) + +prepare_unstable_candidate () ( + worktree=$1 + state=$2 + root_state=$state + stable_candidate=$3 + failure_file=$4 + topics=$state/unstable-topics + unstable_old=$(awk -F '\t' '$1 == "unstable" { print $3 }' \ + "$state/inputs") + mode=$(state_value "$state" unstable-mode) + version=$(state_value "$state" config-version) + + if ! test -s "$topics" + then + if test "$mode" = disable + then + test "$version" = 2 || + die "codex-unstable is not enabled" + test "$unstable_old" = \ + "$(state_value "$state" published-unstable-oid)" || + die "cannot disable codex-unstable after its published output changed" + printf '%s\n' "$(null_oid)" >"$state/unstable-output-oid" || + die "could not disable the empty codex-unstable output" + return 0 + fi + if test "$version" = 1 && test "$mode" != enable + then + return 0 + fi + test -n "$unstable_old" || + die "the enabled codex-unstable output was not snapshotted" + mkdir -p "$state/unstable" || + die "could not prepare empty codex-unstable state" + : >"$state/unstable/topics" + : >"$state/unstable/topic-updates" + if ! is_null_oid "$unstable_old" && + test "$(git show -s --format=%P "$unstable_old")" = \ + "$stable_candidate" && + test "$(git rev-parse "$unstable_old^{tree}")" = \ + "$(git rev-parse "$stable_candidate^{tree}")" + then + unstable_candidate=$unstable_old + else + unstable_candidate=$(create_unstable_sentinel \ + "$stable_candidate") + fi + printf '%s\n' "$unstable_candidate" \ + >"$state/unstable-output-oid" || + die "could not retain the empty codex-unstable sentinel" + return 0 + fi + test "$mode" != disable || + die "cannot disable codex-unstable while enrolled topics remain" + + test -n "$unstable_old" || + die "active unstable topics were not included in the input snapshot" + if test "$version" = 1 && ! is_null_oid "$unstable_old" + then + die "$meta_config_path does not describe the existing codex-unstable output" + fi + if test "$version" = 2 && is_null_oid "$unstable_old" + then + die "$meta_config_path records codex-unstable, but its output disappeared" + fi + + unstable_state=$state/unstable + mkdir -p "$unstable_state" || + die "could not prepare unstable reconstruction state" + cp "$state/inputs" "$unstable_state/inputs" || + die "could not pin unstable inputs" + cp "$topics" "$unstable_state/topics" || + die "could not pin unstable topics" + cp "$state/published-unstable-topics" \ + "$unstable_state/published-topics" || + die "could not retain published unstable topology" + printf '%s\n' "$(state_value "$state" controller-oid)" \ + >"$unstable_state/controller-oid" + printf '%s\n' "$(state_value "$state" remote)" \ + >"$unstable_state/remote" + printf '%s\n' codex >"$unstable_state/base-name" + printf '%s\n' "$stable_candidate" >"$unstable_state/base-oid" + printf '%s\n' codex-unstable >"$unstable_state/codex-name" + printf '%s\n' "$unstable_old" >"$unstable_state/codex-oid" + printf '%s\n' "$(state_value "$state" require-automation)" \ + >"$unstable_state/require-automation" + printf '%s\n' "$script_path" >"$unstable_state/helper" + if test "$version" = 2 + then + published_base=$(state_value "$state" published-unstable-base-oid) + published_output=$(state_value "$state" published-unstable-oid) + validate_live_codex_delta "$published_output" "$unstable_old" \ + "$unstable_state/topics" "$unstable_state" + else + published_base=$(state_value "$state" codex-oid) + published_output=$(null_oid) + fi + printf '%s\n' "$published_base" \ + >"$unstable_state/published-base-oid" + printf '%s\n' "$published_output" \ + >"$unstable_state/published-codex-oid" + prepare_stateful_plan codex "$stable_candidate" \ + "$unstable_state/topics" "$unstable_state" + if ! process_plan "$worktree" "$unstable_state" + then + write_unstable_failure "$failure_file" "$unstable_state" "$worktree" + die "conflict while rebasing unstable topic '$(state_value "$unstable_state" failed-owner)'; no refs were updated" + fi + if ! unstable_candidate=$(assemble_candidate "$worktree" \ + "$unstable_state") + then + if test -f "$unstable_state/integration-failed-name" + then + write_integration_failure "$failure_file" \ + "$unstable_state" "$worktree" + die "codex-unstable integration conflicts while merging '$(state_value "$unstable_state" integration-failed-name)'; no refs were updated" + fi + die "codex-unstable candidate validation failed; no refs were updated" + fi + git merge-base --is-ancestor "$stable_candidate" \ + "$unstable_candidate" || + die "codex-unstable candidate does not contain its exact codex base" + test "$stable_candidate" != "$unstable_candidate" || + die "codex-unstable candidate is not strictly ahead of codex" + verify_unstable_control_paths "$stable_candidate" \ + "$unstable_candidate" "$unstable_state" + if ! is_null_oid "$unstable_old" && + git merge-base --is-ancestor "$stable_candidate" "$unstable_old" && + test "$(git rev-parse "$unstable_candidate^{tree}")" = \ + "$(git rev-parse "$unstable_old^{tree}")" && + codex_has_expected_integrations "$unstable_state" "$unstable_old" + then + unstable_candidate=$unstable_old + fi + printf '%s\n' "$unstable_candidate" \ + >"$root_state/unstable-output-oid" || + die "could not retain the unstable output candidate" +) + initialize_config () { remote=origin base_name=master @@ -2493,6 +3121,7 @@ local_refresh () { rerere_name=codex session= require_automation= + unstable_mode= while test $# -gt 0 do case "$1" in @@ -2502,6 +3131,18 @@ local_refresh () { --codex) require_arg "$@"; codex_name=$2; shift 2 ;; --rerere-from) require_arg "$@"; rerere_name=$2; shift 2 ;; --require-automation) require_automation=t; shift ;; + --enable-unstable) + test -z "$unstable_mode" || + die "unstable lane mode was specified more than once" + unstable_mode=enable + shift + ;; + --disable-unstable) + test -z "$unstable_mode" || + die "unstable lane mode was specified more than once" + unstable_mode=disable + shift + ;; -h|--help) usage; exit 0 ;; *) die "unknown refresh option '$1'" ;; esac @@ -2531,6 +3172,7 @@ local_refresh () { --bundle "$session/codex.bundle" \ --failure "$session/codex-conflict.md" test -z "$require_automation" || set -- "$@" --require-automation + test -z "$unstable_mode" || set -- "$@" "--${unstable_mode}-unstable" fetch_heads "$remote" rewrite "$@" say "local refresh session: $session" @@ -2549,6 +3191,7 @@ rewrite () { failure_file= worktree= require_automation= + unstable_mode= while test $# -gt 0 do @@ -2564,6 +3207,18 @@ rewrite () { --failure) require_arg "$@"; failure_file=$2; shift 2 ;; --worktree) require_arg "$@"; worktree=$2; shift 2 ;; --require-automation) require_automation=t; shift ;; + --enable-unstable) + test -z "$unstable_mode" || + die "unstable lane mode was specified more than once" + unstable_mode=enable + shift + ;; + --disable-unstable) + test -z "$unstable_mode" || + die "unstable lane mode was specified more than once" + unstable_mode=disable + shift + ;; -h|--help) usage; exit 0 ;; *) die "unknown rewrite option '$1'" ;; esac @@ -2573,7 +3228,8 @@ rewrite () { require_full_repository inputs=$tmp_dir/inputs topics=$tmp_dir/topics - snapshot_inputs "$remote" "$base_name" "$codex_name" "$inputs" "$topics" + snapshot_inputs "$remote" "$base_name" "$codex_name" "$inputs" \ + "$topics" '' "$unstable_mode" test -z "$inputs_file" || cp "$inputs" "$inputs_file" base_oid=$(awk -F '\t' '$1 == "base" { print $3 }' "$inputs") @@ -2623,6 +3279,15 @@ rewrite () { then candidate=$codex_oid fi + if test -n "$(state_value "$state" unstable-mode)" + then + published_codex=$(state_value "$state" published-codex-oid) + test "$candidate" = "$codex_oid" && + test "$codex_oid" = "$published_codex" || + die "changing the codex-unstable lane requires a clean, unchanged production codex output" + fi + prepare_unstable_candidate "$worktree" "$state" "$candidate" \ + "$failure_file" create_meta_commit "$state" "$candidate" write_complete_updates "$state" "$candidate" "$tmp_dir/updates" @@ -2630,6 +3295,11 @@ rewrite () { test -z "$updates_file" || cp "$tmp_dir/updates" "$updates_file" test -z "$bundle_file" || create_bundle "$bundle_file" "$state" "$candidate" say "rewrote all active topics and assembled codex candidate $candidate" + if test -f "$state/unstable-output-oid" && + ! is_null_oid "$(state_value "$state" unstable-output-oid)" + then + say "assembled codex-unstable candidate $(state_value "$state" unstable-output-oid)" + fi } verify_inputs () { @@ -2657,10 +3327,12 @@ verify_inputs () { "$expected") test -n "$expected_controller" || die "input snapshot has no controller commit" + expected_mode=$(awk -F '\t' '$1 == "lane-mode" { print $3 }' \ + "$expected") actual=$tmp_dir/actual-inputs topics=$tmp_dir/actual-topics snapshot_inputs "$remote" "$base_name" "$codex_name" "$actual" "$topics" \ - "$expected_controller" + "$expected_controller" "$expected_mode" if ! cmp -s "$expected" "$actual" then diff -u "$expected" "$actual" >&2 || : @@ -2674,8 +3346,15 @@ prepare_input_graph () { mkdir -p "$graph" || die "could not prepare input graph verification" awk -F '\t' ' $1 == "controller" || $1 == "base" || $1 == "codex" || - $1 == "topic" { if (NF != 3) exit 1; next } - $1 == "admission" { if (NF != 5) exit 1; next } + $1 == "topic" || $1 == "unstable" || + $1 == "unstable-topic" || $1 == "lane-mode" { + if (NF != 3) exit 1 + next + } + $1 == "admission" || $1 == "unstable-admission" { + if (NF != 5) exit 1 + next + } { exit 1 } ' "$inputs" || die "input snapshot contains an invalid record" for kind in controller base codex @@ -2684,6 +3363,23 @@ prepare_input_graph () { END { print count + 0 }' "$inputs")" = 1 || die "input snapshot must contain exactly one '$kind' record" done + for kind in unstable lane-mode admission unstable-admission + do + test "$(awk -F '\t' -v kind="$kind" '$1 == kind { count++ } + END { print count + 0 }' "$inputs")" -le 1 || + die "input snapshot contains more than one '$kind' record" + done + lane_mode=$(awk -F '\t' '$1 == "lane-mode" { print $3 }' "$inputs") + case "$lane_mode" in + ''|enable|disable) ;; + *) die "input snapshot contains an invalid unstable lane mode" ;; + esac + if test -n "$lane_mode" + then + test "$(awk -F '\t' '$1 == "lane-mode" { print $2 }' \ + "$inputs")" = refs/heads/codex-unstable || + die "input snapshot contains an invalid unstable lane ref" + fi awk -F '\t' '$1 == "topic" { name=$2 sub("^refs/heads/", "", name) @@ -2710,7 +3406,32 @@ prepare_input_graph () { base_name=${base_ref#refs/heads/} codex_name=${codex_ref#refs/heads/} read_meta_config "$controller_oid" "$base_name" "$codex_name" "$graph" + config_version=$(state_value "$graph" config-version) + unstable_count=$(awk -F '\t' '$1 == "unstable" { count++ } + END { print count + 0 }' "$inputs") + case "$config_version:$lane_mode" in + 1:) + test "$unstable_count" = 0 || + die "version 1 input snapshot invents an unstable output" + ;; + 1:enable|2:|2:disable) + test "$unstable_count" = 1 || + die "enabled unstable lane is missing its snapshotted output" + test "$(awk -F '\t' '$1 == "unstable" { print $2 }' \ + "$inputs")" = refs/heads/codex-unstable || + die "input snapshot records an invalid unstable output ref" + ;; + *) die "input snapshot changes the unstable lane without authorization" ;; + esac codex_oid=$(awk -F '\t' '$1 == "codex" { print $3 }' "$inputs") + awk -F '\t' '$1 == "unstable-topic" { + name=$2 + sub("^refs/heads/", "", name) + printf "%s\t%s\n", name, $3 + }' "$inputs" | LC_ALL=C sort >"$graph/unstable-current-topics" || + die "could not prepare unstable ownership verification" + validate_lane_isolation "$graph" "$graph/topics" \ + "$graph/unstable-current-topics" "$codex_oid" published_codex=$(state_value "$graph" published-codex-oid) awk -F '\t' '$1 == "admission" { print }' "$inputs" \ >"$graph/snapshot-admissions" || @@ -2776,6 +3497,14 @@ prepare_input_graph () { done <"$graph/remotes" test -n "$graph_remote" || die "could not locate the snapshotted Codex remote" + if test "$config_version" = 1 && + git rev-parse --verify \ + "$(remote_ref "$graph_remote" codex-unstable)^{commit}" \ + >/dev/null 2>&1 + then + die "$meta_config_path does not describe the existing codex-unstable output" + fi + printf '%s\n' "$graph_remote" >"$graph/remote" reject_unadmitted_topic_history "$graph_remote" "$graph/topics" \ "$published_codex" "$base_oid" validate_live_codex_delta "$published_codex" \ @@ -2783,6 +3512,126 @@ prepare_input_graph () { prepare_stateful_plan "$base_name" "$base_oid" "$graph/topics" "$graph" } +prepare_unstable_input_graph () ( + inputs=$1 + stable_candidate=$2 + stable_graph=$3 + graph=$4 + mkdir -p "$graph" || + die "could not prepare unstable input graph verification" + awk -F '\t' '$1 == "unstable-topic" { + name=$2 + sub("^refs/heads/", "", name) + printf "%s\t%s\n", name, $3 + }' "$inputs" | LC_ALL=C sort >"$graph/topics" || + die "could not read unstable topics from the input snapshot" + test "$(cut -f1 "$graph/topics" | sort -u | wc -l | tr -d ' ')" = \ + "$(wc -l <"$graph/topics" | tr -d ' ')" || + die "input snapshot contains duplicate unstable Codex topics" + while IFS="$tab" read -r name oid + do + is_unstable_topic_name "$name" || + die "input snapshot contains invalid unstable topic '$name'" + require_full_commit_oid "$oid" + done <"$graph/topics" + cp "$stable_graph/published-unstable-topics" \ + "$graph/published-topics" || + die "could not retain published unstable topology for verification" + version=$(state_value "$stable_graph" config-version) + mode=$(awk -F '\t' '$1 == "lane-mode" { print $3 }' "$inputs") + unstable_oid=$(awk -F '\t' '$1 == "unstable" { print $3 }' "$inputs") + test -n "$unstable_oid" || + die "unstable topics have no snapshotted output ref" + awk -F '\t' '$1 == "unstable-admission" { print }' "$inputs" \ + >"$graph/snapshot-admissions" || + die "could not inspect reviewed unstable Codex admissions" + if test "$version" = 2 + then + ! is_null_oid "$unstable_oid" || + die "published codex-unstable output is missing" + published_base=$(state_value "$stable_graph" \ + published-unstable-base-oid) + published_output=$(state_value "$stable_graph" \ + published-unstable-oid) + if test "$published_output" = "$unstable_oid" + then + test ! -s "$graph/snapshot-admissions" || + die "input snapshot records an unstable admission without a pending merge" + admitted_name= + else + test "$(wc -l <"$graph/snapshot-admissions" | tr -d ' ')" = 1 || + die "input snapshot does not contain exactly one reviewed unstable admission" + IFS="$tab" read -r kind admitted_ref admitted_oid \ + admitted_number admitted_merge <"$graph/snapshot-admissions" || + die "could not inspect the reviewed unstable Codex admission" + admitted_name=${admitted_ref#refs/heads/} + test "$admitted_ref" = "refs/heads/$admitted_name" && + is_unstable_topic_name "$admitted_name" && + test "$admitted_merge" = "$unstable_oid" || + die "input snapshot contains an invalid reviewed unstable admission" + snapshot_head=$(awk -F '\t' -v name="$admitted_name" \ + '$1 == name { print $2 }' "$graph/topics") + test "$snapshot_head" = "$admitted_oid" || + die "input snapshot contains an invalid reviewed unstable admission" + authenticate_pending_codex_merge - "$published_output" \ + "$unstable_oid" "$graph/verified-admission" \ + "$snapshot_head" codex-unstable + printf '%s\t%s\t%s\t%s\n' "$admitted_name" \ + "$admitted_oid" "$admitted_number" "$admitted_merge" \ + >"$graph/expected-admission" + cmp -s "$graph/expected-admission" \ + "$graph/verified-admission" || + die "input snapshot does not match the reviewed unstable admission" + fi + else + test "$mode" = enable && is_null_oid "$unstable_oid" || + die "$meta_config_path does not describe the existing codex-unstable output" + test ! -s "$graph/topics" && + test ! -s "$graph/snapshot-admissions" || + die "a newly enabled unstable lane cannot enroll unreviewed topics" + published_base=$(awk -F '\t' '$1 == "codex" { print $3 }' \ + "$inputs") + published_output=$(null_oid) + admitted_name= + fi + while IFS="$tab" read -r name oid + do + if ! awk -F '\t' -v name="$name" '$1 == name { found=1 } + END { exit !found }' "$graph/published-topics" && + test "$name" != "$admitted_name" + then + die "input snapshot contains unadmitted unstable topic '$name'" + fi + done <"$graph/topics" + if test "$mode" = disable + then + test "$version" = 2 && ! test -s "$graph/topics" && + test ! -s "$graph/snapshot-admissions" || + die "cannot disable codex-unstable while reviewed topics remain" + fi + if test -s "$graph/topics" + then + graph_remote=$(state_value "$stable_graph" remote) + codex_oid=$(awk -F '\t' '$1 == "codex" { print $3 }' "$inputs") + reject_unadmitted_topic_history "$graph_remote" "$graph/topics" \ + "$published_output" "$codex_oid" codex-unstable + validate_live_codex_delta "$published_output" "$unstable_oid" \ + "$graph/topics" "$graph" + fi + printf '%s\n' codex >"$graph/base-name" + printf '%s\n' "$stable_candidate" >"$graph/base-oid" + printf '%s\n' codex-unstable >"$graph/codex-name" + printf '%s\n' "$published_base" >"$graph/published-base-oid" + printf '%s\n' "$published_output" >"$graph/published-codex-oid" + if test -s "$graph/topics" + then + prepare_stateful_plan codex "$stable_candidate" \ + "$graph/topics" "$graph" + else + : >"$graph/plan" + fi +) + topic_control_paths_unchanged () ( base_oid=$1 head_oid=$2 @@ -2791,6 +3640,7 @@ topic_control_paths_unchanged () ( .github/rulesets/codex-branch.json \ .github/rulesets/codex-meta.json \ .github/rulesets/codex-topics.json \ + .github/rulesets/codex-unstable-branch.json \ .github/workflows/codex-admission.yml \ .github/workflows/codex-topic.yml \ .github/workflows/codex.yml \ @@ -2807,6 +3657,44 @@ topic_control_paths_unchanged () ( ':(exclude).github/workflows/codex-release.yml' ) +verify_unstable_control_paths () ( + base_oid=$1 + unstable_candidate=$2 + state=$3 + + topic_control_paths_unchanged "$base_oid" "$unstable_candidate" || + die "codex-unstable changes a protected controller or CI file" + git diff --quiet "$base_oid" "$unstable_candidate" -- \ + ':(glob).github/workflows/*.yml' \ + ':(glob).github/workflows/*.yaml' || + die "codex-unstable changes a GitHub Actions workflow" + release_workflow_is_codex_only "$unstable_candidate" || + die "codex-unstable changes the production-only release trigger" + while IFS="$tab" read -r name old prerequisite old_base prerequisite_tip + do + is_unstable_topic_name "$name" || + die "unstable integration contains stable topic '$name'" + if test "$prerequisite" != codex + then + is_unstable_topic_name "$prerequisite" || + die "unstable topic '$name' has non-unstable prerequisite '$prerequisite'" + dependency=$(result_lookup "$state/results" "$prerequisite") + test -n "$dependency" || + die "unstable topic '$name' has no rewritten prerequisite" + else + dependency=$base_oid + fi + tip=$(result_lookup "$state/results" "$name") + test -n "$tip" || die "unstable topic '$name' has no rewritten tip" + topic_control_paths_unchanged "$dependency" "$tip" || + die "unstable topic '$name' changes a protected controller or CI file" + git diff --quiet "$dependency" "$tip" -- \ + ':(glob).github/workflows/*.yml' \ + ':(glob).github/workflows/*.yaml' || + die "unstable topic '$name' changes a GitHub Actions workflow" + done <"$state/plan" +) + verify_control_paths () { inputs=$1 updates=$2 @@ -2825,6 +3713,11 @@ verify_control_paths () { then die "candidate downgrades the canonical Codex admission workflow" fi + if automation_workflow_is_reviewed "$published_codex" && + ! automation_workflow_is_reviewed "$candidate" + then + die "candidate downgrades the canonical Codex admission workflow" + fi fi if test -z "$require_automation" @@ -2845,7 +3738,8 @@ verify_control_paths () { while IFS="$tab" read -r ref old new do case "$ref" in - refs/heads/meta|refs/heads/codex) continue ;; + refs/heads/meta|refs/heads/codex|refs/heads/codex-unstable) continue ;; + refs/heads/??/codex/*-unstable) continue ;; esac name=${ref#refs/heads/} plan_row=$(awk -F '\t' -v name="$name" '$1 == name { print; exit }' \ @@ -2898,6 +3792,8 @@ verify_meta_update () ( updates=$2 candidate=$3 graph=$4 + unstable_graph=${5:-} + unstable_candidate=${6:-} old_meta=$(awk -F '\t' '$1 == "controller" { print $3 }' "$inputs") new_meta=$(awk -F '\t' '$1 == "refs/heads/meta" { print $3 }' "$updates") test -n "$new_meta" || die "updates contain no meta state commit" @@ -2951,8 +3847,39 @@ verify_meta_update () ( codex_ref=$(awk -F '\t' '$1 == "codex" { print $2 }' "$inputs") base_name=${base_ref#refs/heads/} codex_name=${codex_ref#refs/heads/} - write_meta_config "$base_name" "$base_oid" "$codex_name" "$candidate" \ - "$graph/expected-meta-topics" "$graph/expected-meta-config" + if test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" + then + test -n "$unstable_graph" || + die "unstable output has no verified topic graph" + : >"$graph/expected-meta-unstable-topics" + while IFS="$tab" read -r name old + do + ref=refs/heads/$name + new=$(awk -F '\t' -v ref="$ref" \ + '$1 == ref { print $3 }' "$updates") + test -n "$new" || + die "updates contain no rewritten unstable tip for '$name'" + prerequisite=$(awk -F '\t' -v name="$name" \ + '$1 == name { value=$3 } END { if (value != "") print value }' \ + "$unstable_graph/plan") + test -n "$prerequisite" || + die "verified unstable graph contains no prerequisite for '$name'" + printf '%s\t%s\t%s\n' "$name" "$new" "$prerequisite" \ + >>"$graph/expected-meta-unstable-topics" + done <"$unstable_graph/topics" + LC_ALL=C sort -o "$graph/expected-meta-unstable-topics" \ + "$graph/expected-meta-unstable-topics" + write_meta_config "$base_name" "$base_oid" "$codex_name" \ + "$candidate" "$graph/expected-meta-topics" \ + "$graph/expected-meta-config" \ + "$graph/expected-meta-unstable-topics" \ + "$candidate" "$unstable_candidate" + else + write_meta_config "$base_name" "$base_oid" "$codex_name" \ + "$candidate" "$graph/expected-meta-topics" \ + "$graph/expected-meta-config" + fi git show "$new_meta:$meta_config_path" >"$graph/actual-meta-config" \ 2>/dev/null || die "next meta state has no $meta_config_path" cmp -s "$graph/expected-meta-config" "$graph/actual-meta-config" || @@ -2964,6 +3891,7 @@ verify_output () { updates= result= require_automation= + stable_recovery= while test $# -gt 0 do case "$1" in @@ -2971,6 +3899,7 @@ verify_output () { --updates) require_arg "$@"; updates=$2; shift 2 ;; --result) require_arg "$@"; result=$2; shift 2 ;; --require-automation) require_automation=t; shift ;; + --stable-recovery) stable_recovery=t; shift ;; *) die "unknown verify-output option '$1'" ;; esac done @@ -2983,37 +3912,142 @@ verify_output () { candidate=$(resolve_commit "$(sed -n '1p' "$result")") test "$(awk -F '\t' '$1 == "refs/heads/codex" { print $3 }' "$updates")" = "$candidate" || die "candidate does not match the codex update" + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' "$updates") + unstable_input=$(awk -F '\t' '$1 == "unstable" { print $3 }' \ + "$inputs") + lane_mode=$(awk -F '\t' '$1 == "lane-mode" { print $3 }' "$inputs") + unstable_topic_count=$(awk -F '\t' \ + '$1 == "unstable-topic" { count++ } END { print count + 0 }' \ + "$inputs") + if test -n "$stable_recovery" + then + test -n "$unstable_input" && + ! is_null_oid "$unstable_input" && + test -z "$unstable_candidate" && + test -z "$lane_mode" || + die "production recovery cannot change its enabled unstable lane" + test "$(awk -F '\t' '$1 == "unstable-admission" { count++ } + END { print count + 0 }' "$inputs")" = 0 || + die "production recovery cannot bypass a pending unstable admission" + elif test -z "$unstable_input" + then + test -z "$unstable_candidate" && test "$unstable_topic_count" = 0 && + test -z "$lane_mode" || + die "disabled unstable lane has unsnapshotted output or topics" + elif test "$lane_mode" = enable + then + is_null_oid "$unstable_input" && + test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" && + test "$unstable_topic_count" = 0 || + die "enabling codex-unstable requires an empty reviewed sentinel" + elif test "$lane_mode" = disable + then + ! is_null_oid "$unstable_input" && + is_null_oid "$unstable_candidate" && + test "$unstable_topic_count" = 0 || + die "disabling codex-unstable requires an empty published lane" + else + ! is_null_oid "$unstable_input" && + test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" || + die "enabled unstable lane must retain its generated output" + fi LC_ALL=C sort -c "$updates" || die "updates are not canonically sorted" test "$(cut -f1 "$updates" | sort -u | wc -l | tr -d ' ')" = \ "$(wc -l <"$updates" | tr -d ' ')" || die "updates contain duplicate refs" - awk -F '\t' '$1 == "controller" || $1 == "codex" || $1 == "topic" { print $2 }' "$inputs" \ + awk -F '\t' -v recovery="$stable_recovery" ' + $1 == "controller" || $1 == "codex" || $1 == "topic" || + (recovery == "" && ($1 == "unstable" || $1 == "unstable-topic")) { + print $2 + } + ' "$inputs" \ | LC_ALL=C sort >"$tmp_dir/expected-update-refs" cut -f1 "$updates" | LC_ALL=C sort >"$tmp_dir/actual-update-refs" cmp -s "$tmp_dir/expected-update-refs" "$tmp_dir/actual-update-refs" || - die "updates do not cover exactly codex and every snapshotted topic" + die "updates do not cover exactly every snapshotted output and topic" base_oid=$(awk -F '\t' '$1 == "base" { print $3 }' "$inputs") test -n "$base_oid" || die "input snapshot has no base commit" prepare_input_graph "$inputs" "$tmp_dir/topic-graph" + if test -n "$lane_mode" + then + input_codex=$(awk -F '\t' '$1 == "codex" { print $3 }' "$inputs") + published_codex=$(state_value "$tmp_dir/topic-graph" \ + published-codex-oid) + test "$candidate" = "$input_codex" && + test "$input_codex" = "$published_codex" || + die "changing the codex-unstable lane cannot publish a production update" + fi + unstable_graph= + if test -n "$unstable_input" && test -z "$stable_recovery" + then + unstable_graph=$tmp_dir/unstable-topic-graph + prepare_unstable_input_graph "$inputs" "$candidate" \ + "$tmp_dir/topic-graph" "$unstable_graph" + fi git merge-base --is-ancestor "$base_oid" "$candidate" || die "candidate is not based on the snapshotted master" - verify_meta_update "$inputs" "$updates" "$candidate" \ - "$tmp_dir/topic-graph" + if test -n "$stable_recovery" + then + test "$(state_value "$tmp_dir/topic-graph" config-version)" = 2 && + test "$unstable_input" = "$(state_value \ + "$tmp_dir/topic-graph" published-unstable-oid)" || + die "production recovery cannot change the published unstable output" + old_meta=$(awk -F '\t' '$1 == "controller" { print $3 }' \ + "$inputs") + new_meta=$(awk -F '\t' '$1 == "refs/heads/meta" { print $3 }' \ + "$updates") + test "$old_meta" = "$new_meta" || + die "production recovery cannot change published meta state" + else + verify_meta_update "$inputs" "$updates" "$candidate" \ + "$tmp_dir/topic-graph" "$unstable_graph" "$unstable_candidate" + fi verify_control_paths "$inputs" "$updates" "$candidate" \ "$tmp_dir/topic-graph" "$require_automation" while IFS="$tab" read -r ref old new do git check-ref-format "$ref" >/dev/null || die "invalid update ref '$ref'" expected_old=$(awk -F '\t' -v ref="$ref" \ - '$1 != "admission" && $2 == ref { print $3 }' \ + '($1 == "controller" || $1 == "codex" || + $1 == "topic" || $1 == "unstable" || + $1 == "unstable-topic") && $2 == ref { print $3 }' \ "$inputs") test "$old" = "$expected_old" || die "old value for '$ref' does not match the input snapshot" - resolve_commit "$old" >/dev/null - require_full_commit_oid "$new" + if ! is_null_oid "$old" + then + resolve_commit "$old" >/dev/null + fi + if ! is_null_oid "$new" + then + require_full_commit_oid "$new" + fi case "$ref" in refs/heads/meta) ;; refs/heads/codex) ;; + refs/heads/codex-unstable) + if ! is_null_oid "$new" + then + git merge-base --is-ancestor "$candidate" "$new" || + die "codex-unstable does not contain its exact codex candidate" + test "$new" != "$candidate" || + die "codex-unstable is not strictly ahead of codex" + fi + ;; + refs/heads/??/codex/*-unstable) + test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" || + die "unstable topic update has no unstable output" + git merge-base --is-ancestor "$candidate" "$new" || + die "rewritten '$ref' is not based on codex" + git merge-base --is-ancestor "$new" "$unstable_candidate" || + die "codex-unstable does not contain '$ref'" + ;; refs/heads/??/codex/?*) + is_stable_topic_name "${ref#refs/heads/}" || + die "unstable topic '$ref' entered the stable update set" git merge-base --is-ancestor "$base_oid" "$new" || die "rewritten '$ref' is not based on master" git merge-base --is-ancestor "$new" "$candidate" || @@ -3049,6 +4083,8 @@ verify_output () { die "could not prepare integration verification" printf '%s\n' "$base_oid" >"$tmp_dir/topic-graph/base-oid" || die "could not prepare integration base verification" + printf '%s\n' codex >"$tmp_dir/topic-graph/codex-name" || + die "could not prepare integration output verification" : >"$tmp_dir/topic-graph/results" || die "could not prepare rewritten topic verification" while IFS="$tab" read -r name old @@ -3062,6 +4098,52 @@ verify_output () { write_integration_topics "$tmp_dir/topic-graph" codex_has_expected_integrations "$tmp_dir/topic-graph" "$candidate" || die "candidate does not contain one canonical integration merge per topic" + if test -n "$unstable_graph" + then + if is_null_oid "$unstable_candidate" + then + return 0 + fi + : >"$unstable_graph/results" || + die "could not prepare rewritten unstable topic verification" + while IFS="$tab" read -r name old + do + ref=refs/heads/$name + new=$(awk -F '\t' -v ref="$ref" \ + '$1 == ref { print $3 }' "$updates") + test -n "$new" || + die "updates contain no rewritten unstable tip for '$name'" + result_record "$unstable_graph/results" "$name" "$new" + done <"$unstable_graph/topics" + while IFS="$tab" read -r name old prerequisite old_base prerequisite_tip + do + new=$(result_lookup "$unstable_graph/results" "$name") + if test "$prerequisite" = codex + then + new_prerequisite=$candidate + else + new_prerequisite=$(result_lookup \ + "$unstable_graph/results" "$prerequisite") + test -n "$new_prerequisite" || + die "unstable topic '$name' has no rewritten prerequisite '$prerequisite'" + fi + git merge-base --is-ancestor "$new_prerequisite" "$new" || + die "unstable rewrite lost dependency '$prerequisite' -> '$name'" + done <"$unstable_graph/plan" + if test -s "$unstable_graph/topics" + then + write_integration_topics "$unstable_graph" + codex_has_expected_integrations "$unstable_graph" \ + "$unstable_candidate" || + die "codex-unstable does not contain one canonical integration merge per topic" + else + unstable_sentinel_is_canonical "$candidate" \ + "$unstable_candidate" || + die "empty codex-unstable output is not its canonical sentinel" + fi + verify_unstable_control_paths "$candidate" \ + "$unstable_candidate" "$unstable_graph" + fi } push_updates () { @@ -3073,17 +4155,27 @@ push_updates () { while IFS="$tab" read -r ref old new do case "$filter:$ref" in - topics:refs/heads/meta|topics:refs/heads/codex) continue ;; + topics:refs/heads/meta|topics:refs/heads/codex|topics:refs/heads/codex-unstable) continue ;; esac - set -- "$@" "--force-with-lease=$ref:$old" + if is_null_oid "$old" + then + set -- "$@" "--force-with-lease=$ref:" + else + set -- "$@" "--force-with-lease=$ref:$old" + fi done <"$updates" set -- "$@" "$remote" while IFS="$tab" read -r ref old new do case "$filter:$ref" in - topics:refs/heads/meta|topics:refs/heads/codex) continue ;; + topics:refs/heads/meta|topics:refs/heads/codex|topics:refs/heads/codex-unstable) continue ;; esac - set -- "$@" "$new:$ref" + if is_null_oid "$new" + then + set -- "$@" ":$ref" + else + set -- "$@" "$new:$ref" + fi done <"$updates" "$@" } @@ -3093,7 +4185,7 @@ staging_ref () { git check-ref-format "refs/heads/$name" >/dev/null 2>&1 || die "invalid staging branch '$name'" case "$name" in - ??/codex/*|codex|master|meta) + ??/codex/*|codex|codex-unstable|master|meta) die "staging branch '$name' overlaps a protected input or output" ;; esac @@ -3129,10 +4221,22 @@ stage_candidate () { test -f "$inputs" || die "input snapshot '$inputs' does not exist" test -f "$updates" || die "update manifest '$updates' does not exist" make_tmp_dir - candidate=$(awk -F '\t' '$1 == "refs/heads/codex" { print $3 }' \ - "$updates") - test -n "$candidate" || die "update manifest has no codex candidate" - printf '%s\n' "$candidate" >"$tmp_dir/stage-result" || + stable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex" { print $3 }' "$updates") + test -n "$stable_candidate" || + die "update manifest has no codex candidate" + if test "$staging" = codex-unstable-staging + then + candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' \ + "$updates") + test -n "$candidate" && ! is_null_oid "$candidate" || + die "update manifest has no codex-unstable candidate" + else + candidate=$stable_candidate + fi + stage_target=$candidate + printf '%s\n' "$stable_candidate" >"$tmp_dir/stage-result" || die "could not prepare staging verification" set -- --inputs "$inputs" --updates "$updates" \ --result "$tmp_dir/stage-result" @@ -3144,14 +4248,14 @@ stage_candidate () { # A failed CI run intentionally leaves staging behind. Recreate an # identical ref so a new user-authenticated push starts fresh push CI. - if test -n "$old" && test "$old" = "$candidate" + if test -n "$old" && test "$old" = "$stage_target" then git -c core.hooksPath=/dev/null push --atomic --porcelain \ "--force-with-lease=$ref:$old" "$remote" ":$ref" old= fi git -c core.hooksPath=/dev/null push --atomic --porcelain \ - "--force-with-lease=$ref:$old" "$remote" "$candidate:$ref" + "--force-with-lease=$ref:$old" "$remote" "$stage_target:$ref" } promote_updates () { @@ -3159,18 +4263,37 @@ promote_updates () { updates=$2 ref=$3 candidate=$4 + unstable_ref=${5:-} + unstable_candidate=${6:-} set -- git -c core.hooksPath=/dev/null push --atomic --porcelain while IFS="$tab" read -r update_ref old new do - set -- "$@" "--force-with-lease=$update_ref:$old" + if is_null_oid "$old" + then + set -- "$@" "--force-with-lease=$update_ref:" + else + set -- "$@" "--force-with-lease=$update_ref:$old" + fi done <"$updates" - set -- "$@" "--force-with-lease=$ref:$candidate" "$remote" + set -- "$@" "--force-with-lease=$ref:$candidate" + if test -n "$unstable_ref" + then + set -- "$@" \ + "--force-with-lease=$unstable_ref:$unstable_candidate" + fi + set -- "$@" "$remote" while IFS="$tab" read -r update_ref old new do - set -- "$@" "$new:$update_ref" + if is_null_oid "$new" + then + set -- "$@" ":$update_ref" + else + set -- "$@" "$new:$update_ref" + fi done <"$updates" set -- "$@" ":$ref" + test -z "$unstable_ref" || set -- "$@" ":$unstable_ref" "$@" } @@ -3211,7 +4334,33 @@ promote () { staged=$(remote_head_oid "$remote" "$ref") test "$staged" = "$candidate" || die "staging ref '$ref' moved or disappeared before promotion" - promote_updates "$remote" "$updates" "$ref" "$candidate" + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' "$updates") + unstable_ref= + unstable_staged= + if test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" + then + unstable_ref=$(staging_ref codex-unstable-staging) + test "$ref" != "$unstable_ref" || + die "stable and unstable staging refs must be distinct" + unstable_staged=$(remote_head_oid "$remote" "$unstable_ref") + test "$unstable_staged" = "$unstable_candidate" || + die "unstable staging ref '$unstable_ref' moved or disappeared before promotion" + elif test -n "$unstable_candidate" && + test "$(awk -F '\t' '$1 == "lane-mode" { print $3 }' \ + "$inputs")" = disable + then + candidate_unstable_ref=$(staging_ref codex-unstable-staging) + unstable_staged=$(remote_head_oid "$remote" \ + "$candidate_unstable_ref") + if test -n "$unstable_staged" + then + unstable_ref=$candidate_unstable_ref + fi + fi + promote_updates "$remote" "$updates" "$ref" "$candidate" \ + "$unstable_ref" "$unstable_staged" base_ref=$(awk -F '\t' '$1 == "base" { print $2 }' "$inputs") base_oid=$(awk -F '\t' '$1 == "base" { print $3 }' "$inputs") current_base=$(remote_head_oid "$remote" "$base_ref") @@ -3331,14 +4480,27 @@ wait_for_refresh_run () ( rebuild_codex () { local_preparation= - case $# in - 0) ;; - 1) - test "$1" = --local || { usage >&2; exit 129; } - local_preparation=t - ;; - *) usage >&2; exit 129 ;; - esac + rebuild_unstable_mode= + for option in "$@" + do + case "$option" in + --local) + test -z "$local_preparation" || { usage >&2; exit 129; } + local_preparation=t + ;; + --enable-unstable) + test -z "$rebuild_unstable_mode" || { usage >&2; exit 129; } + rebuild_unstable_mode=enable + ;; + --disable-unstable) + test -z "$rebuild_unstable_mode" || { usage >&2; exit 129; } + rebuild_unstable_mode=disable + ;; + *) usage >&2; exit 129 ;; + esac + done + test -z "$rebuild_unstable_mode" || test -n "$local_preparation" || + die "changing the codex-unstable lane requires Meta/rebuild --local" require_operator_context if test -z "$local_preparation" then @@ -3434,22 +4596,28 @@ wait_for_staging_ci () ( repository=$2 candidate=$3 baseline=$4 - workflow_runs="repos/$repository/actions/workflows/main.yml/runs?branch=codex-staging&event=push&head_sha=$candidate&per_page=100" + staging=${5:-codex-staging} + workflow_runs="repos/$repository/actions/workflows/main.yml/runs?branch=$staging&event=push&head_sha=$candidate&per_page=100" - say "Waiting for staging CI for $candidate..." + if test "$staging" = codex-staging + then + say "Waiting for staging CI for $candidate..." + else + say "Waiting for $staging CI for $candidate..." + fi run_id= attempt=0 while test "$attempt" -lt 60 do attempt=$((attempt + 1)) run_id=$("$gh_command" api --hostname github.com "$workflow_runs" --jq \ - ".workflow_runs | map(select(.id > ($baseline | tonumber) and .head_branch == \"codex-staging\" and .head_sha == \"$candidate\" and .event == \"push\" and .path == \".github/workflows/main.yml\")) | sort_by(.id) | .[0].id // empty") || + ".workflow_runs | map(select(.id > ($baseline | tonumber) and .head_branch == \"$staging\" and .head_sha == \"$candidate\" and .event == \"push\" and .path == \".github/workflows/main.yml\")) | sort_by(.id) | .[0].id // empty") || die "could not query staging CI" test -z "$run_id" || break sleep 5 done test -n "$run_id" || - die "no new CI run appeared for codex-staging at $candidate" + die "no new CI run appeared for $staging at $candidate" case "$run_id" in ''|*[!0-9]*) die "staging CI returned an invalid run ID" ;; esac @@ -3471,7 +4639,7 @@ wait_for_staging_ci () ( IFS="$tab" read -r actual_id event branch sha path status \ conclusion url <"$tmp_dir/ci-run" test "$actual_id" = "$run_id" && test "$event" = push && - test "$branch" = codex-staging && test "$sha" = "$candidate" && + test "$branch" = "$staging" && test "$sha" = "$candidate" && test "$path" = .github/workflows/main.yml || die "staging CI run $run_id no longer identifies the exact candidate" "$gh_command" api --hostname github.com --paginate \ @@ -3534,8 +4702,13 @@ wait_for_staging_ci () ( die "could not inspect jobs for staging CI run $run_id" test "$(printf '%s\n' "$config_conclusion" | sed '/^$/d' | wc -l | tr -d ' ')" = 1 && test "$config_conclusion" = success || - die "CI config did not run successfully on codex-staging" - say "Full staging CI passed." + die "CI config did not run successfully on $staging" + if test "$staging" = codex-staging + then + say "Full staging CI passed." + else + say "Full $staging CI passed." + fi ) freeze_local_candidate () { @@ -3576,6 +4749,14 @@ verify_candidate_bundle () { printf '%s refs/codex-output/meta\n' "$new_meta" \ >>"$tmp_dir/expected-bundle-heads" fi + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' "$updates") + if test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" + then + printf '%s refs/codex-output/unstable\n' "$unstable_candidate" \ + >>"$tmp_dir/expected-bundle-heads" + fi LC_ALL=C sort -o "$tmp_dir/bundle-heads" "$tmp_dir/bundle-heads" LC_ALL=C sort -o "$tmp_dir/expected-bundle-heads" \ "$tmp_dir/expected-bundle-heads" @@ -3660,10 +4841,12 @@ prepare_local_candidate () { say "Local preparation session: $session" ( cd "$prepare_repository" || exit 1 - with_isolated_git_environment \ - "$frozen_helper" refresh \ + set -- "$frozen_helper" refresh \ --session "$session" --remote origin --base master \ --codex codex --rerere-from codex --require-automation + test -z "${rebuild_unstable_mode:-}" || + set -- "$@" "--${rebuild_unstable_mode}-unstable" + with_isolated_git_environment "$@" ) || die "local Codex preparation failed; inspect '$session'" local_candidate_dir=$tmp_dir/local-candidate freeze_local_candidate "$session" "$local_candidate_dir" @@ -3674,7 +4857,8 @@ stage_and_wait_for_ci () { candidate=$2 inputs=$3 updates=$4 - workflow_runs="repos/$repository/actions/workflows/main.yml/runs?branch=codex-staging&event=push&head_sha=$candidate&per_page=100" + staging=codex-staging + workflow_runs="repos/$repository/actions/workflows/main.yml/runs?branch=$staging&event=push&head_sha=$candidate&per_page=100" baseline=$(gh api --hostname github.com "$workflow_runs" --jq \ '[.workflow_runs[].id] | max // 0') || die "could not record the staging CI baseline" @@ -3686,9 +4870,28 @@ stage_and_wait_for_ci () { test -n "$publisher" || die "GitHub CLI returned no authenticated user" say "Publishing the prepared candidate with the credentials for origin." say "GitHub API user: $publisher" - stage_candidate --remote origin --staging codex-staging \ + stage_candidate --remote origin --staging "$staging" \ --inputs "$inputs" --updates "$updates" --require-automation - wait_for_staging_ci gh "$repository" "$candidate" "$baseline" + wait_for_staging_ci gh "$repository" "$candidate" "$baseline" \ + "$staging" + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' "$updates") + if test -n "$unstable_candidate" && + ! is_null_oid "$unstable_candidate" + then + staging=codex-unstable-staging + workflow_runs="repos/$repository/actions/workflows/main.yml/runs?branch=$staging&event=push&head_sha=$unstable_candidate&per_page=100" + baseline=$(gh api --hostname github.com "$workflow_runs" --jq \ + '[.workflow_runs[].id] | max // 0') || + die "could not record the unstable staging CI baseline" + case "$baseline" in + ''|*[!0-9]*) die "unstable staging CI baseline is not a numeric run ID" ;; + esac + stage_candidate --remote origin --staging "$staging" \ + --inputs "$inputs" --updates "$updates" --require-automation + wait_for_staging_ci gh "$repository" "$unstable_candidate" \ + "$baseline" "$staging" + fi } rebuild_codex_locally () { @@ -4012,7 +5215,16 @@ publish_topics () { fi die "rewritten topic graph failed candidate validation; no refs were updated" fi - create_meta_commit "$state" "$candidate" + stable_recovery= + if test "$(state_value "$state" config-version)" = 2 + then + stable_recovery=t + printf '%s\n' "$(state_value "$state" controller-oid)" \ + >"$state/meta-oid" || + die "could not preserve published meta state during recovery" + else + create_meta_commit "$state" "$candidate" + fi write_complete_updates "$state" "$candidate" "$tmp_dir/updates" printf '%s\n' "$candidate" >"$tmp_dir/result" || die "could not prepare topic verification" @@ -4022,6 +5234,7 @@ publish_topics () { set -- --inputs "$state/inputs" --updates "$tmp_dir/updates" \ --result "$tmp_dir/result" test -z "$require_automation" || set -- "$@" --require-automation + test -z "$stable_recovery" || set -- "$@" --stable-recovery verify_output "$@" ) || die "rewritten topic graph failed output verification" git -C "$worktree" -c core.fsmonitor=false worktree remove --force \ diff --git a/.github/workflows/codex.yml b/.github/workflows/codex.yml index 2e37dd1efe8ed1..80997f5a0fa1c7 100644 --- a/.github/workflows/codex.yml +++ b/.github/workflows/codex.yml @@ -13,7 +13,7 @@ concurrency: jobs: prepare: - name: Rebase topics and assemble codex + name: Rebase topics and assemble Codex branches runs-on: ubuntu-24.04 steps: - name: Check out the pinned controller @@ -41,7 +41,7 @@ jobs: test "$(git rev-parse HEAD)" = "$WORKFLOW_SHA" test "$(git rev-parse refs/remotes/origin/codex)" = "$GITHUB_SHA" - - name: Rebase every topic and assemble codex + - name: Rebase topics and assemble the enabled Codex branches id: rewrite env: GH_TOKEN: ${{ github.token }} @@ -61,6 +61,18 @@ jobs: --failure "$RUNNER_TEMP/codex-conflict.md" candidate=$(cat "$RUNNER_TEMP/codex-candidate") + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' \ + "$RUNNER_TEMP/codex-updates") + case "$unstable_candidate" in + '') ;; + *[!0]*) + test "$unstable_candidate" != "$candidate" + git merge-base --is-ancestor "$candidate" "$unstable_candidate" + echo "unstable-candidate=$unstable_candidate" >>"$GITHUB_OUTPUT" + ;; + *) ;; + esac artifact_name="codex-candidate-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" { printf 'repository\t%s\n' "$GITHUB_REPOSITORY" @@ -85,11 +97,23 @@ jobs: old_codex=$(awk -F '\t' '$1 == "codex" { print $3 }' \ "$RUNNER_TEMP/codex-inputs") candidate=$(cat "$RUNNER_TEMP/codex-candidate") + unstable_candidate=$(awk -F '\t' \ + '$1 == "refs/heads/codex-unstable" { print $3 }' \ + "$RUNNER_TEMP/codex-updates") { echo '## Codex candidate' echo echo "Old codex: \`$old_codex\`" echo "Candidate: \`$candidate\`" + case "$unstable_candidate" in + '') ;; + *[!0]*) + echo "Unstable candidate: \`$unstable_candidate\`" + ;; + *) + echo 'Unstable candidate: branch will be removed' + ;; + esac echo "Controller: \`$CONTROLLER_OID\`" echo echo 'Publish this exact prepared run from a clean local clone:' diff --git a/t/t9905-codex-branch.sh b/t/t9905-codex-branch.sh index ca9aaaaac94e02..eddf08e2ba491b 100755 --- a/t/t9905-codex-branch.sh +++ b/t/t9905-codex-branch.sh @@ -54,7 +54,8 @@ manifest_has () { snapshot_without_staging () { snapshot_refs "$1" | - sed '/^refs\/heads\/codex-staging[[:space:]]/d' + sed '/^refs\/heads\/codex-staging[[:space:]]/d + /^refs\/heads\/codex-unstable-staging[[:space:]]/d' } write_automation_workflow () { @@ -77,6 +78,59 @@ write_reviewed_automation_workflow () { cat >"$1" <<-'EOF' name: Refresh codex + on: + workflow_dispatch: + pull_request: + branches: + - codex + - codex-unstable + types: + - opened + - reopened + - synchronize + - ready_for_review + merge_group: + types: + - checks_requested + + permissions: + actions: read + contents: read + pull-requests: read + + jobs: + refresh: + if: github.event_name == 'workflow_dispatch' + uses: openai/git/.github/workflows/codex.yml@meta + admission: + name: Codex admission + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex') + permissions: + contents: read + pull-requests: read + uses: openai/git/.github/workflows/codex-admission.yml@meta + unstable_admission: + name: Codex unstable admission + if: >- + (github.event_name == 'pull_request' && + github.event.pull_request.base.ref == 'codex-unstable') || + (github.event_name == 'merge_group' && + github.event.merge_group.base_ref == 'refs/heads/codex-unstable') + permissions: + contents: read + pull-requests: read + uses: openai/git/.github/workflows/codex-admission.yml@meta + EOF +} + +write_stable_reviewed_automation_workflow () { + cat >"$1" <<-'EOF' + name: Refresh codex + on: workflow_dispatch: pull_request: @@ -179,7 +233,7 @@ install_admission_gh () { merged_at=2026-08-04T00:00:00Z merge=$FAKE_ADMISSION_MERGE base_repository=openai/git - base=codex + base=${FAKE_ADMISSION_BASE:-codex} head_repository=openai/git head_ref=$FAKE_ADMISSION_BRANCH head=$FAKE_ADMISSION_HEAD @@ -280,6 +334,9 @@ install_admission_gate_gh () { repos/openai/git/git/ref/heads/codex) printf '%s\n' "$FAKE_GATE_CODEX" ;; + repos/openai/git/git/ref/heads/codex-unstable) + printf '%s\n' "$FAKE_GATE_UNSTABLE" + ;; repos/openai/git/git/ref/heads/??/codex/*) if test "${FAKE_GATE_MODE:-}" = changed-topic then @@ -289,13 +346,31 @@ install_admission_gate_gh () { fi ;; repos/openai/git/contents/codex.config\?ref=*) - published=$FAKE_GATE_CODEX - test "${FAKE_GATE_MODE:-}" != pending || - published=$FAKE_GATE_OTHER - printf '[codex]\n\toutput-tip = %s\n' "$published" + published_stable=$FAKE_GATE_CODEX + published_unstable=$FAKE_GATE_UNSTABLE + if test "${FAKE_GATE_MODE:-}" = pending + then + if test "${FAKE_GATE_LANE:-codex}" = codex-unstable + then + published_unstable=$FAKE_GATE_OTHER + else + published_stable=$FAKE_GATE_OTHER + fi + fi + printf '[codex]\n\toutput-tip = %s\n' \ + "$published_stable" + printf '[codex-unstable]\n\tbase-tip = %s\n\toutput-tip = %s\n' \ + "$published_stable" "$published_unstable" + if test "${FAKE_GATE_MODE:-}" = hidden-enrolled-unstable + then + printf '[branch "cc/codex/private-parent-unstable"]\n' + printf '\tcodex-tip = %s\n' "$FAKE_GATE_OTHER" + fi ;; repos/openai/git/git/commits/*) first=$FAKE_GATE_CODEX + test "${FAKE_GATE_LANE:-codex}" != codex-unstable || + first=$FAKE_GATE_UNSTABLE test "${FAKE_GATE_MODE:-}" != wrong-parent || first=$FAKE_GATE_OTHER printf '%s\t%s\t%s\t%s\n' \ @@ -358,6 +433,14 @@ install_admission_gate_gh () { printf '%s\t%s\n' "cc/codex/private-parent-$suffix" \ "$FAKE_GATE_OTHER" ;; + hidden-enrolled-unstable) + printf '%s\t%s\n' cc/codex/private-parent-unstable \ + "$FAKE_GATE_OTHER" + ;; + descendant-checkpoint) + printf '%s\t%s\n' cc/codex/checkpoint-unstable \ + "$FAKE_GATE_OTHER" + ;; same-tip-alias) printf '%s\t%s\n' cc/codex/same-tip \ "$FAKE_GATE_TOPIC" @@ -367,7 +450,12 @@ install_admission_gate_gh () { repos/openai/git/compare/*) case "$endpoint" in *"$FAKE_GATE_OTHER...$FAKE_GATE_TOPIC") - printf '%s\n' "$FAKE_GATE_SHARED" + if test "${FAKE_GATE_MODE:-}" = descendant-checkpoint + then + printf '%s\n' "$FAKE_GATE_TOPIC" + else + printf '%s\n' "$FAKE_GATE_SHARED" + fi ;; *"master...$FAKE_GATE_SHARED") if test "${FAKE_GATE_MODE:-}" = newer-master @@ -396,25 +484,34 @@ run_admission_gate () { mode=$1 && event=$2 && branch=${3:-bb/codex/reviewed} && + lane=${4:-codex} && directory=$TRASH_DIRECTORY/admission-gate-bin && meta=1111111111111111111111111111111111111111 && codex=2222222222222222222222222222222222222222 && + unstable=7777777777777777777777777777777777777777 && topic=3333333333333333333333333333333333333333 && candidate=4444444444444444444444444444444444444444 && other=5555555555555555555555555555555555555555 && shared=6666666666666666666666666666666666666666 && + if test "$lane" = codex-unstable + then + base=$unstable + else + base=$codex + fi && env PATH="$directory:$PATH" GH_TOKEN=not-a-real-token \ GITHUB_REPOSITORY=openai/git GITHUB_EVENT_NAME="$event" \ GITHUB_SHA="$candidate" WORKFLOW_REPOSITORY=openai/git \ WORKFLOW_SHA="$meta" EVENT_ACTION=checks_requested \ - GROUP_BASE_REF=refs/heads/codex GROUP_BASE_SHA="$codex" \ - GROUP_HEAD_REF=refs/heads/gh-readonly-queue/codex/pr-42 \ + GROUP_BASE_REF="refs/heads/$lane" GROUP_BASE_SHA="$base" \ + GROUP_HEAD_REF="refs/heads/gh-readonly-queue/$lane/pr-42" \ GROUP_HEAD_SHA="$candidate" PULL_NUMBER=42 \ - PULL_BASE_REF=codex PULL_BASE_SHA="$codex" \ + PULL_BASE_REF="$lane" PULL_BASE_SHA="$base" \ PULL_HEAD_REF="$branch" PULL_HEAD_SHA="$topic" \ PULL_HEAD_REPOSITORY=openai/git PULL_DRAFT=false \ FAKE_GATE_MODE="$mode" FAKE_GATE_META="$meta" \ - FAKE_GATE_CODEX="$codex" FAKE_GATE_TOPIC="$topic" \ + FAKE_GATE_LANE="$lane" FAKE_GATE_CODEX="$codex" \ + FAKE_GATE_UNSTABLE="$unstable" FAKE_GATE_TOPIC="$topic" \ FAKE_GATE_CANDIDATE="$candidate" FAKE_GATE_OTHER="$other" \ FAKE_GATE_SHARED="$shared" FAKE_GATE_BRANCH="$branch" \ bash "$TRASH_DIRECTORY/admission-gate.sh" @@ -477,11 +574,13 @@ admission_command () { fixture=$1 && mode=${2:-success} && shift 2 && + base=${ADMISSION_BASE:-codex} && branch=${ADMISSION_TOPIC:-bb/codex/reviewed} && - merge=$(git rev-parse refs/remotes/origin/codex) && + merge=$(git rev-parse "refs/remotes/origin/$base") && head=$(git rev-parse "refs/remotes/origin/$branch") && other=$(git rev-parse refs/remotes/origin/master) && env PATH="$TRASH_DIRECTORY/$fixture-bin:$PATH" \ + FAKE_ADMISSION_BASE="$base" \ FAKE_ADMISSION_MERGE="$merge" \ FAKE_ADMISSION_HEAD="$head" \ FAKE_ADMISSION_OTHER="$other" \ @@ -641,6 +740,181 @@ install_explicit_meta_state () ( rm -f "$state_bases" "$state_config" "$state_index" ) +install_unstable_meta_state () ( + meta_branch=$1 + base_branch=$2 + stable_branch=$3 + unstable_branch=$4 + install_meta_state "$meta_branch" "$base_branch" "$stable_branch" && + meta_parent=$(git rev-parse "$meta_branch") && + stable_tip=$(git rev-parse "$stable_branch") && + unstable_tip=$(git rev-parse "$unstable_branch") && + state_topics=.codex-unstable-state-topics && + state_config=.codex-unstable-state-config && + state_index=.codex-unstable-state-index && + git show "$meta_parent:codex.config" | + awk -v stable="$stable_tip" -v unstable="$unstable_tip" ' + /^\tversion = 1$/ { + $0 = "\tversion = 2" + } + /^\toutput-tip = / && !added { + print + print "" + print "[codex-unstable]" + print "\tbase-ref = refs/heads/codex" + print "\tbase-tip = " stable + print "\toutput-ref = refs/heads/codex-unstable" + print "\toutput-tip = " unstable + added = 1 + next + } + { print } + ' >"$state_config" && + git for-each-ref --format="%(refname:short)%09%(objectname)" \ + refs/heads | + while IFS="$(printf '\t')" read -r name oid + do + case "$name" in + ??/codex/?*-unstable) ;; + *) continue ;; + esac + case "$name" in + ??/codex/*/*) continue ;; + esac + if git merge-base --is-ancestor "$oid" "$unstable_tip" + then + printf "%s\t%s\n" "$name" "$oid" + fi + done | LC_ALL=C sort >"$state_topics" && + while IFS="$(printf '\t')" read -r name oid + do + prerequisite=$stable_branch && + prerequisite_tip=$stable_tip && + while IFS="$(printf '\t')" read -r other_name other_oid + do + test "$name" = "$other_name" && continue + test "$oid" = "$other_oid" && continue + if git merge-base --is-ancestor "$other_oid" "$oid" && + git merge-base --is-ancestor \ + "$prerequisite_tip" "$other_oid" + then + prerequisite=$other_name && + prerequisite_tip=$other_oid + fi + done <"$state_topics" && + { + printf "\n[branch \"%s\"]\n" "$name" && + printf "\tremote = .\n" && + printf "\tmerge = refs/heads/%s\n" "$prerequisite" && + printf "\tcodex-tip = %s\n" "$oid" + } >>"$state_config" + done <"$state_topics" && + blob=$(git hash-object -w "$state_config") && + helper_blob=$(git hash-object -w "$codex_branch") && + rm -f "$state_index" && + GIT_INDEX_FILE=$state_index git read-tree "$meta_parent^{tree}" && + GIT_INDEX_FILE=$state_index git update-index --add --cacheinfo \ + 100644,"$blob",codex.config && + GIT_INDEX_FILE=$state_index git update-index --add --cacheinfo \ + 100755,"$helper_blob",.github/workflows/codex-branch.sh && + tree=$(GIT_INDEX_FILE=$state_index git write-tree) && + meta_tip=$(printf "%s\n" "meta: initialize unstable Codex topic state" | + git commit-tree "$tree" -p "$meta_parent") && + git update-ref "refs/heads/$meta_branch" "$meta_tip" "$meta_parent" && + rm -f "$state_topics" "$state_config" "$state_index" +) + +create_unstable_sentinel () ( + base=$(git rev-parse "$1") && + tree=$(git rev-parse "$base^{tree}") && + sentinel=$(printf '%s\n' 'Initialize codex-unstable' | + GIT_AUTHOR_NAME=$codex_bot_name \ + GIT_AUTHOR_EMAIL=$codex_bot_email \ + GIT_COMMITTER_NAME=$codex_bot_name \ + GIT_COMMITTER_EMAIL=$codex_bot_email \ + git -c commit.gpgSign=false commit-tree "$tree" -p "$base") && + git branch codex-unstable "$sentinel" +) + +setup_pending_unstable () ( + fixture=$1 + topic=${2:-bb/codex/reviewed-unstable} + style=${3:-merge} + + git init --bare "$fixture.git" && + test_create_repo "$fixture-source" && + ( + cd "$fixture-source" && + git remote add origin "../$fixture.git" && + write base shared && + git add shared && + install_rerere_train && + git commit -m "unstable admission base" && + git switch -c aa/codex/enrolled master && + write enrolled enrolled-file && + git add enrolled-file && + git commit -m "already enrolled production topic" && + git branch codex && + create_unstable_sentinel codex && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + if test "$style" = sentinel + then + git push origin master meta codex codex-unstable \ + aa/codex/enrolled + else + git switch -c "$topic" codex && + write reviewed reviewed-unstable-file && + git add reviewed-unstable-file && + git commit -m "reviewed unstable topic" && + git switch codex-unstable && + case "$style" in + merge) + git merge --no-ff "$topic" \ + -m "Merge pull request #42 from openai/$topic" + ;; + squash) + git merge --squash "$topic" && + git commit -m "squash an unadmitted unstable topic" + ;; + *) return 1 ;; + esac && + git push origin master meta codex codex-unstable \ + aa/codex/enrolled "$topic" + fi + ) && + git clone "$fixture.git" "$fixture-runner" && + install_admission_gh "$TRASH_DIRECTORY/$fixture-bin" +) + +unstable_admission_rewrite () { + fixture=$1 && + mode=${2:-success} && + shift 2 && + ADMISSION_BASE=codex-unstable \ + ADMISSION_TOPIC=${ADMISSION_TOPIC:-bb/codex/reviewed-unstable} \ + admission_rewrite "$fixture" "$mode" "$@" +} + +apply_test_updates () ( + remote=$1 + updates=$2 + set -- git push --atomic --force "$remote" + while IFS="$(printf '\t')" read -r ref old new + do + case "$new" in + 0000000000000000000000000000000000000000|\ + 0000000000000000000000000000000000000000000000000000000000000000) + set -- "$@" ":$ref" + ;; + *) + set -- "$@" "$new:$ref" + ;; + esac || return 1 + done <"$updates" + "$@" +) + updated_tip () { awk -F "$(printf '\t')" -v ref="refs/heads/$1" \ '$1 == ref { print $3 }' "$2" @@ -798,7 +1072,7 @@ test_expect_success 'reviewed admission requires one app-authenticated queue ent test_grep "pull_request)" "$codex_admission_workflow" && test_grep "merge_group)" "$codex_admission_workflow" && test_grep "codex.output-tip" "$codex_admission_workflow" && - test_grep "refs/heads/gh-readonly-queue/codex/" \ + test_grep "refs/heads/gh-readonly-queue/" \ "$codex_admission_workflow" && test_grep "pull-requests: read" "$codex_admission_workflow" && ! grep -E "contents: write|pull-requests: write|statuses: write|id-token|actions/checkout|git push" \ @@ -855,7 +1129,7 @@ test_expect_success 'admission workflow executes both pull-request and queue che test_grep "Approved pull request #42" newer-master.out && for mode in pending wrong-parent unapproved multiple-pulls \ changed-topic changed-workflow hidden-prerequisite \ - hidden-wip hidden-stale hidden-unstable + hidden-wip hidden-stale hidden-unstable hidden-enrolled-unstable do test_expect_code 1 run_admission_gate "$mode" merge_group \ >"queue-$mode.out" 2>"queue-$mode.err" || return 1 @@ -1419,7 +1693,8 @@ test_expect_success 'topics cannot change the convenience wrappers' ' ' test_expect_success 'the reviewed automation trampoline cannot be downgraded' ' - for direction in upgrade downgrade + for direction in upgrade stable-upgrade downgrade \ + stable-downgrade dual-downgrade do fixture="automation-$direction" && git init --bare "$fixture.git" && @@ -1433,27 +1708,39 @@ test_expect_success 'the reviewed automation trampoline cannot be downgraded' ' git commit -m "automation migration base" && git switch -c aa/codex/automation master && mkdir -p .github/workflows && - if test "$direction" = upgrade - then + case "$direction" in + upgrade) write_automation_workflow \ .github/workflows/codex.yml - else + ;; + stable-upgrade|stable-downgrade) + write_stable_reviewed_automation_workflow \ + .github/workflows/codex.yml + ;; + downgrade|dual-downgrade) write_reviewed_automation_workflow \ .github/workflows/codex.yml - fi && + ;; + esac && git add .github/workflows/codex.yml && git commit -m "published automation" && git branch codex && git branch meta master && install_meta_state meta master codex && - if test "$direction" = upgrade - then + case "$direction" in + upgrade|stable-upgrade) write_reviewed_automation_workflow \ .github/workflows/codex.yml - else + ;; + downgrade|stable-downgrade) write_automation_workflow \ .github/workflows/codex.yml - fi && + ;; + dual-downgrade) + write_stable_reviewed_automation_workflow \ + .github/workflows/codex.yml + ;; + esac && git add .github/workflows/codex.yml && git commit -m "change automation generation" && git push origin master meta codex aa/codex/automation @@ -1462,7 +1749,8 @@ test_expect_success 'the reviewed automation trampoline cannot be downgraded' ' ( cd "$fixture-runner" && fetch_all && - if test "$direction" = upgrade + if test "$direction" = upgrade || + test "$direction" = stable-upgrade then sh "$codex_branch" rewrite --remote origin \ --base master --codex codex \ @@ -4873,4 +5161,1335 @@ test_expect_success PYTHON 'publish-run authenticates the artifact and promotes ) ' +test_expect_success 'unstable rules require one app-authenticated queue entry' ' + rules="$codex_root/.github/rulesets/codex-unstable-branch.json" && + test_path_is_file "$rules" && + jq -e " + (.conditions.ref_name.include == [\"refs/heads/codex-unstable\"]) and + (.rules[] | select(.type == \"merge_queue\") | + .parameters.merge_method == \"MERGE\" and + .parameters.grouping_strategy == \"ALLGREEN\" and + .parameters.max_entries_to_build == 1 and + .parameters.max_entries_to_merge == 1 and + .parameters.min_entries_to_merge == 1) and + (.rules[] | select(.type == \"required_status_checks\") | + .parameters.required_status_checks == [{ + \"context\": \"Codex unstable admission / Verify reviewed topic\", + \"integration_id\": 15368 + }]) + " "$rules" && + test_grep "codex-unstable.output-tip" "$codex_admission_workflow" && + test_grep "codex-unstable.base-tip" "$codex_admission_workflow" +' + +test_expect_success 'unstable admission executes both lane-specific checks' ' + install_admission_gate_gh "$TRASH_DIRECTORY/admission-gate-bin" && + sed -n "/^ run: |\$/,/^ [^ ]/p" \ + "$codex_admission_workflow" | + sed "1d; s/^ //" >"$TRASH_DIRECTORY/admission-gate.sh" && + run_admission_gate success pull_request \ + bb/codex/reviewed-unstable codex-unstable >unstable-pull.out && + test_grep "Approved pull request #42" unstable-pull.out && + run_admission_gate success merge_group \ + bb/codex/reviewed-unstable codex-unstable >unstable-queue.out && + test_grep "Approved pull request #42" unstable-queue.out && + run_admission_gate descendant-checkpoint merge_group \ + bb/codex/reviewed-unstable codex-unstable \ + >unstable-descendant-queue.out && + test_grep "Approved pull request #42" \ + unstable-descendant-queue.out && + for mode in pending wrong-parent unapproved multiple-pulls \ + changed-topic changed-workflow hidden-prerequisite + do + test_expect_code 1 run_admission_gate "$mode" merge_group \ + bb/codex/reviewed-unstable codex-unstable \ + >"unstable-$mode.out" 2>"unstable-$mode.err" || return 1 + done && + test_expect_code 1 run_admission_gate success pull_request \ + bb/codex/reviewed codex-unstable \ + >stable-in-unstable.out 2>stable-in-unstable.err && + test_grep "unstable" stable-in-unstable.err +' + +test_expect_success 'unadmitted unstable checkpoints leave v1 output unchanged' ' + setup_pending_admission inert-unstable-checkpoints && + enrolled=$(git --git-dir=inert-unstable-checkpoints.git \ + rev-parse refs/heads/aa/codex/enrolled) && + git --git-dir=inert-unstable-checkpoints.git update-ref \ + refs/heads/codex "$enrolled" && + ( + cd inert-unstable-checkpoints-runner && + fetch_all && + admission_rewrite inert-unstable-checkpoints api-failure && + apply_test_updates origin updates && + fetch_all && + stable=$(git rev-parse origin/codex) && + meta=$(git rev-parse origin/meta) && + ( + cd ../inert-unstable-checkpoints-source && + git switch -c tb/codex/status-part-01-unstable master && + write one status-one && + git add status-one && + git commit -m "unreviewed status checkpoint one" && + git switch -c tb/codex/status-part-02-unstable && + write two status-two && + git add status-two && + git commit -m "unreviewed status checkpoint two" && + git switch -c tb/codex/status-part-03-unstable master && + write three status-three && + git add status-three && + git commit -m "unreviewed parallel status checkpoint" && + git push origin tb/codex/status-part-01-unstable \ + tb/codex/status-part-02-unstable \ + tb/codex/status-part-03-unstable + ) && + fetch_all && + : >"$TRASH_DIRECTORY/inert-unstable-checkpoints-gh.log" && + snapshot_refs ../inert-unstable-checkpoints.git >before && + CODEX_UNSTABLE_MODE=enable \ + admission_rewrite inert-unstable-checkpoints api-failure && + test "$stable" = "$(cat result)" && + test "$meta" = "$(updated_tip meta updates)" && + ! grep -F -- "-unstable" inputs && + ! grep -F -- "-unstable" updates && + test_must_be_empty \ + "$TRASH_DIRECTORY/inert-unstable-checkpoints-gh.log" && + snapshot_refs ../inert-unstable-checkpoints.git >after && + test_cmp before after + ) +' + +test_expect_success 'enabling unstable creates a strict empty sentinel' ' + setup_pending_admission enable-unstable && + enrolled=$(git --git-dir=enable-unstable.git \ + rev-parse refs/heads/aa/codex/enrolled) && + git --git-dir=enable-unstable.git update-ref \ + refs/heads/codex "$enrolled" && + ( + cd enable-unstable-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result stable.result --updates stable.updates \ + --inputs stable.inputs --failure stable.failure && + apply_test_updates origin stable.updates && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --enable-unstable \ + --result result --updates updates --inputs inputs \ + --bundle candidate.bundle --failure failure && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + meta=$(updated_tip meta updates) && + test -n "$unstable" && + test "$stable" != "$unstable" && + test "$stable" = "$(git rev-parse "$unstable^")" && + test "$(git rev-parse "$stable^{tree}")" = \ + "$(git rev-parse "$unstable^{tree}")" && + test "Initialize codex-unstable" = \ + "$(git show -s --format=%s "$unstable")" && + has_codex_bot_author "$unstable" && + has_codex_bot_committer "$unstable" && + git show "$meta:codex.config" >enabled.config && + test 2 = "$(git config -f enabled.config --get codex.version)" && + test "$stable" = "$(git config -f enabled.config \ + --get codex-unstable.base-tip)" && + test "$unstable" = "$(git config -f enabled.config \ + --get codex-unstable.output-tip)" && + ! grep -F -- "-unstable\"]" enabled.config && + sh "$codex_branch" verify-output \ + --inputs inputs --updates updates --result result && + git bundle verify candidate.bundle && + git bundle list-heads candidate.bundle >bundle-heads && + test_grep "refs/codex-output/unstable" bundle-heads + ) +' + +test_expect_success 'an empty unstable sentinel stays strict across rebuilds' ' + setup_pending_unstable empty-unstable \ + bb/codex/reviewed-unstable sentinel && + ( + cd empty-unstable-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result result --updates updates --inputs inputs \ + --failure failure && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + test "$stable" != "$unstable" && + test "$stable" = "$(git rev-parse "$unstable^")" && + test "$(git rev-parse "$stable^{tree}")" = \ + "$(git rev-parse "$unstable^{tree}")" && + git show "$(updated_tip meta updates):codex.config" \ + >next.config && + ! grep -F -- "-unstable\"]" next.config + ) +' + +test_expect_success 'a reviewed unstable merge enrolls only its retained head' ' + setup_pending_unstable reviewed-unstable && + ( + cd reviewed-unstable-runner && + fetch_all && + unstable_admission_rewrite reviewed-unstable success && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + topic=$(updated_tip bb/codex/reviewed-unstable updates) && + meta=$(updated_tip meta updates) && + test -n "$unstable" && + git merge-base --is-ancestor "$stable" "$unstable" && + test_must_fail git cat-file -e "$stable:reviewed-unstable-file" && + test reviewed = "$(git show "$unstable:reviewed-unstable-file")" && + git show "$meta:codex.config" >next.config && + test "$topic" = "$(git config -f next.config \ + --get branch.bb/codex/reviewed-unstable.codex-tip)" && + test refs/heads/codex = "$(git config -f next.config \ + --get branch.bb/codex/reviewed-unstable.merge)" && + awk -F "$(printf "\t")" '\'' + $1 == "unstable-admission" && + $2 == "refs/heads/bb/codex/reviewed-unstable" && + $4 == 42 { found=1 } + END { exit !found } + '\'' inputs && + test_grep pulls "$TRASH_DIRECTORY/reviewed-unstable-gh.log" && + test_grep reviews "$TRASH_DIRECTORY/reviewed-unstable-gh.log" && + ADMISSION_BASE=codex-unstable \ + ADMISSION_TOPIC=bb/codex/reviewed-unstable \ + admission_command reviewed-unstable success \ + verify-output --inputs inputs --updates updates \ + --result result + ) +' + +test_expect_success 'an unstable admission rejects wrong lane or provenance' ' + setup_pending_unstable unstable-provenance && + ( + cd unstable-provenance-runner && + fetch_all && + snapshot_refs ../unstable-provenance.git >before && + for mode in no-pull-request wrong-merge wrong-base \ + wrong-base-repository wrong-head-repository wrong-head-ref \ + wrong-head draft-pull-request duplicate-pull-request \ + open-pull-request unmerged-pull-request api-failure + do + test_expect_code 1 \ + unstable_admission_rewrite unstable-provenance "$mode" \ + >"$mode.out" 2>"$mode.err" || return 1 + done && + snapshot_refs ../unstable-provenance.git >after && + test_cmp before after + ) +' + +test_expect_success 'an unstable admission requires a current trusted review' ' + setup_pending_unstable unstable-review && + ( + cd unstable-review-runner && + fetch_all && + for mode in no-review outsider-review self-review stale-review \ + rejected-review revoked-review dismissed-review \ + review-api-failure + do + test_expect_code 1 \ + unstable_admission_rewrite unstable-review "$mode" \ + >"$mode.out" 2>"$mode.err" || return 1 + done && + unstable_admission_rewrite unstable-review commented-review + ) +' + +test_expect_success 'a squash cannot enroll an unstable preview' ' + setup_pending_unstable unstable-squash \ + bb/codex/reviewed-unstable squash && + ( + cd unstable-squash-runner && + fetch_all && + test_expect_code 1 \ + unstable_admission_rewrite unstable-squash success \ + >rewrite.out 2>rewrite.err && + test_grep "normal two-parent merge" rewrite.err + ) +' + +test_expect_success 'an unadmitted unstable descendant remains inert' ' + setup_pending_unstable unstable-descendant && + ( + cd unstable-descendant-source && + git switch -c cc/codex/checkpoint-unstable \ + bb/codex/reviewed-unstable && + write unreviewed checkpoint-file && + git add checkpoint-file && + git commit -m "unreviewed descendant checkpoint" && + git push origin cc/codex/checkpoint-unstable + ) && + ( + cd unstable-descendant-runner && + fetch_all && + unstable_admission_rewrite unstable-descendant success && + unstable=$(updated_tip codex-unstable updates) && + test reviewed = "$(git show "$unstable:reviewed-unstable-file")" && + test_must_fail git cat-file -e "$unstable:checkpoint-file" && + ! grep -F "cc/codex/checkpoint-unstable" updates && + ! grep -F "cc/codex/checkpoint-unstable" inputs + ) +' + +test_expect_success 'disabling an empty unstable lane removes it atomically' ' + setup_pending_unstable disable-empty-unstable \ + bb/codex/reviewed-unstable sentinel && + ( + cd disable-empty-unstable-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result stable.result --updates stable.updates \ + --inputs stable.inputs --failure stable.failure && + apply_test_updates origin stable.updates && + fetch_all && + old=$(git rev-parse origin/codex-unstable) && + git --git-dir=../disable-empty-unstable.git update-ref \ + refs/heads/codex-unstable-staging "$old" && + zero=$(printf "%s\n" "$old" | tr "0123456789abcdef" 0) && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --disable-unstable \ + --result result --updates updates --inputs inputs \ + --failure failure && + test "$zero" = "$(updated_tip codex-unstable updates)" && + meta=$(updated_tip meta updates) && + git show "$meta:codex.config" >disabled.config && + test 1 = "$(git config -f disabled.config --get codex.version)" && + test_must_fail git config -f disabled.config \ + --get codex-unstable.output-tip && + sh "$codex_branch" verify-output \ + --inputs inputs --updates updates --result result && + sh "$codex_branch" stage \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates && + race_ref=refs/heads/codex-unstable-staging && + race_new=$(git rev-parse origin/master) && + test "$race_new" != "$old" && + real_git=$(command -v git) && + remote_git=$PWD/../disable-empty-unstable.git && + mkdir stale-preview-race-bin && + write "#!/bin/sh +case \" \$* \" in +*\" push \"*) + \"$real_git\" --git-dir=\"$remote_git\" update-ref \\ + \"$race_ref\" \"$race_new\" \"$old\" || exit + ;; +esac +exec \"$real_git\" \"\$@\"" stale-preview-race-bin/git && + chmod +x stale-preview-race-bin/git && + snapshot_without_staging ../disable-empty-unstable.git \ + >before-stage-race && + test_expect_code 1 env PATH="$PWD/stale-preview-race-bin:$PATH" \ + sh "$codex_branch" promote \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >race.out 2>race.err && + test "$race_new" = \ + "$(git --git-dir=../disable-empty-unstable.git \ + rev-parse "$race_ref")" && + snapshot_without_staging ../disable-empty-unstable.git \ + >after-stage-race && + test_cmp before-stage-race after-stage-race && + git --git-dir=../disable-empty-unstable.git update-ref \ + "$race_ref" "$old" "$race_new" && + GIT_TRACE=1 sh "$codex_branch" promote \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >promote.out 2>promote.trace && + test_grep "push --atomic --porcelain" promote.trace && + test_grep \ + "force-with-lease=refs/heads/codex-unstable-staging:$old" \ + promote.trace && + test_grep ":refs/heads/codex-unstable-staging" promote.trace && + test_must_fail git --git-dir=../disable-empty-unstable.git \ + show-ref --verify refs/heads/codex-unstable && + test_must_fail git --git-dir=../disable-empty-unstable.git \ + show-ref --verify refs/heads/codex-unstable-staging && + test_must_fail git --git-dir=../disable-empty-unstable.git \ + show-ref --verify refs/heads/codex-staging + ) +' + +test_expect_success 'disabling an enrolled unstable lane fails closed' ' + setup_pending_unstable disable-enrolled-unstable && + ( + cd disable-enrolled-unstable-runner && + fetch_all && + unstable_admission_rewrite disable-enrolled-unstable success && + apply_test_updates origin updates && + fetch_all && + snapshot_refs ../disable-enrolled-unstable.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --disable-unstable \ + --result disabled.result --updates disabled.updates \ + --inputs disabled.inputs --failure disabled.failure \ + >disabled.out 2>disabled.err && + test_grep "unstable" disabled.err && + snapshot_refs ../disable-enrolled-unstable.git >after && + test_cmp before after + ) +' + +test_expect_success 'preview lane transitions require unchanged stable output' ' + setup_pending_admission dirty-unstable-enable && + ( + cd dirty-unstable-enable-runner && + fetch_all && + snapshot_refs ../dirty-unstable-enable.git >before && + test_expect_code 1 \ + admission_command dirty-unstable-enable success \ + rewrite --remote origin --base master --codex codex \ + --enable-unstable --result result --updates updates \ + --inputs inputs --failure failure \ + >enable.out 2>enable.err && + test_grep "stable\|codex" enable.err && + snapshot_refs ../dirty-unstable-enable.git >after && + test_cmp before after + ) && + + setup_pending_unstable dirty-unstable-disable \ + bb/codex/reviewed-unstable sentinel && + ( + cd dirty-unstable-disable-source && + git switch aa/codex/enrolled && + write changed changed-stable-file && + git add changed-stable-file && + git commit -m "advance enrolled stable topic" && + git push origin aa/codex/enrolled + ) && + ( + cd dirty-unstable-disable-runner && + fetch_all && + snapshot_refs ../dirty-unstable-disable.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --disable-unstable \ + --result result --updates updates --inputs inputs \ + --failure failure >disable.out 2>disable.err && + test_grep "stable\|codex" disable.err && + snapshot_refs ../dirty-unstable-disable.git >after && + test_cmp before after + ) +' + +test_expect_success 'an empty unstable sentinel cannot impersonate its bot' ' + setup_pending_unstable forged-unstable-sentinel \ + bb/codex/reviewed-unstable sentinel && + ( + cd forged-unstable-sentinel-source && + stable=$(git rev-parse codex) && + forged=$( + GIT_AUTHOR_NAME="Untrusted Author" \ + GIT_AUTHOR_EMAIL=author@example.com \ + GIT_COMMITTER_NAME="Untrusted Committer" \ + GIT_COMMITTER_EMAIL=committer@example.com \ + git commit-tree "$stable^{tree}" -p "$stable" \ + -m "Initialize codex-unstable" + ) && + git update-ref refs/heads/codex-unstable "$forged" && + install_unstable_meta_state meta master codex codex-unstable && + git push --force origin meta codex-unstable + ) && + ( + cd forged-unstable-sentinel-runner && + fetch_all && + snapshot_refs ../forged-unstable-sentinel.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result result --updates updates --inputs inputs \ + --failure failure >rewrite.out 2>rewrite.err && + test_grep "sentinel\|unstable" rewrite.err && + test_path_is_missing result && + snapshot_refs ../forged-unstable-sentinel.git >after && + test_cmp before after + ) +' + +test_expect_success 'a v2 snapshot cannot silently erase its preview lane' ' + setup_pending_unstable missing-unstable-snapshot \ + bb/codex/reviewed-unstable sentinel && + ( + cd missing-unstable-snapshot-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result result --updates updates --inputs inputs \ + --failure failure && + awk -F "$(printf "\t")" '\''$1 != "unstable"'\'' \ + inputs >missing.inputs && + test_expect_code 1 sh "$codex_branch" verify-output \ + --inputs missing.inputs --updates updates --result result \ + >verify.out 2>verify.err && + test_grep "unstable\|snapshot" verify.err + ) +' + +test_expect_success 'stable conflict recovery preserves an enabled preview lane' ' + git init --bare unstable-stable-recovery.git && + test_create_repo unstable-stable-recovery-source && + ( + cd unstable-stable-recovery-source && + git remote add origin ../unstable-stable-recovery.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git switch -c aa/codex/stable master && + write topic shared && + git add shared && + git commit -m "conflicting enrolled stable topic" && + git branch codex && + create_unstable_sentinel codex && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git switch master && + write upstream shared && + git add shared && + git commit -m "conflicting upstream base" && + git push origin master meta codex codex-unstable \ + aa/codex/stable + ) && + + git clone unstable-stable-recovery.git \ + unstable-stable-recovery-runner && + ( + cd unstable-stable-recovery-runner && + fetch_all && + stable=$(git rev-parse origin/codex) && + unstable=$(git rev-parse origin/codex-unstable) && + meta=$(git rev-parse origin/meta) && + old_topic=$(git rev-parse origin/aa/codex/stable) && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex \ + --result result --updates updates --inputs inputs \ + --failure failure >rewrite.out 2>rewrite.err && + test_grep "aa/codex/stable" failure && + digest=$(git hash-object inputs) && + sh "$codex_branch" resolve --remote origin \ + --base master --codex codex --inputs-oid "$digest" \ + --worktree resolution >resolve.out && + write resolved resolution/shared && + git -C resolution add shared && + sh "$codex_branch" continue --worktree resolution \ + >continue.out && + sh "$codex_branch" publish-topics --worktree resolution && + new_topic=$(git --git-dir=../unstable-stable-recovery.git \ + rev-parse refs/heads/aa/codex/stable) && + test "$old_topic" != "$new_topic" && + test resolved = "$(git show "$new_topic:shared")" && + test "$stable" = \ + "$(git --git-dir=../unstable-stable-recovery.git \ + rev-parse refs/heads/codex)" && + test "$unstable" = \ + "$(git --git-dir=../unstable-stable-recovery.git \ + rev-parse refs/heads/codex-unstable)" && + test "$meta" = \ + "$(git --git-dir=../unstable-stable-recovery.git \ + rev-parse refs/heads/meta)" && + git worktree remove --force resolution + ) +' + +test_expect_success 'unstable parent rewrites replace old history when codex advances' ' + git init --bare unstable-parent.git && + test_create_repo unstable-parent-source && + ( + cd unstable-parent-source && + git remote add origin ../unstable-parent.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/parent-unstable codex && + write old unstable-old-parent-file && + git add unstable-old-parent-file && + git commit -m "old unstable parent" && + git switch -c cc/codex/child-unstable && + write child unstable-child-file && + git add unstable-child-file && + git commit -m "preserved unstable child" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + + git switch --detach codex && + write replacement unstable-new-parent-file && + git add unstable-new-parent-file && + git commit -m "replacement unstable parent" && + git branch -f bb/codex/parent-unstable HEAD && + git switch master && + write advanced unstable-advanced-base-file && + git add unstable-advanced-base-file && + git commit -m "advance codex underneath unstable topics" && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/parent-unstable \ + cc/codex/child-unstable + ) && + + git clone unstable-parent.git unstable-parent-runner && + ( + cd unstable-parent-runner && + fetch_all && + old_stable=$(git rev-parse origin/codex) && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --bundle candidate.bundle \ + --failure failure && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + parent=$(updated_tip bb/codex/parent-unstable updates) && + child=$(updated_tip cc/codex/child-unstable updates) && + test "$old_stable" != "$stable" && + test "$stable" = "$(git rev-parse "$parent^")" && + test "$parent" = "$(git rev-parse "$child^")" && + test advanced = "$(git show "$unstable:unstable-advanced-base-file")" && + test replacement = "$(git show "$unstable:unstable-new-parent-file")" && + test child = "$(git show "$unstable:unstable-child-file")" && + test_must_fail git cat-file -e \ + "$unstable:unstable-old-parent-file" && + git log --format=%s "$stable..$child" >subjects && + test_grep "^replacement unstable parent$" subjects && + ! grep -q "^old unstable parent$" subjects && + git merge-base --is-ancestor "$stable" "$unstable" + ) +' + +test_expect_success 'rewinding an unstable parent does not leak its removed commit' ' + git init --bare unstable-rewind.git && + test_create_repo unstable-rewind-source && + ( + cd unstable-rewind-source && + git remote add origin ../unstable-rewind.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/parent-unstable codex && + write kept unstable-kept-file && + git add unstable-kept-file && + git commit -m "kept unstable parent commit" && + kept=$(git rev-parse HEAD) && + write removed unstable-removed-file && + git add unstable-removed-file && + git commit -m "removed unstable parent commit" && + git switch -c cc/codex/child-unstable && + write child unstable-child-file && + git add unstable-child-file && + git commit -m "unstable child after rewind" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git branch -f bb/codex/parent-unstable "$kept" && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/parent-unstable \ + cc/codex/child-unstable + ) && + + git clone unstable-rewind.git unstable-rewind-runner && + ( + cd unstable-rewind-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --bundle candidate.bundle \ + --failure failure && + stable=$(cat result) && + parent=$(updated_tip bb/codex/parent-unstable updates) && + child=$(updated_tip cc/codex/child-unstable updates) && + unstable=$(updated_tip codex-unstable updates) && + test "$stable" = "$(git rev-parse "$parent^")" && + test "$parent" = "$(git rev-parse "$child^")" && + test kept = "$(git show "$unstable:unstable-kept-file")" && + test child = "$(git show "$unstable:unstable-child-file")" && + test_must_fail git cat-file -e \ + "$unstable:unstable-removed-file" && + git log --format=%s "$stable..$child" >subjects && + ! grep -q "^removed unstable parent commit$" subjects && + git bundle verify candidate.bundle && + git bundle list-heads candidate.bundle >bundle-heads && + test_grep "refs/codex-output/candidate" bundle-heads && + test_grep "refs/codex-output/unstable" bundle-heads && + git clone ../unstable-rewind.git ../unstable-rewind-import && + git -C ../unstable-rewind-import bundle unbundle \ + "$PWD/candidate.bundle" >imported-heads && + test_grep "refs/codex-output/candidate" imported-heads && + test_grep "refs/codex-output/unstable" imported-heads && + git -C ../unstable-rewind-import cat-file -e \ + "$unstable^{commit}" + ) +' + +test_expect_success 'a coherent unstable restack can reverse topic dependencies' ' + git init --bare unstable-reorder.git && + test_create_repo unstable-reorder-source && + ( + cd unstable-reorder-source && + git remote add origin ../unstable-reorder.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/a-unstable codex && + write A unstable-a-file && + git add unstable-a-file && + git commit -m "unstable dependency A" && + old_a=$(git rev-parse HEAD) && + git switch -c cc/codex/b-unstable && + write B unstable-b-file && + git add unstable-b-file && + git commit -m "unstable dependency B" && + old_b=$(git rev-parse HEAD) && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + + git switch --detach codex && + git cherry-pick "$old_b" && + new_b=$(git rev-parse HEAD) && + git branch -f cc/codex/b-unstable "$new_b" && + git cherry-pick "$old_a" && + git branch -f bb/codex/a-unstable HEAD && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/a-unstable \ + cc/codex/b-unstable + ) && + + git clone unstable-reorder.git unstable-reorder-runner && + ( + cd unstable-reorder-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure && + stable=$(cat result) && + a=$(updated_tip bb/codex/a-unstable updates) && + b=$(updated_tip cc/codex/b-unstable updates) && + unstable=$(updated_tip codex-unstable updates) && + meta=$(updated_tip meta updates) && + test "$stable" = "$(git rev-parse "$b^")" && + test "$b" = "$(git rev-parse "$a^")" && + git show "$meta:codex.config" >next.config && + test refs/heads/codex = "$(git config -f next.config \ + --get branch.cc/codex/b-unstable.merge)" && + test refs/heads/cc/codex/b-unstable = \ + "$(git config -f next.config \ + --get branch.bb/codex/a-unstable.merge)" && + git rev-list --first-parent --reverse "$stable..$unstable" \ + >integrations && + first=$(sed -n 1p integrations) && + second=$(sed -n 2p integrations) && + test "Merge cc/codex/b-unstable into codex-unstable" = \ + "$(git show -s --format=%s "$first")" && + test "Merge bb/codex/a-unstable into codex-unstable" = \ + "$(git show -s --format=%s "$second")" + ) +' + +test_expect_success 'removing an unstable prerequisite with a stale child fails closed' ' + git init --bare unstable-retired-parent.git && + test_create_repo unstable-retired-parent-source && + ( + cd unstable-retired-parent-source && + git remote add origin ../unstable-retired-parent.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/parent-unstable codex && + write parent unstable-parent-file && + git add unstable-parent-file && + git commit -m "retired unstable prerequisite" && + git switch -c cc/codex/child-unstable && + write child unstable-child-file && + git add unstable-child-file && + git commit -m "stale unstable child" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git push origin master meta codex codex-unstable \ + aa/codex/stable cc/codex/child-unstable + ) && + + git clone unstable-retired-parent.git unstable-retired-parent-runner && + ( + cd unstable-retired-parent-runner && + fetch_all && + snapshot_refs ../unstable-retired-parent.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "prerequisite.*retired" err && + test_path_is_missing result && + snapshot_refs ../unstable-retired-parent.git >after && + test_cmp before after + ) +' + +test_expect_success 'unstable topics can be combined and split into a new prerequisite' ' + git init --bare unstable-combine.git && + test_create_repo unstable-combine-source && + ( + cd unstable-combine-source && + git remote add origin ../unstable-combine.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/parent-unstable codex && + write parent unstable-parent-file && + git add unstable-parent-file && + git commit -m "original unstable parent" && + old_parent=$(git rev-parse HEAD) && + git switch -c cc/codex/child-unstable && + write child unstable-child-file && + git add unstable-child-file && + git commit -m "original unstable child" && + old_child=$(git rev-parse HEAD) && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + + git switch --detach codex && + git cherry-pick --no-commit "$old_parent" && + git commit -m "combined unstable prefix" && + git cherry-pick --no-commit "$old_child" && + git commit -m "combined unstable suffix" && + git branch -f cc/codex/child-unstable HEAD && + git branch -D bb/codex/parent-unstable && + git push origin master meta codex codex-unstable \ + aa/codex/stable cc/codex/child-unstable + ) && + + git clone unstable-combine.git unstable-combine-runner && + ( + cd unstable-combine-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure && + combined=$(updated_tip cc/codex/child-unstable updates) && + meta=$(updated_tip meta updates) && + git show "$meta:codex.config" >combined.config && + test refs/heads/codex = "$(git config -f combined.config \ + --get branch.cc/codex/child-unstable.merge)" && + test_must_fail git config -f combined.config \ + --get branch.bb/codex/parent-unstable.codex-tip && + test parent = "$(git show "$combined:unstable-parent-file")" && + test child = "$(git show "$combined:unstable-child-file")" && + apply_test_updates origin updates && + fetch_all && + + prefix=$(git rev-parse \ + "origin/cc/codex/child-unstable^") && + git branch bb/codex/prefix-unstable "$prefix" && + git push origin bb/codex/prefix-unstable && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result ignored-result \ + --updates ignored-updates --inputs ignored-inputs \ + --failure ignored-failure && + test -z "$(updated_tip bb/codex/prefix-unstable \ + ignored-updates)" && + ignored_meta=$(updated_tip meta ignored-updates) && + git show "$ignored_meta:codex.config" >ignored.config && + test_must_fail git config -f ignored.config \ + --get branch.bb/codex/prefix-unstable.codex-tip && + + old_child=$(git rev-parse origin/cc/codex/child-unstable) && + git switch bb/codex/prefix-unstable && + write reviewed unstable-reviewed-prefix-file && + git add unstable-reviewed-prefix-file && + git commit -m "reviewed unstable prerequisite" && + new_prefix=$(git rev-parse HEAD) && + git switch --detach "$new_prefix" && + git cherry-pick "$old_child" && + git branch -f cc/codex/child-unstable HEAD && + git switch --detach origin/codex-unstable && + git merge --no-ff "$new_prefix" \ + -m "Merge pull request #42 from openai/bb/codex/prefix-unstable" && + git push --force origin \ + HEAD:refs/heads/codex-unstable \ + bb/codex/prefix-unstable \ + cc/codex/child-unstable && + fetch_all && + install_admission_gh "$TRASH_DIRECTORY/unstable-combine-bin" && + ADMISSION_BASE=codex-unstable \ + ADMISSION_TOPIC=bb/codex/prefix-unstable \ + admission_command unstable-combine success rewrite --remote origin \ + --base master --codex codex --result split-result \ + --updates split-updates --inputs split-inputs \ + --failure split-failure && + parent=$(updated_tip bb/codex/prefix-unstable split-updates) && + child=$(updated_tip cc/codex/child-unstable split-updates) && + meta=$(updated_tip meta split-updates) && + test "$parent" = "$(git rev-parse "$child^")" && + git show "$meta:codex.config" >split.config && + test refs/heads/codex = "$(git config -f split.config \ + --get branch.bb/codex/prefix-unstable.merge)" && + test refs/heads/bb/codex/prefix-unstable = \ + "$(git config -f split.config \ + --get branch.cc/codex/child-unstable.merge)" + ) +' + +test_expect_success 'retiring the last unstable topic retains an empty preview lane' ' + git init --bare unstable-delete.git && + test_create_repo unstable-delete-source && + ( + cd unstable-delete-source && + git remote add origin ../unstable-delete.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/last-unstable codex && + write preview unstable-last-file && + git add unstable-last-file && + git commit -m "last unstable topic" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git push origin master meta codex codex-unstable \ + aa/codex/stable + ) && + + git clone unstable-delete.git unstable-delete-runner && + ( + cd unstable-delete-runner && + fetch_all && + old_unstable=$(git rev-parse origin/codex-unstable) && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + test "$stable" = "$(git rev-parse "$unstable^")" && + test "$(git rev-parse "$stable^{tree}")" = \ + "$(git rev-parse "$unstable^{tree}")" && + test "Initialize codex-unstable" = \ + "$(git show -s --format=%s "$unstable")" && + manifest_has codex-unstable "$old_unstable" "$unstable" \ + updates && + meta=$(updated_tip meta updates) && + git show "$meta:codex.config" >next.config && + test 2 = "$(git config -f next.config --get codex.version)" && + test "$unstable" = "$(git config -f next.config \ + --get codex-unstable.output-tip)" && + test_must_fail git config -f next.config \ + --get branch.bb/codex/last-unstable.codex-tip && + sh "$codex_branch" verify-output \ + --inputs inputs --updates updates --result result && + apply_test_updates origin updates && + test "$unstable" = "$(git --git-dir=../unstable-delete.git \ + rev-parse refs/heads/codex-unstable)" + ) +' + +test_expect_success 'an untracked unstable output cannot be silently retired' ' + git init --bare unstable-untracked.git && + test_create_repo unstable-untracked-source && + ( + cd unstable-untracked-source && + git remote add origin ../unstable-untracked.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git branch meta master && + install_meta_state meta master codex && + git switch -c codex-unstable codex && + write manual untracked-unstable-file && + git add untracked-unstable-file && + git commit -m "untracked unstable integration" && + git push origin master meta codex codex-unstable \ + aa/codex/stable + ) && + + git clone unstable-untracked.git unstable-untracked-runner && + ( + cd unstable-untracked-runner && + fetch_all && + snapshot_refs ../unstable-untracked.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "codex-unstable" err && + test_path_is_missing result && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --enable-unstable \ + --result enabled.result --updates enabled.updates \ + --inputs enabled.inputs --failure enabled.failure \ + >enabled.out 2>enabled.err && + test_grep "codex-unstable" enabled.err && + test_path_is_missing enabled.result && + snapshot_refs ../unstable-untracked.git >after && + test_cmp before after + ) +' + +test_expect_success 'retiring unstable output rejects unrecorded direct commits' ' + git init --bare unstable-retire-race.git && + test_create_repo unstable-retire-race-source && + ( + cd unstable-retire-race-source && + git remote add origin ../unstable-retire-race.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/retired-unstable codex && + write preview unstable-retired-file && + git add unstable-retired-file && + git commit -m "recorded unstable topic" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git switch codex-unstable && + write direct unstable-direct-file && + git add unstable-direct-file && + git commit -m "direct commit after recorded unstable output" && + git push origin master meta codex codex-unstable \ + aa/codex/stable + ) && + + git clone unstable-retire-race.git unstable-retire-race-runner && + ( + cd unstable-retire-race-runner && + fetch_all && + snapshot_refs ../unstable-retire-race.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep \ + "codex-unstable\|unstable output\|normal two-parent merge" \ + err && + test_path_is_missing result && + snapshot_refs ../unstable-retire-race.git >after && + test_cmp before after + ) +' + +test_expect_success 'unstable topics cannot hide workflow changes in a dependent revert' ' + git init --bare unstable-workflow.git && + test_create_repo unstable-workflow-source && + ( + cd unstable-workflow-source && + git remote add origin ../unstable-workflow.git && + write base shared && + mkdir -p .github/workflows && + write trusted .github/workflows/main.yml && + git add shared .github/workflows/main.yml && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/parent-unstable codex && + write malicious .github/workflows/main.yml && + git add .github/workflows/main.yml && + git commit -m "unstable topic changes protected workflow" && + git switch -c cc/codex/child-unstable && + write trusted .github/workflows/main.yml && + git add .github/workflows/main.yml && + git commit -m "unstable child hides protected workflow change" && + git diff --quiet codex HEAD -- .github/workflows/main.yml && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/parent-unstable \ + cc/codex/child-unstable + ) && + + git clone unstable-workflow.git unstable-workflow-runner && + ( + cd unstable-workflow-runner && + fetch_all && + snapshot_refs ../unstable-workflow.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "workflow\|controller\|protected" err && + test_path_is_missing result && + snapshot_refs ../unstable-workflow.git >after && + test_cmp before after + ) +' + +test_expect_success 'unstable topics cannot redirect or delete the production release workflow' ' + git init --bare unstable-release.git && + test_create_repo unstable-release-source && + ( + cd unstable-release-source && + git remote add origin ../unstable-release.git && + write base shared && + mkdir -p .github/workflows && + write_release_workflow codex .github/workflows/codex-release.yml && + git add shared .github/workflows/codex-release.yml && + install_rerere_train && + git commit -m base && + git branch codex && + git branch aa/codex/stable codex && + git switch -c bb/codex/release-unstable codex && + write_release_workflow codex-unstable \ + .github/workflows/codex-release.yml && + git add .github/workflows/codex-release.yml && + git commit -m "release untrusted preview builds" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git switch -c release-deleted codex && + git rm .github/workflows/codex-release.yml && + git commit -m "delete production release workflow" && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/release-unstable release-deleted + ) && + + git clone unstable-release.git unstable-release-runner && + ( + cd unstable-release-runner && + fetch_all && + snapshot_refs ../unstable-release.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "release\|workflow\|protected" err && + test_path_is_missing result && + snapshot_refs ../unstable-release.git >after && + test_cmp before after && + old=$(git rev-parse origin/bb/codex/release-unstable) && + deleted=$(git rev-parse origin/release-deleted) && + git --git-dir=../unstable-release.git update-ref \ + refs/heads/bb/codex/release-unstable "$deleted" "$old" && + fetch_all && + snapshot_refs ../unstable-release.git >before-deletion && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result deleted-result \ + --updates deleted-updates --inputs deleted-inputs \ + --failure deleted-failure >deleted.out 2>deleted.err && + test_grep "release\|workflow\|protected" deleted.err && + test_path_is_missing deleted-result && + snapshot_refs ../unstable-release.git >after-deletion && + test_cmp before-deletion after-deletion + ) +' + +test_expect_success 'both candidates stage independently and publish under one atomic lease' ' + git init --bare unstable-promotion.git && + test_create_repo unstable-promotion-source && + ( + cd unstable-promotion-source && + git remote add origin ../unstable-promotion.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git switch -c aa/codex/stable master && + write stable stable-file && + git add stable-file && + git commit -m "atomic stable topic" && + git branch codex && + git switch -c bb/codex/preview-unstable codex && + write preview preview-file && + git add preview-file && + git commit -m "atomic unstable topic" && + git branch codex-unstable && + git branch meta master && + install_unstable_meta_state meta master codex codex-unstable && + git push origin master meta codex codex-unstable \ + aa/codex/stable bb/codex/preview-unstable + ) && + + git clone unstable-promotion.git unstable-promotion-runner && + ( + cd unstable-promotion-runner && + fetch_all && + sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure && + stable=$(cat result) && + unstable=$(updated_tip codex-unstable updates) && + snapshot_without_staging ../unstable-promotion.git >primary-before && + GIT_TRACE=1 sh "$codex_branch" stage \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >stable-stage.out 2>stable-stage.trace && + test "$stable" = "$(git --git-dir=../unstable-promotion.git \ + rev-parse refs/heads/codex-staging)" && + test_expect_code 1 sh "$codex_branch" promote \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >missing-stage.out 2>missing-stage.err && + snapshot_without_staging ../unstable-promotion.git \ + >after-missing-stage && + test_cmp primary-before after-missing-stage && + + GIT_TRACE=1 sh "$codex_branch" stage \ + --remote origin --staging codex-unstable-staging \ + --inputs inputs --updates updates \ + >unstable-stage.out 2>unstable-stage.trace && + test "$unstable" = "$(git --git-dir=../unstable-promotion.git \ + rev-parse refs/heads/codex-unstable-staging)" && + snapshot_without_staging ../unstable-promotion.git \ + >after-both-stages && + test_cmp primary-before after-both-stages && + git --git-dir=../unstable-promotion.git update-ref \ + refs/heads/codex-unstable-staging "$stable" "$unstable" && + test_expect_code 1 sh "$codex_branch" promote \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >moved-stage.out 2>moved-stage.err && + snapshot_without_staging ../unstable-promotion.git \ + >after-stage-race && + test_cmp primary-before after-stage-race && + git --git-dir=../unstable-promotion.git update-ref \ + refs/heads/codex-unstable-staging "$unstable" "$stable" && + + GIT_TRACE=1 sh "$codex_branch" promote \ + --remote origin --staging codex-staging \ + --inputs inputs --updates updates \ + >promote.out 2>promote.trace && + test_grep "push --atomic --porcelain" promote.trace && + test_grep \ + "force-with-lease=refs/heads/codex-staging:$stable" \ + promote.trace && + test_grep \ + "force-with-lease=refs/heads/codex-unstable-staging:$unstable" \ + promote.trace && + test_grep ":refs/heads/codex-staging" promote.trace && + test_grep ":refs/heads/codex-unstable-staging" promote.trace && + while IFS="$(printf "\t")" read -r ref old new + do + lease_old=$old && + case "$lease_old" in + 0000000000000000000000000000000000000000|\ + 0000000000000000000000000000000000000000000000000000000000000000) + lease_old= + ;; + esac && + test_grep "force-with-lease=$ref:$lease_old" promote.trace && + test "$new" = "$(git --git-dir=../unstable-promotion.git \ + rev-parse "$ref")" || return 1 + done before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "bb/codex/conflict-unstable" err && + test_path_is_missing result && + if test -f failure + then + test_grep "bb/codex/conflict-unstable" failure && + ! grep -E "resolve .*--base(=| )codex .*--codex(=| )codex-unstable" \ + failure + else + test_grep "restack\|rebase\|resolve" err + fi && + snapshot_refs ../unstable-conflict.git >after && + test_cmp before after + ) +' + +test_expect_success 'published stable topics cannot all disappear behind unstable previews' ' + git init --bare unstable-stable-retired.git && + test_create_repo unstable-stable-retired-source && + ( + cd unstable-stable-retired-source && + git remote add origin ../unstable-stable-retired.git && + write base shared && + git add shared && + install_rerere_train && + git commit -m base && + git switch -c aa/codex/stable master && + write stable stable-file && + git add stable-file && + git commit -m "last published stable topic" && + git branch codex && + git branch meta master && + install_meta_state meta master codex && + git switch -c bb/codex/preview-unstable codex && + write preview preview-file && + git add preview-file && + git commit -m "unstable preview cannot replace stable" && + git push origin master meta codex bb/codex/preview-unstable + ) && + + git clone unstable-stable-retired.git unstable-stable-retired-runner && + ( + cd unstable-stable-retired-runner && + fetch_all && + snapshot_refs ../unstable-stable-retired.git >before && + test_expect_code 1 sh "$codex_branch" rewrite --remote origin \ + --base master --codex codex --result result \ + --updates updates --inputs inputs --failure failure \ + >out 2>err && + test_grep "all enrolled Codex topics were removed" err && + test_path_is_missing result && + snapshot_refs ../unstable-stable-retired.git >after && + test_cmp before after + ) +' + test_done