From 65065e7427129e5e13fc82a212d43d09ebd33d33 Mon Sep 17 00:00:00 2001 From: Trask Stalnaker Date: Wed, 5 Aug 2026 11:18:41 -0700 Subject: [PATCH 1/5] Pin the pull request dashboard stable channel to v0.5.0 The staged rollout landed with the stable jobs still calling the local reusable workflow, so both channels ran the code from the triggering commit and a change still reached every configured repository the moment it merged. The first promotion could not be written before a release commit existed. Point every stable job at v0.5.0, in both the `uses:` ref and the matching `code_ref` input, so the pinned scripts match the workflow YAML that invoked them. Canary repositories keep running from the triggering commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13be8112-8ee4-4bc9-b6d0-0ad0a07ad80d --- .github/scripts/pull-request-dashboard/RATIONALE.md | 8 ++++---- .github/workflows/pull-request-dashboard.yml | 13 +++++++------ pull-request-dashboard/README.md | 4 +--- 3 files changed, 12 insertions(+), 13 deletions(-) diff --git a/.github/scripts/pull-request-dashboard/RATIONALE.md b/.github/scripts/pull-request-dashboard/RATIONALE.md index ca79dc5fa79..98c6615ea39 100644 --- a/.github/scripts/pull-request-dashboard/RATIONALE.md +++ b/.github/scripts/pull-request-dashboard/RATIONALE.md @@ -63,10 +63,10 @@ the implementation understandable and operationally cheap. branch. The version that each repository runs is then visible in the workflow file, a rollback is a revert, and the reference stays hash pinned the way every other action reference in this repository is. -- Until the first promotion the stable jobs call the local workflow, so both - channels run the same code. A cross-repository reference cannot be introduced - in the same change that adds the `code_ref` input, because the commit it would - have to pin does not exist yet. +- The first promotion could not land in the change that added the `code_ref` + input, because the release commit it had to pin did not exist yet. Until it + landed the stable jobs called the local workflow and both channels ran the + same code. ## Workflow Concurrency diff --git a/.github/workflows/pull-request-dashboard.yml b/.github/workflows/pull-request-dashboard.yml index 2288b2dc5b0..2ae835d1772 100644 --- a/.github/workflows/pull-request-dashboard.yml +++ b/.github/workflows/pull-request-dashboard.yml @@ -188,9 +188,7 @@ jobs: # on the canary repositories. # # `uses` cannot take an expression, so each entry path has one job per - # channel. The stable jobs below still call the local workflow, which means - # both channels currently run the same code. Promoting for the first time, and - # every time after, means cutting a release and opening a pull request that + # channel. Promoting means cutting a release and opening a pull request that # points every stable job at that release: # # uses: open-telemetry/shared-workflows/.github/workflows/pull-request-dashboard-repo.yml@ # vX.Y.Z @@ -230,9 +228,10 @@ jobs: fail-fast: false max-parallel: 3 matrix: ${{ fromJSON(needs.resolve-targets.outputs.stable_matrix) }} - uses: ./.github/workflows/pull-request-dashboard-repo.yml + uses: open-telemetry/shared-workflows/.github/workflows/pull-request-dashboard-repo.yml@94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 with: repository: ${{ matrix.name }} + code_ref: 94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 secrets: PR_DASHBOARD_PRIVATE_KEY: ${{ secrets.PR_DASHBOARD_PRIVATE_KEY }} SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} @@ -268,10 +267,11 @@ jobs: permissions: contents: write copilot-requests: write - uses: ./.github/workflows/pull-request-dashboard-repo.yml + uses: open-telemetry/shared-workflows/.github/workflows/pull-request-dashboard-repo.yml@94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 with: repository: ${{ inputs.repository }} pr_number: ${{ inputs.pr_number }} + code_ref: 94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 secrets: PR_DASHBOARD_PRIVATE_KEY: ${{ secrets.PR_DASHBOARD_PRIVATE_KEY }} SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} @@ -302,10 +302,11 @@ jobs: permissions: contents: write copilot-requests: write - uses: ./.github/workflows/pull-request-dashboard-repo.yml + uses: open-telemetry/shared-workflows/.github/workflows/pull-request-dashboard-repo.yml@94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 with: repository: ${{ inputs.repository }} pr_number: ${{ needs.resolve-head-sha.outputs.pr_number }} + code_ref: 94425809b433630371efafff2067f2c5b19f3fc2 # v0.5.0 secrets: PR_DASHBOARD_PRIVATE_KEY: ${{ secrets.PR_DASHBOARD_PRIVATE_KEY }} SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} diff --git a/pull-request-dashboard/README.md b/pull-request-dashboard/README.md index d5c4dd71e10..9db902be33d 100644 --- a/pull-request-dashboard/README.md +++ b/pull-request-dashboard/README.md @@ -261,9 +261,7 @@ its settings, takes effect on the next run in both channels. as a comment the way actions are pinned elsewhere in this repository. `test_rollout.py` fails if the two disagree. -Rolling back before a promotion means not promoting; after a promotion it means -reverting the bump. Until the first promotion the stable jobs call the workflow -at the triggering commit, so both channels run the same code. +Rolling back means reverting the bump to an earlier release commit. Running two versions at once adds two rules for changes to this workflow: From 9b6a79086ca38507dad0a81205713a80f4266da3 Mon Sep 17 00:00:00 2001 From: Trask Stalnaker Date: Wed, 5 Aug 2026 11:35:01 -0700 Subject: [PATCH 2/5] Drop the rationale bullet for the pre-promotion state this change ends Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13be8112-8ee4-4bc9-b6d0-0ad0a07ad80d --- .github/scripts/pull-request-dashboard/RATIONALE.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.github/scripts/pull-request-dashboard/RATIONALE.md b/.github/scripts/pull-request-dashboard/RATIONALE.md index 98c6615ea39..50b0f1d0523 100644 --- a/.github/scripts/pull-request-dashboard/RATIONALE.md +++ b/.github/scripts/pull-request-dashboard/RATIONALE.md @@ -63,10 +63,6 @@ the implementation understandable and operationally cheap. branch. The version that each repository runs is then visible in the workflow file, a rollback is a revert, and the reference stays hash pinned the way every other action reference in this repository is. -- The first promotion could not land in the change that added the `code_ref` - input, because the release commit it had to pin did not exist yet. Until it - landed the stable jobs called the local workflow and both channels ran the - same code. ## Workflow Concurrency From 2ba61c7068d87ce5c0e57f550596778ee053423d Mon Sep 17 00:00:00 2001 From: Trask Stalnaker Date: Wed, 5 Aug 2026 11:37:05 -0700 Subject: [PATCH 3/5] Document what rolling back a promotion actually does Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13be8112-8ee4-4bc9-b6d0-0ad0a07ad80d --- pull-request-dashboard/README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/pull-request-dashboard/README.md b/pull-request-dashboard/README.md index 9db902be33d..ede960eec50 100644 --- a/pull-request-dashboard/README.md +++ b/pull-request-dashboard/README.md @@ -261,7 +261,21 @@ its settings, takes effect on the next run in both channels. as a comment the way actions are pinned elsewhere in this repository. `test_rollout.py` fails if the two disagree. -Rolling back means reverting the bump to an earlier release commit. +### Rolling back + +Reverting the bump to an earlier release commit is the rollback path, and it is +safe, but it is not always self-healing. Each repository's state branch records +the delivery versions of the last worker that ran, and a worker behind on any of +them skips delivery rather than delivering an older shape. Reverting across a +version bump therefore stops delivery for the rolled-back repositories, and +state files whose version no longer matches are regenerated. + +Reverting a change that bumped no version resumes normally. Reverting one that +did needs `delivery-versions.json` deleted from the affected repositories' state +branches, which lets the older code claim the versions it does understand. +Canary repositories are unaffected either way, since they never run pinned code. + +### Changing the workflow Running two versions at once adds two rules for changes to this workflow: From f2537936220f9f3ada6931489ca816b22a4e56a1 Mon Sep 17 00:00:00 2001 From: Trask Stalnaker Date: Wed, 5 Aug 2026 11:38:41 -0700 Subject: [PATCH 4/5] Explain rollback as stopping a bad change rather than restoring delivery Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13be8112-8ee4-4bc9-b6d0-0ad0a07ad80d --- .../pull-request-dashboard/RATIONALE.md | 5 ++++ pull-request-dashboard/README.md | 26 ++++++++++++------- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/.github/scripts/pull-request-dashboard/RATIONALE.md b/.github/scripts/pull-request-dashboard/RATIONALE.md index 50b0f1d0523..cc18dfb4e47 100644 --- a/.github/scripts/pull-request-dashboard/RATIONALE.md +++ b/.github/scripts/pull-request-dashboard/RATIONALE.md @@ -63,6 +63,11 @@ the implementation understandable and operationally cheap. branch. The version that each repository runs is then visible in the workflow file, a rollback is a revert, and the reference stays hash pinned the way every other action reference in this repository is. +- A rollback stops a bad change rather than restoring delivery. Downgraded code + reads newer delivery state as empty, so delivering from it would repeat + reminders and re-review requests already sent; the delivery version check + makes it skip delivery instead. Rolling forward is the way out, and a paused + dashboard is the cheaper failure. ## Workflow Concurrency diff --git a/pull-request-dashboard/README.md b/pull-request-dashboard/README.md index ede960eec50..478d6d8b473 100644 --- a/pull-request-dashboard/README.md +++ b/pull-request-dashboard/README.md @@ -263,17 +263,23 @@ its settings, takes effect on the next run in both channels. ### Rolling back -Reverting the bump to an earlier release commit is the rollback path, and it is -safe, but it is not always self-healing. Each repository's state branch records -the delivery versions of the last worker that ran, and a worker behind on any of -them skips delivery rather than delivering an older shape. Reverting across a -version bump therefore stops delivery for the rolled-back repositories, and -state files whose version no longer matches are regenerated. - +Reverting the bump to an earlier release commit stops the fleet from running a +bad change. What it does not do is resume delivery from newer state, and that is +deliberate: delivered author reminders and Copilot re-review requests live in +versioned state files that read as empty when their version does not match, so +downgraded code would see nothing as delivered and send it all again. + +Each repository's state branch therefore records the delivery versions of the +last worker that ran, and a worker behind on any of them skips delivery instead. Reverting a change that bumped no version resumes normally. Reverting one that -did needs `delivery-versions.json` deleted from the affected repositories' state -branches, which lets the older code claim the versions it does understand. -Canary repositories are unaffected either way, since they never run pinned code. +did leaves the stable repositories not delivering, which is the intended +outcome: a paused maintainer aid is cheap, and duplicate reminders across the +fleet are not. + +Recovery from that state is to roll forward — fix the change and promote a new +release — rather than to clear the recorded versions, which would re-enable the +duplicate delivery the check exists to prevent. Canary repositories are +unaffected either way, since they never run pinned code. ### Changing the workflow From acb3f260f2a8fe4b8289fd3a42534d28f9ab3002 Mon Sep 17 00:00:00 2001 From: Trask Stalnaker Date: Wed, 5 Aug 2026 11:45:43 -0700 Subject: [PATCH 5/5] Simplify the rollback section Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13be8112-8ee4-4bc9-b6d0-0ad0a07ad80d --- pull-request-dashboard/README.md | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-) diff --git a/pull-request-dashboard/README.md b/pull-request-dashboard/README.md index 478d6d8b473..b5221f3d597 100644 --- a/pull-request-dashboard/README.md +++ b/pull-request-dashboard/README.md @@ -263,22 +263,10 @@ its settings, takes effect on the next run in both channels. ### Rolling back -Reverting the bump to an earlier release commit stops the fleet from running a -bad change. What it does not do is resume delivery from newer state, and that is -deliberate: delivered author reminders and Copilot re-review requests live in -versioned state files that read as empty when their version does not match, so -downgraded code would see nothing as delivered and send it all again. - -Each repository's state branch therefore records the delivery versions of the -last worker that ran, and a worker behind on any of them skips delivery instead. -Reverting a change that bumped no version resumes normally. Reverting one that -did leaves the stable repositories not delivering, which is the intended -outcome: a paused maintainer aid is cheap, and duplicate reminders across the -fleet are not. - -Recovery from that state is to roll forward — fix the change and promote a new -release — rather than to clear the recorded versions, which would re-enable the -duplicate delivery the check exists to prevent. Canary repositories are +Revert the bump to an earlier release commit. If the reverted change bumped no +delivery version, the stable channel resumes normally. If it did, those +repositories stop delivering rather than repeat reminders they can no longer see +as sent, so recovery is to roll forward with a fix. Canary repositories are unaffected either way, since they never run pinned code. ### Changing the workflow