Skip to content

fix: preserve benchmark previews when publishing fails - #2

Merged
cpunion merged 1 commit into
mainfrom
fix/publish-soft-failure
Jul 30, 2026
Merged

fix: preserve benchmark previews when publishing fails#2
cpunion merged 1 commit into
mainfrom
fix/publish-soft-failure

Conversation

@cpunion

@cpunion cpunion commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • degrade missing/unwritable benchmark data repositories to an artifact preview with a PR warning
  • keep artifact validation, trusted configuration, rendering, preview upload, and PR comments as hard failures
  • optionally dispatch a repository event after benchmark data is ready, without adding a caller-side runner job
  • document the optional external repository/token behavior and cover workflow invariants with tests

Tests

  • npm run check
  • YAML parse and Bash syntax checks for the reusable publisher
  • empty history fallback exercised with an unavailable origin

@fennoai fennoai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: preserve benchmark previews when publishing fails

The core degrade-to-preview design is sound. continue-on-error is applied to the right steps (writable checkout, commit, dispatch), trust-establishing steps (artifact download, trusted-config checkout, render) correctly remain hard failures, and the HISTORY_AVAILABLE gating of the Pages URL holds across the writable-success, writable-fail, and not-writable paths. Token scoping is correct — github.token is only used for the same-data-repository path and never sent to an external repo — and all fork-controlled values feeding jq/gh api are passed as --arg/env/form fields rather than interpolated, so no injection vector was found. README and the new tests accurately match the implementation.

Two findings worth addressing are left inline. One additional note below.

Note — redundant fetch of the data repo on every run (.github/workflows/publish.yml:182-224): Neither the writable checkout (line 182, no ref:) nor the preview clone (line 203, git clone --depth=1 with no --branch) targets inputs.data_branch (default pages); both fetch the data repo's default branch, which Select data branch then discards and re-fetches via a second network round-trip. Passing ref:/--branch "$DATA_BRANCH" (with the existing orphan fallback for a fresh repo) would fetch the right branch once and avoid downloading default-branch content that is never used. Pre-existing pattern, not introduced by this PR, but relevant to the touched steps.

echo "history-loaded=false" >> "$GITHUB_OUTPUT"
fi

- name: Select data branch

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Select data branch runs unconditionally with no continue-on-error, unlike every other data-side step in this PR. preview-data guarantees a benchmark-data git repo exists, but the git fetch origin ... / git checkout -B / git checkout --orphan / git rm calls here run under set -euo pipefail and are unguarded. A transient failure (e.g. the remote becoming unreachable after the initial clone) will hard-fail the entire publish job, which defeats the PR's goal of degrading to a preview. Consider whether this step should also tolerate failure, or confirm the hard failure is intended.

DISPATCH_EVENT: ${{ inputs.data_dispatch_event }}
GH_TOKEN: >-
${{ steps.source.outputs.same-data-repository == 'true' &&
github.token || secrets.data_token }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When same-data-repository == 'true', the dispatch uses github.token. Events triggered by the default GITHUB_TOKEN do not start new workflow runs (GitHub's recursion guard). Since the documented use case for data_dispatch_event is to kick off a Pages deployment workflow in the data repo, that downstream workflow will silently never fire on the same-repo path. Worth verifying against the README recommendation (README:420-424); a PAT/app token may be required for the dispatch to be actionable even for the same repository.

@cpunion
cpunion merged commit b7eae35 into main Jul 30, 2026
0 of 2 checks passed
@cpunion
cpunion deleted the fix/publish-soft-failure branch July 30, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant