Pr multi review - #14
Open
ikethecoder wants to merge 20 commits into
Open
Conversation
…w skills Replace github.workflow_ref parsing (unreliable when called via workflow_call from another repo) with an explicit source_ref input, hardcoding bcgov/aps-devops as the source repo for the shared review scripts and skills. Also commit the two skills the workflow depends on (pr-review-process, review-response-format), which were never pushed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
prepare job never emitted a source_repo output, and source_ref was never emitted either, so the shared-scripts checkouts in review-gpt/review-claude/synthesize-and-implement silently resolved to empty strings. Hardcode source_repo to bcgov/aps-devops (the repo housing the shared scripts always, regardless of caller repo) and reference inputs.source_ref directly instead of round-tripping it through prepare's outputs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Anthropic rejects `temperature` for the current Claude models used here (400: "temperature is deprecated for this model"), so drop it from the review call. Also move review/implement models from claude-opus-5 to claude-sonnet-5. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
call-review-model.mjs now normalizes each provider's token usage into
a common { input_tokens, output_tokens, total_tokens } shape and
attaches it to gpt.json/claude.json as an optional `usage` field
(absent if the provider didn't return usage data). The workflow now
also passes the triggering user (github.triggering_actor) into the
triage prompt. pr-review-process's Step 4 picks both up to add an
optional "Run info" line to the final PR comment: who kicked off the
run and each reviewer's token usage, omitting any reviewer that has
no usage data rather than guessing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
anthropics/claude-code-action supports claude_code_oauth_token (a long-lived token from `claude setup-token` under a Pro/Max subscription) as an alternative to anthropic_api_key; the CLI prefers it when both are present. Declare CLAUDE_CODE_OAUTH_TOKEN as an optional reusable-workflow secret and pass it through — direct workflow_dispatch use picks it up automatically from a repo/org secret of the same name, no declaration needed there. ANTHROPIC_API_KEY stays required since review-claude's direct API call needs it regardless. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
actions/checkout and actions/upload-artifact go v4 -> v7 (both now run on Node 24; checkout v7's only other behavior change — blocking fork PR checkout on pull_request_target/workflow_run — doesn't apply, this workflow only uses workflow_dispatch/workflow_call). actions/download-artifact goes v4 -> v8 (its latest; v7 isn't the newest release for this action). v5's breaking change only affects downloading by artifact-ids, and every download here is by name, so no workflow changes needed beyond the version bump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Neither the OpenAI nor Anthropic Messages APIs return dollar cost, only token counts, so call-review-model.mjs now estimates cost_usd from a hardcoded $/MTok pricing table and attaches it to each reviewer's usage object. Anthropic figures are from the official pricing page; GPT-5.5 figures are from third-party trackers and are flagged as such since OpenAI doesn't publish a stable page for it — cost is omitted entirely for any model missing from the table. The Claude Code implement/triage step tracks its own exact spend (total_cost_usd) internally and writes it to the execution_file it outputs. Added a follow-up step that reads that file after the triage step finishes and posts the exact cost as a short addendum comment on the original PR — this can't be included in Claude's own comment since the cost isn't known until the run completes. The step is best-effort (continue-on-error, runs even on a failed triage step) so it never turns an otherwise-successful run red. pr-review-process's Step 4 now reports each reviewer's estimated cost next to its token usage, and is told not to estimate the implement step's cost itself since the follow-up step reports the exact figure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…al PR # pr-review-process now reports which model each agent used (reviewers' models come from their findings files' model field; the implement model is passed into the prompt since the running agent can't otherwise know its own model id) in the Run info line of the final summary comment. Task PR titles now start with "[PR #<original_pr_number>]" so it's immediately obvious from the PR list which original PR each task PR feeds back into, without opening it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ption
Adds a third review-claude-code job alongside review-gpt/review-claude:
instead of a raw Anthropic Messages API call, it installs and invokes
the `claude` CLI directly (claude -p --tools "" --output-format json),
so it authenticates via CLAUDE_CODE_OAUTH_TOKEN (falling back to
ANTHROPIC_API_KEY) rather than always drawing on the API key like
review-claude. --tools "" keeps it a plain one-shot completion with no
tool/agentic surface, matching the other reviewers' shape; --system-prompt
replaces Claude Code's own default system prompt so the reviewer isn't
carrying unrelated agentic framing. Its output feeds into
synthesize-and-implement as a third findings file alongside gpt.json
and claude.json.
The CLI's own result JSON carries exact total_cost_usd/usage, so unlike
the raw-API reviewers this one doesn't need the estimated-pricing-table
lookup — call-review-model.mjs now only falls back to the pricing-table
estimate when a caller hasn't already supplied usage.cost_usd directly.
Also fixes a real bug this surfaced: the final result object spread
`{ reviewer, model, usage, ...parsed.value }` let a reviewer's own
(frequently wrong) self-reported model id in its JSON response silently
overwrite our own authoritative, actually-invoked model string. Flipped
the spread order so our own values always win.
pr-review-process and review-response-format now document the third
reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed directly against the claude CLI: when both ANTHROPIC_API_KEY
and an OAuth/subscription login are present, the API key silently
takes precedence — even an invalid key wins ("claude.ai connectors
are disabled because ANTHROPIC_API_KEY ... takes precedence"), then
bills against it. On an account with a Pro/Max subscription but no
loaded API credits, this surfaces as "Credit balance is too low"
instead of drawing on the subscription's included usage.
Both review-claude-code and the triage step were passing
ANTHROPIC_API_KEY and CLAUDE_CODE_OAUTH_TOKEN unconditionally, so
whenever CLAUDE_CODE_OAUTH_TOKEN was set, ANTHROPIC_API_KEY (a
required secret, always present) silently won anyway. Both now only
pass ANTHROPIC_API_KEY through when CLAUDE_CODE_OAUTH_TOKEN is unset,
confirmed empirically that an empty-string env var is treated as
unset by the CLI (falls through to the OAuth login correctly).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds review_claude_api (boolean, default true) to both workflow_dispatch
and workflow_call inputs, gating the review-claude job's `if:`. Useful
to skip the ANTHROPIC_API_KEY-billed reviewer entirely — e.g. when
relying on review-gpt + the CLAUDE_CODE_OAUTH_TOKEN-billed
review-claude-code and there's no API credit balance to spend.
Skipping a needed job doesn't satisfy synthesize-and-implement's
implicit `if: success()`, so it now has an explicit
`if: ${{ !failure() && !cancelled() }}` to tolerate review-claude
being skipped while still blocking on a genuine failure anywhere in
the dependency chain. The review-claude artifact download is now
conditional on the same input too, since review-claude never uploads
it when skipped — pr-review-process's Step 4 already treats a missing
findings file the same as a parse_error one, so no skill change needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
review-claude (raw Anthropic API, billed via ANTHROPIC_API_KEY) is now opt-in rather than opt-out — review-gpt and review-claude-code (billed via CLAUDE_CODE_OAUTH_TOKEN) run by default; pass review_claude_api: true to also spend API credits on the third reviewer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run-name is evaluated once at trigger time with only inputs/github context available — no job has run yet, so the PR title (fetched via gh pr view inside the prepare job) can't be included; there's also no API to rename a run after it starts. PR number is known immediately from the trigger input, so that's what the run list now shows instead of a generic "Multi-Model PR Review #N". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a "Label PR as reviewed" step after the triage step, applying a pr-multi-reviewed label to the original PR. Creates the label (idempotently, via --force) if it doesn't already exist in the repo being reviewed. Runs with the default `if:` (implicit success()), so the label only lands once triage actually finished rather than after every attempt. Uses the job's existing issues: write permission, no new grants needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clarifies that this job is the raw-Anthropic-API reviewer, as distinct from review-claude-code (billed via the claude CLI/subscription). Updates the job id, its artifact name (both upload and download sides), the needs array, and every prose comment/description referencing the job by name. File names (claude.json) and the --provider claude script flag are unchanged — those are tied to the provider, not the job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.