Skip to content

fix(checkup): auto-degrade to same-family fixer when one provider family is down (#1941)#1942

Open
Serhan-Asad wants to merge 3 commits into
mainfrom
fix/issue-1941-checkup-auto-degrade
Open

fix(checkup): auto-degrade to same-family fixer when one provider family is down (#1941)#1942
Serhan-Asad wants to merge 3 commits into
mainfrom
fix/issue-1941-checkup-auto-degrade

Conversation

@Serhan-Asad

Copy link
Copy Markdown
Collaborator

Fixes #1941.

Root cause

The checkup review loop deadlocked instead of degrading when only one AI provider family was alive. With the cloud's config --reviewer codex --fixer claude --fallback-reviewer-on-failure:

  1. codex (reviewer) hard-fails (auth dead pool-wide, pdd_cloud#2897).
  2. _maybe_run_fallback_reviewer promotes claude — the fixer's own role — to review the PR (this is the only fallback reviewer available).
  3. claude reports real findings.
  4. The loop hit break at the _maybe_run_fallback_reviewer "findings" branch with stop reason "secondary reviewer claude reported findings (fallback)"no fix round ever ran, because the only fixer (claude) was now the active reviewer and same-role fixing was blocked.

The findings were legitimate (the gate worked); the self-heal was structurally impossible. Result on both staging PRs: same-role-review-fix: false, max-rounds-reached: false, terminal review_findings_remain.

Note the asymmetry that made this a bug: the reviewer_fallback path (a distinct 3rd reviewer) falls through to a fix round, but the _maybe_run_fallback_reviewer path (fixer-as-reviewer) terminated.

Fix

In the fallback-reviewer "findings" branch of run_checkup_review_loop, instead of terminating, auto-degrade: hand the actionable findings to the fixer on the next iteration (this mirrors the pre-existing fallback-clean gate-findings handoff a few lines above — pending_findings = ...; continue), reassign the active reviewer to the fallback role, and let the normal fix + fresh-verify path run a fresh same-family session.

Disclosed honestly:

  • new ReviewLoopState.role_independence: "independent" | "degraded (<role> unavailable)" — rendered as a role-independence: report line and a role_independence key in final-state.json;
  • same_role_review_fix stamped true for the degraded run;
  • the fix-failure stop reason names the vacancy explicitly (e.g. Fixer claude could not address claude's findings [role-independence degraded (codex unavailable)].) instead of a bare "could not address".

The superseded primary already renders (optional, superseded by <fallback>) (because state.active_reviewer points at the fallback), so the pdd_cloud verdict adapter drops it from the required-reviewer set and ships on the degraded-but-clean result — verified against the real adapter; no cloud change needed (this is exactly what the issue asked the CLI to make automatic + disclosed).

Behavior is unchanged when both families are alive (this path only triggers on primary-reviewer failure).

Acceptance criteria

  • ✅ codex dead, claude fallback-reviews + reports findings → fix round RUNS via fresh same-family claude session; fresh-final verify runs; report carries same-role-review-fix: true + role-independence: degraded (codex unavailable).
  • ✅ both families alive → unchanged (cross-family, no stamp, fallback role never invoked).
  • ✅ degraded same-family fixer ALSO fails → stop reason names the vacancy explicitly.
  • ✅ reproduces the #4234/#4235 shape end-to-end in a fixture.

Tests

tests/test_checkup_review_loop.py:

  • test_issue_1941_auto_degrades_to_same_family_fixer_on_fallback_findings — the deadlock→ship path; asserts the review-loop Machine Verdict passes (ships) plus final-state.json disclosure fields.
  • test_issue_1941_both_families_alive_stays_independent
  • test_issue_1941_degraded_same_family_fixer_failure_names_vacancy

Full test_checkup_review_loop.py suite: 306 passed. Broader test_agentic_checkup.py / test_checkup_report.py / commands/test_checkup.py: 91 passed, 1 skipped. pyflakes + pylint(E) clean on the changed file.

Known limitations (out of scope, documented)

  • A distinct entry path — pre-seeded pending_findings (layer1_step5_evidence, epic-only) + provider down at verify time — still terminates; this change is neutral there (it neither helps nor worsens), and fixing it means relaxing the deliberately-disabled verify-path independence guard. Separate concern.
  • If a live independent fixer_fallback is configured alongside fallback_reviewer_on_failure, the degrade runs the configured primary fixer (same-family) first and only consults the independent fallback if it fails; disclosure then errs conservative (claims a weaker guarantee than reality). Reordering to prefer a live independent fallback would be a policy change beyond Checkup review loop deadlocks when one provider family is down: auto-degrade to same-family fixer with report disclosure instead of terminating with findings #1941.
  • The degrade consumes one loop round (identical to the existing fallback-clean handoff), so it needs max_rounds >= 2 (prod default is 5).

Reviewer notes

Independently adversarially reviewed (2 rounds) — approved, no blockers, no new defects. (codex-work was dead pool-wide during this work — the exact failure this PR fixes — so a Claude Opus reviewer stood in.)

🤖 Generated with Claude Code

…ily is down (#1941)

The review loop dead-locked instead of degrading when only one AI provider
family was alive. With `--reviewer codex --fixer claude
--fallback-reviewer-on-failure`, a codex hard-failure promoted claude (the
fixer's own role) to fallback-review the PR; when claude reported real
findings the loop hit `break` with "secondary reviewer ... reported findings
(fallback)" and never ran a fix round — even though a fresh same-family claude
session could execute the concrete findings and be re-reviewed.

Fix: in the fallback-reviewer "findings" branch of `run_checkup_review_loop`,
instead of terminating, hand the actionable findings to the fixer on the next
iteration (mirrors the existing fallback-clean gate-findings handoff just
above), reassign the active reviewer to the fallback role, and let the normal
fix + fresh-verify path run. Disclose the weaker guarantee honestly:

- new `ReviewLoopState.role_independence` (`"independent"` |
  `"degraded (<role> unavailable)"`), rendered as a `role-independence:` report
  line and a `role_independence` key in final-state.json;
- `same_role_review_fix` is stamped true for the degraded run;
- the fix-failure stop reason names the vacancy explicitly instead of a bare
  "could not address".

The superseded primary already renders `(optional, superseded by <fallback>)`,
so the pdd_cloud verdict adapter drops it from the required-reviewer set and
ships on the degraded-but-clean result — no cloud change needed. Behavior is
unchanged when both families are alive.

Adds regression tests: the deadlock→ship path (asserts the review-loop Machine
Verdict passes + final-state disclosure), both-families-alive stays
independent, and degraded-fixer-also-fails names the vacancy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread context/checkup_review_loop_example.py Fixed
Address review of #1942: the code change added runtime same-role degradation
and a new `role_independence` state/report/final-state field, but the
prompt-driven source-of-truth surfaces still described the pre-#1941 contract.
Left unreconciled, `pdd sync` / the auto-heal bot would regenerate the code
from the stale prompt and drop the fix.

Reconcile all four surfaces with the implemented behavior:
- pdd/prompts/checkup_review_loop_python.prompt: specify the auto-degrade rule
  under `fallback_reviewer_on_failure` (fallback-reviewer findings → fresh
  same-family fix + required verify instead of dead-end), the `role-independence:`
  report header line, the `role_independence` final-state.json key + revised
  `same_role_review_fix` semantics (config-time OR runtime), the appended-field
  ordering, and the degraded fix-failure stop-reason suffix.
- architecture.json: add the #1941 auto-degrade sideEffect + `role_independence`
  to the final-state.json field list for run_checkup_review_loop.
- README.md: document the auto-degrade behavior + disclosure under
  `--fallback-reviewer-on-failure`.
- context/checkup_review_loop_example.py: add `role_independence` to the
  ReviewLoopState example, the final-state payload, the report-header comment,
  and the rendered-report example.

No code/behavior change; docs-only reconciliation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Serhan-Asad Serhan-Asad force-pushed the fix/issue-1941-checkup-auto-degrade branch from 36665ac to 9bee527 Compare July 8, 2026 23:44
@Serhan-Asad

Copy link
Copy Markdown
Collaborator Author

Review responses (changes requested)

Thanks — all three failure modes addressed.

Failure Mode 1 — source-of-truth prompt/doc drift (BLOCKING): fixed. Reconciled every generated/authored surface with the implemented behavior so a pdd sync / auto-heal regen reproduces the fix instead of overwriting it (commit docs(checkup): reconcile source-of-truth surfaces):

  • pdd/prompts/checkup_review_loop_python.prompt — added the auto-degrade rule under fallback_reviewer_on_failure, the role-independence: report line, the role_independence final-state key + revised same_role_review_fix semantics (config-time OR runtime), the appended-field ordering, and the degraded stop-reason suffix.
  • architecture.json — added the Checkup review loop deadlocks when one provider family is down: auto-degrade to same-family fixer with report disclosure instead of terminating with findings #1941 auto-degrade sideEffect + role_independence to the final-state field list.
  • README.md — documented the auto-degrade + disclosure under --fallback-reviewer-on-failure.
  • context/checkup_review_loop_example.py — added role_independence to the state example, final-state payload, header comment, and rendered-report example.

(Note: the auto-heal bot pushed a heal commit based on the code-only commit before the prompt was reconciled, so it regenerated the docs to the pre-#1941 contract. I force-pushed the #1941-aware prompt so a re-heal derives from the correct source.)

Failure Mode 2 — cloud adapter disclosure: addressed via companion PR promptdriven/pdd_cloud#3126. The checkup_verdict_adapter now parses the role-independence: line into a structured VerificationVerdict.role_independence field and routes an otherwise-clean degraded run to the existing ship_degraded verdict (never a silent ship). Adapter suite 82 passed; verifier+engine consumers 115 passed; end-to-end (CLI degraded report → adapter) verified.

Failure Mode 3 — degrade with a live independent fixer_fallback: documented (err-safe). Confirmed fixer_fallback is plumbed, so a future caller could reach it. In this path the configured primary fixer equals the fallback reviewer's family by construction, so the degrade runs same-family first and the degraded stamp is honest for the primary attempt; if an independent fallback later fixes, disclosure over-states degradation (safe direction). Re-ordering to prefer a live independent fallback is a policy change beyond #1941; noted in the PR's Known Limitations.

…gnostics (#1941)

Review hardening for #1942: the cloud verdict adapter reads a
`role-independence:` header line to route a degraded run to `ship_degraded`,
but untrusted reviewer stderr rendered in `### Reviewer Diagnostics` could echo
`role-independence: independent` and — since the adapter previously took the
last match in the whole report — override the authoritative header, silently
downgrading a degraded disclosure back to a plain ship (or the reverse).

Add `_defang_role_independence_inline` to `_defang_adapter_trip_wires` so any
`role-independence:`/`role_independence=` marker leaking from reviewer stderr is
neutralized at the render boundary, exactly like the existing
`reviewer-status:` / `fresh-final-review:` / `max-*-reached:` markers. The
on-disk `final-state.json` and `reviewer_status_details` keep the original text.

Prompt spec (27b) updated to list the new marker. Companion adapter-side
defense (header-scoped parse) is in promptdriven/pdd_cloud#3126.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants