Skip to content

fix(review): surface per-model terminal errors and readable diagnostics in review-failure telemetry#8302

Open
JSONbored wants to merge 3 commits into
mainfrom
fix/ai-review-exhausted-observability
Open

fix(review): surface per-model terminal errors and readable diagnostics in review-failure telemetry#8302
JSONbored wants to merge 3 commits into
mainfrom
fix/ai-review-exhausted-observability

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Two observability defects made the 2026-07-23 ORB review outage (every block-mode dual review inconclusive → "AI review could not be completed" holds) harder to diagnose from Sentry alone:

  • ai_review_provider_exhausted masked the primary's real failure. The exhausted summary only logged the LAST error across all models × attempts, so the fallback's circuit_open (codex, structurally disabled) hid the primary's genuinely distinct terminal error (claude_code_error_429) — the single Sentry event pointed at the wrong provider (LOOPOVER-2A). runWorkersOpinion now also tracks a per-model terminal-error map (errorsByModel) and includes it in the exhausted log. The existing error field is unchanged, so Sentry grouping/fingerprints are unaffected.
  • review_diagnostics reached Sentry as unreadable "[Object]" entries. Both captureReviewFailure sites passed the raw AiReviewDiagnostic[]; the entries sit at normalize-depth 4, so the SDK's default normalizeDepth: 3 flattened each one to the literal string "[Object]", erasing exactly the model/attempt/status/error detail they exist to carry (LOOPOVER-2B). A new formatReviewDiagnosticsForCapture helper renders them as compact model#attempt:status[:error] strings, which survive normalization verbatim. The error field is errorMessage() output, never raw provider text, so the "withholds unsafe provider text" boundary is intact.

Validation

  • npx vitest run test/unit/ai-review.test.ts test/unit/ai-review-advisory.test.ts — 264 passed, including:
    • a regression test reproducing the exact outage shape (primary rate-limits, fallback circuit-opens) asserting the exhausted log carries BOTH models' own terminal errors while error stays the last one;
    • direct formatReviewDiagnosticsForCapture cases covering the with-error and without-error arms and the empty array.
  • npm run typecheck clean.
  • npm run test:coverage (unsharded) green.

…cs in review-failure telemetry

The ai_review_provider_exhausted summary only carried the LAST error across
all models x attempts, so a fallback's circuit_open masked the primary's
distinct terminal failure (a rate-limit 429) during the 2026-07-23 outage
(LOOPOVER-2A) -- the single Sentry event pointed at the wrong provider.
runWorkersOpinion now also tracks each model's own terminal error and logs
the map alongside the unchanged last-error field.

Both captureReviewFailure sites passed raw AiReviewDiagnostic[] into Sentry
context, where the SDK's default normalizeDepth flattened every entry to the
literal string "[Object]" (LOOPOVER-2B), erasing the model/attempt/status/
error detail. formatReviewDiagnosticsForCapture renders them as compact
model#attempt:status[:error] strings that survive normalization.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (5a5a4f1) and head (2c894f3) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.95%. Comparing base (1dba0ec) to head (2c894f3).
⚠️ Report is 17 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8302      +/-   ##
==========================================
- Coverage   92.43%   91.95%   -0.48%     
==========================================
  Files         791      791              
  Lines       79320    79337      +17     
  Branches    23952    23961       +9     
==========================================
- Hits        73317    72952     -365     
- Misses       4866     5305     +439     
+ Partials     1137     1080      -57     
Flag Coverage Δ
shard-1 54.91% <25.00%> (-2.85%) ⬇️
shard-2 49.10% <50.00%> (+1.44%) ⬆️
shard-3 56.29% <100.00%> (-0.30%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/ai-review-orchestration.ts 100.00% <ø> (ø)
src/services/ai-review.ts 96.90% <100.00%> (+0.01%) ⬆️

... and 7 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 24, 2026
@loopover-orb

loopover-orb Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-24 14:48:36 UTC

4 files · 2 blockers · CI green · clean

⏸️ Suggested Action - Manual Review

  • AI review could not be completed: The dual-model AI review did not return a usable verdict for this change.

Review summary
AI review could not be completed for this PR head. LoopOver is holding this PR for manual review instead of relying on deterministic signals alone.

Nits — 1 non-blocking
  • AI review could not be completed — The gate is held for a human reviewer rather than passed automatically; it re-evaluates on the next update.

Concerns raised — review before merging

  • No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example `Closes #123`) before opening the PR.

Decision drivers

  • ❌ Code review — 2 blockers (no AI review ran — blocker is from a non-AI gate check)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 14 registered-repo PR(s), 14 merged, 346 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 14 PR(s), 346 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 14 PR(s), 346 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Then work through the remaining 3 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 24, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 24, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui 2c894f3 Commit Preview URL

Branch Preview URL
Jul 24 2026, 02:08 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant