Bound execution during model provider failures

[sentry-junior]

Upstream model latency can make Warden appear stuck for 20–60 minutes. Warden currently compounds this with unbounded normal skill runs and retries at both the Pi provider layer and hunk-analysis layer.

Current behavior

• Pi allows 2 internal provider retries by default in packages/warden/src/sdk/runtimes/pi.ts.
• Hunk analysis allows 3 additional retries in packages/warden/src/sdk/retry.ts.
• Normal skill runs do not pass a timeout to Pi.
• The circuit breaker is global and based on consecutive failures, so unrelated successes can reset it.
• Timed-out or failed hunks can be difficult to distinguish operationally from successful reviews with no findings.

Recent telemetry showed Kimi K2.6 successful calls with p95 around 6.2 minutes and a 42.6-minute maximum; provider errors reached 58 minutes. A single OpenRouter HTTP request lasted roughly 30 minutes. Kimi K3 had a substantially better measured tail.

Proposed execution policy

1. Bound execution

   • Provider request deadline: 2 minutes
   • Hunk budget including retries/fallback: 5 minutes
   • Review budget: 15 minutes
   • Cancel outstanding work when a budget expires

2. Use one retry owner

   • Let Warden own retries instead of stacking Pi and hunk retries
   • Retry at most once and only when enough budget remains
   • Do not retry auth errors, invalid model selectors, or requests that already consumed most of the budget

3. Support ordered model failover

   • Configure primary and fallback routes
   • Fail over only for timeout/provider-unavailable outcomes
   • Record the attempted route and route that produced the result

4. Scope bulkheads and circuit breakers by provider/model

   • Cap concurrency per route
   • Use rolling timeout/error rates rather than one global consecutive-failure count
   • Temporarily open unhealthy routes and send new work to fallback

5. Report honest outcomes

   • Distinguish complete, partial, and provider-unavailable reviews
   • Preserve completed skill results when other work fails
   • Never present timed-out hunks as a clean no-findings review

6. Add operational telemetry

   • Resolved request provider/model on every span
   • Actual provider request duration and time to first token
   • Attempt number, timeout budget, retry/failover reason, and circuit state
   • Timeout, fallback, partial-review, and p95 metrics by route

7. Reduce upstream exposure

   • Consolidate overlapping skills where practical
   • Tighten path filtering before dispatch
   • Prioritize likely-relevant files and skip optional verification when the run budget is low

Suggested implementation order: deadlines, retry de-duplication, fallback routing, provider-scoped concurrency/circuit breaking, then telemetry and outcome cleanup.

Related observability fix: #454.

Requested by David Cramer via Junior.

--

View Junior Session in Sentry