Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

[don-petry]

Why

The canary-ring promotion model (epic #495) advances a release next → ring0 → ring1 → stable, gated at every step by a soak on the prior ring. Today that gate is qualitative — the runbook says "confirm its callers' runs are healthy before advancing" — which can't be automated or applied consistently. The first real promotion of the six #482 reusables is in flight (.github-private#870, next now at v2.1.0), so we need a concrete, mechanical soak standard the promotion can be held to (and that canary-rollout.sh / #501 can eventually enforce for cross-repo agents).

What the data says (why a naive rule fails)

14-day run volume on the next tier (.github-private), executed = success+failure (skips excluded); 0 failures across all six → baseline ≈ 0%:

reusable	exec'd/day	50%/day	notes
agent-shield	~71	~36	high volume
dependency-audit	~71	~36	high volume
pr-review-mention	~52	~26	38% skipped
auto-rebase	9	~5	medium
dependabot-automerge	3	~1.5	96% skipped
dependabot-rebase	0	0	no runs in 14d

A literal "N hours + 50% of daily average" breaks three ways:

1. The time floor and the count aren't independent — at 71 runs/day, 36 runs takes ~12h, so the count silently dominates for busy reusables and the time floor never binds.
2. It collapses at the low end — dependabot-rebase (0 runs) gets no sample gate and may not fire in any short window; dependabot-automerge is 96% skips. Only time can protect these.
3. Skips and volume-capping distort the average — must count executed runs only, and the busy three are so high-volume that 50% balloons the soak.

Proposed standard

A ring advance passes for a reusable when all hold (evaluated per reusable, per ring):

• Time floor: ≥ 12 h since the channel tag moved.
• Sample: executed healthy runs ≥ clamp(round(0.5 × daily_avg₁₄d), 5, 25)
  • daily_avg₁₄d = executed runs (success+failure; exclude skipped/cancelled) over the last 14 days, computed across the tier's repos (next→.github-private; ring0→.github; ring1→TalkTerm+bmad; stable→broad fleet).
  • floor 5 (below that a count is noise), cap 25 (beyond that is diminishing returns and just stalls promotion).
  • count only runs started after the cut (so they exercise the new candidate).
• Health: failure_rate ≤ baseline + ε AND zero startup_failures — the count is the sample size; this is the pass/fail (reuses decide_gate / failure_rate_permille).
• Low-volume fallback: if 0.5 × daily_avg < 5, drop the count gate and require 24 h + ≥ 1 executed healthy run instead (covers dependabot-automerge, dependabot-rebase).

Net effect on the in-flight promotion: busy three gate ~12 h (floor-bound); auto-rebase ~12–13 h (5 runs); the two dependabot reusables on the 24 h fallback — i.e. two waves, not six independent timers.

Open questions (to decide here)

1. Per-ring floor. 12 h is sized for next (first real exposure; doesn't span a daily cycle at 12 h, but the count drags busy ones there anyway). Should ring0/ring1/stable use a shorter floor (e.g. 4–6 h) since the candidate already soaked upstream, or stay uniform at 12 h?
2. Sample bounds. Are MIN=5 / MAX=25 right? (Rationale: ~25 clean runs makes a ~30% regression a <0.1% miss; <5 is statistically meaningless.)
3. ε for the failure-rate comparison — baseline is ~0 today; allowed delta = 0, or a permille threshold (e.g. ≤ baseline + 50‰)?
4. Per-reusable vs batch promotion (batching lets a 0-volume reusable gate the whole cohort into the 24 h path).
5. Run attribution — "since cut" by created-after-timestamp (proxy) vs detecting the run actually resolved the candidate SHA.
6. Where the standard lives — petry-projects/.github/standards/ (org standard) cross-referenced from .github-private/docs/release/{versioning,runbook}.md.
7. Enforcement — a standalone soak-check (reads channels + run history → per-reusable PASS/WAIT + reason) for the interim, then consumed by canary-rollout.sh for cross-repo agents (feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501).

Acceptance

• Soak standard written (numbers + rationale + per-ring floor decision) in standards/ and cross-linked from the release runbook/versioning docs.
• A mechanical soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback).
• Wired into the promotion path (interim: manual soak-check before each cut-release.sh --channel <ring> --push; target: canary-rollout.sh consumes it for cross-repo agents).

References

• Epic docs(standards): add Advanced Security (GHAS) org standard #495; promotion in flight .github-private#870; automated gate .github-private#501; cross-repo cut wiring .github-private#959.
• Current qualitative gate: .github-private/docs/release/runbook.md ("Promotion is gated at every ring") + versioning.md.

Until this lands, the in-flight #870 promotion uses a provisional 12 h + healthy-runs judgment before advancing each ring.

[don-petry]

Dev-Lead Implementation Plan

Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

Scope

Add a concrete, mechanical release soak standard to standards/ plus a pure-bash, unit-tested soak-check (decision core + CLI) that emits per-reusable/per-ring PASS/WAIT with the gating reason (time / sample / health / fallback).

Open-question decisions (made here, documented in the standard)

1. Per-ring floor: next = 12 h; ring0/ring1/stable = 6 h (candidate already soaked upstream; outer rings stay count-bound at their higher volume).
2. Sample bounds: keep MIN=5 / MAX=25.
3. ε (failure-rate delta): 50‰, configurable (--epsilon-permille). Absorbs one transient failure without masking a real regression.
4. Per-reusable vs batch: gate evaluated per reusable, per ring; batching is an operational layer (a 0-volume member drags the cohort to the 24 h fallback) — documented, not enforced in the core.
5. Run attribution: "since cut" via created-after-timestamp proxy (documented limitation; SHA-resolution is the target).
6. Where it lives: standards/release-soak-standard.md, cross-linked from standards/ci-standards.md + the canary-rings ADR. (Cross-link from .github-private/docs/release/{versioning,runbook}.md is a follow-up — that repo isn't editable from here.)
7. Enforcement: standalone scripts/soak-check.sh for the interim; consumed by canary-rollout.sh (feat: implement issue #500 — pr-review: dispatcher fails on review_requested (empty gh api URL) + agent comment runaway #501) later.

Implementation Checklist

• scripts/lib/soak-gate.sh — pure decision core: soak_ring_floor_hours, soak_sample_threshold (clamp/round), soak_is_low_volume, soak_failure_rate_permille, soak_decide_gate
• scripts/soak-check.sh — CLI: gathers 14 d executed-run metrics across the tier's repos + cut-time via gh, supports flag overrides, prints PASS/WAIT
• standards/release-soak-standard.md — numbers + rationale + per-ring decision + soak-check usage
• standards/ci-standards.md + docs/initiatives/agent-canary-rings-adr.md — cross-link the new standard
• .github/workflows/soak-check-tests.yml — shellcheck + bats gate for the new files

Tests to Write (TDD — before implementation)

• test/scripts/lib/soak-gate.bats — per-ring floors, sample clamp at 5/25, half-up rounding, permille, low-volume fallback (24 h + ≥1), and the full WAIT(time/sample/health-startup/health-rate)/PASS decision matrix
• test/scripts/soak-check/soak-check.bats — end-to-end PASS + WAIT scenarios via a gh stub, flag overrides, and usage errors

Test Command

bats --print-output-on-failure test/scripts/lib/soak-gate.bats test/scripts/soak-check/ and the repo lint bash .dev-lead/scripts/dev-lead-lint.sh

[don-petry]

Applied live to the in-flight #870 next soak (2026-06-27)

Ran the proposed gate against all six reusables (next → v2.1.0, cut 2026-06-27T13:57Z). Verdict: all WAIT on the time floor (cut age ~2h; 0 failures; samples already accruing — the busy three had 11/12/148 runs vs a 26 target, so the floor is the binding gate, as intended). Concrete advance windows:

• count-mode (agent-shield, dependency-audit, pr-review-mention): cut+12h = 2026-06-28T01:57Z
• fallback (auto-rebase, dependabot-automerge, dependabot-rebase): cut+24h = 2026-06-28T13:57Z

Edge case the implementation must pin down (open question #2): auto-rebase has a 14-day average of 9.1/day → 0.5 × avg = 4.55. Under clamp(round(4.55), 5, 25) the count target rounds to 5 (count-mode), but the fallback trigger 0.5×avg < 5 is true → it lands in the 24h fallback instead. So round-then-clamp-to-5 and fallback-if-<5 disagree right at the boundary. Decide one rule, e.g.:

• (a) fallback strictly when 0.5×avg < 5 (auto-rebase → 24h fallback) — what the live run did; or
• (b) count-mode whenever round(0.5×avg) ≥ 5 after clamping (auto-rebase → 12h + 5 runs).

Recommend (b) — the clamp floor (MIN=5) already encodes "5 is the smallest usable sample," so a workflow that can realistically produce 5 runs should be count-gated, and fallback should mean "can't even get 5." That makes the fallback condition clamp-target unreachable in a reasonable window, not a raw <5 on the average.

[don-petry]

Dev-Lead: issue #548 implementation failed — will retry

The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

• Cause: engine-error
• Run: https://github.com/petry-projects/.github/actions/runs/28294124951

[don-petry]

Dev-Lead Implementation Plan

Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

Scope

Ship a concrete, mechanical soak standard (numbers + rationale) in standards/, plus a unit-tested soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason, and cross-link it from the canary-rings ADR.

Decisions on the open questions

1. Per-ring floor: next = 12 h, ring0/ring1 = 6 h, stable = 4 h. Downstream rings already soaked upstream, so independent dwell shrinks — but never to 0 (low-volume callers need wall-clock time; some failure modes only surface on a multi-hour cron cadence).
2. Sample bounds: MIN=5, MAX=25 as proposed.
3. ε: default 0‰ (baseline ≈ 0 today), overridable via --epsilon-permille.
4. Per-reusable evaluation; batching is left to the operator/promotion script (so a 0-volume reusable can't drag the cohort).
5. Run attribution: "since cut" via created-after-timestamp proxy (documented limitation).
6. Lives in standards/release-soak.md, cross-linked from docs/initiatives/agent-canary-rings-adr.md (the host-repo runbook/versioning docs live in .github-private and are out of this repo's scope).
7. Enforcement: standalone soak-check.sh for the interim (manual gate before each cut-release.sh --channel <ring> --push); consumable later by canary-rollout.sh.

Implementation Checklist

• standards/release-soak.md — the standard (time floor table, sample clamp, health gate, low-volume fallback, rationale, decisions).
• scripts/lib/soak-gate.sh — pure decision core (no gh): soak_sample_target, soak_time_floor_hours, soak_decide.
• scripts/soak-check.sh — CLI: validates reusable+tier (sources ring-pins.sh), gathers metrics via gh (overridable by flags), prints PASS/WAIT + gating reason, exits 0=PASS / 20=WAIT.
• docs/initiatives/agent-canary-rings-adr.md — cross-link the new standard + soak-check.
• .github/workflows/soak-check-tests.yml — shellcheck + bats gate mirroring standards-deploy-tests.yml.

Tests to Write

• test/scripts/lib/soak-gate.bats — sample clamp (floor/cap/fallback), per-ring floors, time/sample/health gate ordering, fallback path, PASS path.
• test/scripts/soak-check/cli.bats — arg validation, override-flag path, PASS/WAIT exit codes, gh-stub gathering path.

Test Command

bats test/scripts/lib/soak-gate.bats test/scripts/soak-check/ plus bash .dev-lead/scripts/dev-lead-lint.sh

[don-petry]

Dev-Lead: issue #548 implementation failed — will retry

The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 2 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

• Cause: engine-error
• Run: https://github.com/petry-projects/.github/actions/runs/28296340572

[don-petry]

Dev-Lead Implementation Plan

Issue: #548 — Define the release soak standard (ring-promotion gate): time floor + traffic-scaled sample + health

Scope

Add a mechanical release-soak standard to standards/ (resolving the open questions with rationale) plus a standalone, pure-cored soak-check that, given a reusable + tier, emits PASS/WAIT with the gating reason (time / sample / health / fallback) — the interim gate before each ring promotion, later consumable by canary-rollout.sh.

Decisions baked into the standard (open questions)

1. Per-ring floor — next=12h, ring0/ring1=6h, stable=4h (inner-soaked candidates re-confirm on local traffic but don't re-prove from scratch). Low-volume fallback floor is a uniform 24h (time is the only protection for near-zero-volume reusables).
2. Sample bounds — keep MIN=5 / MAX=25.
3. ε — 0 permille while baseline ≈ 0 (zero new regressions tolerated during soak); ε is a documented, configurable knob.
4. Per-reusable evaluation (batch is a wrapper concern; the gate is per reusable/per ring).
5. Run attribution — "since cut" by created-after channel-tag timestamp (proxy), documented as such.
6. Location — standards/release-soak.md, cross-linked from ci-standards.md and the canary-rings ADR.

Implementation Checklist

• standards/release-soak.md — the standard (numbers + rationale + per-ring floor table)
• standards/ci-standards.md — cross-link the new standard from the staged-promotion section
• docs/initiatives/agent-canary-rings-adr.md — cross-link the soak standard
• scripts/lib/soak-gate.sh — pure decision core (ring floor, required-sample clamp, low-volume detection, failure-rate permille, soak_decide → PASS/WAIT+reason)
• scripts/soak-check.sh — CLI wrapper: gather run metrics + cut time via gh, call the core, print per-reusable PASS/WAIT

Tests to Write

• test/scripts/lib/soak-gate.bats — clamp bounds (floor 5 / cap 25), low-volume boundary (avg<10), per-ring floor, failure-rate permille rounding, and soak_decide for each reason (time / sample / health / fallback) + PASS
• test/scripts/soak-check/cli.bats — usage/arg validation + a stubbed-gh PASS and WAIT path

Test Command

npx bats test/scripts/lib/soak-gate.bats test/scripts/soak-check (+ bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)

[don-petry]

Dev-Lead: cannot implement issue #548 — needs human attention

Automatic retries are exhausted (attempt 3 of 3). The most recent failure was engine-error.

• Run: https://github.com/petry-projects/.github/actions/runs/28299778090

Please review the failures above. To retry anyway, remove the dev-lead:needs-human label and re-apply the dev-lead label.