Clarify candidate validation and make final-audit sampling adaptive

[Kevinjohn]

The final audit should validate every discovered candidate while sampling only apparently clean, lower-risk work—and expand automatically when the sample reveals a miss.

Context

The current contract requires the independent final auditor to inspect all Tier A units and a deterministic sample of completed non-Tier-A units:

min(N, audit_cap, max(25, ceil(0.01 * N)))

with a default audit_cap of 200.

That sampling rule can be read as limiting issue verification, even though sampling and candidate validation serve different purposes:

• Every discovered candidate should be validated and dispositioned.
• Sampling should apply only to completed Tier B/C units that appear clean after primary review.
• The sample is an independent check on review quality, not a filter for deciding which known issues deserve verification.

The existing wording also does not define how audit coverage should expand when a sampled unit reveals something the primary review missed. A fixed ceiling can therefore become least effective precisely when the audit discovers evidence that the review process may be unreliable.

Proposed handling

1. Validate every discovered candidate

Candidate validation and deduplication should retain 100% coverage. Every candidate from primary review, second review, cross-component review, tail review, or audit must receive an explicit disposition supported by evidence.

Sampling must never be used to select which known candidates are validated.

2. Independently review all mandatory high-risk scope

The final audit should continue to inspect:

• every Tier A unit;
• every required second-review completion;
• every material blocker and unresolved uncertainty;
• every P0/P1 finding;
• disputed, weakly supported, or unusually broad dispositions;
• representative exclusions and not-applicable claims.

3. Sample only the remaining apparently clean scope

After mandatory scope is removed, use deterministic, risk-stratified sampling for completed Tier B/C units with no known reason for mandatory inspection.

Stratification should account for characteristics such as subsystem, risk tier, reviewer identity, language, unit type, and review-evidence quality so that one large homogeneous group cannot dominate the sample.

The sample should remain reproducible from frozen review identities.

4. Expand coverage when the audit finds a miss

Finding a previously undiscovered defensible issue in sampled scope should trigger an explicit escalation policy:

1. Identify the relevant stratum and likely shared failure mode.
2. Audit all directly related units or increase coverage substantially within that stratum.
3. Reassess other work completed by the same reviewer or with the same evidence weakness.
4. Repeat until the expanded audit produces no further misses.
5. Escalate to 100% independent review of the affected stratum—or all remaining eligible units—when misses repeat, cross boundaries, or indicate systematic review failure.

The audit report should record why coverage expanded, which population was affected, and why stopping was justified.

5. Treat the 25–200 formula as a default, not a hard safety ceiling

The initial sample formula can remain a reproducible starting point, but audit_cap=200 should not prevent evidence-driven expansion. If retained, the cap should apply only before any miss is found.

Consider making the initial policy configurable while preserving a conservative default and recording the effective policy in canonical review state.

Suggested contract distinction

Known candidates           -> validate and disposition 100%
Tier A and mandatory scope -> independently inspect 100%
Apparently clean Tier B/C  -> deterministic stratified sample
Miss found in sample       -> expand affected scope adaptively
Repeated/systematic misses -> independent review up to 100%

Acceptance criteria

• The contract explicitly states that every discovered candidate is validated and dispositioned.
• Sampling is explicitly limited to apparently clean, completed non–Tier A work.
• Mandatory final-audit scope is enumerated separately from sampled scope.
• A deterministic escalation policy expands audit coverage when a sample reveals a miss.
• The default sample cap cannot block evidence-driven expansion.
• Canonical state records the initial population, strata, sampled units, expansion triggers, expanded scope, and stopping rationale.
• Tests cover the initial sample, a single-miss expansion, repeated misses, and escalation to full affected-scope review.

Discussion points

• Should one missed issue expand the entire stratum immediately, or begin with a defined multiplier?
• Which dimensions must always be used for stratification?
• What evidence should trigger full-repository independent review rather than stratum-level expansion?
• Should the initial 25–200 formula remain the default, be revised, or be replaced by a risk-weighted policy?