Investigate token efficiency in exhaustive multi-review runs

[Kevinjohn]

Exhaustive review runs may spend avoidable tokens on repeated reviewer orientation while correctly satisfying the required coverage and independence gates.

Context

This project intentionally favors complete, evidence-backed review over minimum token consumption. A run this extensive is not expected to be cheap, and optimization must not weaken semantic coverage, reviewer independence, candidate preservation, validation, tail review, or the final audit.

However, obvious avoidable costs should be measured and, where supported by evidence, reduced.

Example run

A substantial repository review produced the following checkpoint:

• 77 semantic units primary-reviewed and imported
• 267 observations and 200 validations in canonical state
• 34 Tier-A independent second-review assignments, with one risk angle per assignment
• At the observed snapshot: 20 of 34 second reviews completed in 8m 23s
• Approximately 1.9 million aggregate specialist tokens reported at that snapshot
• Controller context was 31% occupied
• Five-hour usage allowance was at 35% after 4h 8m
• The run used the highest-capability model with a high reasoning setting

The aggregate token count may include work from assignments still in progress, so dividing it by the 20 completed assignments is only a rough diagnostic. It nevertheless suggests that assignment construction and repeated repository orientation are worth investigating.

Questions to investigate

1. Are specialist packets actually compact?

The Skill requires packets to contain only assigned scope, immutable manifest identities, relevant extracted references, output schemas, and the mandatory specialist block. It explicitly excludes the full governing documents, unrelated findings, other ledgers, conversation history, and orchestration mechanics.

Measure whether dispatched packets follow that rule in practice, including inherited conversation context and any broad repository material loaded before targeted inspection.

2. Does each second-review requirement need a cold reviewer context?

The contract requires a second-review execution identity distinguishable from the contributing primary reviewers. It does not appear to require a globally unique reviewer for every second-review requirement.

Investigate whether a smaller independent reviewer pool could process multiple related assignments in warm contexts while preserving separate attempt manifests, persisted results, and all independence checks.

3. Is Tier-A classification calibrated correctly?

In the example, 34 second reviews were required for 77 units. Confirm that Tier A is limited to the critical boundaries defined in the Reference Pack rather than being applied broadly to ordinary production code.

This matters because each Tier-A classification adds deeper primary evidence requirements, focused validation, independent second review, and mandatory inspection during the final audit.

4. Can model and reasoning effort be routed by task?

Assess whether the highest-capability/high-reasoning configuration is necessary for:

• semantic primary and second review;
• adversarial candidate validation;
• cross-component reconciliation;
• deterministic imports, schema checks, ID allocation, report generation, polling, and bookkeeping.

Prefer deterministic tooling for mechanical work. If model routing is introduced, establish which tasks can safely use a less expensive configuration without reducing review quality.

5. Are later phases repeating whole-unit review?

Verify that each overlapping phase stays narrow:

• second review checks its assigned risk angle;
• candidate validation validates concrete observations;
• cross-component reconciliation checks seams and composed behavior;
• tail review covers the explicitly named underrepresented classes;
• final audit challenges evidence and satisfies its prescribed scope.

The phases intentionally overlap, but they should not accidentally become repeated general reviews of the same units.

6. What should be measured?

Add or document enough observability to compare runs, where the host exposes the data:

• input, output, and cached tokens per attempt;
• assignment-packet size;
• tokens spent before the first scope-specific inspection;
• files and bytes read per attempt, including repeated reads;
• tokens per unit, angle, candidate, and validated finding;
• Tier-A versus Tier-B/C cost;
• cold-context versus warm-pool second-review cost;
• incomplete/retried attempt overhead;
• bookkeeping/tooling tokens versus semantic-review tokens.

Do not treat findings per token as the sole quality metric: a correct clean review is still valuable.

Possible changes to evaluate

• Enforce or mechanically generate minimal phase-specific assignment packets.
• Reuse independent reviewer contexts across compatible assignments, with conflict checks against primary reviewer identities.
• Add a Tier-A calibration check during the representative pilot.
• Route deterministic orchestration and bookkeeping away from the most expensive reasoning configuration.
• Add phase-specific instructions that prohibit restarting a general review during validation, reconciliation, tail, and audit work.
• Record a lightweight per-attempt efficiency ledger when host usage data is available.

Acceptance criteria

• Capture measurements from at least one representative exhaustive run.
• Identify which costs are required by the review contract and which are avoidable implementation overhead.
• Propose changes only where independence, full observation preservation, ten-angle coverage, validation, and audit gates remain intact.
• Compare the current and proposed approach on equivalent scope.
• Document expected savings, quality risks, and rollback conditions.
• Add tests for any changes to packet construction, identity conflict checking, tiering gates, or phase scoping.

Non-goals

• Setting an arbitrary token cap for exhaustive reviews.
• Removing required Tier-A second reviews or the independent final audit.
• Reducing reported observations to improve an efficiency metric.
• Trading review completeness for lower headline usage without explicit user authority.

[Kevinjohn]

Interim benchmark from two exhaustive runs

We now have one detailed resource-usage package and one earlier completed review record. This is an interim comparison while additional configurations run. It should not yet be interpreted as a like-for-like efficiency result.

Run A: earlier, lower-detail execution

• 657 repository paths covered at the reviewed revision
• 26 semantic work units
• 9 Tier-A units, all independently second-reviewed
• 17 non-Tier-A units
• All 26 units independently inspected by the final audit
• 28 total observations:
  • 13 validated code findings: 8 P2, 3 P3, 2 P4
  • 8 documentation observations
  • 4 test gaps
  • 2 duplicates
  • 1 rejected candidate
• Verdict: CHANGES REQUIRED
• Host-visible token usage, per-attempt usage, cache fields, agent count, and phase-level resource allocation were not captured

Run B: instrumented high-detail execution

Directly measured from persisted transcripts and in-session completion metadata:

• 77 primary work units
• 34 Tier-A independent second reviews
• 132 specialist workers launched and completed
• 0 interrupted, blocked, or abandoned workers
• 1 workflow-level retry caused by a zero-worker misfire, with approximately zero token usage
• Elapsed initialization-to-conclusion time: 1h 44m 49s
• Host-reported specialist usage: 11,301,010 tokens

Host-reported usage by phase:

Phase	Tokens	Share
Map	1.70M	15.0%
Pilot	0.72M	6.4%
Bulk primary	5.89M	52.1%
Independent second review	1.94M	17.2%
Candidate verification	0.72M	6.4%
Reconciliation	0.20M	1.8%
Final audit	0.07M	0.6%
Tooling-related specialist work	0.07M	0.6%

Transcript fields recorded:

• 2,168,358 generated output tokens
• 39,897,612 cache-creation tokens
• 267,208,749 cache-read tokens
• 4,896 API calls

The host-reported figure and transcript fields do not reconcile to a stable formula; observed ratios vary materially. Cache-read volume is therefore treated as a reread-volume proxy, not incremental billing or directly additive token cost. Controller usage, allowance consumption, and per-workflow host-token composition were unavailable.

Confirmed positives

• Canonical hashing, identifier allocation, transactional imports, gapless finding-ID assignment, integrity checks, and report generation were performed deterministically rather than through model reasoning.
• Specialist packets were already compact and compliant: median size approximately 2,527 bytes.
• No full contract, full Reference Pack, conversation history, or unrelated finding ledger was sent to specialists.
• Failed/interrupted worker overhead was effectively zero.

These results rule out several initial suspects: oversized dispatch packets, model-driven canonical bookkeeping, and large retry waste were not material causes in Run B.

Strongest current efficiency hypothesis

The contract requires an independent second reviewer relative to a unit's contributing primary reviewer. It does not require a globally unique cold reviewer context for every second-review requirement.

Run B used 34 cold, single-angle second-review contexts. Those attempts were associated with approximately 5.3M cache-creation tokens in the transcript-derived accounting. A conflict analysis found that a small warm pool—approximately 2–4 independent reviewer contexts—may be able to process multiple compatible requirements while preserving:

• reviewer independence;
• separate immutable attempts;
• exact requirement scope;
• separately persisted results;
• all canonical completion checks.

This is an experiment candidate, not an adopted optimization. It needs an equivalent-scope warm-versus-cold comparison and explicit conflict checking.

Other opportunities

1. Repeated authority reads: many reviewers independently reread common authority material. Test a compact invariant digest or narrower extracted references without weakening the governing rules.
2. Configuration routing: test a less expensive configuration for Tier-C or tail-only assignments while retaining stronger reasoning for critical semantic review, adversarial validation, reconciliation, and audit work.
3. Workflow guards: retain the new non-empty-scope assertion and defensive argument parsing that prevent zero-worker dispatches. Avoid rerunning state-changing commands merely to inspect their error behavior.
4. Automatic instrumentation: future runs should capture per-worker host tokens, start/stop timestamps, phase/unit/tier/reviewer tags, controller usage, and a cold-versus-warm flag.

Important comparison caveat

The two runs partitioned the same broad review problem very differently:

Measure	Run A	Run B
Work units	26	77
Tier-A second reviews	9	34
Total observations	28	267
Validated P2–P4 findings	13	43
Host-visible tokens	Not captured	11.301M

Raw token totals, elapsed time, and finding counts are therefore not directly comparable. Work-unit granularity and Tier-A classification may be as consequential as the execution configuration itself. Future comparisons must normalize for revision, paths, implementation size, unit partitioning, tier mix, required second reviews, completed phases, concurrency, cache conditions, and retries.

Current takeaways

• Exhaustive semantic coverage remains intentionally expensive.
• The dominant measured cost was primary semantic review, followed by independent second review.
• Compact packets and deterministic bookkeeping are already working as intended.
• The best quality-preserving optimization candidate is warm reuse of independent second-review contexts.
• Tier-A calibration and semantic-unit construction need to be held stable—or explicitly treated as experimental variables—when contrasting configurations.
• Findings per token must not be the sole metric: clean independent verification still supplies assurance value.
• No monetary estimate is needed; host-visible tokens, transcript fields, elapsed time, scope, and completion quality provide the trend baseline.

Additional contrasting runs should be appended here using the same normalized benchmark fields before implementation changes are selected.