Skip to content

feat: harden diagnostic claim audits#5

Merged
tmusser merged 46 commits into
mainfrom
agent/diagnostic-claim-traps
Jul 12, 2026
Merged

feat: harden diagnostic claim audits#5
tmusser merged 46 commits into
mainfrom
agent/diagnostic-claim-traps

Conversation

@tmusser

@tmusser tmusser commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Why

The new QQ, ECDF, and residual intents render correctly, but diagnostics need stronger claim discipline than ordinary descriptive charts. This follow-up adds evidence-backed traps and visual contracts so a plausible-looking diagnostic cannot silently support an overconfident absence claim.

What this PR adds

  • QQ claim support checks for severe outer-tail departure from the fitted normal line.
  • Residual claim support checks for strong fitted-value correlation and ordered-thirds mean shifts, including curved structure.
  • Dedicated QQ fitted-reference and residual zero-reference contracts for chart and first-party spec audits.
  • First-party statistical intent metadata in generated Vega-Lite specs, scoped to QQ, ECDF, and residual so older usermeta contracts remain unchanged.
  • Four CLI trap fixtures: severe QQ tails, missing QQ reference line, obvious residual pattern, and a four-row diagnostic sample.
  • Diagnostic claim guidance with weak/strong QQ, residual, and ECDF wording examples.
  • Unit and end-to-end CI coverage for the new verdicts and rule IDs.

Deterministic boundaries

  • QQ tail warning: maximum outer-20% departure of at least 0.8 sample standard deviations.
  • Residual pattern warning: absolute fitted/residual correlation of at least 0.5, or ordered-thirds mean shift of at least 1.0 residual standard deviation.
  • Diagnostic sample gate: FAIL below 5 complete observations, WARN for 5–19, PASS at 20+.

These are explainable guardrails for obvious structure, not formal normality or model-adequacy tests.

Trap outcomes

  • qq_heavy_tails: REVIEW via claim.qq.normality_support
  • qq_missing_reference_line: BLOCK via visual.qq.reference_line
  • residual_obvious_pattern: REVIEW via claim.residual.pattern_support
  • diagnostic_tiny_sample: BLOCK via data.residual.sample_size

Verification

  • Full pytest suite: passed
  • Existing CLI REVIEW/BLOCK/warnings-as-errors smoke checks: passed
  • All four diagnostic trap checks, including exact verdict and rule assertions: passed
  • Final GitHub Actions run #162 on head 401c0ea: passed
  • Final review diff: 24 intended files with no temporary bootstrap or debugging files
  • artifacts/VERIFY.md and artifacts/HANDOFF.md are aligned with the PR

@tmusser
tmusser marked this pull request as ready for review July 12, 2026 10:45
@tmusser
tmusser merged commit 994e087 into main Jul 12, 2026
2 checks passed
@tmusser
tmusser deleted the agent/diagnostic-claim-traps branch July 12, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant