Skip to content

feat(audit): add CoV flagging with --max-cov threshold#693

Draft
kaihowl wants to merge 1 commit into
masterfrom
claude/open-issues-review-vNkea
Draft

feat(audit): add CoV flagging with --max-cov threshold#693
kaihowl wants to merge 1 commit into
masterfrom
claude/open-issues-review-vNkea

Conversation

@kaihowl
Copy link
Copy Markdown
Owner

@kaihowl kaihowl commented May 24, 2026

Summary

  • Adds --max-cov <percentage> CLI flag to git perf audit to flag measurements with high Coefficient of Variation (CoV = σ/μ × 100%)
  • When tail or head CoV exceeds the threshold, a ⚠️ High CoV: tail=X.X%, head=Y.Y% (threshold: Z.Z%) warning is appended to the audit output
  • Configurable via .gitperfconfig as max_cov under [measurement] or per-measurement tables, with standard CLI > config precedence
  • Warning appears regardless of pass/fail status — it's informational about measurement stability

Closes #326

Test plan

  • cargo nextest run -- --skip slow --skip test_remove — 382/382 pass
  • cargo clippy — clean
  • cargo fmt — applied
  • Manpages regenerated via ./scripts/generate-manpages.sh
  • 6 new unit tests covering: warning fires above threshold, absent below threshold, absent with no threshold, not shown for single measurement, shows both tail/head values, shown on failing audit

https://claude.ai/code/session_01RN3hLyQDXCQVLBGC3XcYCM


Generated by Claude Code

Adds Coefficient of Variation (CoV = σ/μ × 100%) flagging to the audit
command. When tail or head CoV exceeds the configured threshold, a
⚠️ High CoV warning is appended to the audit output to indicate low
measurement stability that may affect z-score reliability.

Configuration:
- CLI: --max-cov <percentage>
- Config: max_cov = 50.0 under [measurement] or per-measurement table
- Precedence: CLI > per-measurement config > global config > no flagging

Closes #326

https://claude.ai/code/session_01RN3hLyQDXCQVLBGC3XcYCM
@github-actions
Copy link
Copy Markdown
Contributor

Performance Report

Performance Results

Audit Results

Error: No measurement for HEAD.
Audit failed, but continuing...

Measurement Storage Size

Live Measurement Size Report
============================

⚠️  Shallow clone detected - measurement counts may be incomplete (see FAQ)

Number of commits with measurements: 325
Total measurement data size (on-disk (compressed)): 451.5kB

Repository Statistics (for context):
-------------------------------------
  Loose objects: 0 (0B)
  Packed objects: 10809 (2.7MB)
  Total repository size: 2.7MB

Created by git-perf

@github-actions
Copy link
Copy Markdown
Contributor

🧬 PR Mutation Testing Results

Status: ❌ Missed mutants detected - tests need improvement

Mutation Testing Metrics (Changed Code Only)

Metric Count
Total Mutants 29
✅ Caught 12
❌ Missed 14
⏱️ Timeout 0
🚫 Unviable 3

Mutation Score: 46.1% (12/26 viable mutants caught)

⚠️ Action Required

This PR has 14 uncaught mutant(s) in the changed code.

What this means: The tests do not adequately verify the behavior of the code changes.

Next steps:

  1. Review the mutation testing artifacts (download below)
  2. Add or improve tests to catch the missed mutants
  3. Run cargo mutants --in-diff <diff-file> locally to iterate faster

Incremental mutation testing only tests code modified in this PR.
Full mutation testing runs weekly to catch issues in unchanged code.

📦 Artifacts

Download full mutation report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(report): add stddev flagging and display in results

2 participants