Skip to content

METRICS.md

Latest commit

 

History

History
108 lines (92 loc) · 5.49 KB

METRICS.md

File metadata and controls

108 lines (92 loc) · 5.49 KB
Raw

Metrics

File-level

  • ai_share — rolling, lines-changed-weighted average of the AI-authored percentage across every commit that has touched this file. Weighted by diff size of each commit, not raw file LOC, so one large AI-authored initial commit doesn't get diluted to nothing by a hundred one-line human edits, and vice versa.
  • last_touched_by — "ai" or "human", whichever authored the most recent change.
  • days_since_ai_touch (report-only, derived) — for files currently owned by AI (last_touched_by == "ai"), days since the most recent AI-authored commit touched them. null once a human has touched the file since — a feedback-loop-latency number, not just the binary "unreviewed debt" flag below.
  • confidence — "notes" (seeded from git-ai/whogitit line-level data), "self-attested" (from this session's own commit trailers), or "unknown" (no attribution data available for a historical commit — never guessed from code style).
  • fix_count — number of commits against this file whose trailer class field starts with bugfix.
  • revert_count — number of commits with class=revert.
  • incident_count — number of commits explicitly marked incident=true (opt-in, separate from fix_count, which counts any bugfix-classed commit whether or not it was tied to a real incident).
  • risk_score — see below.

Risk score (starting heuristic — tune per project)

churn_norm    = min(1, total_lines_seen / 200)
fix_rate_norm = min(1, fix_count / 5)
risk_score    = ai_share * churn_norm * fix_rate_norm

The intent: a file is only a "hotspot" if it is both AI-heavy and demonstrably costing rework. A file that's 100% AI-written but has never needed a fix scores low. A file that's 10% AI-written but churns constantly scores low too — the AI attribution isn't the thing driving the cost. Only the intersection surfaces. The constants (200 lines, 5 fixes) are arbitrary normalization points, not calibrated against real data yet — treat them as a starting point to adjust once you have a few months of history.

Relationship to DORA's Change Failure Rate

risk_score is a git-only leading indicator, not a substitute for DORA's Change Failure Rate or MTTR — it has no visibility into deploys or production incidents, only fix/revert commits in git history, and a bugfix commit class is a weak proxy for "this caused a production failure." Report it as "worth investigating," never as a reliability metric. If you want something closer to a real, attribution-segmented Change Failure Rate, use incident_count below, which requires explicitly marking a commit as incident-linked rather than inferring it from class alone.

Project-level (from ai-codekeep report)

  • Project AI share — total-lines-weighted average of every tracked file's ai_share. Reported, but deliberately shown below the hotspot list in output — it's the least actionable number in the report.
  • Hotspot count — files with risk_score >= 0.3 (see HOTSPOT_THRESHOLD in bin/ledger.mjs).
  • Unreviewed AI debt — files where last_touched_by == "ai" and every entry in history was authored by AI (no human has touched the file since it was introduced). Reported alongside the oldest days_since_ai_touch among them, so "2 files" becomes "2 files, oldest 30d untouched" — the actual DevEx feedback-loop-latency signal, not just a count.
  • Incident rate by attribution — of all AI-authored commits across the project, what share were marked incident=true, versus the same for human-authored commits. Omitted from the report entirely until at least one commit has ever been marked incident-linked, so it never shows a misleading 0%/0% when the field simply isn't in use yet.

Change-level

  • reason — free text, inferred from a linked issue/PR/failing test or supplied explicitly by the user. Never fabricated.
  • class — one of feature, refactor, bugfix, bugfix-of-ai-code, revert. bugfix-of-ai-code is the category the rest of this system exists to produce: it's the difference between "AI wrote this" and "AI wrote this and it needed a human fix."

Design constraint: file-level, not author-level

history[].author is deliberately always "ai" or "human" — never a real identity. This is intentional, not a missing feature: per SPACE's warning against Activity metrics doubling as individual performance signals, this system should never grow a per-engineer dimension. If you're extending ai-codekeep, keep it that way — see also SECURITY.md.

To be precise about the boundary: this constraint governs what the tool computes, not what git makes possible. Trailers sit in commits that carry author names, so per-person correlation is inherently derivable with plain git by anyone determined to do it. The constraint's value is that ai-codekeep never produces, normalizes, or legitimizes that view — a policy line for contributors, not a technical safeguard for teams.

Explicitly not measured

  • Raw lines of code as a standalone metric — see "Lines Don't Matter" for why this is a weak proxy on its own; it's only used here as a weighting factor, never reported by itself.
  • Style-based AI detection on code with no attribution history. If confidence is unknown, it stays unknown.