Remove vocabulary gates from performance eval

[artl93]

Part of #886

Purpose

Remove vocabulary overfit from scenarios 8 and 9 without weakening or broadening the concrete performance findings they measure. This PR changes only the evaluator; it does not change analyzing-dotnet-performance itself.

The expected outcomes remain directly verifiable against the fixture code: repeated enumeration, delegate and array allocation, per-match string allocation, params-array allocation, static initialization allocation, and existing span-based positive patterns. Correct responses no longer have to use the words LINQ, All, or Where to receive credit.

Changes

• remove literal LINQ, All, and Where output gates
• retain atomic, fixture-specific rubric criteria for the original performance findings
• avoid requiring a preferred remediation where identifying the defect is the measurable outcome
• add an independent criterion that analysis-only responses do not modify the fixture, matching the prompt and comparable evals
• rename the two scenario labels around the measured outcomes rather than implementation vocabulary

Evidence

A matched evaluator A/B held the skill and workflow byte-identical across three repetitions and five model families:

• current-eval control
• vocabulary-gate-free treatment

The aggregate skilled-vs-baseline result remained effectively unchanged: +0.258 control versus +0.265 treatment. Unchanged scenarios differed by +0.027, establishing a larger stochastic-noise floor than the aggregate +0.007 movement.

The control produced one concrete vocabulary-only false negative: a successful MAI scenario 9 response passed exit-success and all 5/5 substantive rubric criteria but failed solely because its final prose omitted Where. This directly supports removing the hard vocabulary gate.

That A/B preceded the final rubric refinement in e4f6235e; it supports the gate removal, not a claim that the exact final wording improves scores. The final wording was deliberately moved back toward the original code-grounded findings after review.

Validation

• targeted skill-validator check --plugin plugins/dotnet-diag passes
• YAML and focused scenario structure validation passes
• git diff --check passes
• branch diff contains only tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

[Copilot]

Pull request overview

This PR updates the dotnet-diag performance skill evaluation to reduce overfitting in scenarios 8 and 9 by making rubrics more outcome-focused, and adjusts the skill guidance to reduce unnecessary reference loading and tool-call overhead during focused reviews.

Changes:

• Rewrote scenarios 8 and 9 prompts/rubrics in both canonical and Vally evals toward independently evaluable, outcome-focused criteria.
• Updated analyzing-dotnet-performance skill workflow guidance to emphasize “read source first”, load references only when needed, and batch confirmation searches.

Show a summary per file

File	Description
tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml	Updates scenario 8/9 prompts and rubrics to be less technique/vocabulary-specific.
tests/dotnet-diag/analyzing-dotnet-performance/eval.vally.yaml	Mirrors the scenario 8/9 prompt/rubric updates for Vally parity.
plugins/dotnet-diag/skills/analyzing-dotnet-performance/SKILL.md	Adjusts workflow guidance to reduce reference loading and tool-call overhead.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 3/3 changed files
• Comments generated: 6

[Copilot]

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0 new

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
analyzing-dotnet-performance	Detects compiled regex startup budget and regex chain allocations	4.0/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅
analyzing-dotnet-performance	Detects CurrentCulture comparer and compiled regex budget in inflection rules	5.0/5 → 5.0/5	✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌ [1]
analyzing-dotnet-performance	Finds per-call Dictionary allocation not hoisted to static	3.7/5 ⏰ → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [2]
analyzing-dotnet-performance	Catches compound allocations in recursive number converter with ToLower	3.7/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill, bash, grep	✅ 0.13	✅
analyzing-dotnet-performance	Finds StringComparison.Ordinal missing and FrozenDictionary opportunities	4.7/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌ [3]
analyzing-dotnet-performance	Detects Aggregate+Replace chain and struct missing IEquatable	3.7/5 → 4.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep, bash / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [4]
analyzing-dotnet-performance	Finds branched Replace chain in format string manipulation	3.3/5 → 4.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [5]
analyzing-dotnet-performance	Catches LINQ on hot-path string processing and All(char.IsUpper)	4.0/5 → 4.7/5 🟢	✅ analyzing-dotnet-performance; tools: skill, bash, grep / ✅ analyzing-dotnet-performance; tools: skill, grep, bash	✅ 0.13	✅ [6]
analyzing-dotnet-performance	Detects LINQ pipeline in TimeSpan formatting and collection processing	3.7/5 → 2.3/5 ⏰ 🔴	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	❌
analyzing-dotnet-performance	Flags Span inconsistencies and compound method chains in truncation library	3.0/5 ⏰ → 2.3/5 ⏰ 🔴	✅ analyzing-dotnet-performance; tools: report_intent, skill, view / ✅ analyzing-dotnet-performance; tools: report_intent, view, grep, skill	✅ 0.13	❌ [7]
analyzing-dotnet-performance	Identifies unsealed leaf classes and locale hierarchy patterns	3.0/5 → 5.0/5 🟢	✅ analyzing-dotnet-performance; tools: skill, grep / ✅ analyzing-dotnet-performance; tools: skill	✅ 0.13	✅ [8]

[1] ⚠️ High run-to-run variance (CV=67%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -6.0% due to: tokens (29261 → 80436), tool calls (2 → 7), time (19.5s → 49.4s)
[2] ⚠️ High run-to-run variance (CV=157%) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=402%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -9.0% due to: tokens (29583 → 80500), tool calls (2 → 6), time (26.8s → 53.9s)
[4] ⚠️ High run-to-run variance (CV=70%) — consider re-running with --runs 5
[5] ⚠️ High run-to-run variance (CV=107%) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=13524%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=94%) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=61%) — consider re-running with --runs 5

⏰ timeout — run(s) hit the (120s, 180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 905 in dotnet/skills, download eval artifacts with gh run download 29626270559 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0b7ae60bec7c7fa37b98ae6c3fb4f286d70f426a/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for 0b7ae60. cc @dotnet/dotnet-diag @dotnet/skills-diag-reviewers — please review.

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
❌	dotnet-diag	analyzing-dotnet-performance	10/21	47.6%

Uncovered: dotnet-diag/analyzing-dotnet-performance

• [Validation] All critical patterns were checked (from reference files or inline recipes) (line 178)
• [Validation] Topic-specific recipes run only when matching signals detected (line 179)
• [Validation] Each finding includes a concrete code fix (line 180)
• [Validation] Scan execution checklist is complete (all recipes run) (line 181)
• [Validation] Summary table included at end (line 182)
• [Pitfall] Suggesting Span in async methods (line 189)
• [Pitfall] Suggesting ConfigureAwait(false) in app code (line 191)
• [Pitfall] Recommending ValueTask everywhere (line 192)
• [Pitfall] Flagging new HttpClient() in DI services (line 193)
• [Pitfall] Suggesting CollectionsMarshal.AsSpan broadly (line 195)
• [WorkflowStep] Step 1: Load Reference Files (if available) (line 38)

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml:160

• Scenario name still includes technique/vocabulary-specific terms (LINQ/All) even though the grader gates were removed and the rubric is now outcome-focused. Renaming the scenario helps keep the evaluator aligned with the PR’s intent and avoids reintroducing vocabulary bias in reporting/diagnostics.

  - name: Catches LINQ on hot-path string processing and All(char.IsUpper)

tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml:178

• Scenario name still calls out LINQ specifically, but the updated evaluation criteria no longer depend on LINQ terminology. Consider renaming to describe the measured outcomes (avoidable intermediate enumeration/materialization) so the scenario label matches the new rubric.

  - name: Detects LINQ pipeline in TimeSpan formatting and collection processing

• Files reviewed: 1/1 changed files
• Comments generated: 0 new

[AbhitejJohn]

/evaluate

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — 1 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
analyzing-dotnet-performance	✅	+32.7% [+16.5%, +48.9%]	10/0/1	4.9/5	4.7/5

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

👋 @artl93 — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[AbhitejJohn]

Hi @artl93 — friendly ping on this PR when you get a chance! Happy to help if needed.

[Copilot]

Copilot's findings

Suppressed comments (1)

tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml:173

• This rubric item references match.Value.ToLower(), but in the fixture (fixtures/string-humanize-and-dehumanize.cs) the per-match allocation is value.ToLower() (where value is assigned from match.Value). Using the exact expression from the fixture keeps the criterion code-grounded and avoids confusion during grading.

      - Identifies match.Value.ToLower() inside the projection as a per-match string allocation

• Files reviewed: 1/1 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

• Files reviewed: 1/1 changed files
• Comments generated: 0 new

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 1 improved, ❌ 0 no credible change, 🔻 0 regressed.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
analyzing-dotnet-performance	✅	+72.7%	0.011	+29.1%	9/1/1	4.7/5	4.6/5	✅ 0.13	⚠️ 11/11 · 6/11 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

Per-scenario details for 1 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for c792d20. cc @dotnet/dotnet-diag @dotnet/skills-diag-reviewers — please review.