Flag reference skills that carry a direct-activation eval (#899)

[Evangelink]

Summary

Reviewing #899 for closure turned up one defect that is still live on main, in the gate rather than in a skill.

disable-model-invocation: true drops a skill from the CLI's <available_skills> menu — skill-validator excludes such skills from the menu budget entirely, so the model never sees the name or the description. The experiment's skilled variant loads exactly one skill:

skilled:
  environment:
    skills:
      - plugins/${eval.grandparent}/skills/${eval.parent}

so for a reference skill the skilled arm is the baseline arm. eng/eval-quality/README.md and check_eval_quality.py::_is_reference_skill both already spell this out.

The gate acts on only half of it. report_uncovered() skipped any skill that had an eval:

if not os.path.isdir(skill_dir) or skill in evals:
    continue

That reports the better case — a reference skill with no eval, which is visibly zero evidence — and stays silent on the worse one: a reference skill with a direct eval, which emits a fabricated pass/fail into the plugin's pass rate. Two of those landed after the reasoning was committed:

Eval	Added by	Reported before	Reported now
tests/dotnet-test/platform-detection/eval.yaml	#974	—	warning
tests/dotnet-test/filter-syntax/eval.yaml	#976	—	warning

Both are the only instances repo-wide, so the new warning's blast radius is exactly two lines.

Changes

• check_eval_quality.py — split the skill in evals branch so a reference skill with a direct eval is collected and reported, naming the offending spec. It is a warning, not an error: whether to retire those two evals or keep them pending a real result is a judgement call for the owners, and the gate's convention is that judgement calls report rather than block.
• selftest_eval_quality.py — an output_case proving the warning fires, plus a new silent_case helper proving it stays quiet for a model-invocable skill with the same eval. A warning that fires on well-formed input is worse than no warning, so both sides of the trigger are pinned. 22 → 24 self-tests.
• eng/eval-quality/README.md — the "Skills with no eval" section became "Skill eval coverage" and now documents both halves. Its callout said filter-syntax was the sole exception; that stopped being true when Fix migrate-mstest-v1v2-to-v3 activation and test-skill eval quality #974 landed.
• plugins/dotnet-test/README.md — said three of the four reference skills "deliberately have no tests/dotnet-test/<skill>/eval.yaml", naming platform-detection among them. Close the remaining dotnet-test eval follow-ups (#899) #971 wrote that text but merged after Fix migrate-mstest-v1v2-to-v3 activation and test-skill eval quality #974 had already added the eval, so it shipped stale.

Neither eval is deleted here. #976's stimuli are ordinary user requests graded on whether the answer carries the right syntax, which is a defensible attempt at the problem — it has simply never produced a verdict (it ran during the PAT-pool outage, and no cross-family run has covered either eval since). Surfacing it is the change that lets the owners decide on evidence.

Related issue

Found while reviewing #899. Does not close it — the remaining blocker there is a cross-family re-run on the merged tree, and the executors/runs dispatch inputs still live only on the unmerged #920 branch.

Validation

• python eng/eval-quality/selftest_eval_quality.py — all 24 pass, including the two new cases.
• python eng/eval-quality/check_eval_quality.py — No errors; the new warning lists exactly dotnet-test/filter-syntax and dotnet-test/platform-detection, and the pre-existing "2 reference skill(s) have no eval" warning is unchanged.
• python eng/eval-quality/check_eval_quality.py --base-ref origin/main (what CI runs on a PR) — No errors.
• Swept every plugin for disable-model-invocation: true to confirm the four dotnet-test skills are the only reference skills in the repo, so no other plugin gains a warning.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

On the last three: no ownership moved, no plugin metadata changed (plugin.json untouched, only the plugin README's prose), and no external domains are referenced.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-test	assertion-quality	22/22	100%
✅	dotnet-test	code-testing-agent	4/4	100%
✅	dotnet-test	crap-score	6/6	100%
✅	dotnet-test	detect-static-dependencies	22/22	100%
✅	dotnet-test	filter-syntax	1/1	100%
✅	dotnet-test	generate-testability-wrappers	24/24	100%
✅	dotnet-test	grade-tests	29/29	100%
✅	dotnet-test	migrate-static-to-wrapper	26/26	100%
✅	dotnet-test	mtp-hot-reload	16/16	100%
✅	dotnet-test	run-tests	16/16	100%
✅	dotnet-test	test-anti-patterns	25/25	100%
✅	dotnet-test	test-gap-analysis	30/30	100%
✅	dotnet-test	test-smell-detection	26/26	100%
✅	dotnet-test	test-tagging	28/28	100%
✅	dotnet-test	writing-mstest-tests	45/45	100%

[Copilot]

Pull request overview

This PR tightens the eval-quality gate’s coverage reporting for reference skills (disable-model-invocation: true) by flagging the worse state: reference skills that nevertheless ship a direct-activation eval, which produces a baseline-vs-baseline “verdict” (judge noise) that can pollute plugin pass rates. It also updates documentation and adds self-tests to pin both the “warning fires” and “warning stays quiet” behaviors.

Changes:

• Update report_uncovered() to warn when a reference skill has a direct eval, naming the specific offending tests/<plugin>/<skill>/eval.yaml.
• Extend eval-quality self-tests with a new silent_case() helper and two cases that assert both sides of the warning trigger.
• Refresh documentation to describe “skill eval coverage” and the specific “reference skill with direct eval” warning, and fix the dotnet-test README’s now-stale exception list.

Show a summary per file

File	Description
eng/eval-quality/check_eval_quality.py	Splits the “skill has eval” branch to additionally warn on reference skills that carry direct-activation evals, listing offending specs.
eng/eval-quality/selftest_eval_quality.py	Adds silent_case() and two targeted self-tests to ensure the new warning triggers only when appropriate.
eng/eval-quality/README.md	Updates the warning documentation to cover both “no eval” and “degenerate direct eval” cases for reference skills.
plugins/dotnet-test/README.md	Updates plugin documentation to reflect that platform-detection and filter-syntax are the two current direct-eval exceptions.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 4/4 changed files
• Comments generated: 0