Support classic .NET test projects in dotnet-test - #993
Conversation
Teach dotnet-test skills and agents to preserve non-SDK projects, packages.config dependencies, explicit compile registration, legacy runners, and version-compatible MSTest APIs. Add regression evals for generation, execution, coverage, and authoring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
Skill Coverage Report
Uncovered:
|
GitHub Actions
Inactive
There was a problem hiding this comment.
Pull request overview
This PR expands the dotnet-test plugin to correctly handle classic (non-SDK-style) .NET test projects—especially packages.config-based net472 MSTest solutions—by updating platform/project-system detection, ensuring version-compatible MSTest authoring guidance, and adding regression fixtures/evals to prevent accidental modernization (e.g., SDK-style coverage injection or implicit migration advice).
Changes:
- Add classic non-SDK project-system awareness (including
packages.config) across run/test/coverage guidance and agent workflows. - Gate MSTest authoring recommendations to the installed MSTest version (with explicit fallbacks for MSTest 3.5.x).
- Add new realistic classic net472 MSTest fixtures and regression eval scenarios for running tests, coverage analysis, and test generation/registration.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/writing-mstest-tests/eval.yaml | Adds an eval stimulus to enforce MSTest-version-compatible exception assertions (MSTest 3.5.2). |
| tests/dotnet-test/run-tests/fixtures/classic-mstest/TestProject.csproj | Introduces a classic net472 MSTest packages.config-style test project fixture. |
| tests/dotnet-test/run-tests/fixtures/classic-mstest/packages.config | Pins MSTest adapter/framework dependencies for the classic run-tests fixture. |
| tests/dotnet-test/run-tests/fixtures/classic-mstest/OrderServiceTests.cs | Adds a classic MSTest test file fixture used by run-tests evaluation. |
| tests/dotnet-test/run-tests/eval.yaml | Adds an eval scenario ensuring classic projects route to MSBuild + VSTest/MSTest runner guidance. |
| tests/dotnet-test/coverage-analysis/fixtures/classic-packages-config/packages.config | Adds a classic packages.config fixture for coverage-analysis scenarios. |
| tests/dotnet-test/coverage-analysis/fixtures/classic-packages-config/LegacyTests.cs | Adds a basic MSTest test file for the classic coverage-analysis fixture. |
| tests/dotnet-test/coverage-analysis/fixtures/classic-packages-config/Legacy.Tests.csproj | Adds a classic net472 test project fixture for coverage-analysis. |
| tests/dotnet-test/coverage-analysis/eval.yaml | Adds a regression scenario ensuring classic projects are not modified when coverage is absent. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/tests/packages.config | Adds classic test dependencies (MSTest/Moq/NBuilder) for code-testing-agent fixture. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/tests/FixtureBase.cs | Adds a base fixture pattern to enforce preserving legacy suite conventions. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/tests/DiscountServiceTests.cs | Adds a representative classic MSTest test using Moq/NBuilder conventions. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/tests/Discounts.Tests.csproj | Adds a classic test project requiring explicit <Compile Include> registration. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/src/DiscountService.cs | Adds production-like code to exercise boundary/missing-product test generation in classic projects. |
| tests/dotnet-test/code-testing-agent/fixtures/classic-mstest/src/Discounts.csproj | Adds a classic net472 production project fixture referenced by the tests project. |
| tests/dotnet-test/code-testing-agent/eval.yaml | Adds a scenario requiring classic test-file registration and MSTest 3.5.2-compatible assertions. |
| plugins/dotnet-test/skills/writing-mstest-tests/SKILL.md | Updates MSTest authoring guidance to respect installed MSTest version and classic project constraints. |
| plugins/dotnet-test/skills/run-tests/SKILL.md | Updates run-tests guidance to support classic non-SDK projects (MSBuild + runner) in addition to dotnet test. |
| plugins/dotnet-test/skills/platform-detection/SKILL.md | Extends detection instructions to classify SDK-style vs classic project systems and use packages.config signals. |
| plugins/dotnet-test/skills/crap-score/SKILL.md | Prevents classic project modification/injection and requires repo-owned coverage workflow or existing Cobertura. |
| plugins/dotnet-test/skills/coverage-analysis/SKILL.md | Updates coverage workflow to partition SDK vs classic projects and avoid SDK-style injection into classic projects. |
| plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md | Updates execution reference to exclude classic projects from automatic provider/test execution paths. |
| plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md | Adds classic vs SDK test project partitioning logic and corresponding branching guidance. |
| plugins/dotnet-test/skills/coverage-analysis/references/guidelines.md | Clarifies modification rules: only SDK-style projects may get provider additions; classic must remain untouched. |
| plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md | Adds .NET extension guidance for classic project detection, command selection, and explicit file registration. |
| plugins/dotnet-test/skills/code-testing-agent/SKILL.md | Adds explicit guardrails for classic non-SDK .NET projects (preserve versions, register files, avoid modernizing). |
| plugins/dotnet-test/README.md | Updates plugin README to document classic non-SDK support and constraints. |
| plugins/dotnet-test/agents/testability-migration.agent.md | Adds an explicit “check repository policy” phase before any testability refactor actions. |
| plugins/dotnet-test/agents/test-quality-auditor.agent.md | Removes default testability-migration handoff and tightens policy gating around production refactors. |
| plugins/dotnet-test/agents/code-testing-tester.agent.md | Updates tester agent to use repository runner commands for classic projects and avoid substituting dotnet test. |
| plugins/dotnet-test/agents/code-testing-researcher.agent.md | Updates researcher agent to detect classic projects and record authoritative classic build/test commands. |
| plugins/dotnet-test/agents/code-testing-implementer.agent.md | Updates implementer agent to explicitly register new files in classic .csproj compile items. |
| plugins/dotnet-test/agents/code-testing-generator.agent.md | Updates generator agent to require repository-appropriate full-workspace build validation (no forced dotnet build). |
| plugins/dotnet-test/agents/code-testing-builder.agent.md | Updates builder agent to use MSBuild for classic projects and avoid implicitly migrating to SDK-style builds. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 34/34 changed files
- Comments generated: 3
|
❌ Evaluation ran but produced no results. The evaluate job completed but no |
Tighten the MSTest version grader, make the runner fixture assertion behavioral, and use nameof for the guarded parameter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
tests/dotnet-test/writing-mstest-tests/eval.yaml:227
- The new stimulus can still pass if the answer includes newer APIs like Assert.ThrowsExactly alongside the required Assert.ThrowsException call, even though the rubric says not to recommend newer APIs. Add an explicit negative grader, and tighten the version regex with word boundaries so it can't match e.g. "13.5.2".
tests/dotnet-test/coverage-analysis/eval.yaml:45 - This new classic-project stimulus relies on the rubric to forbid SDK-style provider injection, but the graders don't currently fail answers that recommend commands like
dotnet add ... coverlet.collector(even if the files remain unchanged). Adding anoutput-not-matchesgrader would prevent false positives.
graders:
- type: run-command
config:
command: sh -c "diff -u .eval-baseline/Legacy.Tests.csproj Legacy.Tests.csproj && diff -u .eval-baseline/packages.config packages.config"
expected_exit_code: 0
- Files reviewed: 34/34 changed files
- Comments generated: 0 new
GitHub Actions
Inactive
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results6 skill(s) evaluated — ✅ 3 improved, ❌ 3 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +33.3% (5W/6T/1L over 12 trial(s), sign test p=0.109), mean preference +13.3% — not credible (sign test p=0.109 > 0.05)
❌ crap-score — detailsReason: Net win +66.7% (4W/2T/0L over 6 trial(s), sign test p=0.063), mean preference +36.7% — not credible — 2 of 6 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ platform-detection — detailsReason: Net win -20.0% (0W/4T/1L over 5 trial(s), sign test p=0.500), mean preference -20.0% — no improvement
Per-scenario details for 3 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
|
✅ Evaluation passed for |
Fix test-generation routing, add discriminating CRAP scenarios, and make platform detection directly measurable with accurate dotnet-test mode and MTP bridge semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
GitHub Actions
Failure
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (3)
plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md:160
- Same issue in the mixed-project snippet:
$major -ge 10isn’t sufficient to decide native-MTP argument placement. In SDK 10 VSTest mode (runner unset/VSTest),--coveragemust still be passed after--. Update this conditional to use aglobal.jsonrunner check (native MTP vs VSTest mode).
if ($major -ge 10) {
dotnet test --project $tp.FullName --results-directory $rawDir --coverage --coverage-output-format cobertura --coverage-output $rawDir
} else {
dotnet test $tp.FullName --results-directory $rawDir -- --coverage --coverage-output-format cobertura --coverage-output $rawDir
}
tests/dotnet-test/writing-mstest-tests/eval.yaml:226
- The MSTest version grader still allows a bare "3.5.2" match because the "MSTest" prefix is optional. This reintroduces the earlier issue where responses can pass without explicitly acknowledging MSTest 3.5.2. Require an explicit MSTest + 3.5.2 mention in the regex.
plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md:126 - This section selects
--coveragevs-- --coveragepurely from the SDK major version, but elsewhere in this PR you distinguish SDK 10 VSTest mode (including the MTP bridge) from native MTP mode. On SDK 10 in VSTest mode, MTP arguments still must go after--. The guidance and snippet should key offglobal.jsontest.runner(native MTP vs VSTest mode), not just$major.
This issue also appears on line 156 of the same file.
The command syntax depends on the .NET SDK version. In .NET 9, Microsoft.Testing.Platform arguments
must be passed after the `--` separator. In .NET 10+, `--coverage` is a top-level `dotnet test` flag.
```powershell
$rawDir = Join-Path "<COVERAGE_DIR>" "raw"
- Files reviewed: 56/56 changed files
- Comments generated: 1
github-actions
Bot
added
the
pr-state/ready-for-eval
📊 Skill Evaluation Results4 skill(s) evaluated — ✅ 2 improved, ❌ 2 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win -16.7% (2W/11T/5L over 18 trial(s), sign test p=0.227), mean preference -10.0% — no improvement
❌ platform-detection — detailsReason: Net win +11.1% (2W/6T/1L over 9 trial(s), sign test p=0.500), mean preference -2.2% — not credible — 6 of 9 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
Per-scenario details for 2 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Reset mutable classic fixtures between trials, remove noisy generation scenarios, clarify MSTest runner bridge semantics, and add discriminating platform precedence cases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Failure
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
tests/dotnet-test/run-tests/eval.yaml:87
- This no-tool NUnit filtering stimulus asks for the exact
dotnet testcommand, but the current graders only check for--filterand the filter expression. An answer that never includesdotnet test(or that provides only the filter snippet) could still pass. Add adotnet testoutput-matches grader to align the checks with the prompt requirement.
tests/dotnet-test/writing-mstest-tests/eval.yaml:48 - The async-timeout stimulus can currently pass while leaving
CancellationToken.Noneuntouched. The regex graders only require that the output mentionCancellationTokensomewhere, and the prompt grader isn’t guaranteed to reject the unchanged argument. Since the prompt explicitly requires flowingTestContext.CancellationToken, add an explicit grader forTestContext.CancellationTokenand rejectCancellationToken.Noneto prevent false positives.
- Files reviewed: 88/92 changed files
- Comments generated: 0 new
📊 Skill Evaluation Results4 skill(s) evaluated — ✅ 1 improved, ❌ 3 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +16.7% (2W/10T/0L over 12 trial(s), sign test p=0.250), mean preference +6.7% — not credible — 10 of 12 trial(s) tied, leaving only 2 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ platform-detection — detailsReason: Net win +25.0% (3W/9T/0L over 12 trial(s), sign test p=0.125), mean preference +20.0% — not credible — 9 of 12 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ writing-mstest-tests — detailsReason: Net win +35.7% (7W/5T/2L over 14 trial(s), sign test p=0.090), mean preference +18.6% — not credible (sign test p=0.090 > 0.05)
Per-scenario details for 1 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Align the generation pipeline contract with its eval, improve MSTest routing and response semantics, increase platform evidence, and isolate coverage and CRAP execution shards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (3)
tests/dotnet-test/crap-score/eval.yaml:5
- The PR description says the evals are split into four execution shards (generation, risk, execution, mstest) with
coverage-analysis + crap-scoreboth in the risk shard. This eval usesexecutionShard: risk-crap, which creates an extra shard name and may defeat the intendedmax-parallel: 4scheduling split. Consider using the sharedriskshard name here to match the described scheduling.
tests/dotnet-test/coverage-analysis/eval.yaml:5 - The PR description calls out a single
riskexecution shard containing bothcoverage-analysisandcrap-score. This eval currently usesexecutionShard: risk-coverage, which creates a separate shard name and may reintroduce the single-shard bottleneck (or exceed the intendedmax-parallel: 4). Rename this shard toriskto align with the described scheduling.
tests/dotnet-test/crap-score/eval.yaml:27 - This grader asserts the classic project files are unchanged, but it doesn't currently prevent the skill from creating additional
*.csprojfiles (for example, a temporary SDK wrapper) or emitting a Cobertura report, both of which would violate the scenario's intent but could still pass this check. Consider extending therun-commandto fail if any extra project files or Cobertura XML are created.
- Files reviewed: 92/96 changed files
- Comments generated: 0 new
📊 Skill Evaluation Results6 skill(s) evaluated — ✅ 4 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win -40.0% (1W/4T/5L over 10 trial(s), sign test p=0.109), mean preference -16.0% — no improvement
|
| Scenario | Net win | Δ Pref | Trials (W/T/L) |
|---|---|---|---|
| ▲ Analyse a CI Cobertura report without re-running tests or installing tools | +50.0% | +20.0% | 1/1/0 |
| ▲ Coverage plateau diagnosis | +100.0% | +40.0% | 2/0/0 |
| = Distinguish partially covered branches from covered lines | +0.0% | +0.0% | 0/2/0 |
| ▲ Preserve a classic packages.config project when coverage data is absent | +50.0% | +20.0% | 1/1/0 |
| ▲ Project-wide coverage analysis with existing Cobertura data | +100.0% | +70.0% | 2/0/0 |
| = Reconcile a coverage target spread across several members | +0.0% | +0.0% | 0/2/0 |
| = Refactoring safety assessment from coverage data | +0.0% | +0.0% | 0/1/0 |
| ▲ Run coverage from scratch without existing data | +100.0% | +40.0% | 2/0/0 |
Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown.
🔍 Full Results - additional metrics and failure investigation steps
To investigate failures, paste this to your AI coding agent:
For PR 993 in dotnet/skills, download eval artifacts with
gh run download 31197234575 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/191ed06c10bca672155fea6f69d89d9daf577500/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.
▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions
github-actions
Bot
added
waiting-on-review
Summary
packages.configbefore choosing build, test, or coverage commands<Compile Include>registration, legacy dependency versions, custom fixture bases, and repository MSBuild/VSTest/MSTest workflowsReview
Three independent review rounds were completed. Findings fixed included unsafe
packages.configclassification, mixed-solution coverage execution, unconditional testability migration routing, legacy validation wording, MSTest version-gating contradictions, and fixture adapter integration.Evaluation follow-up
Run
31012795023produced three non-passes with distinct causes:p=0.109), with activation only 1/5. Its frontmatter simultaneously claimed mandatory test generation and excluded MSTest. The only loss said the baseline was better because it covered the exact0%boundary while the skilled run substituted50%. The description now keeps MSTest generation at this entry point, and the classic stimulus explicitly requires<0,0,100, and>100.p=0.063). Two ties left only four discordant trials, so no record could have passed. Two distinct scenarios now measure classic-project coverage safety and asyncMoveNext-to-source line mapping rather than repeating existing tasks.disable-model-invocation: true; the isolated arm randomly lost the SDK 10 precedence case while the plugin arm answered it correctly. Platform detection is now directly invocable and its expanded eval measures harder project-system, runner, bridge, and precedence decisions. The guidance also now distinguishesdotnet testcommand mode from the platform actually executed through the MTP bridge.Run
31039546010was inconclusive because three trials errored atsession.idle:writing-mstest-testsplugin trials stalled at the exact 5-minute budget—one before any response and one after two successful globs. This was classified as transient reliability; the eval now allows 6 minutes without changing skill content..coverage-runSDK project, collected coverage from the substitute assembly, and timed out while writing that misleading report. The skill now has a prominent hard stop forbidding temporary wrappers or substitute coverage. The eval snapshots every project file and rejects any newly generated Cobertura report.coverage-analysisnow uses two runs instead of three. Its eight distinct scenarios still produce 16 counted trials; the prior measured record was 17W/6T/1L over 24.crap-scorecollection fallback to SDK-style projects.Run
31083033945exhausted the fixed 155-minute Vally step after 233 of 234 expected trajectories; all recorded trajectories succeeded and no verdict phase ran. Runtime analysis showed the largest costs in broad generation/build scenarios. The evals now retain every unique requirement while replacing expensive near-ties with focused no-tool stimuli:code-testing-agent, including cooperative timeout syntax and correctIsInRange(min, max, value)ordering; the unchanged measured subset was 11W/1TValidation
dotnet run --no-build --no-restore --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/dotnet-testpython eng/eval-quality/check_eval_quality.pygit diff --checkRun
31112078844confirmed the structural bottleneck: all six changed skills were still assigned to the syntheticdefaultexecution shard, so one Vally job owned every baseline/isolated/plugin trajectory. The workflow already supportsexecutionShard:and merges shard artifacts downstream. The six evals are now split into four domain shards aligned withmax-parallel: 4:generation: code-testing-agentrisk: coverage-analysis + crap-scoreexecution: platform-detection + run-testsmstest: writing-mstest-testsThis changes only scheduling; each skill retains the same scenarios, variants, comparison, and statistical gate. The obsolete monolithic run bound to
8c638d083was cancelled after the sharded commit was pushed.Run
31112078844completed comparisons and narrowed the remaining work to two skills:p=0.250). Only the broad Vitest scenario activated the skill consistently; focused generation was baseline-equivalent. Three distinct project-wide scenarios now exercise the orchestration/checklist advantage across multi-module Python, classic multi-class MSTest, and multi-package Go. At two runs each, the eval now has 18 trials and six new broad trial opportunities without relying on extra repetition.p=0.312). The sole loss was semantically correct—“VSTest command mode executing MTP”—but put VSTest on the requestedPlatform:line. The skill now definesPlatformas the executed platform and reports command mode separately. Two baseline-easy framework cases were replaced by harder, valid SDK 9 boundaries: MTP runner without the dotnet-test bridge, and MSTest.Sdk explicitly opting back into VSTest withUseVSTest=true.Both replacement platform fixtures restore successfully, and the generation fixtures compile/build. The generation shard now evaluates 18 trials; platform detection remains at 9 trials.
Run
31117103578, the first sharded retry, never reached Vally:prepare-validatorfailed while downloading a GitHub Action with repeatedService Unavailableerrors. This was transient infrastructure, not an eval result.