Add golden end-to-end test generation scenarios - #997
Conversation
Add reusable scaffolding and testability skills plus eval fixtures for zero-to-one, partial-suite gap, and ambient-dependency workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
Use the SDK-style project type GUID in the solution fixture and keep the discount lower bound explicitly decimal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
|
/evaluate ca8f82d |
There was a problem hiding this comment.
Pull request overview
This PR adds “golden” end-to-end evaluation scenarios and fixtures for the dotnet-test plugin so test generation and testability workflows are exercised against realistic starting states (zero-to-one scaffolding, partial suites, and ambient time/filesystem dependencies), and extends skill/agent guidance to route those scenarios through the intended helpers.
Changes:
- Introduces a new
testability-obstacleskill plus corresponding eval/fixtures covering time + filesystem seams, existing seams, and static-API clock overrides. - Adds new C# fixture solutions and expands
code-testing-agenteval coverage for zero-to-one scaffolding and gap-filling in an existing suite. - Updates agent/skill docs and plugin README to reflect the new orchestration behavior and helper skills.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/testability-obstacle/fixtures/static-receipt-namer/tests/ReceiptNamer.Tests.csproj | New MTP/xUnit v3 test project fixture for static clock-seam scenario. |
| tests/dotnet-test/testability-obstacle/fixtures/static-receipt-namer/src/ReceiptNamer.csproj | New production project fixture for static receipt naming utility. |
| tests/dotnet-test/testability-obstacle/fixtures/static-receipt-namer/src/ReceiptNamer.cs | Static API using UTC time to force a scoped, async-safe clock seam. |
| tests/dotnet-test/testability-obstacle/fixtures/python-report-writer/report_writer.py | Non-.NET fixture used as a dormancy/off-target routing guard. |
| tests/dotnet-test/testability-obstacle/fixtures/expiration-policy/tests/ExpirationPolicy.Tests.csproj | New test project fixture for TimeProvider migration boundary testing. |
| tests/dotnet-test/testability-obstacle/fixtures/expiration-policy/src/ExpirationPolicy.csproj | New production project fixture for expiration policy. |
| tests/dotnet-test/testability-obstacle/fixtures/expiration-policy/src/ExpirationPolicy.cs | Production code using DateTime.UtcNow to motivate a TimeProvider seam. |
| tests/dotnet-test/testability-obstacle/fixtures/config-loader/tests/ConfigLoader.Tests.csproj | New test project fixture for filesystem seam request. |
| tests/dotnet-test/testability-obstacle/fixtures/config-loader/src/ConfigLoader.csproj | New production project fixture for config loader. |
| tests/dotnet-test/testability-obstacle/fixtures/config-loader/src/ConfigLoader.cs | Production code using File.* to motivate a minimal filesystem seam. |
| tests/dotnet-test/testability-obstacle/fixtures/archive-writer/tests/ArchiveWriter.Tests.csproj | New test project fixture for combined time+filesystem deterministic testing. |
| tests/dotnet-test/testability-obstacle/fixtures/archive-writer/src/ArchiveWriter.csproj | New production project fixture for archive writer. |
| tests/dotnet-test/testability-obstacle/fixtures/archive-writer/src/ArchiveWriter.cs | Production code combining UTC timestamp + filesystem write. |
| tests/dotnet-test/testability-obstacle/fixtures/already-seamed/tests/SnapshotExporter.Tests.csproj | New test project fixture verifying reuse of existing seams. |
| tests/dotnet-test/testability-obstacle/fixtures/already-seamed/src/SnapshotExporter.csproj | New production project fixture for already-seamed exporter. |
| tests/dotnet-test/testability-obstacle/fixtures/already-seamed/src/SnapshotExporter.cs | Production code already taking TimeProvider + storage seam. |
| tests/dotnet-test/testability-obstacle/eval.yaml | New eval spec for the testability-obstacle skill across multiple seam types. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/src/Commerce.Pricing/PriceCalculator.cs | New production behavior used to validate zero-to-one scaffolding + useful tests. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/src/Commerce.Pricing/Commerce.Pricing.csproj | New pricing project fixture referenced by generated tests. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/src/Commerce.Domain/Money.cs | New domain value type used by pricing behavior. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/src/Commerce.Domain/Commerce.Domain.csproj | New domain project fixture in the solution. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/README.md | Fixture README encoding scaffolding conventions (tests dir, CPM, xUnit v3). |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/Directory.Packages.props | Central package management for the fixture (xUnit v3 version). |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/Directory.Build.props | Shared fixture build defaults (net10.0, nullable, implicit usings). |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-zero-to-one-scaffolding/Commerce.sln | New solution used to validate dotnet test discovery at solution level. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-partial-discount-suite/tests/DiscountRulesTests.cs | Baseline partial suite used to validate gap discovery without duplication. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-partial-discount-suite/tests/DiscountRules.Tests.csproj | New test project fixture for partial-suite gap discovery scenario. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-partial-discount-suite/src/DiscountRules.csproj | New production project fixture for discount rules. |
| tests/dotnet-test/code-testing-agent/fixtures/csharp-partial-discount-suite/src/DiscountRules.cs | Production logic with edge cases intended to be discovered by the agent. |
| tests/dotnet-test/code-testing-agent/eval.yaml | Adds new scenarios and adjusts runs to keep the eval verdict reachable. |
| tests/dotnet-test/agent.testability-migration/eval.yaml | Updates eval shape to defaults: and refines prompts toward bounded planning/implementation intent. |
| plugins/dotnet-test/skills/testability-obstacle/SKILL.md | New public skill guidance for minimal seams + deterministic tests. |
| plugins/dotnet-test/skills/scaffold-dotnet-test-project/SKILL.md | New internal helper skill for zero-to-one .NET test-project scaffolding. |
| plugins/dotnet-test/skills/code-testing-agent/SKILL.md | Updates code-testing workflow to invoke scaffolding/gap-analysis/testability-obstacle as prerequisites. |
| plugins/dotnet-test/README.md | Documents new skill and internal helper; updates entry-point agent description. |
| plugins/dotnet-test/agents/testability-migration.agent.md | Extends orchestration pipeline to include deterministic test writing when requested. |
| plugins/dotnet-test/agents/code-testing-generator.agent.md | Adds explicit prerequisites for .NET scaffolding and testability-obstacle routing. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Suppressed comments (2)
tests/dotnet-test/testability-obstacle/eval.yaml:32
- This grep alternation is double-escaped (
\\|) inside single quotes, so it won't behave as an OR. That makes the grader much weaker (and it can pass even if tests don't introduce any fake/in-memory implementation naming signal). Use a single\|for alternation.
command: sh -c "grep -R 'TimeProvider' fixtures/archive-writer/tests --include='*.cs' | grep -q . && grep -R 'Fake\\|Stub\\|InMemory\\|Recording' fixtures/archive-writer/tests --include='*.cs' | grep -q ."
tests/dotnet-test/testability-obstacle/eval.yaml:89
- Both grep alternations are double-escaped (
\\|) inside single quotes, so the negative checks won't reliably detectFile.ReadAllText/File.ExistsorIServiceCollection/ServiceCollection. This can let the scenario pass even when the production code still uses System.IO directly or a DI container was added. Use a single\|for alternation.
command: sh -c "! grep -q 'File.ReadAllText\\|File.Exists' fixtures/config-loader/src/ConfigLoader.cs && ! grep -R 'IServiceCollection\\|ServiceCollection' fixtures/config-loader --include='*.cs'"
- Files reviewed: 37/37 changed files
- Comments generated: 1
📊 Skill Evaluation Results2 skill(s) evaluated — ✅ 0 improved, ❌ 2 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +42.9% (3W/4T/0L over 7 trial(s), sign test p=0.125), mean preference +25.7% — not credible — 4 of 7 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ testability-obstacle — detailsReason: Net win -16.7% (0W/5T/1L over 6 trial(s), sign test p=0.500), mean preference -6.7% — no improvement
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
👋 @Evangelink — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Use extended regular expressions so forbidden-call and fake-name graders reliably match each alternative without YAML escape ambiguity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
|
/evaluate b3e6b17 |
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
tests/dotnet-test/code-testing-agent/eval.yaml:15
- The stimulus name uses "Csharp" rather than the conventional "C#" (or "CSharp"), which makes the scenario title look like a typo in reports/logs. Consider updating it for clarity and consistency with other .NET references in this repo.
- name: Create and wire the first test project for a Csharp solution
- Files reviewed: 37/37 changed files
- Comments generated: 0 new
📊 Skill Evaluation Results2 skill(s) evaluated — ✅ 0 improved, ❌ 2 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +14.3% (1W/6T/0L over 7 trial(s), sign test p=0.500), mean preference +5.7% — not credible — 6 of 7 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ testability-obstacle — detailsReason: Net win +16.7% (1W/5T/0L over 6 trial(s), sign test p=0.500), mean preference +6.7% — not credible — 5 of 6 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
|
✅ Evaluation passed for |
Bundle isolated-run guidance, sharpen plugin routing, and add solution-filter, nested-scope, and fake-time delay decisions. Repeat the expensive end-to-end scenarios twice to clear the observed tie rate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
|
/evaluate 32f01da |
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
tests/dotnet-test/code-testing-agent/eval.yaml:15
- Scenario name uses "Csharp" which reads like a typo/inconsistent casing for C#; this shows up in logs/reports and is easy to misread. Consider spelling it as "C#" (or "CSharp") for clarity.
- name: Create and wire the first test project for a Csharp solution
- Files reviewed: 45/45 changed files
- Comments generated: 0 new
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results4 skill(s) evaluated — ✅ 0 improved, ❌ 4 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win -18.8% (1W/11T/4L over 16 trial(s), sign test p=0.188), mean preference -7.5% — no improvement
❌ generate-testability-wrappers — detailsReason: Net win +26.7% (8W/3T/4L over 15 trial(s), sign test p=0.194), mean preference +10.7% — not credible (sign test p=0.194 > 0.05)
❌ migrate-static-to-wrapper — detailsReason: Net win +16.7% (3W/1T/2L over 6 trial(s), sign test p=0.500), mean preference -3.3% — not credible (sign test p=0.500 > 0.05)
❌ testability-obstacle — detailsReason: Net win +14.3% (3W/10T/1L over 14 trial(s), sign test p=0.312), mean preference +5.7% — not credible — 10 of 14 trial(s) tied, leaving only 4 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Restore code-testing and migration skills to main, promote zero-to-one scaffolding to its own skill, extend verified gap analysis through focused test additions, and concentrate testability evaluation on nested ambient-scope behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
Skill Coverage Report
Uncovered:
|
GitHub Actions
Inactive
|
⏳ Commit |
github-actions
Bot
added
waiting-on-review
Fix xUnit v3 discovery and no-op scaffolding, constrain smoke-test scope, double the clean gap-analysis evidence, and add two deterministic static-identity scenarios matching the strongest testability delta. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
|
/evaluate 3159000 |
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (6)
tests/dotnet-test/testability-obstacle/eval.yaml:63
- This grader requires AsyncLocal to appear in LegacyConfig.cs specifically. If the implementation introduces a separate in-memory filesystem provider helper under src/ (while keeping LegacyConfig static), this check would fail even though the behavior is correct. Grep the src directory instead of the single file.
tests/dotnet-test/testability-obstacle/eval.yaml:129 - This check hard-codes CorrelationIds.cs as the only acceptable location for AsyncLocal. If the seam is factored into another file under src/, the grader will fail a correct solution. Consider grepping across fixtures/static-correlation-ids/src instead.
tests/dotnet-test/testability-obstacle/eval.yaml:162 - This grader forces both AsyncLocal and Random.Shared to appear in CouponCodes.cs. A correct implementation could keep CouponCodes static but place the AsyncLocal random provider into a helper file. Grepping across src keeps the intent while avoiding false negatives.
tests/dotnet-test/testability-obstacle/eval.yaml:196 - This grader requires AsyncLocal to be in MachineLabels.cs specifically, which can reject equivalent solutions that factor the ambient seam into a helper file. Grep across src for the seam markers, and use -q for the negative SetEnvironmentVariable check to avoid dumping output on failure.
tests/dotnet-test/testability-obstacle/eval.yaml:96 - Like the earlier scenarios, this assumes AsyncLocal must be implemented inside FeatureFlags.cs. A helper-file based seam would be rejected even if the public static API stays intact. Grep recursively under src to make the grader accept equivalent implementations.
tests/dotnet-test/testability-obstacle/eval.yaml:30 - These graders assume the seam’s AsyncLocal (and even the static class declaration) must live in a specific source file. A valid implementation could keep the public static API but move the AsyncLocal-backed provider into a separate internal helper file, which would incorrectly fail this check. Consider grepping across the whole src directory instead of a single .cs file.
This issue also appears in the following locations of the same file:
- line 63
- line 96
- line 129
- line 162
- line 196
- Files reviewed: 59/59 changed files
- Comments generated: 0 new
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results4 skill(s) evaluated — ✅ 0 improved, ❌ 4 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win -6.3% (2W/11T/3L over 16 trial(s), sign test p=0.500), mean preference -2.5% — no improvement
❌ generate-testability-wrappers — detailsReason: Net win +20.0% (7W/4T/4L over 15 trial(s), sign test p=0.274), mean preference +20.0% — not credible (sign test p=0.274 > 0.05)
❌ migrate-static-to-wrapper — detailsReason: Net win +16.7% (2W/3T/1L over 6 trial(s), sign test p=0.500), mean preference +16.7% — not credible — 3 of 6 trial(s) tied, leaving only 3 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
❌ testability-obstacle — detailsReason: Net win +7.1% (1W/13T/0L over 14 trial(s), sign test p=0.500), mean preference +2.9% — not credible — 13 of 14 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
🔍 Full Results - additional metrics and failure investigation steps
|
📊 Skill Evaluation Results3 skill(s) evaluated — ✅ 1 improved, ❌ 2 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ scaffold-dotnet-test-project — detailsReason: Net win +42.9% (8W/4T/2L over 14 trial(s), sign test p=0.055), mean preference +17.1% — not credible (sign test p=0.055 > 0.05)
❌ test-gap-analysis — detailsReason: Net win +28.6% (8W/2T/4L over 14 trial(s), sign test p=0.194), mean preference +2.9% — not credible (sign test p=0.194 > 0.05)
Per-scenario details for 1 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results3 skill(s) evaluated — ✅ 1 improved, ❌ 2 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ scaffold-dotnet-test-project — detailsReason: Net win +14.3% (5W/6T/3L over 14 trial(s), sign test p=0.363), mean preference +5.7% — not credible (sign test p=0.363 > 0.05)
❌ testability-obstacle — detailsReason: Net win +31.3% (6W/9T/1L over 16 trial(s), sign test p=0.063), mean preference +12.5% — not credible (sign test p=0.063 > 0.05)
Per-scenario details for 1 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
Drop consistently losing or inert scenarios, add SDK solution registration and empirical retry-policy mutation coverage, and sharpen activation for specialized workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
|
/evaluate 757f99e |
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results3 skill(s) evaluated — ✅ 2 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ test-gap-analysis — detailsReason: Net win +25.0% (8W/4T/4L over 16 trial(s), sign test p=0.194), mean preference +2.5% — not credible (sign test p=0.194 > 0.05)
Per-scenario details for 2 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
📊 Skill Evaluation Results3 skill(s) evaluated — ✅ 2 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ scaffold-dotnet-test-project — detailsReason: Net win +21.4% (4W/9T/1L over 14 trial(s), sign test p=0.188), mean preference +8.6% — not credible (sign test p=0.188 > 0.05)
Per-scenario details for 2 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
AbhitejJohn
left a comment
AbhitejJohn
left a comment
There was a problem hiding this comment.
I rechecked the latest PR head. The four inline findings still apply.
(Copilot, commenting on Abhitej's behalf.)
| Create the first .NET test project. USE FOR: "solution has no tests", first | ||
| xUnit tests, Tests.csproj/ProjectReference, add an omitted test project to | ||
| .sln/.slnx/.slnf, central packages, or tests missing from CI. DO NOT USE FOR: | ||
| a suitable existing project (stop; use code-testing-agent) or migration. |
There was a problem hiding this comment.
Thanks for adding the omitted-wiring scenarios. Could we narrow this exclusion to a suitable project that is already registered in the requested build entry point? As written, the routing text says to stop for any suitable existing project, but the eval at lines 129-187 expects this skill to activate when an existing project is missing from Catalog.sln or TextTools.slnf. The body already has the correct narrower stop condition at lines 62-65, but the runtime routes from this frontmatter.
(Copilot, commenting on Abhitej's behalf.)
| ### Phase 4: Test | ||
|
|
||
| When the request includes tests, use `testability-obstacle` or | ||
| `code-testing-agent` to: |
There was a problem hiding this comment.
Could we split these two paths? After Phase 3, the seam is already injected, while testability-obstacle explicitly says not to use it when a dependency is already injected and to use code-testing-agent instead. Would it work to use only code-testing-agent here, and treat testability-obstacle as an alternative targeted path that skips Generate/Migrate rather than as the final phase after them?
(Copilot, commenting on Abhitej's behalf.)
|
|
||
| ### Phase 4: Test | ||
|
|
||
| When the request includes tests, use `testability-obstacle` or |
There was a problem hiding this comment.
Could we add one agent-level scenario that requests migration and deterministic tests? This PR adds the full Phase 4 path and removes the old inter-phase confirmation behavior, but tests/dotnet-test/agent.testability-migration/eval.yaml still has only an analysis-only scenario and a targeted TimeProvider migration scenario. Neither exercises this new path, so the stated agent capability can regress without an eval noticing.
(Copilot, commenting on Abhitej's behalf.)
| --- | ||
| name: testability-obstacle | ||
| description: >- | ||
| Make a static C# API testable and add tests. USE FOR: deterministic Guid, |
There was a problem hiding this comment.
Small routing consistency question: could this lead describe C# behavior with ambient dependencies rather than only a static API? The body supports constructor injection for instance classes, and the eval at lines 144-177 expects this skill for the instance LeaseRenewer/TimeProvider case. USE FOR: time may still route correctly, so I see this as an ambiguity rather than a proven activation failure.
(Copilot, commenting on Abhitej's behalf.)
github-actions
Bot
added
waiting-on-author
Why
The test plugin needs stable end-to-end scenarios for three workflows: creating and wiring the first .NET test project, empirically finding and closing gaps in an existing suite, and safely testing public static APIs that depend on ambient state.
What changed
scaffold-dotnet-test-projectfor central packages, bounded project references,.sln/.slnf/.slnxregistration, no-op reuse, and missing CI wiring.test-gap-analysisso it can add focused tests only for empirically survived mutations. Its implementation scenario replays concrete mutations after generation.testability-obstaclefor nested, async-safe ambient overrides across time, filesystem, environment, and identity, plus built-in fake-time delays and a Python dormancy guard.Evaluation history and response
Early consumer-oriented revisions mostly tied and then regressed
code-testing-agent; they also pulled stable sibling skills into evaluation. Those changes were removed.code-testing-agent, its generator,generate-testability-wrappers, andmigrate-static-to-wrapperremain byte-for-byte identical tomain.Specialized runs then established repeatable signal:
31178542305: gap analysis 3W/4T/0L; testability 3W/9T/0L.31185716948and31188608506showed substantial judge variance, but also stable decisions.31185716948: testability passed 9W/7T/0L; scaffold was one p-value step short at 8W/4T/2L.31188608506: gap analysis passed 9W/4T/1L; testability was one win short at 6W/9T/1L.This revision removes only stable negative/inert signal and adds independent proven-family evidence:
.slnxproject registration, complementing the repeatedly winning.sln/.slnfCI-wiring scenarios.Validation
skill-validator check --plugin ./plugins/dotnet-testpython eng/eval-quality/check_eval_quality.py.slnxand retry-policy fixtures