Skip to content

test(benchmarks): add BenchmarkDotNet baseline for the .slnx parser#259

Merged
ChrisonSimtian merged 2 commits into
mainfrom
feat/slnx-parser-benchmarks
May 28, 2026
Merged

test(benchmarks): add BenchmarkDotNet baseline for the .slnx parser#259
ChrisonSimtian merged 2 commits into
mainfrom
feat/slnx-parser-benchmarks

Conversation

@ChrisonSimtian

Copy link
Copy Markdown
Collaborator

First benchmark project in the repo — establishes a baseline for the `.slnx` parser inlined in #248. Once landed, optimisation experiments (e.g. "replace the inlined parser with a proper XmlSerializer impl") can compare against concrete numbers instead of guessing.

Summary

New project at `tests/Benchmarks/Fallout.Persistence.Solution.Benchmarks/`:

Aspect Choice
Framework BenchmarkDotNet 0.14.0 (added to `Directory.Packages.props`)
Scenarios `[Params(1, 10, 100, 1000)]` project counts × `[Params(false, true)]` folder layout = 8 measurements
Fixtures Generated programmatically in `[GlobalSetup]` — deterministic, no committed binary blobs
Allocations `[MemoryDiagnoser]` reports allocs + Gen0/1/2 counts alongside wall-time
What's measured `SolutionSerializers.GetSerializerByMoniker(path).OpenAsync(...)` end-to-end — the parser path; facade wrapping excluded
CI behaviour Build-only (the benchmark project compiles via `dotnet build fallout.slnx`); execution is local-only via `dotnet run -c Release`

Why this shape

  • Programmatic fixtures: deterministic across machines + future-proof if the .slnx format gets more attributes; no `tests/Benchmarks/.../fixtures/*.slnx` to maintain.
  • Skip CI execution: GitHub Actions shared runners are noisy. Numbers from CI would shift run-to-run for reasons unrelated to code. If we ever want continuous benchmark tracking, a dedicated machine + results-database is the right shape — explicit non-goal here.
  • `tests/Benchmarks/` precedent: matches the `tests/Consumers/` shape from test(consumers): add NUKE + Fallout consumer compatibility sentinels #256 — a sibling "this is a non-test test" directory.

How to run locally

```sh

all scenarios:

dotnet run --project tests/Benchmarks/Fallout.Persistence.Solution.Benchmarks/ -c Release --filter '*'

one method, full sweep:

dotnet run --project tests/Benchmarks/Fallout.Persistence.Solution.Benchmarks/ -c Release --filter 'SlnxParseBenchmarks'
```

`Release` is required — BenchmarkDotNet refuses Debug builds anyway. Full sweep takes ~3-5 minutes.

What this is NOT

Test plan

  • `dotnet build fallout.slnx` — green
  • BenchmarkDotNet enumerates all 8 scenarios on local quick-run
  • CI green on PR
  • (after merge) maintainer runs the full sweep locally + captures baseline numbers somewhere — the README on the benchmark project, a follow-up issue, or a wiki page.

ChrisonSimtian and others added 2 commits May 28, 2026 20:08
Establishes a performance baseline for the .slnx parser inlined from
Microsoft's vs-solutionpersistence (see #248). Now that we own this
code, future optimisation work — e.g. replacing the inlined parser
with a proper XmlSerializer implementation — has numbers to compare
against rather than vibes.

Project: tests/Benchmarks/Fallout.Persistence.Solution.Benchmarks/
- BenchmarkDotNet via Directory.Packages.props (0.14.0).
- Single benchmark method ParseSlnx with [Params(1, 10, 100, 1000)] ×
  [Params(false, true)] (folder layout) = 8 scenarios.
- Fixtures generated programmatically in [GlobalSetup] — deterministic,
  no committed binary blobs.
- [MemoryDiagnoser] enabled — allocations + GC pressure reported
  alongside wall-time (allocation pressure is often the bigger win
  in parser rewrites).
- README documents how to invoke, what is and isn't measured, and
  why CI doesn't run benchmarks (noisy + slow).

Establishes the precedent for tests/Benchmarks/Fallout.<X>.Benchmarks/
layout for future benchmark projects (tracked in #258).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r the +Benchmarks project

Adding tests/Benchmarks/Fallout.Persistence.Solution.Benchmarks to
fallout.slnx changed the StronglyTypedSolutionGenerator output (which
emits one Solution.<Name> property per project). Snapshot updated to
match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ChrisonSimtian ChrisonSimtian force-pushed the feat/slnx-parser-benchmarks branch from eac2a8b to 550b3fd Compare May 28, 2026 08:12
@ChrisonSimtian ChrisonSimtian merged commit 6b9158e into main May 28, 2026
2 checks passed
@ChrisonSimtian ChrisonSimtian deleted the feat/slnx-parser-benchmarks branch May 28, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant