Public API surface verification & breaking-change detection

[ChrisonSimtian]

Story

As a Fallout maintainer,
I want an automated test that captures our public API surface and fails when it changes unexpectedly,
so that we catch unintended breaking changes before they ship — against our own last-published version (semver discipline) and against the original NUKE surface (migration fidelity + migrator-tool coverage signal).

Status: Draft for contributor discussion (esp. @dennisdoomen). No code cut yet.
Labels: enhancement, RFC, target/2026.

Background / prior art

Inspired by Dennis Doomen's Mockly.ApiVerificationTests. His ApiApproval test:

• Loads the built Mockly.dll and calls assembly.GeneratePublicApi() from PublicApiGenerator to produce a canonical C# text representation of the public surface.
• Drives an xUnit [Theory] off the TargetFrameworks parsed from the csproj, so each TFM gets its own approved snapshot.
• Uses Verify to diff the generated API against a committed ApprovedApi/<framework>.verified.txt. The snapshot in git is the contract.

Well-trodden approach (Andrew Lock, Preventing breaking changes in public APIs with PublicApiGenerator; used by graphql-dotnet, OpenAPI.NET, FluentAssertions).

We already have the building blocks: Verify.Xunit + Verify.DiffPlex are central packages used throughout tests/. Only PublicApiGenerator is new (and, eventually, JetBrains.Annotations).

The two-baselines model (core concept)

There are two distinct baselines doing two distinct jobs — conflating them is the trap:

1. Committed snapshot (.verified.txt, tracks main HEAD). Dennis's model — re-approved each PR. Answers "did this PR move the surface? reviewer, be aware." A change-awareness net. Because it moves forward on every merge, it cannot tell you whether you broke a consumer.
2. Published-package baseline (pinned, only moves when we ship). Answers "does the cumulative unreleased delta break what people actually have?" This is the semver authority — and it is why the baseline must be the latest published version, not main HEAD. Breaking something added to main last week but never published breaks nobody; it is all one unreleased window. The contract consumers depend on is the last thing we shipped. Comparing against main HEAD would flag zero-impact churn and miss cumulative breaks that matter.

Decision: current surface = built DLL (hermetic, matches the branch); breaking-change baseline = latest published package (fetched live, pinned). Committed snapshot = day-to-day net; published-package diff = the authoritative "this needs a major bump" signal that drives the breaking-change label (see D6).

Goals

1. A tests/Fallout.ApiVerificationTests project that snapshots the public API of our publishable packages and surfaces unreviewed API changes in CI.
2. Cover the multi-targeted reality: publishable projects span netstandard2.0 → net10.0; one snapshot per (package × TFM).
3. Authoritative breaking-change detection against the latest published version (per the two-baselines model).
4. A permanent check against the upstream NUKE surface that doubles as a coverage signal for the Fallout.Migrate tool (see D3).

Non-goals

• A public plugin SDK contract (milestone #7, later major).
• Runtime/behavioral compatibility — this is the shape of the API, not its behavior.
• Replacing human judgement: a diff flags change; a maintainer decides whether it is an acceptable break per channel/versioning policy.
• Publishing official public-API documentation. PUBLIC_API.md is not that (see D5); real API docs must be a conscious, [PublicAPI]-coupled decision (see D4).

Resolved decisions

D1/D2 — Current = built DLL; baseline = latest published GA (nuget.org) ✅

Per the two-baselines model. The authoritative breaking-change baseline is the latest GA on nuget.org (the production contract; breaking changes batch to the yearly major, so GA defines stability). Optional secondary: also diff against the latest -preview (GitHub Packages) for earlier awareness, non-authoritative.

D3 — NUKE comparison is permanent and feeds migrator coverage ✅

Kept for the multi-year transition window (≥ ~1 year, likely longer). Namespaces differ (Nuke.* → Fallout.*); we normalize Fallout. ↔ Nuke. prefixes before diffing. New framing: every break detected between NUKE and Fallout that is not absorbed by a transition shim (src/Shims/) or a Fallout.Migrate rule is a migrator backlog item. The test becomes a standing indicator that the migrator has a gap — not a throwaway rebrand gate.

D5 — Retire build/Build.PublicApi.cs (eventually) ✅

It reflects FalloutBuild into a human-readable PUBLIC_API.md — one of Matthias's one-off oddities. Leave it for now, but it is slated for retirement. We will not treat "it happens to be public" as an official API. If we want published API documentation, it must be a deliberate, [PublicAPI]-coupled artifact — not auto-scraped from accidental visibility. This new verification project is not built on top of Build.PublicApi.cs.

D6 — Label-gated breaking changes ✅

No hard CI fail that blocks merge on an intentional break. When the published-package diff shows a breaking delta, the PR gets the breaking-change label → signals a required major bump → we decide when and where to merge (it accumulates toward the yearly major per ADR-0004/AGENTS.md rule #1). The day-to-day committed-snapshot test still fails-and-prompts-re-approval as a normal awareness gate.

Open decisions (need consensus)

D4 — [PublicAPI] annotation: destination vs. phase 1

• Philosophical destination (Chris): the intended public surface should be an explicit [PublicAPI] decision, not "oh look, it's public." This is the right end state and aligns with D5.
• Constraint: PublicApiGenerator has no native [PublicAPI] filter — we reflect for the attribute and feed survivors into its IncludeTypes option. We use zero JetBrains.Annotations today; annotating NUKE's large incidentally-public surface is a repo-wide effort and a design commitment.
• Proposed phasing: Phase 1 snapshots all-public (accept the noise — get the safety net now), explicitly labelled a temporary net, not the endorsed definition of "our API." Phase 2 introduces [PublicAPI] + an annotated-only snapshot as the intended-surface contract, dovetailing with the plugin-architecture work. Open: do we accept the phase-1 noise, or block on annotating first?

D7 — CI network dependency for the published-package baseline

Fetching the latest published package at test time adds a network dependency on CI (and a "feed unreachable" failure mode). Options: (a) fetch live each run; (b) cache/pin the baseline package version and bump it deliberately on each release; (c) generate the baseline snapshot at release time and commit it. Leaning (b) — pinned, deliberate bump — to keep CI hermetic and the baseline auditable in git.

Proposed approach (post-consensus)

1. New tests/Fallout.ApiVerificationTests, registered in fallout.slnx, inheriting auto-injected test deps via Directory.Build.props. Add PublicApiGenerator to Directory.Packages.props.
2. [Theory] over (package, framework) pairs; framework list parsed from each csproj's TargetFrameworks (Dennis's pattern, generalized to N packages).
3. GeneratePublicApi() per assembly → Verify against committed ApprovedApi/<Package>.<framework>.verified.txt (awareness net).
4. Separate breaking-change check: built surface vs. latest published GA package, normalized + scrubbed; breaking delta → breaking-change label workflow.
5. NUKE-comparison check with Nuke.↔Fallout. normalization; uncovered breaks → migrator backlog.
6. Scrub volatile lines (FrameworkDisplayName, calver -preview version/height stamps).

Acceptance criteria (draft — finalize after D4/D7)

• tests/Fallout.ApiVerificationTests exists, in fallout.slnx, green on a clean checkout.
• Public surface of each agreed publishable package is snapshotted per TFM; snapshots committed.
• An unintended public-signature change fails the awareness test with a readable diff.
• An additive change passes after snapshot re-approval, diff visible in the PR.
• A breaking delta against the latest published GA is detected and drives the breaking-change label workflow (no hard merge-block).
• NUKE-comparison check runs with documented namespace normalization; uncovered breaks are reported as migrator gaps.
• Volatile/version-stamped lines scrubbed (no false positives from calver heights).
• Docs: short docs/ note on re-approving snapshots + relation to semver/channel policy.

Questions for @dennisdoomen

• Why net8.0-only for the test project while the SUT multi-targets — was multi-TFM snapshotting on the SUT enough, or did you hit pain generating cross-TFM?
• Did you consider [PublicAPI]/annotation-based filtering and reject it, or was Mockly's surface small enough that all-public was fine?
• Any gotchas with Assembly.LoadFile + GeneratePublicApi across TFMs / on CI runners?
• Have you ever diffed the built surface against a published NuGet package (vs. a committed snapshot), and if so how did you handle the network/pinning question (our D7)?

References

• Mockly: https://github.com/dennisdoomen/mockly/tree/main/Mockly.ApiVerificationTests
• PublicApiGenerator: https://github.com/PublicApiGenerator/PublicApiGenerator
• Andrew Lock: https://andrewlock.net/preventing-breaking-changes-in-public-apis-with-publicapigenerator/
• Existing reflective doc-gen (to retire): build/Build.PublicApi.cs
• Versioning/channel policy: ADR-0004, ADR-0008, AGENTS.md critical rules Security: pin vulnerable transitives (CVE-2026-33116, Scriban) — mirrors upstream #1592 #1–2

[ChrisonSimtian]

@dennisdoomen I threw together some thoughts around this and made a new milestone.
I'd also love to have some "dummy" projects sitting somewhere that we can run integration tests against before we start bigger pieces of work (different story)
Thoughts?

[dennisdoomen]

2. Breaking something added to main last week but never published breaks nobody; it is all one unreleased window. The contract consumers depend on is the last thing we shipped.

Anything that we merge to main should be shippable at any point. That's why I mark the PRs with breaking changes with a specific label, which then gets included in the release notes.

• the intended public surface should be an explicit [PublicAPI] decision, not "oh look, it's public." This is the right end state and aligns with D5.

I don't get that. [PublicAPI] is just to tell the IDE (and ourselves) that something is properly used, even though there is no code using it.

PublicApiGenerator has no native [PublicAPI] filter — we reflect for the attribute and feed survivors into its IncludeTypes option. We use zero JetBrains.Annotations today; annotating NUKE's large incidentally-public surface is a repo-wide effort and a design commitment.

Not sure what this means.

CI network dependency for the published-package baseline

IMO, a non-issue. A dependency on NuGet.org is already there.

• Why net8.0-only for the test project while the SUT multi-targets — was multi-TFM snapshotting on the SUT enough, or did you hit pain generating cross-TFM?

What do you mean? There are snapshots for all frameworks.

Did you consider [PublicAPI]/annotation-based filtering and reject it, or was Mockly's surface small enough that all-public was fine?

Don't understand this question. Something that public is public.

• Any gotchas with Assembly.LoadFile + GeneratePublicApi across TFMs / on CI runners?

I've seen some quirks with dependencies, which requires you to add them explicitly to the verification project. But that's it.

• Have you ever diffed the built surface against a published NuGet package (vs. a committed snapshot), and if so how did you handle the network/pinning question (our D7)?

Nope. But it's a great idea. In my OSS projects, I never needed that.