Onboard the Build Failure Analysis agentic workflow

[YuliiaKovalova]

Onboards the Build Failure Analysis agentic workflow from microsoft/testfx to dotnet/sdk.

When the Azure Pipelines PR build fails, the workflow reuses the binlogs that build already produced — it does not rebuild — and delegates to the build-failure-analyst agent, which reads them live through the containerized binlog-mcp MCP server and posts an advisory PR comment plus inline suggestion blocks. It is advisory, never gating.

Files

File	Role
.github/workflows/build-failure-analysis.md	Auto trigger on the failing check_run (+ workflow_dispatch for reruns)
.github/workflows/build-failure-analysis-command.md	/analyze-build-failure slash command for manual reruns
.github/workflows/shared/build-failure-analysis-shared.md	Shared prompt body
.github/agents/build-failure-analyst.agent.md	The analyst playbook
*.lock.yml	Generated by gh aw compile (v0.82.9, matching the rest of the repo) — do not hand-edit

Adaptations vs. the testfx original

Aspect	testfx	dotnet/sdk
ADO pipeline	microsoft.testfx, definitionId 209	dotnet-sdk-public-ci, definitionId 101
Logs artifacts	Logs_Build_*	<Leg>_Logs_Attempt<N> → filter _Logs_Attempt[0-9]+$
Base-branch scope	main, rel/*	main, release/*
Copilot auth	single COPILOT_GITHUB_TOKEN	COPILOT_PAT_0..9 pool via shared/pat_pool.md + copilot-pat-pool environment
Tools	default GitHub MCP	cli-proxy: true + github.mode: gh-proxy

The check-name match is against the rollup dotnet-sdk-public-ci check, so the analysis runs once per build rather than once per leg (the pipeline also emits dotnet-sdk-public-ci (Build <leg>) checks).

Validation

I extracted the compiled fetch-binlog script and ran it unmodified against real failed build 1533008 (PR #55527):

PR #55527 base 'main' is in scope.
ADO build 1533008: result='failed' definition='101' sourceBranch='refs/pull/55527/merge'
Extracted 20 binlog(s) from 10/10 legs into /tmp/binlogs
binlog-found=true

All 10 legs resolved and produced binlogs (~650 MB total), comfortably inside the existing per-artifact (500 MB compressed / 2 GB uncompressed) and cumulative (4 GB) guards. The workflow fails closed if any leg is missing a binlog, so this full-coverage result matters.

Security review note

gh aw compile flagged newly-referenced restricted secrets, as required by safe-update mode. Reviewed:

• New secrets: COPILOT_PAT_0 … COPILOT_PAT_9. These are not new to the repository — they are the existing Copilot PAT pool already consumed by issue-triage.lock.yml through .github/workflows/shared/pat_pool.md. This PR reuses that shared import verbatim (uses: shared/pat_pool.md) with the same copilot-pat-pool environment gate; it does not introduce a new secret, widen the pool, or change how a PAT is selected. The selected PAT is exposed only as COPILOT_GITHUB_TOKEN to the Copilot engine, exactly as in issue-triage.md.
• New actions: none beyond what gh-aw generates (actions/checkout, actions/download-artifact, actions/upload-artifact, github/gh-aw-actions/setup), all SHA-pinned by the compiler and already used by existing workflows in this repo.
• New container: mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64, a first-party dotnet-buildtools image, mounted read-only at /data/binlogs.
• Redirects / network: network.allowed is defaults + dotnet. Binlogs are downloaded anonymously from the public dnceng-public/public project.
• Permissions: the agent job is read-only (contents: read, pull-requests: read). All PR writes go through gh-aw safe-outputs in a separate job.

Nothing here I could not verify; the one item worth a second pair of eyes is the deliberate roles: all + min-integrity: none, which are required so the workflow can analyse failing PRs from external contributors. The agent never builds or executes PR code — it only reads binlog data.

[azure-pipelines]

Azure Pipelines:
Successfully started running 2 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR ports a new agentic workflow into dotnet/sdk to automatically (or manually) analyze failed Azure DevOps PR builds by downloading existing per-leg MSBuild binlogs and delegating analysis to the build-failure-analyst agent via a binlog-mcp MCP server—without rebuilding the PR.

Changes:

• Adds an auto-triggered build-failure analysis workflow (check_run + workflow_dispatch) that downloads <Leg>_Logs_Attempt<N> artifacts and stages *.binlog files for analysis.
• Adds a /analyze-build-failure slash-command workflow intended to rerun analysis on demand.
• Introduces shared prompt content + a dedicated agent playbook, and adds the compiled .lock.yml workflow output.

Reviewed changes

Copilot reviewed 4 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
.github/workflows/shared/build-failure-analysis-shared.md	Shared prompt/instructions for the build-failure analyst runs.
.github/workflows/build-failure-analysis.md	Auto workflow: fetches ADO binlogs and runs the analyst agent; includes safety/guard logic.
.github/workflows/build-failure-analysis.lock.yml	Compiled workflow output generated by gh aw compile.
.github/workflows/build-failure-analysis-command.md	Slash-command workflow intended to rerun analysis manually.
.github/agents/build-failure-analyst.agent.md	Analyst playbook describing the methodology and safe-output expectations.

[YuliiaKovalova]

Same fix applied here in f730952 (dotnet/arcade#17228 got the identical change), addressing the suppressed review comments about the permission check. The review is right that admin|write was an assumption about how the endpoint reports roles, and it was load-bearing for maintainers. Rather than depend on which field carries the role, the check now reads both fields the API returns and accepts the union: bash resp=$(gh api "repos/${GITHUB_REPOSITORY}/collaborators/${COMMENTER}/permission" 2>/dev/null) role=$(printf '%s' "${resp}" | jq -r '.role_name // empty' 2>/dev/null) perm=$(printf '%s' "${resp}" | jq -r '.permission // empty' 2>/dev/null) authorized=false for r in "${role}" "${perm}"; do case "${r}" in admin|maintain|maintainer|write) authorized=true ;; esac done role_name carries the precise role, so maintain is authorized and triage stays denied; permission still reports push access for a custom org role whose role_name is a non-standard string. The stale comment about the legacy field collapsing maintain was rewritten to describe this instead. Exercised against every response shape: | response | authorized | |---|---| | admin / write | ✅ | | maintain in role_name (permission: write) | ✅ | | maintain in permission | ✅ | | maintainer spelling | ✅ | | custom org role + permission: write | ✅ | | triage (explicit or collapsed to read) | ❌ | | read / none | ❌ | | custom org role + permission: read | ❌ | | 404 error document / empty / garbage | ❌ | Re-validated in a fork on real issue_comment runs: admin commenter -> 'YuliiaKovalova' has 'admin' access ...; proceeding with the download running; non-write commenter -> ##[warning]'octocat' does not have write access ... (resolved role 'read') with Download binlogs ... => skipped and every downstream job inert.

[nagilson]

Hey @YuliiaKovalova!

I'm excited that we seem to have similar interests; I've been working on a ci failure analysis agentic workflow which is similar in vain to what you're doing for build failure analysis of the main SDK build.

Our approach seems to be different, and I think there's power in having both or even combining our workflows eventually.

This approach is very focused on 'build failure' specifically from the MSBuild side, reading binlogs. I did not use a binlog reader but instead focused on broad root cause analysis of CI failures - in other words, for a specific failure in CI, is it a transitive network flake, a test flake (and why), a nuget config issue, a build issue, etc. Our long-term plan is to extend this skill out over time. I think we can combine the two to make something really cool, and we'd likely want the build failure analysis to be part of the CI overall failure analysis in the long term, although the command invocation logic could remain separate.

I'm unsure how much effort you want to invest in this, but right now there's a very substantial amount of scripted javascript code that seems to be duplicated across the command.md and analysis.md. I would suggest pushing this code out into separate files and invoking them with node. Having duplicated logic will make bugs difficult to sort out, and as it is now, having this complex + large javascript code as embedded run script lines will be difficult to scale and combine. I assume some of the implementation can be modularized and abstracted into concrete components to reduce the amount of code to maintain as well. If there's a bug fix, I don't want to worry about whether the fix was copied to all of the copies.

I recognize that this is intended to only be a port-forward of code from testfx, so I understand if this refactor is not of interest, but I think for us to combine these efforts into something bigger we'd want to have that change 👀

Left two small remarks based on your PR description otherwise, thanks!

[nagilson]

Why was the command and analysis file put into separate workflows? A workflow can have multiple triggers. If this was for different security contexts I think that's more interesting but I'd try to consolidate if at all possible.

[YuliiaKovalova]

You guessed right — it is the different security contexts, and it turns out gh-aw makes consolidation impossible rather than just awkward.

roles: is workflow-scoped, not per-trigger. In the schema it is a sibling of the trigger keys inside the single on: object (pkg/parser/schemas/main_workflow_schema.json), and the compiler reads it exactly once: pkg/workflow/role_checks.go pulls frontmatter["on"]["roles"] into a single []string, emits it as one GH_AW_REQUIRED_ROLES env var, and generates one pre_activation job for the entire workflow (compiler_pre_activation_job.go). There is no syntax for attaching a different role set to a different trigger.

Combining the triggers would be fine — slash_command coexists happily with check_run/workflow_dispatch, nothing in the validator rejects it. It is only the security context that cannot be shared:

workflow	trigger	roles	why
build-failure-analysis	check_run (automatic)	all	must fire on every failing PR, including external contributors'
build-failure-analysis-command	/build-failure-analysis comment	[admin, maintainer, write]	downloads ~600 MB of binlogs on demand

Merging them forces one of two bad outcomes: either external contributors lose automatic analysis, or any commenter can trigger a 600 MB download. So they stay split.

The duplication that split caused was a real problem though, and that part is now fixed — see my reply on the review body.

[nagilson]

This is a great improvement. I do wonder if workflow dispatch could create the security scope changes for us and then dispatch the same build failure .md out, though I'm unsure whether this would work.

[YuliiaKovalova]

The mechanism works. The docs are explicit:

When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run, with the following exceptions:

• workflow_dispatch and repository_dispatch events always create workflow runs.

So a command workflow really could authorize, then gh workflow run build-failure-analysis.lock.yml -f pr-number=N with the plain GITHUB_TOKEN. The automatic workflow already exposes exactly those inputs (pr-number, ado-build-id) — that's how I've been fork-testing it.

Why I don't think it's the right trade here. The command file has to keep existing regardless, because it's what hosts the slash_command trigger and roles:. If its only job becomes "authorize and dispatch", it can't be a gh-aw workflow anymore — gh-aw always compiles an agent job — so it becomes plain YAML, and I'd have to hand-write both the role check and the command-position parsing that pre_activation generates for me today. That's the exact hand-written-authorization surface that just got picked apart in review on the arcade copy of this (dotnet/arcade#17228, where a role_name check I'd written turned out to widen the gate). Trading compiler-generated authorization for more hand-written authorization seems like the wrong direction. It also costs a second run per invocation, and the analysis run is no longer linked to the comment that asked for it.

But the duplication you're pointing at was real, and I've now removed most of it (14fce28).

I diffed the two frontmatters key by key. Four blocks were byte-identical and in gh-aw's importable field set: network, mcp-servers, tools, safe-outputs. Those now live in shared/build-failure-analysis-shared.md, which both workflows already imported. Net -60 lines.

What genuinely cannot be shared, and why:

Field	Why it stays per-workflow
on: / roles: / concurrency:	the reason the two files exist at all
permissions:	imports are validated against the caller, not merged into it — each main workflow must re-declare its own
engine: / environment: / timeout-minutes: / steps:	not in gh-aw's importable set; the engine identifier is "always inherited from the importing workflow"

Verified this is a pure refactor: both regenerated .lock.yml files are byte-identical to the previous ones apart from the frontmatter_hash in the metadata comment (normalized that field and diffed — 0 differing lines in each). gh aw compile --strict v0.82.9, 0 errors / 0 warnings.

So the two files are now down to their genuinely per-trigger parts. Happy to revisit the dispatch shape if gh-aw ever grows per-trigger roles: — that's the thing that would actually collapse them into one file.

[nagilson]

RE your question about roles: all - as long as we're not allowing folks to trigger the workflow themselves and spam AI token cost on us, I don't see any issue with this. Triage workflows already do this across the org.

[YuliiaKovalova]

Thanks — and the exposure is narrower than roles: all makes it look, because this workflow is not comment-triggered at all.

It fires on check_run, and the agent job is gated on needs.fetch-binlog.outputs.binlog-found == 'true'. That output only becomes true when there is a genuinely failed dotnet-sdk-public-ci build (definition 101) for an in-scope PR (base main or release/*) that actually published binlog artifacts, and whose head still matches the PR. Nobody can self-trigger it to burn tokens — you would have to push a commit that really breaks the SDK build.

The on-demand path (/build-failure-analysis), which is comment-triggered and does the expensive fetch, is the one restricted to [admin, maintainer, write] — enforced inside fetch-binlog itself, before the download, since gh-aw's own pre_activation role check runs after it.

[YuliiaKovalova]

Thanks @nagilson — this was the right call, and it's now done.

One clarification first: the duplicated code is bash, not JavaScript. Neither .md source contains an actions/github-script step; the JS you're seeing is gh-aw's own generated boilerplate in the compiled .lock.yml, which isn't hand-authored. The real duplication was ~330 of ~470 lines of the fetch-binlog shell job copied between the two workflows — and it had already drifted (comments out of sync between the two copies), which is exactly the failure mode you were worried about.

On node: I looked at it, but fetch-binlog deliberately runs with no actions/checkout (its only uses: is upload-artifact) — it's a cheap pre-gate that decides whether the expensive agent job runs at all. Invoking node ./scripts/*.js would mean checking out dotnet/sdk on every check_run, which is exactly the cost this job exists to avoid. gh-aw has a first-class mechanism for this instead: imports:, which merges a jobs: block from a shared file at compile time. This repo already uses it for shared/pat_pool.md, so it's the established pattern here.

What changed: the entire job now lives in a single shared/build-failure-analysis-fetch.md, imported by both workflows.

Only the first ~35 lines genuinely differed, because the two triggers resolve the build in opposite directions:

• check_run / workflow_dispatch are told the ADO build id, then derive the PR from its sourceBranch;
• the slash command is told the PR, then queries the newest completed build for refs/pull/<n>/merge.

Both branches converge on the same BUILD_ID / build_json / PR_NUMBER, and everything after that is shared verbatim: base-branch scope check, definition/result validation, staleness guards, artifact-layout detection, missing-leg detection off the ADO timeline, download budgets, fail-closed completeness, output emission, artifact upload.

Net: -1,101 / +974 lines.

Verification (rather than assuming it worked):

• Both workflows compile clean on gh-aw v0.82.9 --strict — 0 errors, 0 warnings.

• The generated fetch-binlog job is byte-identical in both .lock.yml files — 563 lines, same SHA-256. Single source of truth actually proven.

• Diffed old vs. new compiled job: identical apart from the intended import wiring.

• bash -n clean on every compiled run: block.

• Re-ran fork validation in real Actions against both artifact layouts pipeline 101 produces, using genuinely failed builds with no test scaffolding:

  PR	build	base	layout	result
  55475	1536573	main	attempt	10 binlogs from 10/10 artifacts
  55449	1536254	release/9.0.3xx	leg	38 binlogs from 15/16 artifacts

  (the 16th is BuildConfiguration, which correctly reports "no binlog inside" instead of failing closed)

On combining efforts — I'd like that. Broad CI root-cause triage and binlog-level build analysis are complementary: yours answers what kind of failure is this, this one answers which line of code broke the build, and on how many legs. Now that the fetch half is a single importable job with a stable contract (binlog-found, pr-number, missing-legs, plus a staged binlog artifact), wiring it in as a component of a broader CI-failure workflow should be straightforward — happy to pair on that once this lands. Feel free to ping me.

[YuliiaKovalova]

Follow-up to the dedup commit, from a self-review of the shared job.

f36b646 — re-read the PR head before the pre-download staleness check.

When I lifted fetch-binlog into shared/build-failure-analysis-fetch.md I
kept the automatic workflow's version of one line: it reused the PR snapshot
taken earlier during the scope check, whereas the slash-command copy re-read it
(PR_JSON2). Selecting the build costs an Azure DevOps round trip in between,
and right after a force-push the "newest completed build" query can still return
the previous failed build — which is exactly the case this check exists to
catch. So the shared job now re-reads.

To be clear about the blast radius: this is not a correctness fix. The
post-download re-read (LATEST_PR / LATEST_HEAD) already re-validates the
head after the artifacts land and skips if it moved, so a stale build was never
analysable either way. Re-reading here just lets the early-out do its actual job
— skipping before paying for the ~600MB download — instead of discovering the
staleness afterwards. Cost is one extra GET /pulls/{n}.

It also makes this job match microsoft/testfx, where the same shared file is
being introduced in microsoft/testfx#10401, so the two repos don't drift.

Re-verified after the change: compiles clean on gh-aw v0.82.9 --strict
(0 errors, 0 warnings), the fetch-binlog job is still byte-identical in both
.lock.yml files, retention-days is still quoted "1" (the imports:
round-trip turns a bare 1 into 1.0), and bash -n is clean on all
three compiled run: blocks.

[YuliiaKovalova]

Pushed two more changes, carried over from review feedback on dotnet/arcade#17228 (thanks @Evangelink), where this pre-gate shipped first.

1. The pre-gate was matching role_name, which widened it. It accepted the union of role_name and permission over admin|maintain|maintainer|write. Per the REST docs, permission returns the legacy base roles admin|write|read|none, "where the maintain role is mapped to write and the triage role is mapped to read", while role_name gives "the name of the assigned role, including custom roles". So .permission in {admin, write} was already exactly "has push access or better" — the set roles: [admin, maintainer, write] describes, maintainers included — and matching role_name only risked letting a custom organization role merely named like a privileged one clear the gate with no push access. Now reads .permission alone.

While validating that, live testing turned up a real wart in the obvious gh api --jq '.permission' form: on a non-2xx response gh prints the error document to stdout, and --jq does not filter it, so perm becomes a raw JSON blob that gets echoed into the workflow log. Reading the response first and extracting with jq yields "" for any error shape.

shell: bash added to every hand-written run: step.

2. The two workflows shared a concurrency group. Both resolved to build-failure-analysis-<pr>. Groups are repository-global with cancel-in-progress: true, so an automatic analysis and an on-demand /analyze-build-failure for the same PR cancelled each other. Reproduced live in a fork (run 30814539759 had pre_activation/activation/agent/safe_outputs cancelled by run 30814558788 for the same PR number). The command workflow now uses build-failure-analysis-cmd-<pr>; each still collapses its own repeats.

Verified: gh aw compile --strict v0.82.9 → 0 errors / 0 warnings; fetch-binlog still byte-identical across both locks; retention-days still "1"; bash -n clean on all generated run blocks; the new gate exercised against the live GitHub API for admin (allow), read (deny), github-actions[bot] (rejected before the call), empty login (rejected), foo/../../bar (rejected) and a nonexistent user (deny).

Same changes in dotnet/arcade#17240 and microsoft/testfx#10407.

[YuliiaKovalova]

Hey @YuliiaKovalova!

I'm excited that we seem to have similar interests; I've been working on a ci failure analysis agentic workflow which is similar in vain to what you're doing for build failure analysis of the main SDK build.

Our approach seems to be different, and I think there's power in having both or even combining our workflows eventually.

This approach is very focused on 'build failure' specifically from the MSBuild side, reading binlogs. I did not use a binlog reader but instead focused on broad root cause analysis of CI failures - in other words, for a specific failure in CI, is it a transitive network flake, a test flake (and why), a nuget config issue, a build issue, etc. Our long-term plan is to extend this skill out over time. I think we can combine the two to make something really cool, and we'd likely want the build failure analysis to be part of the CI overall failure analysis in the long term, although the command invocation logic could remain separate.

I'm unsure how much effort you want to invest in this, but right now there's a very substantial amount of scripted javascript code that seems to be duplicated across the command.md and analysis.md. I would suggest pushing this code out into separate files and invoking them with node. Having duplicated logic will make bugs difficult to sort out, and as it is now, having this complex + large javascript code as embedded run script lines will be difficult to scale and combine. I assume some of the implementation can be modularized and abstracted into concrete components to reduce the amount of code to maintain as well. If there's a bug fix, I don't want to worry about whether the fix was copied to all of the copies.

I recognize that this is intended to only be a port-forward of code from testfx, so I understand if this refactor is not of interest, but I think for us to combine these efforts into something bigger we'd want to have that change 👀

Left two small remarks based on your PR description otherwise, thanks!

Thank you for the nice comments and sharing your perspective. We are not in rush and can work together on this feature to extend it's capabilities. The points you mentioned seem to be relevant to arcade too (it already has this workflow), so feel free to criticize, contribute and edit this PR , I am all for it :)

[nagilson]

@YuliiaKovalova This looks like a great improvement - thanks for responding to my feedback. Left one more question but I'll let you know when I have time to look into integrating our skills together!

[Evangelink]

If you need fast iteration, feel free to use testfx repo as playground, we have been doing that and it's working well! @YuliiaKovalova knows it all.

[YuliiaKovalova]

Thanks @Evangelink — that's exactly what happened in practice. testfx ended up being where the permission gate got proven: I dispatched three fork runs there to exercise the truth table (denied commenter -> download skipped, owner -> download ran, automatic path -> gate skipped entirely), and that's also where the shared-fetch-binlog refactor landed first. The .permission fix from your arcade review shipped to testfx in microsoft/testfx#10407 and to arcade in dotnet/arcade#17240.

[Evangelink]

Note

These review comments were generated with GitHub Copilot and selected for posting by the reviewer.

Six issues found: two correctness/security concerns and four workflow reliability concerns.

[Evangelink]

target: "*" allows the agent to choose any issue or PR in this repository. Since this workflow intentionally analyzes untrusted fork source and binlog content (min-integrity: none), the prompt instruction to use GH_AW_PR_NUMBER is not a security boundary against prompt injection. Please make the safe-output target deterministic from needs.fetch-binlog.outputs.pr-number, or add an equivalent post-processing check that rejects any other requested target.

[Evangelink]

This call shape does not match the binlog-mcp contract: binlog_warnings accepts binlog_file, code, and project, but no top argument. The agent will get a schema-validation failure exactly when it follows this warning-as-error path. Please remove top: 10 or replace it with a supported filter.

[Evangelink]

This substring check runs before gh-aw's command-position validation. A maintainer merely quoting or discussing /analyze-build-failure therefore starts the multi-hundred-MB artifact download even though pre_activation later rejects the comment as not being a command. Please apply equivalent command-position matching in this pre-gate so only an actual slash command can trigger the download.

[Evangelink]

The boolean form only hides older comments from the same GITHUB_WORKFLOW. These two entry points have different workflow names, so an on-demand rerun will leave the previous automatic analysis visible, and an automatic rerun will leave the command analysis visible. Please use the object form with match so either Build Failure Analysis workflow supersedes comments from the other.

[Evangelink]

This includes every retry artifact instead of the final attempt for each leg. Azure build 1535012 contains both Windows_x64_FullFramework_Logs_Attempt1 and ...Attempt2, confirming both can coexist. If that leg succeeds on retry but another leg makes the overall build fail, the agent can report stale Attempt1 errors as current failures. Please group artifacts by the name before _Attempt<N> and retain only the highest attempt number per leg.

[Evangelink]

This custom MCP image is referenced by a mutable, unversioned tag, and the generated manifest consequently records no digest for it. That allows the analyzer executable and its tool schemas to change without any repository change, which can silently break or alter this workflow. Please pin an immutable digest or a versioned image reference.