ci: split self-hosted E2E lanes into their own workflow (EAI-7548)

crates/e2e-report/src/lib.rs

@@ -414,6 +414,12 @@ fn parse_descriptor(name: &str) -> Descriptor {
         "gpu" => ("MI300X", "Linux"),
         "gpu-strix-ubuntu" => ("Strix Halo", "Ubuntu"),
         "gpu-strix-windows" => ("Strix Halo", "Windows"),
+        // `e2e-unknown-report`: a report whose platform.json sidecar was missing
+        // or unrecognized (e.g. a GPU run that errored before writing it). The OS
+        // is genuinely unknown here — a Windows GPU run that erupted early must NOT
+        // be reported as Linux — so render Unknown / Unknown rather than defaulting
+        // OS to Linux the way `fallback_descriptor` does for a titlecased platform.
+        "unknown" => ("Unknown", "Unknown"),
         other => return fallback_descriptor(other, known_bugs),
     };
 
@@ -1999,6 +2005,34 @@ mod tests {
         assert_eq!(s.passed, 0);
     }
 
+    #[test]
+    fn parse_descriptor_maps_known_artifacts() {
+        for (name, platform, os) in [
+            ("e2e-report", "Mock", "Linux"),
+            ("e2e-gpu-report", "MI300X", "Linux"),
+            ("e2e-gpu-strix-ubuntu-report", "Strix Halo", "Ubuntu"),
+            ("e2e-gpu-strix-windows-report", "Strix Halo", "Windows"),
+        ] {
+            let d = parse_descriptor(name);
+            assert_eq!(
+                (d.platform.as_str(), d.os.as_str()),
+                (platform, os),
+                "{name}"
+            );
+        }
+    }
+
+    #[test]
+    fn parse_descriptor_unknown_is_not_falsely_linux() {
+        // A report whose platform.json was missing (e.g. a GPU run that errored
+        // before writing the sidecar) is labeled `e2e-unknown-report`. Its OS is
+        // genuinely unknown — a Windows GPU run must NOT be reported as Linux — so
+        // both platform AND os render "Unknown", never a default "Linux".
+        let d = parse_descriptor("e2e-unknown-report");
+        assert_eq!(d.platform, "Unknown");
+        assert_eq!(d.os, "Unknown");
+    }
+
     #[test]
     fn scenario_status_undefined_step_is_not_passed() {
         // Regression: an undefined step must fail the scenario, not pass it.

docs/ci-hardware-testing.md

@@ -8,9 +8,18 @@ SPDX-License-Identifier: MIT
 
 The hosted CI (`ubuntu-latest`, `windows-latest`) builds and unit-tests every
 shipping target natively, but GitHub-hosted runners have no AMD GPU. A
-dedicated hardware layer in `.github/workflows/ci.yml` covers that gap by
-running the same cucumber-rs E2E suite on dedicated self-hosted runners with
-real AMD GPUs.
+dedicated hardware layer covers that gap by running the same cucumber-rs E2E
+suite on dedicated self-hosted runners with real AMD GPUs.
+
+That hardware layer lives in its **own workflow**, `.github/workflows/e2e-selfhosted.yml`,
+separate from the main `ci.yml`. The split is deliberate: a job queued on an
+**offline** self-hosted runner cannot be cancelled by GitHub, so if it shared
+`ci.yml`'s concurrency group a superseded run would hold that group and the
+newer run's merge-required (GitHub-hosted) checks would sit pending forever
+(observed on PR #138). Giving the self-hosted lanes their own workflow — and
+thus their own concurrency group — means an offline runner can only ever stall
+that workflow's own supersession, never `ci.yml`'s required checks. See
+`EAI-7548`.
 
 ## Platforms
 
@@ -20,12 +29,16 @@ scenario to pass / xfail / skip for that host from its `@id` and
 `@requires-*` tags, a capability probe, and `expectations.toml` — there is no
 separate tier flag or tag filter to maintain.
 
-| Job | Platform | Runner labels |
-|---|---|---|
-| `e2e` | Mock (no GPU) | GitHub-hosted `ubuntu-latest` |
-| `e2e-gpu` | MI300X (AMD Instinct, bare-metal Linux) | self-hosted `[self-hosted, linux, amd-gpu]` |
-| `e2e-gpu-strix-ubuntu` | Strix Halo (gfx1151) on Ubuntu | self-hosted `[self-hosted, linux, strix-halo]` |
-| `e2e-gpu-strix-windows` | Strix Halo (gfx1151) on native Windows 11 | self-hosted `[self-hosted, windows, strix-halo]` |
+| Job | Workflow | Platform | Runner labels |
+|---|---|---|---|
+| `e2e` | `ci.yml` | Mock (no GPU) | GitHub-hosted `ubuntu-latest` |
+| `e2e-gpu` | `e2e-selfhosted.yml` | MI300X (AMD Instinct, bare-metal Linux) | self-hosted `[self-hosted, linux, amd-gpu]` |
+| `e2e-gpu-strix-ubuntu` | `e2e-selfhosted.yml` | Strix Halo (gfx1151) on Ubuntu | self-hosted `[self-hosted, linux, strix-halo, native]` |
+| `e2e-gpu-strix-windows` | `e2e-selfhosted.yml` | Strix Halo (gfx1151) on native Windows 11 | self-hosted `[self-hosted, windows, strix-halo, native]` |
+
+The Strix Halo lanes pin the extra `native` label because two Linux runners
+share the `strix-halo` label (a native host and a WSL host) and the jobs'
+hardcoded `/home/ubuntu/actions-runner` paths exist only on the native one.
 
 `e2e` is the blocking, GitHub-hosted mock job: `@requires-gpu` scenarios
 resolve to skip here, and known bugs resolve to xfail from
@@ -36,50 +49,61 @@ run on dedicated self-hosted runners with a real AMD GPU attached, so they
 exercise host/GPU detection, engine `detect`/`capabilities`, and live serving
 scenarios that the mock job cannot.
 
-An `e2e-report` job consolidates every platform's report — including partial
-or failed runs — into one HTML report and GitHub step summary, joined by
-scenario id, so the (scenario × platform) expectation grid is visible in one
-place.
+Each workflow has its own consolidated report job (both named `e2e-report`
+internally). `ci.yml`'s `E2E consolidated report` covers the mock platform;
+`e2e-selfhosted.yml`'s `E2E consolidated report (self-hosted)` covers the three
+GPU platforms. Each joins its platforms' reports — including partial or failed
+runs — by scenario id into one HTML report and GitHub step summary.
 
 ## Triggers
 
-The GPU jobs run automatically on `push`, `pull_request`, and `merge_group`
-when both of the following hold:
-
-- the `changes` job's `heavy` path filter is `true` (the change touches code
-  that can affect runtime behavior, not just docs or unrelated files), and
-- the hosted `build-and-test` job succeeded.
-
-They can also be triggered manually via `workflow_dispatch`, independent of
-the `heavy` gate, with these inputs:
-
-- `platform` (choice: `all`, `mock`, `app-dev-gpu`, `strix-ubuntu`,
-  `strix-windows`) — which job(s) to run. `mock` maps to `e2e`,
-  `app-dev-gpu` to `e2e-gpu`, `strix-ubuntu` to `e2e-gpu-strix-ubuntu`, and
-  `strix-windows` to `e2e-gpu-strix-windows`.
+The GPU jobs (in `e2e-selfhosted.yml`) run automatically on `push`,
+`pull_request`, and `merge_group` when the workflow's own `changes` job's
+`heavy` path filter is `true` (the change touches code that can affect runtime
+behavior, not just docs or unrelated files). Unlike the pre-split layout they do
+**not** gate on the hosted `build-and-test` job — cross-workflow `needs` is not
+possible, so each GPU job builds the `rocm` binary itself as its first real step
+(a broken build fails that job fast and non-fatally). `ci.yml`'s required
+`build-and-test` and mock `e2e` remain the authoritative pre-merge build gate.
+
+They can also be triggered manually via `e2e-selfhosted.yml`'s
+`workflow_dispatch`, independent of the `heavy` gate, with these inputs:
+
+- `platform` (choice: `all`, `app-dev-gpu`, `strix-ubuntu`, `strix-windows`) —
+  which self-hosted job(s) to run. `app-dev-gpu` maps to `e2e-gpu`,
+  `strix-ubuntu` to `e2e-gpu-strix-ubuntu`, and `strix-windows` to
+  `e2e-gpu-strix-windows`. (The mock lane has its own `platform` input on
+  `ci.yml`; it is not part of this workflow.)
 - `name_filter` (string) — a scenario-name regex forwarded to the cucumber
   harness (`cargo xtask e2e -- --name <regex>`) so a dispatch can run a
-  single scenario instead of the full suite. Only wired into the three
-  self-hosted GPU jobs (`e2e-gpu`, `e2e-gpu-strix-ubuntu`,
-  `e2e-gpu-strix-windows`); `platform=mock` always runs the full mock suite
-  and ignores it. Empty runs everything applicable to the selected
-  platform(s).
+  single scenario instead of the full suite. Empty runs everything applicable
+  to the selected platform(s).
 - `include_nightly` (boolean, default `false`) — opts a dispatch into
   `@nightly`-tagged scenarios (e.g. large-model serves, cold installs) that
-  are otherwise skipped on a normal push/PR run to keep it fast. Same scope
-  as `name_filter`: only the three self-hosted GPU jobs read it;
-  `platform=mock` ignores it.
+  are otherwise skipped on a normal push/PR run to keep it fast.
+
+Dispatch the GPU lanes with, e.g.:
 
-A manual dispatch skips the hosted `build-and-test` job for a faster loop; the
-E2E jobs run directly against the dispatched ref.
+```bash
+gh workflow run e2e-selfhosted.yml --ref <ref> -f platform=app-dev-gpu
+```
 
 ## Blocking vs. non-blocking
 
-Only `e2e` (the GitHub-hosted mock job) is a required, blocking check. The
-three hardware jobs — `e2e-gpu`, `e2e-gpu-strix-ubuntu`, and
+The three hardware jobs — `e2e-gpu`, `e2e-gpu-strix-ubuntu`, and
 `e2e-gpu-strix-windows` — all run with `continue-on-error: true`, so a
-hardware failure never gates a PR merge. Their results still surface in the
-consolidated `e2e-report` for visibility.
+hardware failure that RUNS never gates a PR merge. Their results still surface
+in the self-hosted consolidated report for visibility.
+
+**Required-check caveat.** These three job names (plus, historically, a
+consolidated-report name) are still in `main`'s required-status-check list.
+`continue-on-error` neutralizes a job that ran and failed, but a required check
+that *never reports* — because its self-hosted runner is offline — is treated as
+missing and still blocks the merge. The workflow split removes the catastrophic
+concurrency stall (an offline runner can no longer freeze `ci.yml`'s hosted
+required checks), but fully unblocking merges while a runner is offline
+additionally requires removing these self-hosted checks from the required list —
+a branch-protection change tracked separately from the workflow split.
 
 ## Fork safety
 
@@ -96,6 +120,11 @@ which requires write access to trigger).
   not performance, so this is not a performance benchmark. Release-fidelity,
   `manylinux2014` (glibc 2.17) packaging validation is handled separately by
   the nightly/release pipeline.
-- `e2e-report` collects whatever ran, including partial results from a
-  cancelled or failed job, and renders one HTML report plus a step summary so
-  all platforms are visible together.
+- Each workflow's `e2e-report` job collects whatever ran in that workflow —
+  including partial results from a cancelled or failed job — and renders one
+  HTML report plus a step summary. `download-artifact@v8` flattens a
+  single-match download straight into the artifacts directory (it uses the root
+  path when exactly one artifact matches, regardless of the pattern), so after
+  the split each report job usually has one artifact; `xtask e2e-report`'s
+  discovery handles both the flattened and per-subdirectory layouts, labeling a
+  root-level report from its `platform.json` slug.

tests/e2e-cucumber/README.md

@@ -122,42 +122,48 @@ serve does not compete with the first for device memory, and the failure quotes
 the service log tail plus the device's free-VRAM state, which is where the
 engine's own reason for the stall is recorded.
 
-CI runs one job per platform, each executing the full suite:
-
-| Job | Platform | Blocking |
-|---|---|---|
-| `e2e` | Mock (no GPU, GitHub-hosted) | yes |
-| `e2e-gpu` | MI300X (self-hosted) | no |
-| `e2e-gpu-strix-ubuntu` | Strix Halo / Ubuntu (self-hosted) | no |
-| `e2e-gpu-strix-windows` | Strix Halo / Windows (self-hosted) | no |
+CI runs one job per platform, each executing the full suite. The mock job lives
+in the `CI` workflow (`ci.yml`); the self-hosted GPU jobs live in a separate
+`E2E self-hosted` workflow (`e2e-selfhosted.yml`) so a job queued on an offline
+self-hosted runner can never stall `ci.yml`'s merge-required checks:
+
+| Job | Workflow | Platform | Blocking |
+|---|---|---|---|
+| `e2e` | `ci.yml` | Mock (no GPU, GitHub-hosted) | yes |
+| `e2e-gpu` | `e2e-selfhosted.yml` | MI300X (self-hosted) | no |
+| `e2e-gpu-strix-ubuntu` | `e2e-selfhosted.yml` | Strix Halo / Ubuntu (self-hosted) | no |
+| `e2e-gpu-strix-windows` | `e2e-selfhosted.yml` | Strix Halo / Windows (self-hosted) | no |
 
 The blocking mock job passes when every applicable scenario is pass-or-xfail with
-no XPASS or unexpected failure; the GPU jobs are non-blocking. The `e2e-report`
-job consolidates all platforms' results into one cross-platform report.
+no XPASS or unexpected failure; the GPU jobs are non-blocking. Each workflow has
+its own `e2e-report` job: `ci.yml`'s consolidates the mock platform, and
+`e2e-selfhosted.yml`'s consolidates the self-hosted platforms.
 
 The nightly workflow runs three non-blocking jobs — the existing MI300X job and
 new Strix Halo jobs on Ubuntu and Windows — with `E2E_INCLUDE_NIGHTLY=1`. The
 shared large-model scenario serves `Qwen/Qwen3.6-27B` through vLLM on MI300X and
 the hardware-verified `unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL` checkpoint
 through Lemonade on Strix Halo.
 
-Use the CI workflow dispatch to run either model independently on a ref:
+Use the self-hosted E2E workflow dispatch to run either model independently on a
+ref (the GPU platform / `include_nightly` / `name_filter` inputs live on
+`e2e-selfhosted.yml`, not `ci.yml`):
 
 ```bash
 # MI300X / vLLM / Qwen3.6-27B
-gh workflow run ci.yml --ref <ref> \
+gh workflow run e2e-selfhosted.yml --ref <ref> \
   -f platform=app-dev-gpu \
   -f include_nightly=true \
   -f name_filter='large platform-specific model'
 
 # Strix Halo Linux / Lemonade / Qwen3.6-35B-A3B-GGUF (UD-Q4_K_XL)
-gh workflow run ci.yml --ref <ref> \
+gh workflow run e2e-selfhosted.yml --ref <ref> \
   -f platform=strix-ubuntu \
   -f include_nightly=true \
   -f name_filter='large platform-specific model'
 
 # Strix Halo Windows / Lemonade / Qwen3.6-35B-A3B-GGUF (UD-Q4_K_XL)
-gh workflow run ci.yml --ref <ref> \
+gh workflow run e2e-selfhosted.yml --ref <ref> \
   -f platform=strix-windows \
   -f include_nightly=true \
   -f name_filter='large platform-specific model'