ci: build once, test exact outputs, and promote by digest

.github/workflows/images-precheck.yml

@@ -30,6 +30,12 @@ jobs:
           node --experimental-strip-types scripts/image-matrix.ts homebrew-plan --version v0.73.1
           node --experimental-strip-types scripts/image-matrix.ts npm-matrix --version v0.73.1
           node --experimental-strip-types scripts/image-matrix.ts npm-plan --version v0.73.1 --lane-filter linux-x64
+          node --experimental-strip-types scripts/release-plan.ts \
+            --version v0.73.1 --mesh-ref v0.73.1 --mesh-repository Mesh-LLM/mesh-llm \
+            --image ghcr.io/mesh-llm/mesh-llm --dry-run true \
+            --publish-images false --publish-release-assets false --publish-npm false \
+            --validate-native true --validate-homebrew false --validate-npm false \
+            --native-selector ubuntu-cpu-amd64 --npm-selector all
           if node --experimental-strip-types scripts/image-matrix.ts github-matrix --version v0.73.1 --variant-filter alpine-cpu; then
             echo "blocked Alpine row entered the package matrix" >&2
             exit 1
@@ -39,8 +45,8 @@ jobs:
           node --experimental-strip-types --test --experimental-test-coverage \
             --test-coverage-lines=100 --test-coverage-branches=100 --test-coverage-functions=100 \
             tests/image-matrix.test.ts
-          node --experimental-strip-types --test \
-            tests/client-readiness-smoke.test.ts tests/homebrew-release.test.ts tests/node-sdk-package.test.ts tests/node-sdk-runtime-smoke.test.ts tests/product-contract.test.ts tests/release-evidence.test.ts tests/sbom-subject.test.ts tests/upstream-archive.test.ts tests/workflow-provenance.test.ts
+          node --experimental-strip-types --test --test-concurrency=1 \
+            tests/client-readiness-smoke.test.ts tests/homebrew-release.test.ts tests/node-sdk-package.test.ts tests/node-sdk-runtime-smoke.test.ts tests/product-contract.test.ts tests/release-evidence.test.ts tests/release-index.test.ts tests/release-plan.test.ts tests/release-workflow.test.ts tests/sbom-subject.test.ts tests/upstream-archive.test.ts tests/upstream-node-addon.test.ts tests/workflow-provenance.test.ts
       - name: Lint shell scripts
         run: |
           sudo apt-get update
@@ -50,7 +56,8 @@ jobs:
         run: |
           go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.12
           "$(go env GOPATH)/bin/actionlint" -color
-          if rg -n -i 'blacksmith|buildjet|runs-on:.*self-hosted|git clone|cargo build|pnpm.*build' .github/workflows/images-release.yml docker; then
+          if rg -n -i 'runs-on:.*self-hosted|git clone|cargo build|pnpm.*build' \
+            .github/workflows/images-release.yml .github/workflows/package-image-row.yml docker; then
             echo "legacy runner or source-build path remains in packaging automation" >&2
             exit 1
           fi
@@ -73,9 +80,6 @@ jobs:
           docker buildx build --check --target runtime --file docker/Dockerfile.mesh-llm \
             --build-arg RUNTIME_BASE_IMAGE=ubuntu:24.04 --build-arg DISTRO=ubuntu \
             --build-arg BACKEND=cpu --build-arg MESH_LLM_VERSION=0.73.1 .
-          docker buildx build --check --target runtime-qa --file docker/Dockerfile.mesh-llm \
-            --build-arg RUNTIME_BASE_IMAGE=ubuntu:24.04 --build-arg DISTRO=ubuntu \
-            --build-arg BACKEND=cpu --build-arg MESH_LLM_VERSION=0.73.1 .
           docker buildx build --check --platform linux/amd64 --target runtime --file docker/Dockerfile.mesh-llm \
             --build-arg RUNTIME_BASE_IMAGE=archlinux:base --build-arg DISTRO=arch \
             --build-arg BACKEND=vulkan --build-arg MESH_LLM_VERSION=0.73.1 .

README.md

@@ -55,9 +55,15 @@ rendered Homebrew formula is also published through the canonical
 
 ## Automation
 
-`.github/workflows/images-release.yml` accepts the `mesh-llm-release` repository dispatch event and safe manual backfills. Every manual run defaults to `dry_run=true`. Dry-run mode forces both publish switches off while still downloading, checksumming, packaging, installing, image-building, and Homebrew-testing the selected rows.
+`.github/workflows/images-release.yml` accepts the `mesh-llm-release` repository dispatch event and safe manual backfills. Every manual run defaults to `dry_run=true`. Dry-run mode forces every publication switch off while preserving the explicitly selected validation components.
 
-Publishing uses separate switches for GHCR images, package release assets, and npm. All require `dry_run=false`; npm uses the `npm` environment and the other channels use `release`. `npm_lane_filter` can schedule one or more addon lanes independently and always disables npm publication for that run.
+Publishing uses separate switches for GHCR images, package release assets, and npm. All require `dry_run=false`; npm uses the `npm` environment and the other channels use `release`. Manual runs use typed `validate_native`, `validate_homebrew`, and `validate_npm` switches. `native_selector` and `npm_selector` accept only `all` or exact checked-in IDs; partial selections can validate but cannot publish.
+
+Each native row resolves its package and runtime bases to immutable digests,
+produces one native package, and builds one final image. Dry runs load and test
+that exact local image without registry writes. Publishing runs push a
+run-scoped staging image, test it by digest, assemble a canonical release
+index, and promote the tested digest without rebuilding.
 
 The upstream release repository must send this payload after its GitHub Release is published:
 
@@ -73,7 +79,7 @@ Cross-repository dispatch requires a fine-grained token or GitHub App with Actio
 node --experimental-strip-types scripts/image-matrix.ts validate
 node --experimental-strip-types scripts/image-matrix.ts upstream-matrix --version v0.73.1
 node --experimental-strip-types scripts/image-matrix.ts npm-matrix --version v0.73.1
-node --experimental-strip-types --test tests/*.test.ts
+node --experimental-strip-types --test --test-concurrency=1 tests/*.test.ts
 shellcheck docker/*.sh packaging/native/*.sh scripts/*.sh
 actionlint
 docker buildx build --check --target native-package-artifact -f docker/Dockerfile.mesh-llm .

TODO.md

@@ -1,5 +1,36 @@
 # Production Readiness TODO
 
+- [x] Replace widening workflow filters with a typed, fail-closed release plan.
+  Final result: manual runs select native, Homebrew, and npm validation through
+  typed booleans plus exact checked-in IDs or `all`; publication implications
+  require the complete matching producer set, and readiness accepts skips only
+  for disabled components.
+  QA: planner tests cover native-only, Homebrew-only, npm-only, full release,
+  exact selectors, invalid/duplicate selectors, publish implications, and
+  readiness-required versus expected-skipped results; actionlint and the
+  workflow provenance suite pass.
+
+- [x] Stage each runtime image once, test its exact registry digest, and promote
+  only that tested digest without another Docker build.
+  Final result: each row resolves immutable bases and produces one package plus
+  one final image; dry runs test a locally loaded image without registry writes,
+  while publish runs test a run-scoped staging digest, assemble a canonical
+  index, and retag only that digest with immutable-tag guards and a rollback
+  ledger.
+  QA: workflow tests prove one image build per publishing row, digest-bound QA,
+  immutable version-tag conflict rejection, convenience-tag rollback evidence,
+  zero Docker builds in promotion, and a complete deterministic release index.
+
+- [x] Consume upstream-produced Node addon artifacts instead of compiling addon
+  source in mesh-packaging.
+  Final result: the upstream release owns five platform-native addon producers;
+  packaging downloads their versioned archives and checksum sidecars, rejects
+  unsafe layouts or manifest/digest drift, and assembles npm without Cargo or a
+  native source build.
+  QA: upstream release tests prove all five target artifacts are checksummed and
+  published; packaging tests reject unsafe, missing, or digest-mismatched addons
+  and the workflow contains no downstream native compilation path.
+
 - [x] Make native release evidence exact and publication immutable.
   Final result: every package row preserves a uniquely named BuildKit statement,
   scans only the exact package file into SPDX, verifies the package basename and

docker/Dockerfile.mesh-llm

@@ -64,16 +64,3 @@ COPY docker/entrypoint.sh /usr/local/bin/mesh-llm-entrypoint
 RUN chmod +x /usr/local/bin/mesh-llm-entrypoint /usr/local/bin/mesh-llm
 EXPOSE 3131 9337
 ENTRYPOINT ["/usr/local/bin/mesh-llm-entrypoint"]
-
-# Dry-run validation extends the exact final runtime stage but emits no image
-# tarball. This avoids duplicating multi-gigabyte GPU layers into the classic
-# Docker store merely to execute package and loader checks.
-FROM runtime AS runtime-qa
-ARG DISTRO=ubuntu
-ARG BACKEND=cpu
-ARG MESH_LLM_VERSION
-COPY docker/qa-runtime-image.sh /usr/local/bin/qa-runtime-image
-COPY scripts/client-readiness-smoke.sh /usr/local/bin/client-readiness-smoke
-RUN sh /usr/local/bin/qa-runtime-image "${DISTRO}" "${BACKEND}" "${MESH_LLM_VERSION}" && \
-    MESH_LLM_SMOKE_BIN=/usr/local/bin/mesh-llm-entrypoint sh /usr/local/bin/client-readiness-smoke && \
-    rm /usr/local/bin/qa-runtime-image /usr/local/bin/client-readiness-smoke

docs/matrix.md

@@ -31,9 +31,10 @@ Windows upstream archives are not repackaged because this repository has no Wind
 ## npm lanes
 
 The `npm.lanes` entries use the same `matrix_enabled` and `release_enabled`
-controls as package rows. They build `@mesh-llm/sdk` addons for macOS arm64/x64,
-Linux arm64/x64, and Windows x64. `npm-matrix` expands the enabled lanes;
-`npm_lane_filter` selects lane IDs or targets for isolated CI dry runs.
+controls as package rows. Upstream builds and smoke-tests checksummed addon
+archives for macOS arm64/x64, Linux arm64/x64, and Windows x64. Packaging safely
+extracts those immutable release assets and assembles `@mesh-llm/sdk` without
+Cargo or native source compilation. `npm_selector` accepts exact lane IDs.
 
 ## Archive deduplication
 

docs/native-packages.md

@@ -31,17 +31,19 @@ the distro package manager, proves ownership of the host plus the versioned
 runtime directory, and runs `mesh-llm --version` plus `mesh-llm runtime list`
 without a GPU device or driver. It then uses the shared readiness helper with
 unique API/console ports and cache/runtime roots to start
-`--log-format json --no-console client --auto`, require either the JSON
+`--log-format json --no-console client` without public discovery, require either the JSON
 `Client ready` message or the structured
 `passive_mode`/`status=ready`/`role=client` event while the process is alive,
 and require bounded SIGINT shutdown.
 
-The Dockerfile's `runtime-qa` stage extends the exact final runtime stage. It
-verifies package ownership, rejects backend imports or unresolved libraries
-from the host executable, and runs that same no-driver client readiness smoke
-without device access. Backend libraries may reference their driver interface
-only from inside the native runtime. Hardware-qualified serving is separate
-additive coverage.
+The per-row workflow builds the Dockerfile's final `runtime` target once, then
+runs external QA against that exact image. It verifies package ownership,
+rejects backend imports or unresolved libraries from the host executable, and
+runs the same no-driver client readiness smoke without device access.
+Publishing QA pulls the run-scoped staging image by digest; promotion retags
+that tested digest without rebuilding. Backend libraries may reference their
+driver interface only from inside the native runtime. Hardware-qualified
+serving is separate additive coverage.
 
 Packages install the host at `/usr/local/bin/mesh-llm` and the selected runtime
 at `/usr/local/lib/mesh-llm/<version>/native-runtimes/<runtime-id>`, alongside

docs/packaging-readiness-gaps.md

@@ -8,11 +8,12 @@ The repository implementation is archive-first: it verifies already-built upstre
 - [x] Verify digest, exact sidecar filename, safe tar entries, expected bundle layout, extraction, and provenance. QA: `tests/upstream-archive.test.ts`.
 - [x] Enforce package-first OCI construction with no direct runtime binary path. QA: Dockerfile target checks plus runtime workflow smoke.
 - [x] Make dry-run execute all validation while forcibly skipping publication. QA: workflow policy and final readiness job.
-- [x] Remove Blacksmith/self-hosted orchestration. QA: precheck scans workflow/Docker paths for legacy runner/source-build strings.
+- [x] Remove external/self-hosted runner orchestration. QA: precheck scans workflow/Docker paths for legacy runner/source-build strings.
 - [x] Correct channel claims: block Alpine/musl and Intel macOS; enable upstream Linux Vulkan and arm64 CUDA 13. QA: matrix tests.
 - [ ] Re-certify GPU package and image QA against product-v2 bundles. QA:
   backend-neutral hosts pass `--version`, `--help`, `runtime list`, and an
-  isolated JSON `client --auto` readiness/clean-SIGINT smoke without devices or
+  isolated JSON `client` readiness/clean-SIGINT smoke without public discovery,
+  devices, or
   driver stubs; packages own versioned runtime trees; all product rows per
   OS/architecture attest the same host SHA-256; and a new full dry run
   succeeds. The earlier static-host baseline is preserved in

docs/publishing.md

@@ -2,21 +2,23 @@
 
 The release workflow has four publication states:
 
-1. `dry_run=true`: all required archive, package, image, and Homebrew validation
-   runs; publication is forcibly disabled. An unfiltered dry run also assembles
-   and verifies the exact release asset set and aggregate package provenance.
-2. `dry_run=false,publish_images=true`: validated OCI images are pushed to GHCR and receive build-provenance attestations.
+1. `dry_run=true`: publication is forcibly disabled while the explicitly
+   enabled native, Homebrew, and npm validation components run.
+2. `dry_run=false,publish_images=true`: every final OCI image is pushed once to
+   a run-scoped staging tag, tested by immutable digest, attested, indexed, and
+   promoted without another build.
 3. `dry_run=false,publish_release_assets=true`: native packages, checksums, SPDX SBOMs, and the rendered Homebrew formula are attached to a `packaging-v<version>` release in this repository.
 4. `dry_run=false,publish_npm=true`: the install-tested `@mesh-llm/sdk`
    tarball is published with provenance; stable versions use `latest` and
    prereleases use `next`.
 
 Non-npm publish jobs use the `release` GitHub environment, whose deployment policy accepts only `main`, while npm publishing uses the separate `npm` environment described below. Publish jobs have job-local write permissions, and all build and validation jobs are read-only. Add required reviewers when the repository plan supports environment reviewers. The upstream tag must already have a non-draft GitHub Release, the repository must be exactly `Mesh-LLM/mesh-llm`, the ref and version must match, and the tag is resolved to an immutable commit SHA for provenance labels.
 
-Filtered runs are validation-only. If any native variant, platform, or npm lane
-filter is present while a publication switch is enabled, planning fails before
-build or publication. Native release publication therefore always represents
-the checked-in complete 11-row package matrix.
+Planning is typed and fail-closed. Native and npm selectors accept only `all` or
+exact checked-in artifact/lane IDs; aliases, empty tokens, duplicates, and
+unknown IDs fail before scheduling. Publication requires the complete matching
+producer set: images require all native rows, package release assets require all
+native rows plus Homebrew, and npm requires all addon lanes.
 
 Every enabled native package row preserves its BuildKit statement under an
 artifact-specific filename, scans the exact package file into SPDX, and verifies
@@ -59,9 +61,12 @@ to `@mesh-llm/sdk`. All other package names are rejected, and subsequent
 upstream releases declare the canonical name directly.
 
 Moving convenience OCI tags are published alongside immutable version tags.
-Rollback must never mutate a versioned host, runtime, product, package, or image
-silently: stop the affected row, preserve both input digests as evidence, and
-publish a new upstream version or explicit correction record.
+The canonical image release index binds the upstream and packaging SHAs,
+digest-qualified runtime bases, package/product hashes, exact QA-tested image
+digests, and destination tags. Promotion rejects a conflicting version tag and
+records each convenience tag's previous digest in a rollback ledger. Promotion
+contains no Docker build. Rollback must never mutate a versioned host, runtime,
+product, package, or image silently.
 
 This repository is the sole GHCR producer. A successful non-canary
 `Mesh-LLM/mesh-llm` release with the complete GPU bundle set dispatches

docs/release-checklist.md

@@ -4,6 +4,8 @@
 
 - [ ] The upstream tag and non-draft GitHub Release exist and the tag resolves to one immutable SHA.
 - [ ] Every generated archive and `.sha256` sidecar exists in that release.
+- [ ] All five Node addon archives and sidecars exist; safe extraction verifies
+  their exact target path, manifest identity, and inner addon digest.
 - [ ] Archive verification accepts only the product-v2 host, host-import report,
   product manifest, and exactly one runtime tree. It verifies every recorded
   digest and rejects unexpected or traversal-prone entries.
@@ -40,11 +42,15 @@
 
 - [ ] A reviewer confirms the selected publish switches and the `release` environment gate.
 - [ ] GHCR tags match `docs/tagging.md`; pushed digests receive provenance attestations.
+- [ ] The canonical image index binds every selected row to its digest-qualified
+  base, package/product hashes, and exact QA-tested staging digest.
+- [ ] Image promotion performs no Docker build, rejects version-tag drift, and
+  preserves previous moving-tag digests in the rollback ledger.
 - [ ] Package release assets contain exact packages, SHA256 manifests, SPDX SBOMs, and the rendered formula.
 - [ ] Every package SPDX document names the exact `.deb` or `.pkg.tar.zst`
   basename and its verified sidecar SHA-256. Every uniquely named per-row
   BuildKit statement names the same subject before aggregate assembly.
-- [ ] No variant, platform, or npm lane filter is present on a publish run.
+- [ ] Every publish channel uses its complete exact selector (`all`).
 - [ ] A new `packaging-v<version>` release does not already exist. If it does,
   the workflow may no-op only when tag target, title, body, state, exact asset
   names, and GitHub asset digests all match; otherwise publication must fail.