ci(qualification): run #693 on fresh macOS 26 arm64 VMs - #694
Draft
mohanagy wants to merge 3 commits into
Draft
Conversation
Diagnostics-only workflow. Runs the frozen qualification commit b1300f8 on three freshly provisioned GitHub-hosted macos-26 VMs to determine whether the workstation's Vitest worker-start failures reproduce on the same OS and architecture. The workstation experiment already rejected CPU contention as the cause: the quietest run produced the most signatures and saturating 16 cores produced fewer. This workflow tests the remaining boundary question. Per attempt: assert Darwin 25 arm64, assert the tested commit equals the frozen SHA, pin Node 22.22.3 and npm 12.0.2, verify the lockfile checksum and that npm ci does not mutate the checkout, prove the raw-log scanner is active with a poisoned positive control, then run quiet baseline A, 64 idle Node processes B1, CPU-saturating load B2, and recovery D through the authoritative guarded commands only. Diagnostics only; this branch must not merge. Refs #693. Related parent: #654. #654 remains open.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Run 31718841614 could never reach condition D. The cleanup verification used `IDLE=$(pgrep -f ... | wc -l)` under `set -euo pipefail`; pgrep exits 1 when it correctly finds no survivors, pipefail propagates that, and the assignment aborted the step precisely when cleanup had SUCCEEDED. Both completed attempts failed there and skipped D, so that run cannot satisfy the A/B/D contract. Corrections, all confined to the diagnostics workflow: - count survivors by redirecting pgrep to a file and swallowing its exit code, so a zero-survivor result is a normal success - prove load liveness with kill -0 against the PID file rather than counting intended PIDs, record expected vs live and the live PID list, and mark a B condition invalid when the intended load was not established - add cleanup traps to B1 and B2 so load is removed even if the guarded suite or metadata collection fails, then verify zero live PIDs after cleanup - source signature counts from the raw guard log when it survives, because the wrapper echoes matching lines into its own report and the outer transcript overcounts; fall back to the outer log when the guard deletes a clean log, and record which source was used plus both checksums - record handshake signatures and a readable marker for isolated historical files - syntax-check every generated helper in-workflow The frozen source under test is unchanged at b1300f8. Diagnostics only; this branch must not merge. Refs #693. Related parent: #654. #654 remains open.
… runner The repository's own policy test — tests/unit/vitest-guard-policy.test.ts, 'contains no raw vitest run invocation in any workflow' — correctly failed all six lanes on the previous diagnostics head because this workflow invoked `npx vitest run` for the isolated historical-file diagnostic. The guard from #690 caught a real violation in a file added by this branch, so the test is not weakened. The isolated runs now go through `npm run test:run -- <path>`, which keeps them single-file while also applying the canonical signature scan. Diagnostics only; this branch must not merge. Refs #693. Related parent: #654. #654 remains open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnostics only. This pull request must not merge.
Adds one temporary workflow to run the frozen #693 qualification commit on fresh GitHub-hosted macOS 26 arm64 VMs. It changes no product source, no package metadata, no dependencies, and no normal CI.
Why
The workstation experiment already rejected the CPU-contention hypothesis on the same frozen commit:
The quietest run produced the most signatures, and saturating 16 cores produced fewer. Counts stayed in a 13–20 band across four orders of magnitude of load. So contention is not the cause, and the remaining question is narrower: does the frozen commit fail on a fresh machine of the same OS and architecture, or only on this workstation?
Design
macos-26, asserted at runtime to bearm64with Darwin major25. The job fails rather than silently substituting a different image.ref: b1300f8fcc2758404abc5e6064433c4d8b2ab40bwithpersist-credentials: false, and the job fails if the testedHEADis not that SHA. Branch code never enters the frozen checkout; helpers are generated in$RUNNER_TEMP.max-parallel: 1, each a newly provisioned VM.--ignore-scripts; lockfile checksum verified before and afternpm ci; the job fails ifnpm cimutates the checkout. No dependency cache.npm run test:runandnpm run test:coverage. Rawvitest runis used only for the isolated historical-file diagnostic, never for accepted qualification evidence.continue-on-errorbecause either outcome is evidence; A and D must be clean.if: always(), 30-day retention, artifact name carrying run id, attempt, matrix index, OS, architecture, and tested commit.Security
permissions: contents: read. No secrets, no environment, no OIDC, no cache, no publish. Branch-push trigger only — noworkflow_dispatch(this workflow is not on the default branch), and none ofpull_request_target,repository_dispatch,workflow_run,schedule, or tag push. All three actions pinned to full commit SHAs already verified in this repository.Disposition
This branch exists to produce evidence for #693. It will be closed as diagnostics-only once the issue comments link every run and artifact. If the capability proves worth keeping, that becomes its own issue after the #654 decision.
Refs #693.
Related parent: #654.
#654 remains open.
Correction — initial run invalidated, workflow fixed
Run
31718841614is not counted toward final qualification. The cleanup-verification step I wrote usedIDLE=$(pgrep -f 'idle-load.mjs' | wc -l)underset -euo pipefail.pgrepexits1when it correctly finds nothing,pipefailpropagates it, and the assignment aborted the step precisely when cleanup had succeeded. Both completed attempts failed there and skipped condition D, making the A → B → D contract unreachable.That is a defect in this diagnostics workflow. No Madar product source was affected, and the frozen commit under test is unchanged at
b1300f8fcc2758404abc5e6064433c4d8b2ab40b— zero non-workflow files differ from it on this branch.The invalidated run did produce preliminary observations, recorded on #693 but not counted: on a fresh Darwin 25 arm64 VM (3 logical CPUs, 7 GiB), quiet
test:run, quiettest:coverage, 64 idle Node processes, and CPU saturation at 293% aggregate each completed with 0/0 signatures.What changed
pgrepto a file and swallows its exit code, so zero survivors is a normal success.kill -0against the PID file; expected versus live counts and the live PID list recorded; a B condition explicitly marked invalid if the intended load never established.signature_source, both counts, and both checksums are recorded; a clean run whose guard log was deleted falls back to the outer log.Validated with ten deterministic local shell probes before pushing.
A second defect, caught by the repository's own guard
The first corrected head failed all six CI lanes because
tests/unit/vitest-guard-policy.test.ts— "contains no raw vitest run invocation in any workflow" — flagged this workflow'snpx vitest runin the isolated-file step. The guard added by #690 caught a real violation in a file this branch introduced. The test was not weakened; the isolated runs now usenpm run test:run -- <path>, keeping them single-file while applying the canonical scan.Current identity
31718841614173bc6913a7e0538d8f352da7add5fd0a1aac84331722725288b1300f8fcc2758404abc5e6064433c4d8b2ab40b(unchanged)Still diagnostics-only. Still must not merge.