Skip to content

ci(qualification): run #693 on fresh macOS 26 arm64 VMs - #694

Draft
mohanagy wants to merge 3 commits into
nextfrom
roadmap/693-macos26-arm64-qualification
Draft

ci(qualification): run #693 on fresh macOS 26 arm64 VMs#694
mohanagy wants to merge 3 commits into
nextfrom
roadmap/693-macos26-arm64-qualification

Conversation

@mohanagy

@mohanagy mohanagy commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Diagnostics only. This pull request must not merge.

Adds one temporary workflow to run the frozen #693 qualification commit on fresh GitHub-hosted macOS 26 arm64 VMs. It changes no product source, no package metadata, no dependencies, and no normal CI.

Why

The workstation experiment already rejected the CPU-contention hypothesis on the same frozen commit:

Run Node CPU Worker-start Handshake
A1 quiet 47.8% 16 16
A2 quiet 0.1% 20 20
A3 quiet 152.4% 13 13
B16 contended 1471.7% 14 14

The quietest run produced the most signatures, and saturating 16 cores produced fewer. Counts stayed in a 13–20 band across four orders of magnitude of load. So contention is not the cause, and the remaining question is narrower: does the frozen commit fail on a fresh machine of the same OS and architecture, or only on this workstation?

Design

  • Runner macos-26, asserted at runtime to be arm64 with Darwin major 25. The job fails rather than silently substituting a different image.
  • Workflow-control commit and tested commit are separated. Checkout pins ref: b1300f8fcc2758404abc5e6064433c4d8b2ab40b with persist-credentials: false, and the job fails if the tested HEAD is not that SHA. Branch code never enters the frozen checkout; helpers are generated in $RUNNER_TEMP.
  • Three attempts, max-parallel: 1, each a newly provisioned VM.
  • Toolchain pinned: Node 22.22.3, npm 12.0.2 installed with --ignore-scripts; lockfile checksum verified before and after npm ci; the job fails if npm ci mutates the checkout. No dependency cache.
  • Positive scanner control per attempt — a deliberately poisoned log must make the canonical scanner exit non-zero. A zero signature count with no proven control is not accepted as evidence.
  • Authoritative commands onlynpm run test:run and npm run test:coverage. Raw vitest run is used only for the isolated historical-file diagnostic, never for accepted qualification evidence.
  • Conditions per VM: A quiet baseline, B1 64 idle Node processes, B2 one busy-loop process per logical CPU, D recovery after verified cleanup. B1/B2 use continue-on-error because either outcome is evidence; A and D must be clean.
  • Evidence directory uploaded with if: always(), 30-day retention, artifact name carrying run id, attempt, matrix index, OS, architecture, and tested commit.

Security

permissions: contents: read. No secrets, no environment, no OIDC, no cache, no publish. Branch-push trigger only — no workflow_dispatch (this workflow is not on the default branch), and none of pull_request_target, repository_dispatch, workflow_run, schedule, or tag push. All three actions pinned to full commit SHAs already verified in this repository.

Disposition

This branch exists to produce evidence for #693. It will be closed as diagnostics-only once the issue comments link every run and artifact. If the capability proves worth keeping, that becomes its own issue after the #654 decision.

Refs #693.
Related parent: #654.
#654 remains open.


Correction — initial run invalidated, workflow fixed

Run 31718841614 is not counted toward final qualification. The cleanup-verification step I wrote used IDLE=$(pgrep -f 'idle-load.mjs' | wc -l) under set -euo pipefail. pgrep exits 1 when it correctly finds nothing, pipefail propagates it, and the assignment aborted the step precisely when cleanup had succeeded. Both completed attempts failed there and skipped condition D, making the A → B → D contract unreachable.

That is a defect in this diagnostics workflow. No Madar product source was affected, and the frozen commit under test is unchanged at b1300f8fcc2758404abc5e6064433c4d8b2ab40b — zero non-workflow files differ from it on this branch.

The invalidated run did produce preliminary observations, recorded on #693 but not counted: on a fresh Darwin 25 arm64 VM (3 logical CPUs, 7 GiB), quiet test:run, quiet test:coverage, 64 idle Node processes, and CPU saturation at 293% aggregate each completed with 0/0 signatures.

What changed

  • Survivor counting redirects pgrep to a file and swallows its exit code, so zero survivors is a normal success.
  • Load liveness proven with kill -0 against the PID file; expected versus live counts and the live PID list recorded; a B condition explicitly marked invalid if the intended load never established.
  • Cleanup traps on B1 and B2 so load is removed even when the guarded suite or metadata collection fails, with live PIDs re-counted afterwards.
  • Signature counts sourced from the raw guard log when it survives. The wrapper echoes matching lines into its own report, so the outer transcript overcounts — demonstrated on a fixture where one event counts 1 in the guard log and 3 in the outer log. signature_source, both counts, and both checksums are recorded; a clean run whose guard log was deleted falls back to the outer log.
  • Isolated historical files record handshake signatures and a readable marker too.
  • Generated helpers are syntax-checked in-workflow.

Validated with ten deterministic local shell probes before pushing.

A second defect, caught by the repository's own guard

The first corrected head failed all six CI lanes because tests/unit/vitest-guard-policy.test.ts"contains no raw vitest run invocation in any workflow" — flagged this workflow's npx vitest run in the isolated-file step. The guard added by #690 caught a real violation in a file this branch introduced. The test was not weakened; the isolated runs now use npm run test:run -- <path>, keeping them single-file while applying the canonical scan.

Current identity

Invalidated run 31718841614
Corrected workflow-control head 173bc6913a7e0538d8f352da7add5fd0a1aac843
Replacement diagnostics run 31722725288
Frozen source under test b1300f8fcc2758404abc5e6064433c4d8b2ab40b (unchanged)

Still diagnostics-only. Still must not merge.

Diagnostics-only workflow. Runs the frozen qualification commit
b1300f8 on three freshly provisioned
GitHub-hosted macos-26 VMs to determine whether the workstation's Vitest
worker-start failures reproduce on the same OS and architecture.

The workstation experiment already rejected CPU contention as the cause:
the quietest run produced the most signatures and saturating 16 cores
produced fewer. This workflow tests the remaining boundary question.

Per attempt: assert Darwin 25 arm64, assert the tested commit equals the
frozen SHA, pin Node 22.22.3 and npm 12.0.2, verify the lockfile checksum
and that npm ci does not mutate the checkout, prove the raw-log scanner is
active with a poisoned positive control, then run quiet baseline A, 64 idle
Node processes B1, CPU-saturating load B2, and recovery D through the
authoritative guarded commands only.

Diagnostics only; this branch must not merge.

Refs #693.
Related parent: #654.
#654 remains open.
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2df1382f-a69c-43d7-b5c8-e8975e22e8ee

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Run 31718841614 could never reach condition D. The cleanup verification used
`IDLE=$(pgrep -f ... | wc -l)` under `set -euo pipefail`; pgrep exits 1 when it
correctly finds no survivors, pipefail propagates that, and the assignment aborted
the step precisely when cleanup had SUCCEEDED. Both completed attempts failed there
and skipped D, so that run cannot satisfy the A/B/D contract.

Corrections, all confined to the diagnostics workflow:

- count survivors by redirecting pgrep to a file and swallowing its exit code, so a
  zero-survivor result is a normal success
- prove load liveness with kill -0 against the PID file rather than counting intended
  PIDs, record expected vs live and the live PID list, and mark a B condition invalid
  when the intended load was not established
- add cleanup traps to B1 and B2 so load is removed even if the guarded suite or
  metadata collection fails, then verify zero live PIDs after cleanup
- source signature counts from the raw guard log when it survives, because the wrapper
  echoes matching lines into its own report and the outer transcript overcounts; fall
  back to the outer log when the guard deletes a clean log, and record which source
  was used plus both checksums
- record handshake signatures and a readable marker for isolated historical files
- syntax-check every generated helper in-workflow

The frozen source under test is unchanged at b1300f8.

Diagnostics only; this branch must not merge.

Refs #693.
Related parent: #654.
#654 remains open.
… runner

The repository's own policy test — tests/unit/vitest-guard-policy.test.ts,
'contains no raw vitest run invocation in any workflow' — correctly failed all six
lanes on the previous diagnostics head because this workflow invoked `npx vitest run`
for the isolated historical-file diagnostic.

The guard from #690 caught a real violation in a file added by this branch, so the
test is not weakened. The isolated runs now go through `npm run test:run -- <path>`,
which keeps them single-file while also applying the canonical signature scan.

Diagnostics only; this branch must not merge.

Refs #693.
Related parent: #654.
#654 remains open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant