Skip to content

feat: prefer specific OS, gate .git on unpacker, probe zst/7z, deploy script - #1074

Merged
coolaj86 merged 4 commits into
mainfrom
fix-builds-cacher-git-gate
May 7, 2026
Merged

feat: prefer specific OS, gate .git on unpacker, probe zst/7z, deploy script#1074
coolaj86 merged 4 commits into
mainfrom
fix-builds-cacher-git-gate

Conversation

@coolaj86

@coolaj86 coolaj86 commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

Five related changes that together make the resolver pick the right binary instead of accidentally installing source archives, git clones, or just dropping versions on the floor:

  1. _webi/builds-cacher.js_enumerateTriplets: prefer specific OS/arch over ANYOS/ANYARCH.
  2. _webi/builds-cacher.jsgetSortedFormats: gate .git on the caller's formats list, mirroring .tar.xz, .tar.zst, .zip, .7z.
  3. webi/webi.sh: probe unzstd/zstd alongside the existing git/unxz/unzip/tar probes.
  4. webi/webi-pwsh.ps1: replace the hardcoded formats=zip,exe,tar,git TODO with Get-Command probes for git, zstd, 7z.
  5. _scripts/deploy-webid: rsync deploy with submodule precondition, _cache symlink restore, anchored worker pkill, smoke tests.
  6. _webi/build-classifier: bump submodule to pull in webi-build-classifier#21 — fixes maybeInstallable rejecting any package version ending in .1.

⚠️ Dependency

Commit 6 bumps the submodule to a SHA on webinstall/webi-build-classifier#21's feature branch. That PR should land first so the bumped SHA is reachable from webi-build-classifier's main. Without #21, serviceman v1.0.1 (and any other patch.1 version of a source-archive-only package) is still silently dropped by the classifier and the user gets v1.0.0 instead.

Motivation

serviceman (and any package with a */*/git fallback alongside per-OS binaries) was resolving to git instead of the binary, even on hosts with git absent:

$ curl -A 'aarch64/unknown Darwin/24.2.0 libc' \
       https://next.webi.sh/api/installers/serviceman@stable.sh
WEBI_EXT='git'

Two distinct bugs:

A. Triplet ordering_enumerateTriplets produced ANYOS-* triplets before specific OS triplets:

1: ANYOS-ANYARCH-none      ← serviceman's */*/git matches here
...
7: posix_2017-ANYARCH-none ← serviceman's tar.gz/zip is here

findMatchingPackages returned the git-only triplet first. The .git-as-unpacker gate (commit 2) is correct but downstream — by the time selectPackage is called the wrong triplet has already been picked.

B. Classifier silent dropTriplet.maybeInstallable checked build.download.endsWith(ext) against TERMS_EXTS_NON_BUILD which contains .1 (intended for manpage.1). GitHub source-archive URLs of the form .../tarball/v1.0.1 end in .1, so the classifier silently rejected the entire v1.0.1 release. Fixed in webi-build-classifier#21 by checking build.name instead.

Behavior

After this PR + the build-classifier PR:

  • serviceman macos/linux any host (default formats): v1.0.1/zip (was git → would fail without git).
  • serviceman Windows: still no Windows binary in the cache (Go-cache content gap, not addressed by this PR — needs therootcompany/serviceman fetched alongside bnnanet/serviceman).
  • vim-commentary (only-git package): unchanged. selectPackage's packages.length === 1 early return covers single-option triplets.
  • webi <pkg> on a host with zstd: now sends ?formats=...,zst.
  • webi-pwsh.ps1 on Windows: probes git/zstd/7z via Get-Command.

Test plan

  • npm test passes.
  • PR branch deployed to both beta.webi.sh and next.webi.sh. Verified /api/installers/serviceman@stable.sh:
    • macos arm64 → v1.0.1/zip (was v1.0.1/git pre-PR; would be v1.0.0/zip without commit 6).
    • linux amd64/arm64/musl → same (v1.0.1/zip).
    • windows amd64 → v0.9.8/git (still broken — cache gap, separately tracked).
  • vim-commentary still resolves to git correctly.
  • Reviewer: check no regression on a binary-package (e.g. node, jq, bat, fd) at the resolved version/ext level.

Notes

  • Companion to fix(builds-cacher): coalesce concurrent getPackages for same name #1073 (race-condition fix in the same file).
  • _scripts/deploy-webid includes a submodule precondition that catches "I forgot to git submodule update --init" before the rsync silently pushes an empty _webi/build-classifier/. That scenario broke deploys repeatedly during this PR's testing.

Commits

c69359f chore(build-classifier): bump to fix-maybe-installable-version-suffix
d9c315d fix(builds-cacher): enumerate specific OS/arch before ANYOS/ANYARCH
19673d8 feat(scripts): add deploy-webid with submodule precondition
2d1c082 feat(webi): probe zst as unpacker; properly probe formats in webi-pwsh
28cd129 ref(builds-cacher): gate .git on client-provided unpacker

Depends on: webinstall/webi-build-classifier#21

coolaj86 added 2 commits May 6, 2026 23:05
`.git` was pushed unconditionally into getSortedFormats's candidate
ext list, while sibling unpacker formats (.tar.xz, .tar.zst, .zip,
.7z) are gated on whether the caller's `formats` argument signals
the client has the corresponding tool.

Make `.git` consistent: only add it to the candidate list when
formats includes 'git'. The default WEBI_FORMATS ('tar,exe,zip,xz,
dmg') doesn't include git, so the change is a no-op for the
current default. Clients that want git-source packages installed
can pass `?formats=tar,exe,zip,xz,dmg,git` (or set the equivalent
in a future client-side probe).

For packages that have only a git-source asset (e.g. some vim
plugins), the existing fallback to `packages[0]` still returns the
git entry — behavior unchanged. The only observable change is for
packages where both a binary and a git fallback exist for the same
triplet: previously the git entry could win over the binary; now
it wins only when the client opts in.
webi/webi.sh: detect unzstd/zstd alongside the existing git/unxz/
unzip/tar probes. Sends `?formats=...,zst` when zstd is available so
the server can pick a .tar.zst build only on hosts that can extract
it.

webi/webi-pwsh.ps1: replace the hardcoded `formats=zip,exe,tar,git`
TODO with real Get-Command probing for git, zstd, and 7z.
@coolaj86 coolaj86 changed the title ref(builds-cacher): gate .git on client-provided unpacker feat(webi): probe git/zst/7z client-side; gate .git in builds-cacher May 7, 2026
@coolaj86 coolaj86 changed the title feat(webi): probe git/zst/7z client-side; gate .git in builds-cacher feat: probe git/zst/7z client-side; gate .git in cacher; add deploy-webid May 7, 2026
@coolaj86 coolaj86 changed the title feat: probe git/zst/7z client-side; gate .git in cacher; add deploy-webid feat: prefer specific OS, gate .git on unpacker, probe zst/7z, deploy script May 7, 2026
@coolaj86
coolaj86 force-pushed the fix-builds-cacher-git-gate branch 2 times, most recently from 4b2deb0 to 0ec38d4 Compare May 7, 2026 06:17
coolaj86 added 2 commits May 7, 2026 00:22
In _enumerateTriplets, the order of `oses` and `arches` was
ANYOS/ANYARCH first, specific second. This caused findMatchingPackages
to pick the most-generic triplet (e.g. ANYOS-ANYARCH-none) before
trying specific OS triplets — and packages that have a wildcard git
fallback alongside per-platform binaries would resolve to the git
source instead of the binary, even when the client never asked for
git as an unpacker.

Reverse the order so specific platforms win:
  - oses: hostTarget.os, posix_2017, posix_2024, ANYOS
  - arches: arches.concat(['ANYARCH'])

Concrete example: serviceman has both posix_2017/*/tar.gz and
*/*/git in the cache. Pre-fix, findMatchingPackages picks
ANYOS-ANYARCH-none (containing only the .git entry). The .git gate
in getSortedFormats then correctly excludes git from format
candidates, but the chosen triplet has nothing else, so selectPackage
falls through to packages[0] = git entry. Post-fix,
findMatchingPackages picks posix_2017-ANYARCH-none first (containing
[tar.gz, zip]), and selectPackage returns tar.gz.
Pulls in webinstall/webi-build-classifier#21 (merged 2026-05-07,
SHA 574eff5) and the host-target x64/win32 fix from #20 (SHA 71c0768)
that landed alongside it.

#21 fixes `maybeInstallable` rejecting any package version ending in
`.1` whose download URL is a GitHub source-archive endpoint
(`/tarball/vX.Y.1` or `/zipball/vX.Y.1`). Without it, this PR's
`_enumerateTriplets` priority fix is undermined: even after picking
the correct posix_2017 triplet, the newest version (e.g. serviceman
v1.0.1) is silently dropped by the classifier and the resolver falls
back to v1.0.0.

Confirmed on next.webi.sh after deploying this branch with the bumped
submodule: `serviceman@stable.sh` now resolves to v1.0.1/zip on macOS
arm64 (was v1.0.0/zip with the pre-rebase pre-fix submodule).
@coolaj86
coolaj86 force-pushed the fix-builds-cacher-git-gate branch from 0ec38d4 to da10371 Compare May 7, 2026 06:22
@coolaj86
coolaj86 merged commit da10371 into main May 7, 2026
3 checks passed
@coolaj86
coolaj86 deleted the fix-builds-cacher-git-gate branch May 7, 2026 06:56
coolaj86 added a commit that referenced this pull request May 8, 2026
Trim the explanatory paragraphs in _classify and findMatchingPackages
to one or two lines stating the WHY (the gotcha or the invariant);
drop time-bound references (#1074), illustrative enumerations, and
restated background that the surrounding code already conveys.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant