Skip to content

fix: disable install scripts in benchmarks for consistent measurement #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
darcyclarke merged 1 commit into from
Mar 18, 2026
Merged

fix: disable install scripts in benchmarks for consistent measurement #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scripts/registry/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ fi
# Base npm install command (without .npmrc setup)
# --prefer-online ensures npm always fetches from the network rather than
# relying on any cached metadata, so we measure actual registry performance.
BENCH_NPM_INSTALL="npm install --prefer-online --no-audit --no-fund --no-update-notifier --loglevel=$BENCH_LOGLEVEL $FORCE_FLAG"
BENCH_NPM_INSTALL="npm install --prefer-online --no-audit --no-fund --no-update-notifier --ignore-scripts --loglevel=$BENCH_LOGLEVEL $FORCE_FLAG"

# Registry definitions
BENCH_REGISTRY_NPM_URL="https://registry.npmjs.org/"
Expand Down
15 changes: 10 additions & 5 deletions scripts/variations/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,15 +79,20 @@ BENCH_SETUP_TURBO=""
BENCH_SETUP_VP=""
BENCH_SETUP_NODE=""

# Bare install commands (no log redirection) — used by strace process counting
BENCH_INSTALL_NPM="npm install --no-audit --no-fund --silent"
BENCH_INSTALL_YARN="corepack yarn@1 install --silent"
# Bare install commands (no log redirection) — used by strace process counting.
# Install scripts are disabled where the PM runs them by default, so benchmarks
# measure dependency resolution + linking only (not arbitrary postinstall work).
# npm, yarn classic, bun: run scripts by default → --ignore-scripts
# berry, zpm, pnpm v10+, vlt: don't run scripts by default → no flag needed
# deno: doesn't run scripts by default → removed --allow-scripts
BENCH_INSTALL_NPM="npm install --no-audit --no-fund --ignore-scripts --silent"
BENCH_INSTALL_YARN="corepack yarn@1 install --ignore-scripts --silent"
BENCH_INSTALL_BERRY="corepack yarn@latest install"
BENCH_INSTALL_ZPM="yarn install --silent"
BENCH_INSTALL_PNPM="corepack pnpm@latest install --silent"
BENCH_INSTALL_VLT="vlt install --view=silent"
BENCH_INSTALL_BUN="bun install --silent"
BENCH_INSTALL_DENO="deno install --allow-scripts --quiet"
BENCH_INSTALL_BUN="bun install --ignore-scripts --silent"
BENCH_INSTALL_DENO="deno install --quiet"

BENCH_COMMAND_NPM="timeout $BENCH_TIMEOUT $BENCH_INSTALL_NPM >> $BENCH_OUTPUT_FOLDER/npm-output-\${HYPERFINE_ITERATION}.log 2>&1"
BENCH_COMMAND_YARN="timeout $BENCH_TIMEOUT $BENCH_INSTALL_YARN > $BENCH_OUTPUT_FOLDER/yarn-output-\${HYPERFINE_ITERATION}.log 2>&1"
Expand Down