Skip to content

feat: performance benchmarking suite, CI comment report, and template tsconfig fix#36

Merged
Asjas merged 5 commits intomainfrom
copilot/add-performance-benchmarking-suite
Mar 23, 2026
Merged

feat: performance benchmarking suite, CI comment report, and template tsconfig fix#36
Asjas merged 5 commits intomainfrom
copilot/add-performance-benchmarking-suite

Conversation

Copy link
Contributor

Copilot AI commented Mar 23, 2026

Implements the performance benchmarking suite from the implementation plan, wires it into CI with a PR comment report mirroring the coverage-report pattern, and fixes a long-standing root tsconfig.json gap that caused ~200 false-positive type errors.

Benchmarks

Five Vitest bench suites under benchmarks/:

File What's measured
renderer/ring-buffer.bench.ts SharedRingBuffer write/read at 64 B / 1 KB / 16 KB + sequential throughput + introspection
renderer/shared-buffer.bench.ts SharedBuffer allocation + JSON payload round-trips at ~100 B / ~2 KB / ~10 KB
utils/safe-redirect.bench.ts safeRedirect valid paths, rejection fast-paths, percent-encoded bypass attempts
utils/promise.bench.ts promiseHash (2–10 entries) + timeout wrapper overhead
utils/ip-address.bench.ts getClientIPAddress across all header patterns including RFC 7239 Forwarded and multi-hop x-forwarded-for

New scripts in package.json:

pnpm bench       # interactive / watch mode
pnpm bench:ci    # single run, writes benchmarks/results.json

A dedicated vitest.bench.config.ts keeps bench files out of pnpm test.

CI workflow (.github/workflows/benchmarks.yml)

Two jobs, same structure as coverage-report:

  • benchmark — runs pnpm bench:ci on Node 24, uploads benchmark-results artifact (30-day retention). Fires on push to main and on PRs touching benchmarks/**, packages/**, vitest.bench.config.ts, or pnpm-lock.yaml.
  • benchmark-report — PR-only. Downloads current + base (main) artifacts via actions/download-artifact + dawidd6/action-download-artifact, then uses actions/github-script to post (or update) a single comment. Uses a hidden HTML marker to avoid comment spam on re-runs.

Comment format — one collapsible <details> block per file:

Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme vs main
SharedRingBuffer – small payload (64 B) — write 64 bytes 545.5 K 0.0018 0.0072 ±0.54% -13.0% ⚠️
SharedRingBuffer – medium payload (1 KB) — write 1 KB 295.1 K 0.0034 0.0128 ±0.53% +13.6% 🚀

Delta thresholds: ⚠️ = >10% slower, 🚀 = >10% faster. Delta column is omitted on the first run (no base artifact on main yet).

Docs

  • docs/benchmarks.md — reference page covering all suites, output columns, config, and how to add a new benchmark
  • VitePress sidebar updated under "Testing & Tooling"
  • README.md gets a Benchmarks section
  • IMPLEMENTATION_PLAN.md item marked done

tsconfig fix

packages/create-scratchy-app/src/template/ imports external packages (@builder.io/qwik, drizzle-orm, etc.) that aren't installed in this monorepo — they're dependencies of the generated app. The per-package tsconfig already excluded src/template; the root one didn't, producing ~200 spurious errors when running tsc --noEmit directly.

// tsconfig.json — added to exclude array
"packages/create-scratchy-app/src/template"

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@socket-security
Copy link

socket-security bot commented Mar 23, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedwrangler@​4.76.0981009296100

View full report

Copilot AI changed the title [WIP] Add performance benchmarking suite to documentation feat: performance benchmarking suite, CI comment report, and template tsconfig fix Mar 23, 2026
Copilot AI requested a review from Asjas March 23, 2026 14:31
@Asjas Asjas marked this pull request as ready for review March 23, 2026 14:32
Copilot AI review requested due to automatic review settings March 23, 2026 14:32
@github-actions
Copy link
Contributor

github-actions bot commented Mar 23, 2026

📚 Docs preview deployed

https://scratchyjs-docs-pr-36.asjas.workers.dev

Updates automatically on every push to this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 23, 2026

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 98.67%
🟰 ±0%
2007 / 2034
🔵 Statements 98.07%
🟰 ±0%
2094 / 2135
🔵 Functions 99.25%
🟰 ±0%
399 / 402
🔵 Branches 92.61%
🟰 ±0%
1104 / 1192
File CoverageNo changed files found.
Generated in workflow #299 for commit d36a04f by the Vitest Coverage Report Action

@github-actions
Copy link
Contributor

github-actions bot commented Mar 23, 2026

⚡ Benchmark Results

benchmarks/renderer/ring-buffer.bench.ts
Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme
SharedRingBuffer – small payload (64 B) — write 64 bytes 518.9 K 0.0019 0.0075 ±1.44%
SharedRingBuffer – small payload (64 B) — write + read 64 bytes 467.0 K 0.0021 0.0078 ±1.42%
SharedRingBuffer – medium payload (1 KB) — write 1 KB 244.4 K 0.0041 0.0142 ±0.58%
SharedRingBuffer – medium payload (1 KB) — write + read 1 KB 210.0 K 0.0048 0.0215 ±1.33%
SharedRingBuffer – large payload (16 KB) — write 16 KB 57.5 K 0.0174 0.0424 ±1.56%
SharedRingBuffer – large payload (16 KB) — write + read 16 KB 83.8 K 0.0119 0.0420 ±18.76%
SharedRingBuffer – sequential throughput (100 × 64 B) — 100 write + read cycles 40.7 K 0.0246 0.0441 ±0.39%
SharedRingBuffer – introspection — availableToRead 14.47 M 0.0001 0.0001 ±0.10%
SharedRingBuffer – introspection — isEmpty 14.88 M 0.0001 0.0001 ±0.10%
SharedRingBuffer – introspection — isFull 14.74 M 0.0001 0.0001 ±0.21%
benchmarks/renderer/shared-buffer.bench.ts
Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme
SharedBuffer – allocation — createSharedBuffer(4 KB) 595.6 K 0.0017 0.0070 ±1.15%
SharedBuffer – allocation — createSharedBuffer(64 KB) 43.3 K 0.0231 0.0426 ±0.75%
SharedBuffer – small payload round-trip — write small JSON 314.4 K 0.0032 0.0120 ±0.43%
SharedBuffer – small payload round-trip — write + read small JSON 246.4 K 0.0041 0.0135 ±0.49%
SharedBuffer – medium payload round-trip — write medium JSON (~2 KB) 99.7 K 0.0100 0.0261 ±0.40%
SharedBuffer – medium payload round-trip — write + read medium JSON (~2 KB) 57.1 K 0.0175 0.0390 ±0.94%
SharedBuffer – large payload round-trip — write large JSON (~10 KB) 15.5 K 0.0647 0.0886 ±0.31%
SharedBuffer – large payload round-trip — write + read large JSON (~10 KB) 7.7 K 0.1293 0.1658 ±0.35%
benchmarks/utils/ip-address.bench.ts
Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme
getClientIPAddress – no IP headers — no IP-related headers → null 2.55 M 0.0004 0.0006 ±0.18%
getClientIPAddress – single header — cf-connecting-ip (Cloudflare) 1.68 M 0.0006 0.0010 ±0.34%
getClientIPAddress – single header — x-forwarded-for (simple) 1.97 M 0.0005 0.0007 ±0.28%
getClientIPAddress – single header — x-real-ip 1.47 M 0.0007 0.0011 ±0.29%
getClientIPAddress – single header — true-client-ip (Akamai / Cloudflare Enterprise) 1.54 M 0.0006 0.0010 ±0.19%
getClientIPAddress – x-forwarded-for multi-hop — 2-hop chain 1.78 M 0.0006 0.0008 ±3.06%
getClientIPAddress – x-forwarded-for multi-hop — 4-hop chain 1.70 M 0.0006 0.0009 ±1.13%
getClientIPAddress – Forwarded header (RFC 7239) — simple for= directive 991.0 K 0.0010 0.0015 ±1.33%
getClientIPAddress – Forwarded header (RFC 7239) — for= with port 749.7 K 0.0013 0.0020 ±0.35%
getClientIPAddress – Forwarded header (RFC 7239) — IPv6 literal 761.4 K 0.0013 0.0017 ±0.27%
getClientIPAddress – Forwarded header (RFC 7239) — multi-hop Forwarded 834.8 K 0.0012 0.0017 ±1.65%
getClientIPAddress – IPv6 addresses — x-forwarded-for IPv6 1.14 M 0.0009 0.0010 ±0.28%
benchmarks/utils/promise.bench.ts
Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme
promiseHash – concurrent resolution — 2 already-resolved promises 869.3 K 0.0012 0.0031 ±1.03%
promiseHash – concurrent resolution — 5 already-resolved promises 524.5 K 0.0019 0.0034 ±1.72%
promiseHash – concurrent resolution — 10 already-resolved promises 311.2 K 0.0032 0.0055 ±1.57%
promiseHash – concurrent resolution — 5 promises with object values 485.4 K 0.0021 0.0026 ±0.27%
timeout – wrapping fast promises — timeout wrapping an already-resolved promise (1 s budget) 965.2 K 0.0010 0.0015 ±0.34%
timeout – wrapping fast promises — timeout wrapping an already-resolved object (5 s budget) 897.6 K 0.0011 0.0017 ±0.31%
benchmarks/utils/safe-redirect.bench.ts
Suite — Benchmark ops/sec mean (µs) p99 (µs) ±rme
safeRedirect – valid paths — root path / 6.89 M 0.0001 0.0002 ±0.12%
safeRedirect – valid paths — simple path /dashboard 3.47 M 0.0003 0.0003 ±0.11%
safeRedirect – valid paths — nested path /settings/profile 3.11 M 0.0003 0.0004 ±0.10%
safeRedirect – valid paths — path with query string /search?q=hello 3.38 M 0.0003 0.0005 ±0.10%
safeRedirect – valid paths — path with hash /docs#section 3.46 M 0.0003 0.0003 ±0.10%
safeRedirect – rejected inputs — absolute URL https://evil.com 4.67 M 0.0002 0.0004 ±0.10%
safeRedirect – rejected inputs — protocol-relative URL //evil.com 4.62 M 0.0002 0.0002 ±0.09%
safeRedirect – rejected inputs — backslash-relative /\evil.com 4.63 M 0.0002 0.0002 ±0.09%
safeRedirect – rejected inputs — path traversal /../etc/passwd 4.07 M 0.0002 0.0003 ±0.11%
safeRedirect – rejected inputs — null input 16.04 M 0.0001 0.0001 ±0.09%
safeRedirect – rejected inputs — undefined input 15.95 M 0.0001 0.0001 ±0.11%
safeRedirect – rejected inputs — empty string 15.82 M 0.0001 0.0001 ±0.17%
safeRedirect – percent-encoded bypass — percent-encoded // (%2F%2F) 4.40 M 0.0002 0.0004 ±0.24%
safeRedirect – percent-encoded bypass — percent-encoded path traversal (%2e%2e) 4.79 M 0.0002 0.0002 ±0.10%
safeRedirect – percent-encoded bypass — mixed percent-encoded absolute URL 4.45 M 0.0002 0.0003 ±0.13%
safeRedirect – custom default redirect — valid path with custom default 3.56 M 0.0003 0.0003 ±0.25%
safeRedirect – custom default redirect — invalid input with custom default 4.79 M 0.0002 0.0002 ±0.10%

ℹ️ No base results from main found — delta column omitted on first run.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Vitest-powered benchmarking suite for key framework hot paths, wires it into CI with an automated PR comment report, and fixes root tsconfig.json exclusions to avoid template-related typecheck noise.

Changes:

  • Added benchmarks/ suites plus a dedicated vitest.bench.config.ts and pnpm bench / pnpm bench:ci scripts.
  • Introduced a new GitHub Actions workflow to run benchmarks and post/update a PR comment with results (and deltas vs main when available).
  • Updated docs/README/sidebar and excluded the create-scratchy-app template directory from the root tsconfig.

Reviewed changes

Copilot reviewed 12 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
vitest.bench.config.ts Dedicated Vitest config to run .bench.ts files and emit JSON results.
package.json Adds bench and bench:ci scripts for local and CI runs.
benchmarks/renderer/ring-buffer.bench.ts Bench suite for SharedRingBuffer write/read throughput and getters.
benchmarks/renderer/shared-buffer.bench.ts Bench suite for SharedBuffer allocation and JSON round-trips.
benchmarks/utils/safe-redirect.bench.ts Bench suite for safeRedirect happy paths and rejection fast-paths.
benchmarks/utils/promise.bench.ts Bench suite for promiseHash and timeout overhead.
benchmarks/utils/ip-address.bench.ts Bench suite for getClientIPAddress across common header patterns.
.github/workflows/benchmarks.yml New CI workflow to run benchmarks, upload artifacts, and post PR comment report.
docs/benchmarks.md Reference documentation for running/adding benchmarks and interpreting output.
docs/.vitepress/config.ts Adds Benchmarks page to the VitePress sidebar.
README.md Adds a Benchmarks section and links to docs.
.gitignore Ignores generated benchmarks/results.json.
tsconfig.json Excludes create-scratchy-app template directory to avoid false-positive type errors.
IMPLEMENTATION_PLAN.md Marks the benchmarking suite item complete.

Asjas and others added 2 commits March 23, 2026 16:36
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: A-J Roos <asjasroos@pm.me>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: A-J Roos <asjasroos@pm.me>
@Asjas Asjas merged commit 50a9265 into main Mar 23, 2026
16 checks passed
@Asjas Asjas deleted the copilot/add-performance-benchmarking-suite branch March 23, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants