Skip to content

feat: test failure scores 0 + set vercel pkg to 1.7.15#67

Merged
kisernl merged 1 commit intomasterfrom
noah/provider-failure-0-fix
Apr 3, 2026
Merged

feat: test failure scores 0 + set vercel pkg to 1.7.15#67
kisernl merged 1 commit intomasterfrom
noah/provider-failure-0-fix

Conversation

@kisernl
Copy link
Copy Markdown
Contributor

@kisernl kisernl commented Apr 3, 2026

No description provided.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Storage Benchmark Results

10MB Files

# Provider Score Download Throughput Upload Status
1 Tigris 93.7 0.12s 704.9 Mbps 1.02s 10/10
2 AWS S3 93.2 0.12s 686.1 Mbps 0.39s 10/10
3 Cloudflare R2 82.0 0.49s 171.6 Mbps 0.98s 10/10

View full run · SVGs available as build artifacts

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 daytona 98.0 0.14s 0.29s 0.29s 10/10
2 e2b 94.8 0.44s 0.64s 0.64s 10/10
3 blaxel 93.3 0.49s 0.94s 0.94s 10/10
4 vercel 82.0 1.76s 1.87s 1.87s 10/10
5 runloop 80.8 1.87s 2.00s 2.00s 10/10
6 codesandbox 74.3 2.36s 2.89s 2.89s 10/10
7 cloudflare 73.7 2.27s 3.18s 3.18s 10/10
8 modal 68.5 2.10s 2.82s 2.82s 9/10
9 namespace 65.4 2.07s 5.54s 5.54s 10/10
10 hopx 53.2 1.13s 15.75s 15.75s 10/10
11 just-bash 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 daytona 98.6 0.12s 0.18s 0.18s 10/10
2 blaxel 94.3 0.49s 0.69s 0.69s 10/10
3 e2b 93.5 0.55s 0.81s 0.81s 10/10
4 hopx 88.3 1.01s 1.40s 1.40s 10/10
5 modal 78.2 1.92s 2.58s 2.58s 10/10
6 runloop 76.7 1.96s 2.89s 2.89s 10/10
7 vercel 76.5 1.74s 3.25s 3.25s 10/10
8 namespace 75.5 2.24s 2.75s 2.75s 10/10
9 codesandbox 72.5 2.62s 2.95s 2.95s 10/10
10 cloudflare 72.0 2.38s 3.44s 3.44s 10/10
11 just-bash 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 daytona 97.5 0.18s 0.37s 0.37s 10/10
2 blaxel 93.2 0.63s 0.77s 0.77s 10/10
3 e2b 93.2 0.51s 0.95s 0.95s 10/10
4 hopx 83.8 1.59s 1.66s 1.66s 10/10
5 vercel 81.4 1.71s 2.09s 2.09s 10/10
6 runloop 81.0 1.87s 1.95s 1.95s 10/10
7 cloudflare 76.1 2.34s 2.47s 2.47s 10/10
8 namespace 76.0 2.33s 2.51s 2.51s 10/10
9 modal 74.7 2.26s 2.95s 2.95s 10/10
10 codesandbox 69.0 2.86s 3.46s 3.46s 10/10
11 just-bash 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates benchmark result handling so provider runs with 0 successful iterations are treated as real failures (score = 0) rather than being marked as “skipped”, and pins the Vercel provider dependency to a specific version.

Changes:

  • Remove “all iterations failed ⇒ skipped” behavior across sandbox/storage/browser benchmarks and result renderers.
  • Ensure stats/time-to-first-ready calculations don’t error on empty successful sets (default to zeros).
  • Pin @computesdk/vercel to 1.7.15 (and update lockfile accordingly).

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/storage/benchmark.ts Stops marking all-failure storage runs as skipped; computes stats from successful-only set.
src/browser/benchmark.ts Stops marking all-failure browser runs as skipped; stats already handle empty successful set.
src/sandbox/benchmark.ts Removes skip-on-all-failures and defaults stats to zeros when no successes.
src/sandbox/concurrent.ts Removes skip-on-all-failures; guards min/stats when no successes.
src/sandbox/staggered.ts Removes skip-on-all-failures; guards min/stats when no successes.
src/sandbox/table.ts Removes special FAILED row; always reports success ratio in “Status”.
src/run.ts Adds (ok/total OK) to printed scores for storage and browser modes.
src/merge-results.ts Removes special FAILED rows in merged storage/browser tables; always reports success ratio.
package.json Pins @computesdk/vercel to 1.7.15.
package-lock.json Updates lockfile to match the pinned @computesdk/vercel version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@kisernl kisernl merged commit 2766b8d into master Apr 3, 2026
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants