chore: Use Codspeed runners for wallclock benches #3194
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3194 +/- ##
==========================================
+ Coverage 93.73% 93.84% +0.11%
==========================================
Files 125 124 -1
Lines 37134 37239 +105
Branches 37134 37239 +105
==========================================
+ Hits 34807 34948 +141
+ Misses 1470 1425 -45
- Partials 857 866 +9
|
Since we're considering their paid plan. CC @sylvestre
larseggert
force-pushed
the
chore-ci-codspeed-runners
branch
from
151befd to
1c0d920
Compare
Co-authored-by: Adrien Cacciaguerra <adrien.caccia@gmail.com> Signed-off-by: Lars Eggert <lars@eggert.org>
larseggert
had a problem deploying
to
claude-review
GitHub Actions
Error
larseggert
had a problem deploying
to
claude-review
GitHub Actions
Error
larseggert
added a commit
to larseggert/neqo
that referenced
this pull request
Dec 15, 2025
larseggert
had a problem deploying
to
claude-review
GitHub Actions
Error
larseggert
had a problem deploying
to
claude-review
GitHub Actions
Error
larseggert
requested a deployment
to
claude-review
GitHub Actions
Waiting
CodSpeed Performance ReportMerging #3194 will not alter performanceComparing Summary
Benchmarks breakdown
Footnotes
|
larseggert
requested review from
KershawChang and
martinthomson
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restructures the benchmarking infrastructure to support Codspeed's paid plan by splitting existing benchmarks into separate wallclock and simulated-time variants, and configuring the CI to run wallclock benchmarks on Codspeed runners while instrumentation benchmarks run on GitHub runners.
Key changes:
- Split
transfer.rsandstreams.rsbenchmarks into_wallclockand_simulatedvariants with shared common modules - Updated Cargo.toml files to specify appropriate Codspeed modes for each benchmark type
- Modified CI workflows to default to Codspeed runners and route wallclock benchmarks to Codspeed infrastructure
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| neqo-transport/benches/transfer_wallclock.rs | New wallclock benchmark using black_box for compute efficiency measurement |
| neqo-transport/benches/transfer_simulated.rs | New simulated-time benchmark measuring network protocol efficiency with throughput metrics |
| neqo-transport/benches/transfer_common.rs | Extracted common setup and configuration logic for transfer benchmarks |
| neqo-transport/benches/transfer.rs | Removed - split into separate wallclock/simulated benchmarks |
| neqo-transport/Cargo.toml | Added benchmark metadata for Codspeed modes and disabled auto-benchmarks |
| neqo-http3/benches/streams_wallclock.rs | New wallclock benchmark for HTTP/3 streams |
| neqo-http3/benches/streams_simulated.rs | New simulated-time benchmark for HTTP/3 streams with throughput measurement |
| neqo-http3/benches/streams.rs | Removed - split into separate wallclock/simulated benchmarks |
| neqo-http3/benches/common.rs | Extracted common setup logic for HTTP/3 benchmarks |
| neqo-http3/Cargo.toml | Updated benchmark metadata to specify Codspeed modes for split benchmarks |
| neqo-common/Cargo.toml | Added criterion to cargo-machete ignore list and disabled auto-benchmarks |
| neqo-bin/Cargo.toml | Added criterion to cargo-machete ignore list and disabled auto-benchmarks |
| .github/workflows/perfcompare.yml | Changed default bencher to Codspeed and added logic for Codspeed runner selection |
| .github/workflows/codspeed.yml | Refactored matrix generation to support per-benchmark runner assignment based on mode |
| .github/workflows/bench.yml | Changed default bencher to Codspeed with appropriate runner selection |
| .github/actions/nss/action.yml | Removed runner.environment == 'github-hosted' checks to support self-hosted runners |
Contributor
Failed Interop TestsQUIC Interop Runner, client vs. server, differences relative to
All resultsSucceeded Interop TestsQUIC Interop Runner, client vs. server neqo-pr as client
neqo-pr as server
Unsupported Interop TestsQUIC Interop Runner, client vs. server neqo-pr as client
neqo-pr as server
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since we're considering their paid plan.
CC @sylvestre