dsv4-fp4-b300-sglang: align env vars to GB300#1682
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
1 similar comment
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — straightforward benchmark recipe alignment (image bump, env var consolidation, fp4-indexer flag).
Extended reasoning...
Overview
PR touches two files: a benchmark shell script (benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh) and a CI config yaml (.github/configs/nvidia-master.yaml). Changes are an image bump (nightly-dev-cu13-20260608-303757cc), addition of --enable-deepseek-v4-fp4-indexer to every concurrency profile, and a refactor that lifts shared DP-attention env vars (SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN, SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW, SGLANG_OPT_USE_ONLINE_COMPRESS, SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK, etc.) into a single block plus a few new vars aligned to GB300.
Security risks
None. This is benchmark/recipe code that runs in CI sweep contexts. No auth, crypto, permissions, network exposure, or user input handling is touched.
Level of scrutiny
Low. This is benchmark configuration aligning a single-node B300 SGLang recipe to its GB300 sibling. Worst case is a tuning regression on that one sweep, which is self-contained and discoverable from sweep results.
Other factors
The bug hunting system found nothing. The refactor is intentional: previously each per-CONC block redundantly set SWA_EVICT_DROP_PAGE_MARGIN, SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW, and DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK (with slightly different values 8320/8320/8256); the consolidated block uses 8192 across the board per GB300 alignment, which is the stated goal. The CONC=512 profile (flashinfer_mxfp4) now also picks up megamoe-specific env vars, but they're inert for non-megamoe backends and consistent with the GB300 reference recipe.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
5 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27128239626 |
yhyang201
added
the
non-canary-full-sweep-enabled
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
yhyang201
removed
the
non-canary-full-sweep-enabled
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27152512947 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27152636345 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27154740922 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27156930191 |
yhyang201
force-pushed
the
align-b300-sglang-env-to-gb300
branch
from
7b37974 to
cc56a5c
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27172624941 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27174016467 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27174016467 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27181483409 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27181483409 |
yhyang201
force-pushed
the
align-b300-sglang-env-to-gb300
branch
from
94906e3 to
0155cc8
Compare
DP-attention mode uses deterministic TCP ports derived from $PORT. If a previous sglang process didn't exit cleanly (e.g. Slurm scancel didn't fully propagate SIGTERM to all subprocesses), the next launch fails with "metrics_port at 9125 is not available". Non-DP-attention mode is unaffected because it uses ipc:// with random temp paths. Add kill_stale_servers() to benchmark_lib.sh and call it from all sglang benchmark scripts that use DP-attention (B300 and B200).
fuser/pkill may not be available in all containers. Use python3+psutil (always present in sglang/vllm images) to precisely kill processes on $PORT and $PORT+237 (the DP-attention metrics_port = PORT + ZMQ_TCP_PORT_DELTA + 4).
yhyang201
force-pushed
the
align-b300-sglang-env-to-gb300
branch
from
0155cc8 to
c27f53f
Compare
The previous kill_stale_servers ran inside the Pyxis container, which cannot see or kill processes in the host PID namespace. Move the cleanup to launch_b300-nv.sh as a bare srun (no --container-image) so pkill runs directly on the compute node and can kill leftover sglang/vllm processes from prior container runs.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27194478439 |
Two fixes: - pkill -f "sglang" matched its own bash -c argv and killed itself before reaching the stale process. Use [s]glang regex trick so pkill skips its own process. - Change PORT from 8888 to 30000 to avoid conflict with stale processes still bound to 8888-derived ports (metrics_port 9125).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27195239810 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 276647b. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27195969331 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27202359539 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27218228998 |
1 participant
Summary
--enable-deepseek-v4-fp4-indexerflag to all concurrency profilesnightly-dev-cu13-20260608-303757ccNote
Medium Risk
Benchmark tuning and runner port changes can shift reported throughput/latency or break CI if clients still assume port 8888; no production auth/data paths, but results comparability is at stake.
Overview
Aligns the
dsv4-fp4-b300-sglangbenchmark recipe with GB300 SGLang settings and bumps the container image innvidia-master.yamltonightly-dev-cu13-20260609-317fc6a9.In
dsv4_fp4_b300_sglang.sh, common launch env switches fromSGLANG_JIT_DEEPGEMM_PRECOMPILE=0to GB300-style vars (FAST_WARMUP,RADIX_FORCE_MISS,DEFAULT_THINKING,DSV4_REASONING_EFFORT=max). For concurrency above 32, DP-attention env (MegaMoE FP4/MXF4, online compress, logging, timeouts, etc.) is centralized in one block instead of repeated per profile; high-conc megamoe blocks drop profile-specificNUM_MAX_TOKENS_PER_RANKoverrides in favor of the shared8192.--enable-deepseek-v4-fp4-indexeris added to every concurrency launch profile.B300 NV runner exports
PORT=30000into the container (was8888) so serve/benchmark clients match the expected listen port.perf-changelog.yamldocuments the config change.Reviewed by Cursor Bugbot for commit bf76d43. Bugbot is set up for automated code reviews on this repo. Configure here.