dsv4-fp4-b300-sglang: enable piecewise cuda graph and mixed chunk#1693
dsv4-fp4-b300-sglang: enable piecewise cuda graph and mixed chunk#1693yhyang201 wants to merge 21 commits into
Conversation
DP-attention mode uses deterministic TCP ports derived from $PORT. If a previous sglang process didn't exit cleanly (e.g. Slurm scancel didn't fully propagate SIGTERM to all subprocesses), the next launch fails with "metrics_port at 9125 is not available". Non-DP-attention mode is unaffected because it uses ipc:// with random temp paths. Add kill_stale_servers() to benchmark_lib.sh and call it from all sglang benchmark scripts that use DP-attention (B300 and B200).
fuser/pkill may not be available in all containers. Use python3+psutil (always present in sglang/vllm images) to precisely kill processes on $PORT and $PORT+237 (the DP-attention metrics_port = PORT + ZMQ_TCP_PORT_DELTA + 4).
The previous kill_stale_servers ran inside the Pyxis container, which cannot see or kill processes in the host PID namespace. Move the cleanup to launch_b300-nv.sh as a bare srun (no --container-image) so pkill runs directly on the compute node and can kill leftover sglang/vllm processes from prior container runs.
Two fixes: - pkill -f "sglang" matched its own bash -c argv and killed itself before reaching the stale process. Use [s]glang regex trick so pkill skips its own process. - Change PORT from 8888 to 30000 to avoid conflict with stale processes still bound to 8888-derived ports (metrics_port 9125).
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| --stream-interval 30 | ||
| --enable-deepseek-v4-fp4-indexer | ||
| --enforce-piecewise-cuda-graph | ||
| --enable-mixed-chunk |
There was a problem hiding this comment.
Port cleanup never invoked
Medium Severity
This commit adds cleanup_server_ports in benchmark_lib.sh to clear the HTTP port and derived SGLang ZMQ/NCCL ports before serve, but dsv4_fp4_b300_sglang.sh still launches sglang serve without calling it, so stale listeners from a prior failed run can still cause “port not available” failures the helper was meant to prevent.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 30211e4. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27218080238 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27218289712 |
…lashinfer-allreduce-fusion
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27219020922 |
…k, set chunked-prefill-size 16384 for high-conc profiles
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f2724b9. Configure here.
| --chunked-prefill-size 8192 | ||
| --disable-flashinfer-autotune | ||
| --enable-deepseek-v4-fp4-indexer | ||
| --enable-mixed-chunk |
There was a problem hiding this comment.
Missing piecewise CUDA graph flag
Medium Severity
The PR adds --enable-mixed-chunk to every concurrency profile but never adds --enforce-piecewise-cuda-graph, even though the title and description call for both on all profiles. Runs therefore omit the intended piecewise CUDA graph enforcement.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit f2724b9. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27246451362 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27253722398 |
1 participant
Summary
--enforce-piecewise-cuda-graphand--enable-mixed-chunkto all concurrency profiles indsv4_fp4_b300_sglang.shNote
Low Risk
Benchmark and CI runner configuration only; no application auth or data-path logic, though perf numbers and which concurrency points run will change.
Overview
Retunes DeepSeek-V4-Pro FP4 on B300 (sglang) benchmarks to match GB300-style launch settings and narrows the published search space.
dsv4_fp4_b300_sglang.shswaps global env (FAST_WARMUP, radix/thinking/reasoning flags), hoists shared DP-attentionSGLANG_*tuning forCONC≥ 512, and adds--enable-deepseek-v4-fp4-indexerand--enable-mixed-chunkacross profiles. High-concurrency megamoe runs usechunked-prefill-size16384 (was 65536), and the 4096-concurrency recipe is removed entirely.nvidia-master.yamlbumps the nightly sglang image and drops the 4096 point from the 8k1k search space.launch_b300-nv.shexportsPORT=30000into the container (was 8888).perf-changelog.yamlrecords the env/image and mixed-chunk/prefill updates.Reviewed by Cursor Bugbot for commit 1bdee89. Bugbot is set up for automated code reviews on this repo. Configure here.