Add vLLM dynamic scheduler reconfigure for single-server sweeps

[JordanNanos]

Summary

• Opt-in VLLM_DYNAMIC_RECONFIGURE=1 hook in run_benchmark_serving that
  calls vLLM /pause → /reconfigure → /resume before each benchmark run
• Reads VLLM_MAX_NUM_BATCHED_TOKENS and VLLM_MAX_NUM_SEQS env vars and
  sends them as a JSON body to POST /reconfigure
• benchmarks/test_reconfigure_sweep.sh — standalone A/B test script that
  compares N cold starts (baseline) vs 1 cold start + N reconfigure cycles
• Documentation in docs/vllm-dynamic-scheduler-reconfigure.md

Pre-built image

docker pull semianalysiswork/vllm-reconfigure:latest

Based on vllm/vllm-openai:v0.18.0 with the reconfigure API overlaid.
Source: JordanNanos/vllm feature/reconfigure-scheduler

Single-node test

docker run --rm --init --network host \
  --runtime nvidia --gpus all --ipc host --privileged \
  --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v $HF_HUB_CACHE:/root/.cache/huggingface \
  -v $(pwd):/workspace -w /workspace \
  -e HF_TOKEN -e PORT=8888 \
  -e MODEL=openai/gpt-oss-120b \
  -e TP=8 -e CONC=32 \
  -e ISL=1024 -e OSL=1024 \
  semianalysiswork/vllm-reconfigure:latest \
  bash benchmarks/test_reconfigure_sweep.sh

Sweeps 3 max_num_batched_tokens × 2 max_num_seqs = 6 configs.
Phase A: 6 cold starts. Phase B: 1 cold start + 5 reconfigure cycles (~1s each).

Test plan

• bash -n benchmarks/benchmark_lib.sh — syntax check
• bash -n benchmarks/test_reconfigure_sweep.sh — syntax check
• Build overlay image (semianalysiswork/vllm-reconfigure:latest)
• Run A/B test on a single GPU node
• Verify benchmark metrics match between baseline and reconfigure phases
• Compare total wall-clock time (expect ~5× reduction in startup overhead)

AI assistance was used to prepare this change.

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[JordanNanos]

Added the requested patched-vLLM distribution paths:

• install_patched_vllm helper in benchmarks/benchmark_lib.sh supporting wheel, git ref, and editable checkout installs.
• docs/vllm-patched-distribution.md with custom image, pinned wheel, mounted editable checkout, and pinned git-ref workflows.

Recommended for cluster sweeps: use a custom image or pinned wheel, then enable VLLM_DYNAMIC_RECONFIGURE=1 only for jobs running a vLLM build with the runtime reconfiguration API.

[claude]

🔴 The three curl calls in reconfigure_vllm_scheduler() are not chained with &&, so a failure of /reconfigure_scheduler is masked by the subsequent /resume success — the function returns 0 even when reconfiguration failed. Additionally, the call site in run_benchmark_serving() ignores the return value entirely, so the benchmark always proceeds regardless of reconfiguration outcome, silently producing incorrect results with stale scheduler settings.

Extended reasoning...

Bug 1 — internal error masking in reconfigure_vllm_scheduler() (lines 38-40):

In bash, a function's return code is the exit code of its last executed command. The three curl calls run unconditionally with no chaining:

curl -fsS -X POST "$base_url/pause?mode=keep"
curl -fsS -X POST -G "$base_url/reconfigure_scheduler" "${params[@]}"
curl -fsS -X POST "$base_url/resume"

The -f flag in -fsS makes curl exit with code 22 on HTTP 4xx/5xx responses. If /reconfigure_scheduler returns an error (e.g., HTTP 400 for an invalid parameter value), curl exits 22 — but execution continues unconditionally to /resume. If /resume succeeds (exit 0), the function returns 0, masking the reconfiguration failure completely.

Bug 2 — return value ignored at the call site in run_benchmark_serving() (~line 361):

if [[ "${VLLM_DYNAMIC_RECONFIGURE:-0}" == "1" && "$backend" == "vllm" ]]; then
    reconfigure_vllm_scheduler "$port"
fi

There is no || return 1 or any check on the return value. There is no set -e in the script (only set +x/set -x). Even if the function were fixed to propagate errors, the benchmark would still proceed unconditionally.

Combined effect — step-by-step proof:

1. User sets VLLM_DYNAMIC_RECONFIGURE=1, VLLM_MAX_NUM_SEQS=999999 (invalid, exceeds server capacity)
2. run_benchmark_serving calls reconfigure_vllm_scheduler "$port"
3. Inside the function: curl -fsS -X POST .../pause succeeds (exit 0)
4. curl -fsS -X POST -G .../reconfigure_scheduler ... → server returns HTTP 400 → curl exits 22
5. Execution continues (no &&, no error check): curl -fsS -X POST .../resume → succeeds (exit 0)
6. Function returns 0 (last command's exit code) — failure masked
7. Back in run_benchmark_serving: return value not checked, benchmark proceeds
8. vLLM server is still running with its original scheduler limits
9. Benchmark results are recorded as if they were obtained with the requested settings — silently incorrect

Fix: Chain the curl calls with && inside the function, and add || return 1 at the call site:

# Inside reconfigure_vllm_scheduler():
curl -fsS -X POST "$base_url/pause?mode=keep" && curl -fsS -X POST -G "$base_url/reconfigure_scheduler" "${params[@]}" && curl -fsS -X POST "$base_url/resume"

# At the call site in run_benchmark_serving():
reconfigure_vllm_scheduler "$port" || return 1

[functionstackx]

does this work with cudagraphs, aiter, amd, flashinfer, torch compile or any other

[JordanNanos]

does this work with cudagraphs, aiter, amd, flashinfer, torch compile or any other

@functionstackx unlikely, only vllm has /pause and /resume from what I can tell