[DNM][AMD] job.slurm: add RDMA library version consistency check across all nodes

[seungrokj]

Summary

• Add _check_rdma_lib_versions() function in job.slurm that runs before job launch
• Checks all nodes in $SLURM_JOB_NODELIST have identical versions of:
  • /usr/lib/x86_64-linux-gnu/libionic.so.1.*
  • /usr/lib/x86_64-linux-gnu/libibverbs.so.1.*
  • /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so
• Resolves symlinks via readlink -f to get the actual versioned library path
• Prints each node's resolved library path for visibility
• Fails the job (exit 1) if any node has a mismatched version
• Also prefer libibverbs.so.1.14.39.0 over libibverbs.so.1.14.57.0 in _find_host_ibverbs()

Test plan

• Run on a cluster where all nodes have matching library versions — job should proceed normally
• Simulate a mismatch — job should abort with a clear error message

🤖 Generated with Claude Code

---

Note

High Risk
Commenting out the Docker/srun launch path would stop all multi-node benchmarks from running; the RDMA check also depends on passwordless SSH to every allocated node.

Overview
Adds a pre-launch _check_rdma_lib_versions gate in job.slurm that SSHs to every host in $SLURM_JOB_NODELIST, resolves three RDMA-related libraries via readlink -f, logs each node’s path, and aborts the job if any node differs from the first.

Important: the diff also comments out the entire downstream launch path (node selection, IP resolution, NFS refresh, Docker srun, router lifecycle, cleanup). After the new check, the script currently ends without starting containers—likely temporary for debugging unless restored before merge.

^{Reviewed by Cursor Bugbot for commit dac1b82. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dac1b82. Configure here.}

[cursor]

Job launch path commented out

High Severity

After _check_rdma_lib_versions succeeds, the script ends because node selection, Docker setup, and the main srun workload are entirely commented out. Slurm jobs appear to succeed but never start benchmarks or containers.

 

^{Reviewed by Cursor Bugbot for commit dac1b82. Configure here.}

[cursor]

Missing libraries pass check

Medium Severity

The RDMA check only compares resolved paths across nodes. If every node returns MISSING or SSH_FAIL, they all match the reference and the function exits successfully without confirming libraries exist or are reachable.

 

^{Reviewed by Cursor Bugbot for commit dac1b82. Configure here.}

[cursor]

Unused ref_versions declaration

Low Severity

declare -A ref_versions is added with a comment about storing per-library reference paths, but the associative array is never read or written anywhere in _check_rdma_lib_versions.

 

^{Reviewed by Cursor Bugbot for commit dac1b82. Configure here.}