Capture Platform Config and warn on CPU settings that aren't optimal - #1097
Open
rpoornac wants to merge 6 commits into
Open
Capture Platform Config and warn on CPU settings that aren't optimal #1097rpoornac wants to merge 6 commits into
rpoornac wants to merge 6 commits into
Conversation
PYTORCH_ROCM_ARCH lists the archs a wheel was compiled for, not the device installed in the node. detect_gfx_arch took the first regex match from it, so every MI355X run using the standard ROCm image recorded gfx90a -- MI200, two generations off -- and because the env hit short-circuits the probe, the rocminfo call that would have answered gfx950 never ran. The value reaches the session manifest and the recipe KB, so runs were filed under the wrong silicon. Skip any env value carrying more than one distinct arch and fall through to the probe. Single-valued overrides keep their precedence. Co-authored-by: Cursor <cursoragent@cursor.com>
Two sessions taken on differently configured nodes are not comparable, and nothing in the report captured that. final.json carried no CPU, GPU, or stack detail, so a throughput delta credited to a code change could equally have been SMT, the NUMA layout, or a frequency governor. ROCm and vLLM versions existed only in the session manifest and raw server logs -- not in the artifact anyone actually reads. Add a platform block to final.json and three lines to final.md's Run summary: CPU model, SMT, NPS, sockets, governor, boost, kernel, the amdgpu-bound GPU count and arch, and the rocm/vllm/aiter/sglang fingerprint. GPU count comes from PCI devices bound to amdgpu so it reflects the hardware present rather than any *_VISIBLE_DEVICES masking, and the stack values reuse detect_stack_fingerprint rather than duplicating detection. Preflight prints the same CPU facts at launch, so a knob toggled mid-session surfaces as a mismatch between the two rather than being attributed to the optimizer. Both probes are WARN-only and fall silent off Linux sysfs. Co-authored-by: Cursor <cursoragent@cursor.com>
The preflight and report probes are sysfs-only because they run inside the
container, where there is no /dev/ipmi0 and no route to the BMC. That leaves
three of the six tuning knobs unreachable: APBDIS, DF C-states, and determinism
have no sysfs or MSR path on kernels without amd_hsmp (upstreamed in 5.18). This
script covers them host-side over Redfish.
Two layers, because neither answers alone. Redfish reports "Auto" for Core
Performance Boost and SMT on this platform, which is not a value; the OS layer
resolves those by measuring achieved clocks and counting threads per core.
Conversely the OS cannot see APBDIS or DF C-states at all. Output reports a BIOS
value, an effective value, and which layer decided.
Attribute names are OEM-defined -- Redfish standardizes the container, not its
contents -- so matching is synonym-based rather than a per-OEM table. On AMD the
names largely derive from AGESA, so Supermicro, Quanta, Gigabyte and AMD
reference designs converge; Dell ("LogicalProc") and Lenovo
("Processors.SMTMode") diverge more. An unmatched knob reports "unknown" rather
than a wrong answer, and --json dumps every candidate so adding an alias is one
line. Verified against Supermicro/AMI only; other OEM patterns come from
published names and are untested.
Reaching Redfish needs credentials the BMC will never disclose, since passwords
are write-only. The script generates one, writes it over KCS -- which is
credential-free by design, as root on the host already carries full BMC
authority through the system interface -- and revokes it on exit. It reuses a
single sentinel slot so repeated runs cannot fill the user table, and passes the
password on stdin so it never lands in /proc/<pid>/cmdline. --no-bmc skips the
BMC entirely. stdlib-only; exits non-zero on any FAIL for CI use.
Co-authored-by: Cursor <cursoragent@cursor.com>
CI E2E report — ✅ Succeeded
|
Screening the value's shape only caught the multi-arch case. A single-valued PYTORCH_ROCM_ARCH=gfx942, which vendor images commonly set, was still returned as the node's arch and still suppressed the rocminfo probe -- wrong in exactly the same way, but plausible enough to survive review. The variable never describes the installed device; framework/targeted_build.py sets it precisely because it is a compile target. Drop it from _GFX_ENVS so the question is answered by the probe rather than inferred from a build flag. Co-authored-by: Cursor <cursoragent@cursor.com>
_platform_fingerprint() samples the process's own node. In a multi-node session the orchestrator usually is not the benchmark node, so the record could describe a machine the numbers never came from -- worse than no record, because it reads as fact when someone explains a delta, and nothing in the output revealed which machine was measured. Add "host" and "multi_node_session" so a reader can tell a whole-truth record from a partial one, and surface the caveat in final.md when the session is multi-node. Preflight prints its hostname for the same reason. Per-node collection remains the real fix; this makes the gap visible rather than hiding it. Also drop the SMT warning. SMT is on by default on EPYC, so it fired on nearly every node, and an alert that always fires trains readers to ignore the rest of the check. The value is still recorded. The alert worth having is a node whose knobs disagree with others in the same session, which needs the per-node collection above. Co-authored-by: Cursor <cursoragent@cursor.com>
Every issue here shared a failure mode: the script could not tell the operator when it did not know something. Revocation ran through a helper that swallows failures, so a failed revoke left an enabled ADMINISTRATOR account reachable over LAN with a known password and said nothing. Each step is now checked, the end state is read back rather than inferred from exit codes, and a failure prints the slot, the user, and the commands to clean it up by hand, then exits 3. This immediately caught a real case: Supermicro rejects privilege=15, so ADMINISTRATOR was never actually dropped -- CALLBACK is the lowest it accepts, and the code now falls back to it. __exit__ still cannot run on SIGKILL or reboot, so a sentinel found already enabled at startup is reported as evidence of exactly that. Measurement assumed a part with at least 69 cores and read the global maximum CPU frequency rather than the pinned core's. On a smaller SKU the affinity call failed, the sysfs reads came back empty, and a meaningless spread still reached the determinism verdict. Cores are now derived from topology (one per physical core, sampled across the part), the pinned core is measured, affinity is set through os.sched_setaffinity so refusal is visible, and an untrustworthy sample reports UNKNOWN instead of a verdict. Also: EPYC generation read the second digit, which happens to work for 9575F and fails for 9755 -- the last digit carries the series. TLS verification is on by default with --ca-cert/--insecure, and the verification error is unwrapped from URLError so it names the flag that fixes it. --bmc-pass is gone; the password comes from $BMC_PASSWORD or a prompt, never argv. --json and text now share one exit-code function: 0 pass, 1 wrong, 2 unresolved, 3 revoke failed. README points at the script, which nothing referenced before. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capture platform fingerprint for reproducibility/record keeping.
Record host CPU tuning state and warn on settings that skew results.