[AMD] remove accuracy wrong sweep point, bump image to sglang-rocm 20260609 by billishyahao · Pull Request #1714 · SemiAnalysisAI/InferenceX

billishyahao · 2026-06-12T05:14:56Z

This patch is to remove accuracy wrong sweep point from dsr1 fp4 sgl disagg on mi355x

Also bumps the dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp and dsr1-fp4-mi355x-sglang-disagg-1k1k-mtp images to lmsysorg/sglang-rocm:v0.5.12.post1-rocm720-mi35x-20260609, which carries the initial accuracy bug fixes (tag verified on Docker Hub, pushed 2026-06-09).

Note

Medium Risk
Benchmark matrix and disagg server env behavior change for DP+EP runs; the decode-side bash condition in the diff looks syntactically invalid and could break multi-node SGLang jobs if not fixed before merge.

Overview
Trims DSR1 FP4 MI355X SGLang disaggregated MTP benchmark coverage and adjusts MoRI MoE env tuning for low-concurrency DP+EP runs.

In amd-master.yaml, drops duplicate 1×DEP8 + 1×DEP8 MTP sweep blocks (conc 128 / 64) and removes the entire dsr1-fp4-mi355x-sglang-disagg-mtp recipe so inaccurate sweep points are no longer scheduled. perf-changelog.yaml records removing conc 128 and 64 for the DEP8 case on dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp.

In server_sglang.sh, MORI_MOE_MAX_INPUT_TOKENS_* is only set when bench max concurrency is > 128 under combined DP+EP; otherwise those variables are unset (prefill and decode). The decode branch in the diff uses a broken test ("decode_max_running_requests" without $ / missing then), which would break the script at runtime if merged as shown.

^{Reviewed by Cursor Bugbot for commit 4b531d9. Bugbot is set up for automated code reviews on this repo. Configure here.}

…i355x

github-actions · 2026-06-12T05:15:05Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T05:17:53Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27396191956
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27396191956

functionstackx · 2026-06-12T05:18:28Z

-dsr1-fp4-mi355x-sglang-disagg-mtp:
-  image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260519
-  model: amd/DeepSeek-R1-0528-MXFP4-v2
-  model-prefix: dsr1
-  runner: mi355x-disagg
-  precision: fp4
-  framework: sglang-disagg
-  multinode: true
-  disagg: true
-  scenarios:
-    fixed-seq-len:
-    - isl: 1024
-      osl: 1024
-      search-space:


hi @billishyahao can u fix the accuracy eval issue instead of just deleting it?

sgl-project/sglang#27194

We are investigating and try to resolve the accuracy issue at the same moment. sgl-project/sglang#27194 . Here we need a clean up to eliminate those incorrect points.

functionstackx · 2026-06-12T05:20:35Z


      # 1P2D TP8
      - spec-decoding: "mtp"
-        conc-list: [ 32, 64 ]


@billishyahao can u fix conc 64 instead of just removing it?

github-actions · 2026-06-12T05:30:18Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27396415083
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27396415083

cursor · 2026-06-12T05:32:26Z

      # 1P2D TP8
      - spec-decoding: "mtp"
-        conc-list: [ 32, 64 ]
+        conc-list: [ 32 ]


Duplicate conc 32 sweep entry

Low Severity

After conc-list for the second # 1P2D TP8 block was narrowed from [ 32, 64 ] to [ 32 ], that entry matches the first # 1P2D TP8 block for concurrency 32 (same topology and DECODE_MTP_SIZE=3), so conc 32 is scheduled twice under dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp.

^{Reviewed by Cursor Bugbot for commit c64a266. Configure here.}

github-actions · 2026-06-12T05:32:46Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27396599603
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27396599603

github-actions · 2026-06-12T05:39:27Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27396690440
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27396690440

billishyahao · 2026-06-12T05:42:15Z

Hi @functionstackx for 0609 sgl nightly image, we still observe other regression so we decide to only remove sweep points in this PR. Once regression is resolved, we will bump the image as a follow-up.

functionstackx · 2026-06-12T05:43:41Z

Hi @functionstackx for 0609 sgl nightly image, we still observe other regression so we decide to only remove sweep points in this PR. Once regression is resolved, we will bump the image as a follow-up.

Instead of removing points, can you update once all points are fixed? i.e. the whole curve is accurate?

functionstackx · 2026-06-12T05:44:51Z

@billishyahao i am sorry but it doesn't make sense to spend time & effort on removing confirmed points with accuracy issues instead of spending time/effort on fixing the root cause of it

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4b531d9. Configure here.}

cursor · 2026-06-12T06:40:03Z

+        MORI_MOE_MAX_INPUT_TOKENS_DECODE=$((MORI_MAX_DISPATCH_TOKENS_DECODE * decode_dp_ranks * 7 / 10))
+    else
+        unset MORI_MOE_MAX_INPUT_TOKENS_DECODE
+    fi


Decode concurrency test wrong

High Severity

The decode guard compares the literal string decode_max_running_requests to 128 instead of "$decode_max_running_requests". The prefill side correctly uses "$prefill_max_running_requests", so the decode MOE override never runs for high bench concurrency and MORI_MOE_MAX_INPUT_TOKENS_DECODE is always unset in this block.

^{Reviewed by Cursor Bugbot for commit 4b531d9. Configure here.}

cursor · 2026-06-12T06:40:03Z

+        MORI_MOE_MAX_INPUT_TOKENS_DECODE=$((MORI_MAX_DISPATCH_TOKENS_DECODE * decode_dp_ranks * 7 / 10))
+    else
+        unset MORI_MOE_MAX_INPUT_TOKENS_DECODE
+    fi


MTP scales unset decode MOE

Medium Severity

When decode concurrency is at most 128, the new branch unsets MORI_MOE_MAX_INPUT_TOKENS_DECODE, but the later MTP block still multiplies that variable. Unset operands in arithmetic expansion become zero, so MTP runs export SGLANG_MORI_MOE_MAX_INPUT_TOKENS=0 instead of omitting the env var like the prefill unset path.

Additional Locations (1)

benchmarks/multi_node/amd_utils/server_sglang.sh#L220-L224

^{Reviewed by Cursor Bugbot for commit 4b531d9. Configure here.}

github-actions · 2026-06-12T06:45:35Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27399274123
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27399274123

[AMD] remove accuracy wrong sweep point from dsr1 fp4 sgl disagg on m…

15ab4c7

…i355x

billishyahao requested a review from a team June 12, 2026 05:14

billishyahao requested review from 1am9trash, chunfangamd, seungrokj and yctseng0211 as code owners June 12, 2026 05:14

github-project-automation Bot added this to InferenceMAX Board Jun 12, 2026

add perf change log

69dd3d8

billishyahao added AMD full-sweep-enabled labels Jun 12, 2026

functionstackx requested changes Jun 12, 2026

View reviewed changes

functionstackx reviewed Jun 12, 2026

View reviewed changes

Merge branch 'main' into amd/mi355x-dsfp4-fixacc

8dec29c

functionstackx changed the title ~~[AMD] remove accuracy wrong sweep point~~ [AMD] remove accuracy wrong sweep point, bump image to sglang-rocm 20260609 Jun 12, 2026

cursor Bot reviewed Jun 12, 2026

View reviewed changes

fix

b37a18d

fix

4b531d9

billishyahao force-pushed the amd/mi355x-dsfp4-fixacc branch from c64a266 to 4b531d9 Compare June 12, 2026 06:39

cursor Bot reviewed Jun 12, 2026

View reviewed changes

Conversation

billishyahao commented Jun 12, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

functionstackx Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

billishyahao Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

functionstackx Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

cursor Bot Jun 12, 2026

Choose a reason for hiding this comment

Duplicate conc 32 sweep entry

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

billishyahao commented Jun 12, 2026

Uh oh!

functionstackx commented Jun 12, 2026

Uh oh!

functionstackx commented Jun 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 12, 2026

Choose a reason for hiding this comment

Decode concurrency test wrong

Uh oh!

cursor Bot Jun 12, 2026

Choose a reason for hiding this comment

MTP scales unset decode MOE

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

billishyahao commented Jun 12, 2026 •

edited by cursor Bot

Loading

functionstackx Jun 12, 2026 •

edited

Loading