Skip to content

vulkan: use mul_mat_vec_id for high-expert-count MoE decode - #25862

Draft
essentialols wants to merge 1 commit into
ggml-org:masterfrom
essentialols:vulkan-moe-highexpert-vecid
Draft

vulkan: use mul_mat_vec_id for high-expert-count MoE decode#25862
essentialols wants to merge 1 commit into
ggml-org:masterfrom
essentialols:vulkan-moe-highexpert-vecid

Conversation

@essentialols

Copy link
Copy Markdown

Extends ggml_vk_use_mul_mat_vec_id() to route MoE models with a large number of experts (>64, e.g. DeepSeek-V3/V4-class 256-expert models) to the mul_mat_vec_id path in the decode regime (batch <= 256).

Motivation

On RADV (Mesa; tested on an AMD Radeon 680M / gfx1035, Mesa 26.1.4) the mul_mat_id shader for >=256-expert MoE models stalls during pipeline compilation at high --n-gpu-layers, wedging the device (SMU becomes unresponsive: SMU: No response, Ring sdma0 reset failed, GPU reset ... ret=-62). Routing these dispatches to the existing mul_mat_vec_id path avoids the stall and is also faster per-dispatch for token generation.

Change

ggml_vk_use_mul_mat_vec_id(): alongside the existing small-batch case (batch <= 8), also return true when src0->ne[2] > 64 (>64 experts) and src2->ne[1] <= 256, restricting the change to the decode regime so large prefill batches are unaffected.

Testing

AMD Radeon 680M (gfx1035), Vulkan/RADV, Mesa 26.1.4, build b10066.

  • Without the change: a 256-expert MoE (Qwen3.6-35B-A3B-class) hangs during shader compilation at -ngl 99 (empty output, device wedge requiring a GPU reset).
  • With the change: the same model loads and generates coherently at -ngl 99.
  • No behavioral change for models with <=64 experts or for batch > 256.

Extend ggml_vk_use_mul_mat_vec_id() to route MoE models with a large number
of experts (>64, e.g. DeepSeek-V3/V4-class 256-expert models) to the
mul_mat_vec_id path in the decode regime (batch <= 256).

On RADV (Mesa, tested on Radeon 680M / gfx1035) the mul_mat_id shader for
>=256 experts stalls during pipeline compilation at high --n-gpu-layers,
hanging the device. Routing these dispatches to the existing mul_mat_vec_id
path avoids the stall and is also faster per-dispatch for token generation.
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jul 18, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Jul 18, 2026

Copy link
Copy Markdown

Hi @essentialols, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@0cc4m

0cc4m commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Open an issue and try to identify why it hangs first. Most likely the validation layers will show some problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants