[Bug]: Llama-4-Maverick-17B-128E-Instruct quantization skips all MoE experts → missing expert weights → vLLM load failure

[shubhra]

⚙️ Your current environment

The output of python collect_env.py

### Environment Information ###
Operating System: `Linux-6.8.0-85-generic-x86_64-with-glibc2.39`
Python Version: `3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]`
llm-compressor Version: `0.8.2.dev54+g6fea8880.d20251119`
compressed-tensors Version: `0.12.3a20251114`
transformers Version: `4.57.1`
torch Version: `2.9.0`
CUDA Devices: `['NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200', 'NVIDIA B200']`
AMD Devices: `None`

🐛 Describe the bug

When running examples/quantization_w4a4_fp4/llama4_example.py to quantize the Llama-4-Maverick-17B-128E-Instruct model, the generated config.json places all MoE experts under the ignore list - none of the routed experts are quantized. These should get quantized.

The shared expert is quantized correctly, but all 128 routed experts remain unquantized, producing no quantized tensors such as w1_weight, w2_weight, w3_weight.

This leads to vLLM failing to load the model with:

 KeyError: 'layers.17.feed_forward.experts.126.w1_weight'

because the expected quantized expert parameters were never created.

Impact:
MoE expert weights are silently omitted during quantization, producing incomplete checkpoints incompatible with vLLM inference.

🛠️ Steps to reproduce

No response

[HDCharles]

I'm seeing

https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a4_fp4/llama4_example.py#L65-L76

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=[
        "re:.*lm_head",
        "re:.*self_attn",
        "re:.*router",
        "re:.*vision_model.*",
        "re:.*multi_modal_projector.*",
        "Llama4TextAttention",
    ],
)

so i'm unsure why its not quantizing e.g. layers.17.feed_forward.experts.126.w1 based on those ignore patterns. Is that part of the vision model or something like that?

Regardless I have a PR up that will fix this issue on the other side, i.e. allowing vllm to load the model. Maybe try that branch out and see if it solves your issue?

[dsikka]

@HDCharles The issue is that it shouldn’t be ignoring those layers at all given they are not in the ignore list. It may because of recent changes to the moe context?

llm-compressor/src/llmcompressor/modeling/llama4.py

Line 18 in db0b68d

@MoECalibrationModule.register("Llama4TextMoe")

The context updates the expert layers and it seems like that may not be happening here.

[HDCharles]

Yeah but it seems like the model should still be runnable since all those layers get ignored in the config but it isn't runnable due to the moe ignore issue fixed by the PR i linked.

There's still the issue that its not quantizing the experts which needs to be debugged, I just wanted to suggest that workaround in the meantime.

[dsikka]

@HDCharles the issue isn't if the model is runnable. This issue is that Shubhra is selecting to quantize all these layers and they are being skipped.

[shivak]

I am encountering this same problem with Qwen3 VL MoE. I suspect the cause is is_permanent = True.

[dsikka]

Fixing through: #2095

[shivak]

Thanks - I can confirm that fixes the issue.

[dsikka]

The linked fix for this has landed