serving-llms-on-instinct: comprehensive behavioral test suite

skills/serving-llms-on-instinct/SKILL.md

@@ -101,6 +101,31 @@ This shallow-clones vllm-project/recipes from GitHub and fetches the latest
 Docker tag from Docker Hub. Takes ~10 seconds. If it fails, the existing
 cache still works.
 
+## Step 3b: Check model-specific overrides (MANDATORY)
+
+Run this script before constructing any Docker command. It reads
+`data/gpu_overrides.json` and prints the env vars and arg changes that MUST
+be applied for this model+GPU combination. Some overrides prevent silent data
+corruption — the model will start and appear healthy but produce wrong output
+without them.
+
+```bash
+python3 scripts/check_overrides.py --model <HF_MODEL_ID> --gfx <gfx_version>
+# Remote:
+python3 scripts/check_overrides.py --model <HF_MODEL_ID> --gfx <gfx_version>
+```
+
+Returns JSON with `env_set` (add these as `--env KEY=VALUE` in docker run),
+`args_remove` (remove these from vLLM args if present), and `summary`
+(explain to the user what changed and why).
+
+Do not skip this step even if you believe you already know the correct
+configuration. Known critical case:
+
+| Model prefix | GPU | Override | Why |
+|---|---|---|---|
+| `openai/gpt-oss` | gfx950 | `VLLM_ROCM_USE_AITER_MOE=0` | AITER MoE kernels corrupt the final answer (word-salad, repetition, unicode junk) at any temperature. The model loads, runs, and returns HTTP 200 without this flag — but the output is wrong. |
+
 ## Step 4: Construct the Docker command
 
 Read `data/recipes_cache.json` and `data/gpu_overrides.json` directly.
@@ -129,6 +154,11 @@ Build the Docker command by combining:
    For MoE models on multi-GPU, also add `--distributed-executor-backend mp`.
 8. **Port arg**: `--port <port>`
 
+**Apply model-specific overrides (final, highest precedence):** Apply the
+`env_set` and `args_remove` values collected in Step 3b. These win over all
+recipe and GPU defaults. If Step 3b found no matching override, skip this
+sub-step (but confirm to the user that no override was needed).
+
 If the exact model ID is not in `recipes_cache.json`, check for a base model
 match by stripping date/version suffixes (e.g., `Kimi-K2-Instruct` matches
 `Kimi-K2-Instruct-0905`). Use the base model's recipe if found.
@@ -315,6 +345,16 @@ with a segfault or illegal instruction: `VLLM_ROCM_USE_AITER_FP4BMM` must be
 `0` on gfx942. This is set correctly in `gpu_overrides.json` for gfx942.
 See vLLM issue #34641.
 
+**AITER MoE kernels corrupt gpt-oss output on gfx950 (MI350X/MI355X)** -- With
+the default AITER MoE path active, `openai/gpt-oss-*` produces coherent
+reasoning but a corrupted final answer: word-salad, repetition loops, stray
+unicode junk -- at any temperature, on both 20b and 120b. The AITER attention
+backend is fine and stays on. Fix: `VLLM_ROCM_USE_AITER_MOE=0`. Note:
+`AITER_MHA=0` and `AITER_FUSED_MOE_A16W4=1` do NOT fix it -- the whole MoE
+AITER path must be disabled. Handled automatically by the `model_overrides`
+entry for `openai/gpt-oss` in `gpu_overrides.json` (gfx950), applied as the
+final, highest-precedence step in Step 4. Observed on ROCm 7.2 / vLLM 0.23.0.
+
 **`HIP error: no kernel image`** -- The Docker image has no compiled kernel
 for your GPU's gfx version. Use `vllm/vllm-openai-rocm:latest`; it includes
 gfx942 and gfx950 kernels.

skills/serving-llms-on-instinct/data/models_to_test.txt

@@ -0,0 +1,70 @@
+meta-llama/Llama-3.1-8B-Instruct
+moonshotai/Kimi-K2-Thinking
+sentence-transformers/all-MiniLM-L6-v2
+maya-research/maya1
+BAAI/bge-m3
+Qwen/Qwen2.5-7B-Instruct
+meta-llama/Llama-3.2-3B-Instruct
+deepseek-ai/DeepSeek-R1
+mistralai/Mistral-7B-Instruct-v0.3
+meta-llama/Llama-3.1-8B
+dphn/Dolphin-Mistral-24B-Venice-Edition
+moonshotai/Kimi-K2-Instruct-0905
+BAAI/bge-reranker-v2-m3
+meta-llama/Meta-Llama-3-8B-Instruct
+MongoDB/mdbr-leaf-ir
+openbmb/MiniCPM4.1-8B
+neuphonic/neutts-air
+WeiboAI/VibeThinker-1.5B
+google/gemma-2-2b-it
+Qwen/Qwen2.5-0.5B-Instruct
+meta-llama/Llama-3.2-1B
+meta-llama/Llama-3.3-70B-Instruct
+thu-pacman/PCMind-2.1-Kaiyuan-2B
+fancyfeast/llama-joycaption-beta-one-hf-llava
+OSS-forge/DeepSeek-Coder-1.3B-cleaned
+ProsusAI/finbert
+joeddav/xlm-roberta-large-xnli
+sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
+mistralai/Mistral-7B-Instruct-v0.2
+TinyLlama/TinyLlama-1.1B-Chat-v1.0
+mistralai/Ministral-8B-Instruct-2410
+ibm-granite/granite-docling-258M
+opendatalab/MinerU2.5-2509-1.2B
+openbmb/InfLLM-V2-Short-Dense-Base
+nvidia/Kimi-K2-Thinking-NVFP4
+facebook/bart-large-cnn
+mistralai/Mixtral-8x7B-Instruct-v0.1
+ministral/Ministral-3b-instruct
+Qwen/Qwen2.5-3B-Instruct
+MongoDB/mdbr-leaf-mt
+ServiceNow-AI/Apriel-1.5-15b-Thinker
+amanrajput/MiniLM-L6-v2-biology-finetuned
+LLM360/K2-V2-Instruct
+openai-community/gpt2
+cross-encoder/ms-marco-MiniLM-L6-v2
+BAAI/bge-reranker-large
+meta-llama/Meta-Llama-3-8B
+meta-llama/Llama-3.1-70B-Instruct
+Qwen/Qwen2.5-1.5B-Instruct
+meta-llama/Llama-3.2-1B-Instruct
+Qwen/Qwen3.6-27B
+Qwen/Qwen3.6-35B-A3B
+Qwen/QwQ-32B
+google/gemma-4-31B-it
+google/gemma-4-26B-A4B-it
+google/gemma-4-E4B-it
+openai/gpt-oss-120b
+openai/gpt-oss-20b
+deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
+zai-org/GLM-4.7-Flash
+ibm-granite/granite-4.1-8b
+poolside/Laguna-XS.2
+Qwen/Qwen2.5-VL-7B-Instruct
+OpenGVLab/InternVL3-78B
+openai/whisper-large-v3-turbo
+black-forest-labs/FLUX.2-dev
+black-forest-labs/FLUX.2-klein-9B
+black-forest-labs/FLUX.1-dev
+Tongyi-MAI/Z-Image-Turbo
+Wan-AI/Wan2.2-T2V-A14B