[Bugfix] Improve DCP error message with backend hint #30174
Conversation
Add a helpful hint to the DCP error message suggesting users try setting VLLM_ATTENTION_BACKEND to a compatible backend such as FLASH_ATTN or FLASHINFER when the current backend doesn't support returning softmax LSE for decode. Fixes vllm-project#28407 Signed-off-by: GeoffreyWang1117 <173976389+GeoffreyWang1117@users.noreply.github.com> Signed-off-by: <>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request improves an error message for Distributed Context Parallelism (DCP) by adding a hint about compatible attention backends. The change is helpful, but the list of suggested backends is incomplete. I've suggested an improvement to make the hint more comprehensive for users on different hardware platforms.
| "does not return the softmax lse for decode." | ||
| "does not return the softmax lse for decode. " | ||
| "Try setting VLLM_ATTENTION_BACKEND to a compatible " | ||
| "backend such as FLASH_ATTN or FLASHINFER." |
There was a problem hiding this comment.
The list of suggested compatible backends is incomplete. For instance, it omits ROCM_FLASH, which is the primary compatible backend for ROCm users. To make this hint more helpful for users on different platforms, I suggest including more compatible backends in the message.
| "backend such as FLASH_ATTN or FLASHINFER." | |
| "backend such as FLASH_ATTN, FLASHINFER, or ROCM_FLASH." |
1 participant
Summary
VLLM_ATTENTION_BACKENDto a compatible backendAssertionError: DCP requires attention impls to return the softmax lse for decode, but the impl FlashInferImpl does not return the softmax lse for decode.Try setting VLLM_ATTENTION_BACKEND to a compatible backend such as FLASH_ATTN or FLASHINFER.Test plan
Fixes #28407
Signed-off-by: GeoffreyWang1117 173976389+GeoffreyWang1117@users.noreply.github.com