Skip to content

[Ascend] Integrate transformer_engine_npu && Fix the GEMM operator bug in the reference backend#89

Open
Darryl233 wants to merge 8 commits into
flagos-ai:mainfrom
Darryl233:dev_lyz
Open

[Ascend] Integrate transformer_engine_npu && Fix the GEMM operator bug in the reference backend#89
Darryl233 wants to merge 8 commits into
flagos-ai:mainfrom
Darryl233:dev_lyz

Conversation

@Darryl233

@Darryl233 Darryl233 commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds Ascend NPU support to the TE-FL plugin system through torch_npu and transformer_engine_npu, and fixes backward-path issues in the reference GEMM implementation.

Changes

Ascend NPU backend

  • Add automatic NPU availability detection and vendor-priority registration.
  • Add support for:
    • FlashAttention with SBHD, BSHD, and THD layouts
    • RMSNorm forward and backward
    • Generic and grouped GEMM
    • Multi-tensor scale and L2-norm operations
  • Add THD ↔ BSHD conversion operators.
  • Keep NPU dependencies lazily imported.

Reference GEMM fixes

  • Fix output shape restoration for transposed inputs.
  • Do not add forward bias in backward mode.
  • Compute fused bias gradients.
  • Apply dGeLU using the saved forward activation.
  • Preserve correct alpha scaling and 3D input behavior.

Testing

Added coverage for:

  • FlashAttention forward/backward accuracy and causal masking
  • RMSNorm forward/backward
  • Generic and grouped GEMM
  • Multi-tensor and FP8 scale operations
  • Reference GEMM backward behavior

Verified on Ascend 910C:

46 passed

Deps

It depends on TransformerEngineNPU. The package natively generated by TransformerEngineNPU is named transformer_engine. Relevant packaging logic needs to be modified so that the generated package is named transformer_engine_npu.

@CLAassistant

CLAassistant commented Jul 21, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@lxd-cumt lxd-cumt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants