54 lines (32 loc) · 1.27 KB

MPSA (Mixed Precision Sparse Attention)

A training free sparse attention implementation. report.

Requirements

torch >= 2.9.0

Installation

pip install -e .

Performance (0.8 sparsity, TFlops, 4090)

Sparse TFlops: calculated by full attention formula.

Sage Attn 2++	MPSA Sparse Attn (0.8)
440 TFlops	1320 TFlops

Example

simple wan 2.2 14B end-to-end example: python example.py
benchmark sparse attn: python test/test_mpsa.py

Code

Sage Attention 2++ Gluon DSL: Folder
sparse attention Gluon DSL: File

Sample Videos

Using Wan 2.2 14B model.

Single Image to Video

Full Attn	Sparse Attn

First-and-Last Image to Video

Full Attn	Sparse Attn