diff --git a/README.md b/README.md index 66b9fb3e9..43962ef52 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,8 @@ We believe that a well-developed open-source code framework can lower the thresh > Currently, the development personnel of this project are limited, with most of the work handled by [Artiprocher](https://github.com/Artiprocher) and [mi804](https://github.com/mi804). Therefore, the progress of new feature development will be relatively slow, and the speed of responding to and resolving issues is limited. We apologize for this and ask developers to understand. +- **August 7, 2026** We add support for Wan-Animate-2 in the Wan series. Given a reference image and a driving video, it makes the reference character perform the motions in the driving video, generating high-quality character animation, with both standard and distilled variants. For details, please refer to the [documentation](/docs/en/Model_Details/Wan.md) and [example code](/examples/wanvideo/). + - **August 3, 2026** MiniMax-H3 open-sourced, welcome a new member to the video model family! Support includes text-to-video-audio generation, keyframe-guided generation, reference-driven generation, low VRAM inference, and NF4-quantized inference. For details, please refer to the [documentation](/docs/en/Model_Details/MiniMax-H3.md) and [example code](/examples/minimax_h3/). - **July 28, 2026** LingBot-Video open-sourced, welcome a new member to the video model family! This release includes two variants, Dense-1.3B and MoE-30B-A3B (30B total parameters, ~3B active per token), both supporting text-to-video, image-to-video and text-to-image generation, low VRAM inference, and LoRA / full training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/LingBot-Video.md) and [example code](/examples/lingbot_video/). Huge thanks to [NancyFyong](https://github.com/NancyFyong) for contributing the integration of this model! @@ -1413,41 +1415,43 @@ Example code for Wan is available at: [/examples/wanvideo/](/examples/wanvideo/) | Model ID | Extra Inputs | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | |-|-|-|-|-|-|-|-| -|[Wan-AI/Wan2.1-T2V-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-T2V-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-T2V-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-1.3B.py)| -|[Wan-AI/Wan2.1-T2V-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-T2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-T2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-14B.py)| -|[Wan-AI/Wan2.1-I2V-14B-480P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-480P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-480P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-480P.py)| -|[Wan-AI/Wan2.1-I2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-720P.py)| -|[Wan-AI/Wan2.1-FLF2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-FLF2V-14B-720P)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-FLF2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-FLF2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-FLF2V-14B-720P.py)| -|[iic/VACE-Wan2.1-1.3B-Preview](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B-Preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B-Preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B-Preview.py)| -|[Wan-AI/Wan2.1-VACE-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-1.3B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B.py)| -|[Wan-AI/Wan2.1-VACE-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-14B.py)| -|[PAI/Wan2.1-Fun-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-InP.py)| -|[PAI/Wan2.1-Fun-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control)|`control_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-Control.py)| -|[PAI/Wan2.1-Fun-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-InP.py)| -|[PAI/Wan2.1-Fun-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control)|`control_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-InP.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-InP.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control-Camera.py)| -|[DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1](https://modelscope.cn/models/DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1)|`motion_bucket_id`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-1.3b-speedcontrol-v1.py)| -|[krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/krea-realtime-video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/krea-realtime-video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/krea-realtime-video.py)| -|[meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video)|`longcat_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/LongCat-Video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/LongCat-Video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/LongCat-Video.py)| -|[ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B)|`vap_video`, `vap_prompt`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Video-As-Prompt-Wan2.1-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Video-As-Prompt-Wan2.1-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Video-As-Prompt-Wan2.1-14B.py)| -|[Wan-AI/Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-T2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-T2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-T2V-A14B.py)| -|[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| -|[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| -|[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| -|[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| -|[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| -|[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| -|[PAI/Wan2.2-Fun-A14B-Control](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control.py)| -|[PAI/Wan2.2-Fun-A14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control-Camera.py)| -|[openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference_low_vram/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/full/MOVA-360P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_full/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/lora/MOVA-360P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_lora/MOVA-360p-I2AV.py)| -|[openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference_low_vram/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/full/MOVA-720P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_full/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/lora/MOVA-720P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_lora/MOVA-720p-I2AV.py)| -|[Wan-AI/Wan-Dancer-14B (global model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan-Dancer-14B-global.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan-Dancer-14B-global.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-global.py)| -|[Wan-AI/Wan-Dancer-14B (local model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan-Dancer-14B-local.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan-Dancer-14B-local.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-local.py)| +|[Wan-AI/Wan2.1-T2V-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B)||[code](/examples/wanvideo/model_inference/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-T2V-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-1.3B.py)| +|[Wan-AI/Wan2.1-T2V-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B)||[code](/examples/wanvideo/model_inference/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-T2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-14B.py)| +|[Wan-AI/Wan2.1-I2V-14B-480P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-480P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-480P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-480P.py)| +|[Wan-AI/Wan2.1-I2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-720P.py)| +|[Wan-AI/Wan2.1-FLF2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-FLF2V-14B-720P)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-FLF2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-FLF2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-FLF2V-14B-720P.py)| +|[iic/VACE-Wan2.1-1.3B-Preview](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B-Preview.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B-Preview.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B-Preview.py)| +|[Wan-AI/Wan2.1-VACE-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-1.3B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B.py)| +|[Wan-AI/Wan2.1-VACE-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-14B.py)| +|[PAI/Wan2.1-Fun-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-InP.py)| +|[PAI/Wan2.1-Fun-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control)|`control_video`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-Control.py)| +|[PAI/Wan2.1-Fun-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-InP.py)| +|[PAI/Wan2.1-Fun-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control)|`control_video`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-InP.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-InP.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control-Camera.py)| +|[DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1](https://modelscope.cn/models/DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1)|`motion_bucket_id`|[code](/examples/wanvideo/model_inference/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-1.3b-speedcontrol-v1.py)| +|[krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video)||[code](/examples/wanvideo/model_inference/krea-realtime-video.py)|[code](/examples/wanvideo/model_inference_low_vram/krea-realtime-video.py)|[code](/examples/wanvideo/model_training/full/krea-realtime-video.sh)|[code](/examples/wanvideo/model_training/validate_full/krea-realtime-video.py)|[code](/examples/wanvideo/model_training/lora/krea-realtime-video.sh)|[code](/examples/wanvideo/model_training/validate_lora/krea-realtime-video.py)| +|[meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video)|`longcat_video`|[code](/examples/wanvideo/model_inference/LongCat-Video.py)|[code](/examples/wanvideo/model_inference_low_vram/LongCat-Video.py)|[code](/examples/wanvideo/model_training/full/LongCat-Video.sh)|[code](/examples/wanvideo/model_training/validate_full/LongCat-Video.py)|[code](/examples/wanvideo/model_training/lora/LongCat-Video.sh)|[code](/examples/wanvideo/model_training/validate_lora/LongCat-Video.py)| +|[ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B)|`vap_video`, `vap_prompt`|[code](/examples/wanvideo/model_inference/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_training/full/Video-As-Prompt-Wan2.1-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_training/lora/Video-As-Prompt-Wan2.1-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Video-As-Prompt-Wan2.1-14B.py)| +|[Wan-AI/Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B)||[code](/examples/wanvideo/model_inference/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-T2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-T2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-T2V-A14B.py)| +|[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| +|[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| +|[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B: Distilled](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py)| +|[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| +|[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| +|[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| +|[PAI/Wan2.2-Fun-A14B-Control](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control.py)| +|[PAI/Wan2.2-Fun-A14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control-Camera.py)| +|[openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p)|`input_image`|[code](/examples/mova/model_inference/MOVA-360p-I2AV.py)|[code](/examples/mova/model_inference_low_vram/MOVA-360p-I2AV.py)|[code](/examples/mova/model_training/full/MOVA-360P-I2AV.sh)|[code](/examples/mova/model_training/validate_full/MOVA-360p-I2AV.py)|[code](/examples/mova/model_training/lora/MOVA-360P-I2AV.sh)|[code](/examples/mova/model_training/validate_lora/MOVA-360p-I2AV.py)| +|[openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p)|`input_image`|[code](/examples/mova/model_inference/MOVA-720p-I2AV.py)|[code](/examples/mova/model_inference_low_vram/MOVA-720p-I2AV.py)|[code](/examples/mova/model_training/full/MOVA-720P-I2AV.sh)|[code](/examples/mova/model_training/validate_full/MOVA-720p-I2AV.py)|[code](/examples/mova/model_training/lora/MOVA-720P-I2AV.sh)|[code](/examples/mova/model_training/validate_lora/MOVA-720p-I2AV.py)| +|[Wan-AI/Wan-Dancer-14B (global model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](/examples/wanvideo/model_inference/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-global.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-global.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-global.py)| +|[Wan-AI/Wan-Dancer-14B (local model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](/examples/wanvideo/model_inference/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-local.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-local.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-local.py)| diff --git a/README_zh.md b/README_zh.md index 91f7e6e90..58ee31fb9 100644 --- a/README_zh.md +++ b/README_zh.md @@ -36,6 +36,8 @@ DiffSynth 目前包括两个开源项目: > 目前本项目的开发人员有限,大部分工作由 [Artiprocher](https://github.com/Artiprocher) 和 [mi804](https://github.com/mi804) 负责,因此新功能的开发进展会比较缓慢,issue 的回复和解决速度有限,我们对此感到非常抱歉,请各位开发者理解。 +- **2026年8月7日** 我们为 Wan 系列新增了 Wan-Animate-2,输入一张参考图和一段驱动视频,即可让参考角色演绎驱动视频中的动作,生成高质量角色动画,包含标准与蒸馏两个变体。详情请参考[文档](/docs/zh/Model_Details/Wan.md)和[示例代码](/examples/wanvideo/)。 + - **2026年8月3日** MiniMax-H3 开源,欢迎加入视频生成模型家族!支持文生音视频、首尾帧引导生成、参考驱动生成、低显存推理以及 NF4 量化推理。详情请参考[文档](/docs/zh/Model_Details/MiniMax-H3.md)和[示例代码](/examples/minimax_h3/)。 - **2026年7月28日** LingBot-Video 开源,欢迎加入视频生成模型家族!本次开源包含 Dense-1.3B 和 MoE-30B-A3B 两个版本(MoE 版总参数量 30B、每个 token 激活约 3B),均支持文生视频、图生视频、文生图推理、低显存推理以及 LoRA / 全量训练能力。详情请参考[文档](/docs/zh/Model_Details/LingBot-Video.md)和[示例代码](/examples/lingbot_video/)。特别感谢 [NancyFyong](https://github.com/NancyFyong) 贡献了本模型的接入代码! @@ -1413,41 +1415,43 @@ Wan 的示例代码位于:[/examples/wanvideo/](/examples/wanvideo/) |模型 ID|额外参数|推理|低显存推理|全量训练|全量训练后验证|LoRA 训练|LoRA 训练后验证| |-|-|-|-|-|-|-|-| -|[Wan-AI/Wan2.1-T2V-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-T2V-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-T2V-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-1.3B.py)| -|[Wan-AI/Wan2.1-T2V-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-T2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-T2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-14B.py)| -|[Wan-AI/Wan2.1-I2V-14B-480P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-480P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-480P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-480P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-480P.py)| -|[Wan-AI/Wan2.1-I2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-720P.py)| -|[Wan-AI/Wan2.1-FLF2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-FLF2V-14B-720P)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-FLF2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-FLF2V-14B-720P.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-FLF2V-14B-720P.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-FLF2V-14B-720P.py)| -|[iic/VACE-Wan2.1-1.3B-Preview](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B-Preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B-Preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B-Preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B-Preview.py)| -|[Wan-AI/Wan2.1-VACE-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-1.3B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B.py)| -|[Wan-AI/Wan2.1-VACE-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-VACE-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-VACE-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-14B.py)| -|[PAI/Wan2.1-Fun-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-InP.py)| -|[PAI/Wan2.1-Fun-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control)|`control_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-Control.py)| -|[PAI/Wan2.1-Fun-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-InP.py)| -|[PAI/Wan2.1-Fun-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control)|`control_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-InP.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-InP.py)| -|[PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)| -|[PAI/Wan2.1-Fun-V1.1-14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control-Camera.py)| -|[DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1](https://modelscope.cn/models/DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1)|`motion_bucket_id`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.1-1.3b-speedcontrol-v1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.1-1.3b-speedcontrol-v1.py)| -|[krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/krea-realtime-video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/krea-realtime-video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/krea-realtime-video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/krea-realtime-video.py)| -|[meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video)|`longcat_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/LongCat-Video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/LongCat-Video.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/LongCat-Video.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/LongCat-Video.py)| -|[ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B)|`vap_video`, `vap_prompt`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Video-As-Prompt-Wan2.1-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Video-As-Prompt-Wan2.1-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Video-As-Prompt-Wan2.1-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Video-As-Prompt-Wan2.1-14B.py)| -|[Wan-AI/Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-T2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-T2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-T2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-T2V-A14B.py)| -|[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| -|[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| -|[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| -|[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| -|[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| -|[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| -|[PAI/Wan2.2-Fun-A14B-Control](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)|`control_video`, `reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control.py)| -|[PAI/Wan2.2-Fun-A14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)|`control_camera_video`, `input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control-Camera.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control-Camera.py)| -|[openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference_low_vram/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/full/MOVA-360P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_full/MOVA-360p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/lora/MOVA-360P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_lora/MOVA-360p-I2AV.py)| -|[openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_inference_low_vram/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/full/MOVA-720P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_full/MOVA-720p-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/lora/MOVA-720P-I2AV.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/mova/model_training/validate_lora/MOVA-720p-I2AV.py)| -|[Wan-AI/Wan-Dancer-14B (global model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan-Dancer-14B-global.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-global.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan-Dancer-14B-global.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-global.py)| -|[Wan-AI/Wan-Dancer-14B (local model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan-Dancer-14B-local.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-local.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan-Dancer-14B-local.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-local.py)| +|[Wan-AI/Wan2.1-T2V-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B)||[code](/examples/wanvideo/model_inference/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-T2V-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-1.3B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-1.3B.py)| +|[Wan-AI/Wan2.1-T2V-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B)||[code](/examples/wanvideo/model_inference/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-T2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-14B.py)| +|[Wan-AI/Wan2.1-I2V-14B-480P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-480P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-480P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-480P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-480P.py)| +|[Wan-AI/Wan2.1-I2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-720P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-720P.py)| +|[Wan-AI/Wan2.1-FLF2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-FLF2V-14B-720P)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-FLF2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-FLF2V-14B-720P.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-FLF2V-14B-720P.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-FLF2V-14B-720P.py)| +|[iic/VACE-Wan2.1-1.3B-Preview](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B-Preview.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B-Preview.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B-Preview.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B-Preview.py)| +|[Wan-AI/Wan2.1-VACE-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-1.3B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B.py)| +|[Wan-AI/Wan2.1-VACE-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-VACE-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-14B.py)| +|[PAI/Wan2.1-Fun-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-InP.py)| +|[PAI/Wan2.1-Fun-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control)|`control_video`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-Control.py)| +|[PAI/Wan2.1-Fun-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-InP.py)| +|[PAI/Wan2.1-Fun-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control)|`control_video`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-InP.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-InP.py)| +|[PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py)| +|[PAI/Wan2.1-Fun-V1.1-14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control-Camera.py)| +|[DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1](https://modelscope.cn/models/DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1)|`motion_bucket_id`|[code](/examples/wanvideo/model_inference/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_training/full/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.1-1.3b-speedcontrol-v1.py)|[code](/examples/wanvideo/model_training/lora/Wan2.1-1.3b-speedcontrol-v1.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.1-1.3b-speedcontrol-v1.py)| +|[krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video)||[code](/examples/wanvideo/model_inference/krea-realtime-video.py)|[code](/examples/wanvideo/model_inference_low_vram/krea-realtime-video.py)|[code](/examples/wanvideo/model_training/full/krea-realtime-video.sh)|[code](/examples/wanvideo/model_training/validate_full/krea-realtime-video.py)|[code](/examples/wanvideo/model_training/lora/krea-realtime-video.sh)|[code](/examples/wanvideo/model_training/validate_lora/krea-realtime-video.py)| +|[meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video)|`longcat_video`|[code](/examples/wanvideo/model_inference/LongCat-Video.py)|[code](/examples/wanvideo/model_inference_low_vram/LongCat-Video.py)|[code](/examples/wanvideo/model_training/full/LongCat-Video.sh)|[code](/examples/wanvideo/model_training/validate_full/LongCat-Video.py)|[code](/examples/wanvideo/model_training/lora/LongCat-Video.sh)|[code](/examples/wanvideo/model_training/validate_lora/LongCat-Video.py)| +|[ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B)|`vap_video`, `vap_prompt`|[code](/examples/wanvideo/model_inference/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_training/full/Video-As-Prompt-Wan2.1-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Video-As-Prompt-Wan2.1-14B.py)|[code](/examples/wanvideo/model_training/lora/Video-As-Prompt-Wan2.1-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Video-As-Prompt-Wan2.1-14B.py)| +|[Wan-AI/Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B)||[code](/examples/wanvideo/model_inference/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-T2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-T2V-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-T2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-T2V-A14B.py)| +|[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| +|[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| +|[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B: Distilled](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py)| +|[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| +|[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| +|[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| +|[PAI/Wan2.2-Fun-A14B-Control](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)|`control_video`, `reference_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control.py)| +|[PAI/Wan2.2-Fun-A14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)|`control_camera_video`, `input_image`|[code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control-Camera.py)|[code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control-Camera.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control-Camera.py)| +|[openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p)|`input_image`|[code](/examples/mova/model_inference/MOVA-360p-I2AV.py)|[code](/examples/mova/model_inference_low_vram/MOVA-360p-I2AV.py)|[code](/examples/mova/model_training/full/MOVA-360P-I2AV.sh)|[code](/examples/mova/model_training/validate_full/MOVA-360p-I2AV.py)|[code](/examples/mova/model_training/lora/MOVA-360P-I2AV.sh)|[code](/examples/mova/model_training/validate_lora/MOVA-360p-I2AV.py)| +|[openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p)|`input_image`|[code](/examples/mova/model_inference/MOVA-720p-I2AV.py)|[code](/examples/mova/model_inference_low_vram/MOVA-720p-I2AV.py)|[code](/examples/mova/model_training/full/MOVA-720P-I2AV.sh)|[code](/examples/mova/model_training/validate_full/MOVA-720p-I2AV.py)|[code](/examples/mova/model_training/lora/MOVA-720P-I2AV.sh)|[code](/examples/mova/model_training/validate_lora/MOVA-720p-I2AV.py)| +|[Wan-AI/Wan-Dancer-14B (global model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](/examples/wanvideo/model_inference/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-global.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-global.py)|[code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-global.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-global.py)| +|[Wan-AI/Wan-Dancer-14B (local model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B)|`wantodance_music_path`, `wantodance_reference_image`, `wantodance_fps`, `wantodance_keyframes`, `wantodance_keyframes_mask`|[code](/examples/wanvideo/model_inference/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-local.sh)|[code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-local.py)|[code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-local.sh)|[code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-local.py)| diff --git a/diffsynth/configs/model_configs.py b/diffsynth/configs/model_configs.py index 2b736e5a4..437c68179 100644 --- a/diffsynth/configs/model_configs.py +++ b/diffsynth/configs/model_configs.py @@ -81,6 +81,13 @@ ] wan_series = [ + { + # Example: ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors") + "model_hash": "4536c21ad8740ba78367af4216ae85bf", + "model_name": "wan_video_dit", + "model_class": "diffsynth.models.wan_animate_2_dit.WanAnimate2Transformer", + "extra_kwargs": {}, + }, { # Example: ModelConfig(model_id="krea/krea-realtime-video", origin_file_pattern="krea-realtime-video-14b.safetensors") "model_hash": "5ec04e02b42d2580483ad69f4e76346a", diff --git a/diffsynth/configs/vram_management_module_maps.py b/diffsynth/configs/vram_management_module_maps.py index 8d83b0fa9..4f874756e 100644 --- a/diffsynth/configs/vram_management_module_maps.py +++ b/diffsynth/configs/vram_management_module_maps.py @@ -114,6 +114,15 @@ "diffsynth.models.wan_video_dit.RMSNorm": "diffsynth.core.vram.layers.AutoWrappedModule", "torch.nn.Conv2d": "diffsynth.core.vram.layers.AutoWrappedModule", }, + "diffsynth.models.wan_animate_2_dit.WanAnimate2Transformer": { + "diffsynth.models.wan_video_dit.MLP": "diffsynth.core.vram.layers.AutoWrappedModule", + "diffsynth.models.wan_animate_2_dit.AttentionBlock": "diffsynth.core.vram.layers.AutoWrappedNonRecurseModule", + "diffsynth.models.wan_animate_2_dit.Head": "diffsynth.core.vram.layers.AutoWrappedModule", + "torch.nn.Linear": "diffsynth.core.vram.layers.AutoWrappedLinear", + "torch.nn.Conv3d": "diffsynth.core.vram.layers.AutoWrappedModule", + "torch.nn.LayerNorm": "diffsynth.core.vram.layers.AutoWrappedModule", + "diffsynth.models.wan_video_dit.RMSNorm": "diffsynth.core.vram.layers.AutoWrappedModule", + }, "diffsynth.models.wan_video_dit.WanModel": { "diffsynth.models.wan_video_dit.MLP": "diffsynth.core.vram.layers.AutoWrappedModule", "diffsynth.models.wan_video_dit.DiTBlock": "diffsynth.core.vram.layers.AutoWrappedNonRecurseModule", diff --git a/diffsynth/core/attention/attention.py b/diffsynth/core/attention/attention.py index 642424cd3..7cff7c14a 100644 --- a/diffsynth/core/attention/attention.py +++ b/diffsynth/core/attention/attention.py @@ -26,6 +26,13 @@ except ModuleNotFoundError: XFORMERS_AVAILABLE = False +try: + from torch.nn.attention.flex_attention import flex_attention as flex_attention_func + flex_attention_func = torch.compile(flex_attention_func, dynamic=False, mode="max-autotune-no-cudagraphs", fullgraph=True, backend="inductor") + FLEX_ATTN_AVAILABLE = True +except (ModuleNotFoundError, ImportError): + FLEX_ATTN_AVAILABLE = False + try: if "enable_gqa" in inspect.signature(torch.nn.functional.scaled_dot_product_attention).parameters: TORCH_SUPPORT_GQA = True @@ -169,9 +176,20 @@ def xformers_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, q_patt return out -def attention_forward(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, q_pattern="b n s d", k_pattern="b n s d", v_pattern="b n s d", out_pattern="b n s d", dims=None, attn_mask=None, scale=None, is_causal=False, compatibility_mode=False, window_size=None): +def flex_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, q_pattern="b n s d", k_pattern="b n s d", v_pattern="b n s d", out_pattern="b n s d", dims=None, attn_mask=None, scale=None, score_mod=None): + assert FLEX_ATTN_AVAILABLE, "Flex Attention is not available. Please upgrade torch to 2.5.0 or later." + required_in_pattern, required_out_pattern = "b n s d", "b n s d" + q, k, v = rearrange_qkv(q, k, v, q_pattern, k_pattern, v_pattern, required_in_pattern, dims) + out = flex_attention_func(query=q, key=k, value=v, block_mask=attn_mask, scale=scale, score_mod=score_mod) + out = rearrange_out(out, out_pattern, required_out_pattern, dims) + return out + + +def attention_forward(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, q_pattern="b n s d", k_pattern="b n s d", v_pattern="b n s d", out_pattern="b n s d", dims=None, attn_mask=None, scale=None, is_causal=False, compatibility_mode=False, window_size=None, use_flex=False, score_mod=None): if compatibility_mode or (attn_mask is not None) or ATTENTION_IMPLEMENTATION == "torch": - if window_size is None: + if use_flex or score_mod is not None: + return flex_attention(q, k, v, q_pattern, k_pattern, v_pattern, out_pattern, dims, attn_mask=attn_mask, scale=scale, score_mod=score_mod) + elif window_size is None: return torch_sdpa(q, k, v, q_pattern, k_pattern, v_pattern, out_pattern, dims, attn_mask=attn_mask, scale=scale, is_causal=is_causal) else: # Sliding Window Attention is not compatible with `is_causal` and `attn_mask`. diff --git a/diffsynth/models/wan_animate_2_dit.py b/diffsynth/models/wan_animate_2_dit.py new file mode 100644 index 000000000..54750db90 --- /dev/null +++ b/diffsynth/models/wan_animate_2_dit.py @@ -0,0 +1,913 @@ +import numpy as np +import torch +import torch.nn as nn +import math +from torch.nn.attention.flex_attention import create_block_mask +from .wan_video_dit import sinusoidal_embedding_1d, RMSNorm, MLP +from ..core.attention.attention import attention_forward +from ..core.gradient import gradient_checkpoint_forward +from functools import lru_cache, partial + + +def _score_mod_impl(score, b_idx, h_idx, q_idx, kv_idx, hw: int, log_scale: float): + condition = (kv_idx >= hw) & (kv_idx < 2 * hw) + return torch.where(condition, score + log_scale, score) + + +@lru_cache(maxsize=32) +def _get_score_mod(hw: int, log_scale: float = -1.0): + return partial(_score_mod_impl, hw=hw, log_scale=log_scale) + + +def rope_params(max_seq_len, dim, theta=10000, offset=0): + assert dim % 2 == 0 + freqs = torch.outer( + torch.arange(max_seq_len)+offset, + 1.0 / torch.pow(theta, torch.arange(0, dim, 2).to(torch.float64).div(dim)) + ) + freqs = torch.polar(torch.ones_like(freqs), freqs) + return freqs + + +def rope_apply(x, grid_sizes, freqs, time_stride=1): + n, c = x.size(2), x.size(3) // 2 + + # split freqs + freqs = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1) + + # loop over samples + output = [] + for i, (f, h, w) in enumerate(grid_sizes.tolist()): + seq_len = f * h * w + + # precompute multipliers + x_i = torch.view_as_complex( + x[i, :seq_len].to(torch.float64).reshape(seq_len, n, -1, 2) + ) + freqs_i = torch.cat([ + freqs[0][:f*time_stride:time_stride].view(f, 1, 1, -1).expand(f, h, w, -1), + freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1), + freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1) + ], dim=-1).reshape(seq_len, 1, -1) + + # apply rotary embedding + x_i = torch.view_as_real(x_i * freqs_i).flatten(2) + x_i = torch.cat([x_i, x[i, seq_len:]]) + + # append to collection + output.append(x_i) + return torch.stack(output).to(x.dtype) + + +def pad_freqs(original_tensor, target_len): + seq_len, s1, s2 = original_tensor.shape + pad_size = target_len - seq_len + padding_tensor = torch.ones( + pad_size, + s1, + s2, + dtype=original_tensor.dtype, + device=original_tensor.device) + padded_tensor = torch.cat([original_tensor, padding_tensor], dim=0) + return padded_tensor + + +class LayerNorm(nn.LayerNorm): + + def __init__(self, dim, eps=1e-6, elementwise_affine=False): + super().__init__(dim, elementwise_affine=elementwise_affine, eps=eps) + + def forward(self, x): + return super().forward(x).type_as(x) + + +class SelfAttention(nn.Module): + + def __init__( + self, + dim, + num_heads, + window_size=(-1, -1), + qk_norm=True, + eps=1e-6 + ): + assert dim % num_heads == 0 + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.window_size = window_size + self.qk_norm = qk_norm + self.eps = eps + + # layers + self.q = nn.Linear(dim, dim) + self.k = nn.Linear(dim, dim) + self.v = nn.Linear(dim, dim) + self.o = nn.Linear(dim, dim) + self.norm_q = RMSNorm(dim, eps=eps) if qk_norm else nn.Identity() + self.norm_k = RMSNorm(dim, eps=eps) if qk_norm else nn.Identity() + + def forward(self, *args, method, **kwargs): + return getattr(self, method)(*args, **kwargs) + + def pre_attention(self, x): + b, s, n, d = *x.shape[:2], self.num_heads, self.head_dim + + # query, key, value function + def qkv_fn(x): + q = self.norm_q(self.q(x)).view(b, s, n, d) + k = self.norm_k(self.k(x)).view(b, s, n, d) + v = self.v(x).view(b, s, n, d) + return q, k, v + + q, k, v = qkv_fn(x) + + return q, k, v + + def post_attention(self, x): + # output + x = x.flatten(2) + x = self.o(x) + return x + + +class CrossAttention(SelfAttention): + def __init__(self, dim, num_heads, window_size=(-1, -1), qk_norm=True, eps=1e-6, use_img_emb=True): + super().__init__(dim, num_heads, window_size, qk_norm, eps) + self.use_img_emb = use_img_emb + if use_img_emb: + self.k_img = nn.Linear(dim, dim) + self.v_img = nn.Linear(dim, dim) + self.norm_k_img = RMSNorm(dim, eps=eps) if qk_norm else nn.Identity() + + def forward(self, x, context, context_lens, counter=0): + if self.use_img_emb: + context_img = context[:, :257] + context = context[:, 257:] + + b, n, d = x.size(0), self.num_heads, self.head_dim + + # compute query, key, value + q = self.norm_q(self.q(x)).view(b, -1, n, d) + k = self.norm_k(self.k(context)).view(b, -1, n, d) + v = self.v(context).view(b, -1, n, d) + + if self.use_img_emb: + k_img = self.norm_k_img(self.k_img(context_img)).view(b, -1, n, d) + v_img = self.v_img(context_img).view(b, -1, n, d) + img_x = attention_forward( + q, k_img, v_img, + q_pattern="b s n d", k_pattern="b s n d", v_pattern="b s n d", out_pattern="b s n d", + ) + # compute attention + x = attention_forward( + q, k, v, + q_pattern="b s n d", k_pattern="b s n d", v_pattern="b s n d", out_pattern="b s n d", + ) + + # output + x = x.flatten(2) + if self.use_img_emb: + img_x = img_x.flatten(2) + x = x + img_x + x = self.o(x) + return x + + +class AttentionBlock(nn.Module): + + def __init__( + self, + dim, + ffn_dim, + num_heads, + window_size=(-1, -1), + qk_norm=True, + cross_attn_norm=False, + eps=1e-6, + use_img_emb=True + ): + super().__init__() + self.dim = dim + self.ffn_dim = ffn_dim + self.num_heads = num_heads + self.window_size = window_size + self.qk_norm = qk_norm + self.cross_attn_norm = cross_attn_norm + self.eps = eps + + # layers + self.norm1 = LayerNorm(dim, eps) + + self.self_attn = SelfAttention(dim, num_heads, window_size, qk_norm, eps) + + self.norm3 = LayerNorm( + dim, eps, elementwise_affine=True + ) if cross_attn_norm else nn.Identity() + + self.cross_attn = CrossAttention(dim, num_heads, (-1, -1), qk_norm, eps, use_img_emb=use_img_emb) + + self.norm2 = LayerNorm(dim, eps) + self.ffn = nn.Sequential( + nn.Linear(dim, ffn_dim), + nn.GELU(approximate='tanh'), + nn.Linear(ffn_dim, dim) + ) + # modulation + self.modulation = nn.Parameter(torch.randn(1, 6, dim) / dim ** 0.5) + + def forward(self, *args, method, **kwargs): + return getattr(self, method)(*args, **kwargs) + + def pre_self_attention(self, x, e): + e = (self.modulation.to(x.device) + e).chunk(6, dim=1) + q, k, v = self.self_attn(self.norm1(x) * (1 + e[1]) + e[0], method="pre_attention") + return q, k, v, e + + def post_self_attention(self, x): + x = self.self_attn(x, method="post_attention") + return x + + def cross_attention(self, x, context, context_lens, e): + x = x + self.cross_attn(self.norm3(x), context, context_lens) + y = self.ffn(self.norm2(x) * (1 + e[4]) + e[3]) + x = x + y * e[5] + return x + + +class Incontext_AttentionBlock(nn.Module): + def __init__( + self, + dim, + ffn_dim, + num_heads, + window_size=(-1, -1), + qk_norm=True, + cross_attn_norm=False, + eps=1e-6, + refer_stride=1, + use_img_emb=True, + use_context_parallel=False, + sparse_type=0, + ): + super().__init__() + self.dim = dim + self.ffn_dim = ffn_dim + self.num_heads = num_heads + self.window_size = window_size + self.qk_norm = qk_norm + self.cross_attn_norm = cross_attn_norm + self.eps = eps + self.refer_stride = refer_stride + self.use_context_parallel = use_context_parallel + self.sparse_type = sparse_type + + self.block = AttentionBlock( + dim, ffn_dim, num_heads, window_size, qk_norm, cross_attn_norm, eps, use_img_emb=use_img_emb + ) + + def forward(self, *args, method, **kwargs): + return getattr(self, method)(*args, **kwargs) + + def forward_ref( + self, + x_ref, + index, + k_cache, + v_cache, + context_ref, + freqs_ref, + grid_sizes_ref, + e_ref, + context_lens, + animate2_offload_kv=False, + use_context_parallel=False, + ): + q_ref, k_ref, v_ref, e_ref = self.block(x_ref, e_ref, method='pre_self_attention') + + if use_context_parallel: + from ..utils.xfuser import all_to_all_4d, is_evenly_divisible, get_sequence_parallel_world_size + assert is_evenly_divisible(q_ref.shape[2]), ( + f"num_heads ({q_ref.shape[2]}) must be divisible by sequence parallel world size ({get_sequence_parallel_world_size()}) " + ) + qkv_ref = torch.cat([q_ref, k_ref, v_ref], dim=0) + qkv_ref = all_to_all_4d(qkv_ref, scatter_dim=2, gather_dim=1) + q_ref, k_ref, v_ref = qkv_ref.chunk(3, dim=0) + + k_cache[index] = k_ref if not animate2_offload_kv else k_ref.to('cpu') + v_cache[index] = v_ref if not animate2_offload_kv else v_ref.to('cpu') + q_ref_add_rope = rope_apply(q_ref, grid_sizes_ref, freqs_ref, self.refer_stride) + k_ref_add_rope = rope_apply(k_ref, grid_sizes_ref, freqs_ref, self.refer_stride) + + ref_f, ref_h, ref_w = grid_sizes_ref[0].tolist() + ref_vail_len = ref_f * ref_h * ref_w + + xout_ref = attention_forward( + q_ref_add_rope.to(v_ref.dtype), + k_ref_add_rope[:, :ref_vail_len].to(v_ref.dtype), + v_ref[:, :ref_vail_len], + q_pattern="b s n d", k_pattern="b s n d", v_pattern="b s n d", out_pattern="b s n d", + ).to(q_ref_add_rope.dtype) + + if use_context_parallel: + xout_ref = all_to_all_4d(xout_ref, scatter_dim=1, gather_dim=2) + + y_ref = self.block(xout_ref, method='post_self_attention') + + x_ref = x_ref + y_ref * e_ref[2] + + x_ref = self.block(x_ref, context_ref, context_lens, e_ref, method='cross_attention') + + return x_ref + + def forward_gen( + self, + x, + index, + k_cache, + v_cache, + attn_mask, + context, + freqs, + freqs_ref, + grid_sizes, + grid_sizes_ref, + origin_len, + origin_area, + e, + context_lens, + use_context_parallel=False, + log_scale=0.0, + ): + + origin_latent_f = origin_len // 4 + 1 + origin_latent_hw = origin_area[0] * origin_area[1] // 256 + origin_max_len = (origin_latent_f + 1) * origin_latent_hw + origin_ref_max_len = origin_latent_f * origin_latent_hw + + f, h, w = grid_sizes[0].tolist() + vail_len = f * h * w + hw = h * w + + ref_f, ref_h, ref_w = grid_sizes_ref[0].tolist() + ref_vail_len = ref_f * ref_h * ref_w + ref_hw = ref_h * ref_w + + q, k, v, e = self.block(x, e, method='pre_self_attention') + if use_context_parallel: + from ..utils.xfuser import all_to_all_4d, is_evenly_divisible, get_sequence_parallel_world_size + assert is_evenly_divisible(q.shape[2]), ( + f"num_heads ({q.shape[2]}) must be divisible by sequence parallel world size ({get_sequence_parallel_world_size()}) " + ) + qkv = torch.cat([q, k, v], dim=0) + qkv = all_to_all_4d(qkv, scatter_dim=2, gather_dim=1) + q, k, v = qkv.chunk(3, dim=0) + + q = rope_apply(q, grid_sizes, freqs) + k = rope_apply(k, grid_sizes, freqs) + k_ref, v_ref = k_cache[index].to(x.device), v_cache[index].to(x.device) + k_ref = rope_apply(k_ref, grid_sizes_ref, freqs_ref, self.refer_stride) + + B, _, N, C = q.shape + device, dtype = q.device, q.dtype + + target_q_len = math.ceil(origin_max_len / 128) * 128 + target_ref_len = math.ceil(origin_ref_max_len / 128) * 128 + target_kv_len = target_q_len + target_ref_len + + q_padding = q[:, vail_len:].clone() + + q_incontext = torch.zeros(B, target_q_len, N, C, device=device, dtype=dtype) + k_incontext = torch.zeros(B, target_kv_len, N, C, device=device, dtype=dtype) + v_incontext = torch.zeros(B, target_kv_len, N, C, device=device, dtype=dtype) + + q_src = q[:, :vail_len].view(B, f, hw, N, C) + k_src = k[:, :vail_len].view(B, f, hw, N, C) + v_src = v[:, :vail_len].view(B, f, hw, N, C) + + q_incontext[:, :f * origin_latent_hw].view(B, f, origin_latent_hw, N, C)[:, :, :hw] = q_src + k_incontext[:, :f * origin_latent_hw].view(B, f, origin_latent_hw, N, C)[:, :, :hw] = k_src + v_incontext[:, :f * origin_latent_hw].view(B, f, origin_latent_hw, N, C)[:, :, :hw] = v_src + + k_ref_src = k_ref[:, :ref_vail_len].view(B, ref_f, ref_hw, N, C) + v_ref_src = v_ref[:, :ref_vail_len].view(B, ref_f, ref_hw, N, C) + + k_incontext[:, target_q_len : target_q_len + ref_f * origin_latent_hw]\ + .view(B, ref_f, origin_latent_hw, N, C)[:, :, :ref_hw] = k_ref_src + v_incontext[:, target_q_len : target_q_len + ref_f * origin_latent_hw]\ + .view(B, ref_f, origin_latent_hw, N, C)[:, :, :ref_hw] = v_ref_src + + score_mod = _get_score_mod(hw=int(origin_latent_hw), log_scale=log_scale) + + xout_full = attention_forward( + q_incontext, k_incontext, v_incontext, + q_pattern="b s n d", k_pattern="b s n d", v_pattern="b s n d", out_pattern="b s n d", + use_flex=True, attn_mask=attn_mask, score_mod=score_mod + ) + + xout_valid = xout_full[:, :f * origin_latent_hw] + xout_valid = xout_valid.view(B, f, origin_latent_hw, N, C) + xout_vail = xout_valid[:, :, :hw] + xout_vail = xout_vail.reshape(B, f * hw, N, C) # [B, f*hw, N, C] + xout = torch.cat([xout_vail, q_padding], dim=1) + if use_context_parallel: + xout = all_to_all_4d(xout, scatter_dim=1, gather_dim=2) + + y = self.block(xout, method='post_self_attention') + + x = x + y * e[2] + + x = self.block(x, context, context_lens, e, method='cross_attention') + return x + + def forward_origin(self, x, x_ref, ref_args, gen_args): + k_cache, v_cache = {}, {} + x_ref = self.forward_ref(x_ref, 0, k_cache, v_cache, **ref_args) + x = self.forward_gen(x, 0, k_cache, v_cache, **gen_args) + return x, x_ref + + +class Head(nn.Module): + + def __init__(self, dim, out_dim, patch_size, eps=1e-6): + super().__init__() + self.dim = dim + self.out_dim = out_dim + self.patch_size = patch_size + self.eps = eps + + # layers + out_dim = math.prod(patch_size) * out_dim + self.norm = LayerNorm(dim, eps) + self.head = nn.Linear(dim, out_dim) + + # modulation + self.modulation = nn.Parameter(torch.randn(1, 2, dim) / dim**0.5) + + def forward(self, x, e): + e = (self.modulation + e.unsqueeze(1)).chunk(2, dim=1) + x = self.head(self.norm(x) * (1 + e[1]) + e[0]) + return x + + +class WanAnimate2Transformer(nn.Module): + + def __init__( + self, + patch_size=(1, 2, 2), + text_len=512, + in_dim=36, + dim=5120, + ffn_dim=13824, + freq_dim=256, + text_dim=4096, + out_dim=16, + num_heads=40, + num_layers=40, + window_size=(-1, -1), + qk_norm=True, + cross_attn_norm=True, + eps=1e-6, + use_img_emb=True, + refer_offset_t=1, + refer_offset_h=0, + refer_offset_w=-1, + refer_stride=1, + sparse_type=0, + ): + super().__init__() + self.patch_size = patch_size + self.text_len = text_len + self.in_dim = in_dim + self.dim = dim + self.ffn_dim = ffn_dim + self.freq_dim = freq_dim + self.text_dim = text_dim + self.out_dim = out_dim + self.num_heads = num_heads + self.num_layers = num_layers + self.window_size = window_size + self.qk_norm = qk_norm + self.cross_attn_norm = cross_attn_norm + self.eps = eps + self.use_img_emb = use_img_emb + self.refer_offset_t = refer_offset_t + self.refer_offset_h = refer_offset_h + self.refer_offset_w = refer_offset_w + self.refer_stride = refer_stride + self.sparse_type = sparse_type + + # [Denoising Transformer] + # embeddings + self.patch_embedding = nn.Conv3d( + in_dim, dim, kernel_size=patch_size, stride=patch_size + ) + self.text_embedding = nn.Sequential( + nn.Linear(text_dim, dim), + nn.GELU(approximate='tanh'), + nn.Linear(dim, dim) + ) + + self.time_embedding = nn.Sequential( + nn.Linear(freq_dim, dim), + nn.SiLU(), + nn.Linear(dim, dim) + ) + self.time_projection = nn.Sequential( + nn.SiLU(), + nn.Linear(dim, dim * 6) + ) + + # blocks + self.blocks = nn.ModuleList([Incontext_AttentionBlock( + dim, ffn_dim, num_heads, window_size, qk_norm, cross_attn_norm, eps, refer_stride, use_img_emb=use_img_emb, sparse_type=self.sparse_type + ) for _ in range(num_layers)]) + + # head + self.head = Head(dim, out_dim, patch_size, eps) + + if use_img_emb: + self.img_emb = MLP(1280, dim, has_pos_emb=False) + + self.block_masks = dict() + self.block_mask_grid_sizes = dict() + + def create_mask(self, origin_len, origin_area, device): + origin_latent_f = origin_len // 4 + 1 + hw = int(np.prod(origin_area).item() // 256) + + q_len = (origin_latent_f + 1) * hw + k_len = origin_latent_f * hw + + q_len_total = math.ceil(q_len / 128) * 128 + k_extra_len_total = math.ceil(k_len / 128) * 128 + k_len_total = q_len_total + k_extra_len_total + + q_limit = q_len + k_limit = k_len + q_total = q_len_total + + def attention_mask_logic(b, h, q_idx, kv_idx): + q_valid = q_idx < q_limit + is_base_attention = kv_idx < q_limit + + q_frame = q_idx // hw + is_first_part = kv_idx < q_total + + kv_frame_1 = kv_idx // hw + kv_is_valid_1 = kv_idx < q_limit + + rel_kv_idx = kv_idx - q_total + kv_frame_2 = (rel_kv_idx // hw) + 1 + kv_is_valid_2 = rel_kv_idx < k_limit + + kv_frame = torch.where(is_first_part, kv_frame_1, kv_frame_2) + kv_is_valid = torch.where(is_first_part, kv_is_valid_1, kv_is_valid_2) + + is_cond_attention = (q_frame == kv_frame) & kv_is_valid + + return q_valid & (is_base_attention | is_cond_attention) + + block_mask = create_block_mask( + attention_mask_logic, + B=None, + H=None, + Q_LEN=q_len_total, + KV_LEN=k_len_total, + device=device, + _compile=True + ) + return block_mask + + def forward_ref( + self, + x_ref, + grid_sizes, + k_cache, + v_cache, + clip_fea_ref, + y_ref, + context_ref, + seq_len_ref, + t, + animate2_offload_kv=False, + use_unified_sequence_parallel: bool = False, + ): + # [reference] + x_ref = [torch.cat([u, v], dim=0) for u, v in zip(x_ref, y_ref)] + # embeddings + x_ref = [self.patch_embedding(u.unsqueeze(0)) for u in x_ref] + grid_sizes_ref = torch.stack([ + torch.tensor(u.shape[2:], dtype=torch.long) for u in x_ref + ]) + x_ref = [u.flatten(2).transpose(1, 2) for u in x_ref] + seq_lens_ref = torch.tensor([u.size(1) for u in x_ref], dtype=torch.long) + assert seq_lens_ref.max() <= seq_len_ref + x_ref = torch.cat([torch.cat([ + u, u.new_zeros(1, seq_len_ref - u.size(1), u.size(2)) + ], dim=1) for u in x_ref]) + + assert (self.dim % self.num_heads) == 0 and (self.dim // self.num_heads) % 2 == 0 + d = self.dim // self.num_heads + + if self.refer_offset_t < 0: + self.refer_offset_t = grid_sizes[0][0].item() + if self.refer_offset_h < 0: + self.refer_offset_h = grid_sizes[0][1].item() + if self.refer_offset_w < 0: + self.refer_offset_w = grid_sizes[0][2].item() + + self.freqs_ref = torch.cat([ + rope_params(512, d - 4 * (d // 6), offset=self.refer_offset_t), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_h), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_w) + ], dim=1).to(x_ref.device) + + # time embeddings ref + e_ref = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t*0+1).to(x_ref.dtype)) + e0_ref = self.time_projection(e_ref).unflatten(1, (6, self.dim)) + + # [context_ref] + context_ref = self.text_embedding(torch.stack([torch.cat([ + u, u.new_zeros(self.text_len - u.size(0), u.size(1)) + ]) for u in context_ref])) + + if self.use_img_emb: + context_clip_ref = self.img_emb(clip_fea_ref) # bs x 257 x dim + context_ref = torch.concat([context_clip_ref, context_ref], dim=1) + + context_lens = None + # arguments + kwargs = dict( + e_ref=e0_ref, + grid_sizes_ref=grid_sizes_ref, + freqs_ref=self.freqs_ref, + context_ref=context_ref, + context_lens=context_lens, + animate2_offload_kv=animate2_offload_kv, + use_context_parallel=use_unified_sequence_parallel, + ) + if use_unified_sequence_parallel: + from ..utils.xfuser import get_current_chunk, is_evenly_divisible, get_sequence_parallel_world_size + assert is_evenly_divisible(x_ref.shape[1]), ( + f"x_ref sequence length ({x_ref.shape[1]}) must be divisible by sequence parallel world size ({get_sequence_parallel_world_size()}) " + ) + x_ref = get_current_chunk(x_ref, dim=1) + for idx, block in enumerate(self.blocks): + x_ref = block(x_ref, idx, k_cache, v_cache, method='forward_ref', **kwargs) + + def forward_gen( + self, + x, + k_cache, + v_cache, + clip_fea, + y, + context, + seq_len, + t, + grid_sizes_ref, + origin_len, + origin_area, + is_uncondtion=False, + use_unified_sequence_parallel: bool = False, + log_scale=0.0, + ): + # params + x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)] + # embeddings + x = [self.patch_embedding(u.unsqueeze(0)) for u in x] + grid_sizes = torch.stack([ + torch.tensor(u.shape[2:], dtype=torch.long) for u in x + ]) + x = [u.flatten(2).transpose(1, 2) for u in x] + seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long) + assert seq_lens.max() <= seq_len + x = torch.cat([torch.cat([ + u, u.new_zeros(1, seq_len - u.size(1), u.size(2)) + ], dim=1) for u in x]) + + assert (self.dim % self.num_heads) == 0 and (self.dim // self.num_heads) % 2 == 0 + d = self.dim // self.num_heads + self.freqs = torch.cat([ + rope_params(512, d - 4 * (d // 6)), + rope_params(512, 2 * (d // 6)), + rope_params(512, 2 * (d // 6)) + ], dim=1).to(x.device) + + if self.refer_offset_t < 0: + self.refer_offset_t = grid_sizes[0][0].item() + if self.refer_offset_h < 0: + self.refer_offset_h = grid_sizes[0][1].item() + if self.refer_offset_w < 0: + self.refer_offset_w = grid_sizes[0][2].item() + + self.freqs_ref = torch.cat([ + rope_params(512, d - 4 * (d // 6), offset=self.refer_offset_t), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_h), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_w) + ], dim=1).to(x.device) + + # time embeddings + e = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t).to(x.dtype)) + e0 = self.time_projection(e).unflatten(1, (6, self.dim)) + + # [context] + context_lens = None + context = self.text_embedding(torch.stack([torch.cat([ + u, u.new_zeros(self.text_len - u.size(0), u.size(1)) + ]) for u in context])) + + if self.use_img_emb: + context_clip = self.img_emb(clip_fea) # bs x 257 x dim + context = torch.concat([context_clip, context], dim=1) + + block_mask_id = (origin_len, origin_area[0], origin_area[1]) + if block_mask_id not in self.block_masks: + self.block_masks[block_mask_id] = self.create_mask(origin_len, origin_area, x.device) + attn_mask = self.block_masks[block_mask_id] + + # arguments + kwargs = dict( + e=e0, + attn_mask=attn_mask, + grid_sizes=grid_sizes, + freqs=self.freqs, + context=context, + grid_sizes_ref=grid_sizes_ref, + freqs_ref=self.freqs_ref, + context_lens=context_lens, + origin_area=origin_area, + origin_len=origin_len, + use_context_parallel=use_unified_sequence_parallel, + log_scale=log_scale, + ) + if use_unified_sequence_parallel: + from ..utils.xfuser import get_current_chunk, is_evenly_divisible, get_sequence_parallel_world_size + assert is_evenly_divisible(x.shape[1]), ( + f"sequence length ({x.shape[1]}) must be divisible by sequence parallel world size ({get_sequence_parallel_world_size()}) " + ) + x = get_current_chunk(x, dim=1) + + for idx, block in enumerate(self.blocks): + if is_uncondtion and idx==9: + continue + x = block(x, idx, k_cache, v_cache, method='forward_gen', **kwargs) + + # head + x = self.head(x, e) + if use_unified_sequence_parallel: + from ..utils.xfuser import gather_all_chunks + x = gather_all_chunks(x, dim=1) + + # unpatchify + x = self.unpatchify(x, grid_sizes) + return [u for u in x] + + def forward_origin( + self, + x, + clip_fea, + y, + context, + seq_len, + x_ref, + clip_fea_ref, + y_ref, + context_ref, + seq_len_ref, + t, + origin_len, + origin_area, + log_scale=0.0, + use_gradient_checkpointing: bool = False, + use_gradient_checkpointing_offload: bool = False, + ): + # params + x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)] + # embeddings + x = [self.patch_embedding(u.unsqueeze(0)) for u in x] + grid_sizes = torch.stack([ + torch.tensor(u.shape[2:], dtype=torch.long) for u in x + ]) + x = [u.flatten(2).transpose(1, 2) for u in x] + seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long) + assert seq_lens.max() <= seq_len + x = torch.cat([torch.cat([ + u, u.new_zeros(1, seq_len - u.size(1), u.size(2)) + ], dim=1) for u in x]) + + # [reference] + # params + x_ref = [torch.cat([u, v], dim=0) for u, v in zip(x_ref, y_ref)] + # embeddings + x_ref = [self.patch_embedding(u.unsqueeze(0)) for u in x_ref] + grid_sizes_ref = torch.stack([ + torch.tensor(u.shape[2:], dtype=torch.long) for u in x_ref + ]) + x_ref = [u.flatten(2).transpose(1, 2) for u in x_ref] + seq_lens_ref = torch.tensor([u.size(1) for u in x_ref], dtype=torch.long) + assert seq_lens_ref.max() <= seq_len_ref + x_ref = torch.cat([torch.cat([ + u, u.new_zeros(1, seq_len_ref - u.size(1), u.size(2)) + ], dim=1) for u in x_ref]) + + assert (self.dim % self.num_heads) == 0 and (self.dim // self.num_heads) % 2 == 0 + d = self.dim // self.num_heads + self.freqs = torch.cat([ + rope_params(512, d - 4 * (d // 6)), + rope_params(512, 2 * (d // 6)), + rope_params(512, 2 * (d // 6)) + ], dim=1).to(x.device) + + if self.refer_offset_t < 0: + self.refer_offset_t = grid_sizes[0][0].item() + if self.refer_offset_h < 0: + self.refer_offset_h = grid_sizes[0][1].item() + if self.refer_offset_w < 0: + self.refer_offset_w = grid_sizes[0][2].item() + + self.freqs_ref = torch.cat([ + rope_params(512, d - 4 * (d // 6), offset=self.refer_offset_t), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_h), + rope_params(512, 2 * (d // 6), offset=self.refer_offset_w) + ], dim=1).to(x.device) + + # time embeddings + e = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t).to(x.dtype)) + e0 = self.time_projection(e).unflatten(1, (6, self.dim)) + + # time embeddings ref + e_ref = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t*0+1).to(x.dtype)) + e0_ref = self.time_projection(e_ref).unflatten(1, (6, self.dim)) + + # [context] + context_lens = None + context = self.text_embedding(torch.stack([torch.cat([ + u, u.new_zeros(self.text_len - u.size(0), u.size(1)) + ]) for u in context])) + + if self.use_img_emb: + context_clip = self.img_emb(clip_fea) # bs x 257 x dim + context = torch.concat([context_clip, context], dim=1) + + # [context_ref] + context_ref = self.text_embedding(torch.stack([torch.cat([ + u, u.new_zeros(self.text_len - u.size(0), u.size(1)) + ]) for u in context_ref])) + + if self.use_img_emb: + context_clip_ref = self.img_emb(clip_fea_ref) # bs x 257 x dim + context_ref = torch.concat([context_clip_ref, context_ref], dim=1) + + block_mask_id = (origin_len, origin_area[0], origin_area[1]) + if block_mask_id not in self.block_masks: + self.block_masks[block_mask_id] = self.create_mask(origin_len, origin_area, x.device) + attn_mask = self.block_masks[block_mask_id] + + # arguments + ref_args = dict( + e_ref=e0_ref, + grid_sizes_ref=grid_sizes_ref, + freqs_ref=self.freqs_ref, + context_ref=context_ref, + context_lens=context_lens, + ) + gen_args = dict( + e=e0, + attn_mask=attn_mask, + grid_sizes=grid_sizes, + freqs=self.freqs, + context=context, + grid_sizes_ref=grid_sizes_ref, + freqs_ref=self.freqs_ref, + context_lens=context_lens, + origin_area=origin_area, + origin_len=origin_len, + log_scale=log_scale, + ) + for idx, block in enumerate(self.blocks): + x, x_ref = gradient_checkpoint_forward( + block, + use_gradient_checkpointing, + use_gradient_checkpointing_offload, + x, x_ref, ref_args, gen_args, method='forward_origin' + ) + + # head + x = self.head(x, e) + + # unpatchify + x = self.unpatchify(x, grid_sizes) + return [u.float() for u in x] + + def unpatchify(self, x, grid_sizes): + c = self.out_dim + out = [] + for u, v in zip(x, grid_sizes.tolist()): + u = u[:math.prod(v)].view(*v, *self.patch_size, c) + u = torch.einsum('fhwpqrc->cfphqwr', u) + u = u.reshape(c, *[i * j for i, j in zip(v, self.patch_size)]) + out.append(u) + return out diff --git a/diffsynth/pipelines/wan_video.py b/diffsynth/pipelines/wan_video.py index c1e4dfb3a..e2ab76f9c 100644 --- a/diffsynth/pipelines/wan_video.py +++ b/diffsynth/pipelines/wan_video.py @@ -1,4 +1,4 @@ -import torch, types +import torch, types, math import numpy as np from PIL import Image from einops import repeat @@ -27,6 +27,7 @@ from ..models.wan_video_mot import MotWanModel from ..models.wav2vec import WanS2VAudioEncoder from ..models.longcat_video_dit import LongCatVideoTransformer3DModel +from ..models.wan_animate_2_dit import WanAnimate2Transformer class WanVideoPipeline(BasePipeline): @@ -78,9 +79,14 @@ def __init__(self, device=get_device_type(), torch_dtype=torch.bfloat16): WanVideoUnit_WanToDance_ProcessInputs(), WanVideoUnit_WanToDance_RefImageEmbedder(), WanVideoUnit_WanToDance_ImageKeyframesEmbedder(), + WanVideoUnit_Animate2RefPromptEmbedder(), + WanVideoUnit_Animate2CLIPEmbedder(), + WanVideoUnit_Animate2VAEEmbedder(), + WanVideoUnit_Animate2RefKVCacheEmbedder(), ] self.post_units = [ WanVideoPostUnit_S2V(), + WanVideoPostUnit_Animate2(), ] self.model_fn = model_fn_wan_video self.compilable_models = ["dit", "dit2"] @@ -88,6 +94,10 @@ def __init__(self, device=get_device_type(), torch_dtype=torch.bfloat16): def enable_usp(self): from ..utils.xfuser import get_sequence_parallel_world_size, usp_attn_forward, usp_dit_forward, usp_vace_forward + self.sp_size = get_sequence_parallel_world_size() + self.use_unified_sequence_parallel = True + if isinstance(self.dit, WanAnimate2Transformer): + return for block in self.dit.blocks: block.self_attn.forward = types.MethodType(usp_attn_forward, block.self_attn) @@ -104,8 +114,6 @@ def enable_usp(self): for block in self.vace2.vace_blocks: block.self_attn.forward = types.MethodType(usp_attn_forward, block.self_attn) self.vace2.forward = types.MethodType(usp_vace_forward, self.vace2) - self.sp_size = get_sequence_parallel_world_size() - self.use_unified_sequence_parallel = True @staticmethod @@ -223,6 +231,13 @@ def __call__( animate_face_video: list[Image.Image] = None, animate_inpaint_video: list[Image.Image] = None, animate_mask_video: list[Image.Image] = None, + # Wan-Animate-2 + animate2_prompt_ref: str = " ", + animate2_reference_image: Image.Image = None, + animate2_reference_video: list[Image.Image] = None, + animate2_refert_images: list[Image.Image] = None, + animate2_offload_kv: bool = False, + animate2_log_scale: float = 0.0, # VAP vap_video: list[Image.Image] = None, vap_prompt: str = " ", @@ -273,11 +288,13 @@ def __call__( # Inputs inputs_posi = { "prompt": prompt, + "positive": True, "vap_prompt": vap_prompt, "tea_cache_l1_thresh": tea_cache_l1_thresh, "tea_cache_model_id": tea_cache_model_id, "num_inference_steps": num_inference_steps, } inputs_nega = { "negative_prompt": negative_prompt, + "positive": False, "negative_vap_prompt": negative_vap_prompt, "tea_cache_l1_thresh": tea_cache_l1_thresh, "tea_cache_model_id": tea_cache_model_id, "num_inference_steps": num_inference_steps, } @@ -298,6 +315,7 @@ def __call__( "sliding_window_size": sliding_window_size, "sliding_window_stride": sliding_window_stride, "input_audio": input_audio, "audio_sample_rate": audio_sample_rate, "s2v_pose_video": s2v_pose_video, "audio_embeds": audio_embeds, "s2v_pose_latents": s2v_pose_latents, "motion_video": motion_video, "animate_pose_video": animate_pose_video, "animate_face_video": animate_face_video, "animate_inpaint_video": animate_inpaint_video, "animate_mask_video": animate_mask_video, + "animate2_prompt_ref": animate2_prompt_ref, "animate2_reference_image": animate2_reference_image, "animate2_reference_video": animate2_reference_video, "animate2_refert_images": animate2_refert_images, "animate2_offload_kv": animate2_offload_kv, "animate2_log_scale": animate2_log_scale, "vap_video": vap_video, "wantodance_music_path": wantodance_music_path, "wantodance_reference_image": wantodance_reference_image, "wantodance_fps": wantodance_fps, "wantodance_keyframes": wantodance_keyframes, "wantodance_keyframes_mask": wantodance_keyframes_mask, @@ -376,12 +394,14 @@ def process(self, pipe: WanVideoPipeline, height, width, num_frames): class WanVideoUnit_NoiseInitializer(PipelineUnit): def __init__(self): super().__init__( - input_params=("height", "width", "num_frames", "seed", "rand_device", "vace_reference_image"), + input_params=("height", "width", "num_frames", "seed", "rand_device", "vace_reference_image", "animate2_reference_video"), output_params=("noise",) ) - def process(self, pipe: WanVideoPipeline, height, width, num_frames, seed, rand_device, vace_reference_image): + def process(self, pipe: WanVideoPipeline, height, width, num_frames, seed, rand_device, vace_reference_image, animate2_reference_video): length = (num_frames - 1) // 4 + 1 + if animate2_reference_video is not None: + length += 1 if vace_reference_image is not None: f = len(vace_reference_image) if isinstance(vace_reference_image, list) else 1 length += f @@ -390,18 +410,17 @@ def process(self, pipe: WanVideoPipeline, height, width, num_frames, seed, rand_ if vace_reference_image is not None: noise = torch.concat((noise[:, :, -f:], noise[:, :, :-f]), dim=2) return {"noise": noise} - class WanVideoUnit_InputVideoEmbedder(PipelineUnit): def __init__(self): super().__init__( - input_params=("input_video", "noise", "tiled", "tile_size", "tile_stride", "vace_reference_image", "framewise_decoding"), + input_params=("input_video", "noise", "tiled", "tile_size", "tile_stride", "vace_reference_image", "framewise_decoding", "animate2_reference_image", "animate2_reference_video"), output_params=("latents", "input_latents"), onload_model_names=("vae",) ) - def process(self, pipe: WanVideoPipeline, input_video, noise, tiled, tile_size, tile_stride, vace_reference_image, framewise_decoding): + def process(self, pipe: WanVideoPipeline, input_video, noise, tiled, tile_size, tile_stride, vace_reference_image, framewise_decoding, animate2_reference_image, animate2_reference_video): if input_video is None: return {"latents": noise} pipe.load_models_to_device(self.onload_model_names) @@ -417,6 +436,12 @@ def process(self, pipe: WanVideoPipeline, input_video, noise, tiled, tile_size, vace_reference_latents = pipe.vae.encode(vace_reference_image, device=pipe.device).to(dtype=pipe.torch_dtype, device=pipe.device) input_latents = torch.concat([vace_reference_latents, input_latents], dim=2) if pipe.scheduler.training: + if animate2_reference_image is not None and animate2_reference_video is not None: + vh, vw = input_video.shape[-2], input_video.shape[-1] + ref_img = pipe.preprocess_image(animate2_reference_image.resize((vw, vh))).to(pipe.device) # (1, C, H, W) + ref_pixel = ref_img.transpose(0, 1) # (C, 1, H, W) + ref_latent = pipe.vae.encode([ref_pixel.to(pipe.torch_dtype)], device=pipe.device).to(dtype=pipe.torch_dtype, device=pipe.device) + input_latents = torch.concat([ref_latent, input_latents], dim=2) return {"latents": noise, "input_latents": input_latents} else: latents = pipe.scheduler.add_noise(input_latents, noise, timestep=pipe.scheduler.timesteps[0]) @@ -1151,6 +1176,160 @@ def process(self, pipe: WanVideoPipeline, wantodance_keyframes, wantodance_keyfr return {"clip_feature": clip_context, "y": y} +class WanVideoUnit_Animate2RefPromptEmbedder(PipelineUnit): + def __init__(self): + super().__init__( + input_params=("animate2_prompt_ref", "animate2_reference_video"), + output_params=("context_ref",), + onload_model_names=("text_encoder",) + ) + + def process(self, pipe: WanVideoPipeline, animate2_prompt_ref, animate2_reference_video): + if animate2_reference_video is None or animate2_prompt_ref is None: + return {} + pipe.load_models_to_device(self.onload_model_names) + context_ref = WanVideoUnit_PromptEmbedder().encode_prompt(pipe, animate2_prompt_ref) + return {"context_ref": context_ref} + + +class WanVideoUnit_Animate2CLIPEmbedder(PipelineUnit): + def __init__(self): + super().__init__( + input_params=("animate2_reference_image", "animate2_reference_video", "height", "width"), + output_params=("clip_feature", "clip_fea_ref"), + onload_model_names=("image_encoder",) + ) + + def process(self, pipe: WanVideoPipeline, animate2_reference_image, animate2_reference_video, height, width): + if animate2_reference_image is None or animate2_reference_video is None: + return {} + pipe.load_models_to_device(self.onload_model_names) + ref = pipe.preprocess_image(animate2_reference_image.resize((width, height))).to(pipe.device) + clip_feature = pipe.image_encoder.encode_image([ref]).to(dtype=pipe.torch_dtype, device=pipe.device) + ref_video_0 = pipe.preprocess_image(animate2_reference_video[0].resize((width, height))).to(pipe.device) + clip_fea_ref = pipe.image_encoder.encode_image([ref_video_0]).to(dtype=pipe.torch_dtype, device=pipe.device) + return {"clip_feature": clip_feature, "clip_fea_ref": clip_fea_ref} + + +class WanVideoUnit_Animate2VAEEmbedder(PipelineUnit): + def __init__(self): + super().__init__( + input_params=("animate2_reference_image", "animate2_reference_video", "animate2_refert_images", "num_frames", "height", "width", "tiled", "tile_size", "tile_stride"), + output_params=("y", "condition_latents", "condition_y", "grid_sizes", "grid_sizes_ref", "seq_len", "seq_len_ref", "origin_len", "origin_area"), + onload_model_names=("vae",) + ) + + @staticmethod + def animate2_get_i2v_mask(lat_t, lat_h, lat_w, mask_len=1, device="cuda"): + msk = torch.zeros(1, (lat_t - 1) * 4 + 1, lat_h, lat_w, device=device) + msk[:, :mask_len] = 1 + msk = torch.concat([torch.repeat_interleave(msk[:, 0:1], repeats=4, dim=1), msk[:, 1:]], dim=1) + msk = msk.view(1, msk.shape[1] // 4, 4, lat_h, lat_w) + msk = msk.transpose(1, 2)[0] + return msk + + def process(self, pipe: WanVideoPipeline, animate2_reference_image, animate2_reference_video, animate2_refert_images, num_frames, height, width, tiled, tile_size, tile_stride): + if animate2_reference_image is None or animate2_reference_video is None: + return {} + pipe.load_models_to_device(self.onload_model_names) + device, dtype = pipe.device, pipe.torch_dtype + H, W = height, width + T = num_frames + 1 + lat_h, lat_w = H // 8, W // 8 + lat_t = T // 4 + 1 + 1 + + # Reference image -> y_ref (mask + latent), single latent frame + ref_img = pipe.preprocess_image(animate2_reference_image.resize((W, H))).to(device) # (1, C, H, W) + ref_pixel = ref_img.transpose(0, 1) # (C, 1, H, W) + ref_latents = pipe.vae.encode([ref_pixel.to(dtype)], device, tiled, tile_size, tile_stride).to(dtype=dtype, device=device) # (1, 16, 1, lat_h, lat_w) + mask_ref = self.animate2_get_i2v_mask(1, lat_h, lat_w, mask_len=1, device=device) + y_ref = torch.concat([mask_ref, ref_latents[0]]).to(dtype=dtype, device=device) # (20, 1, lat_h, lat_w) + + # Reference-temporal slot: for multi-clip long video, the first `mask_reft_len` temporal + # positions carry the previous clip's tail frames (continuation); the rest are zeros. + # Single/first clip: animate2_refert_images is None -> mask_reft_len == 0 (all zeros). + if animate2_refert_images is not None and len(animate2_refert_images) > 0: + mask_reft_len = len(animate2_refert_images) + refert = pipe.preprocess_video([f.resize((W, H)) for f in animate2_refert_images]).to(device) # (1, 3, mask_reft_len, H, W) + zeros_tail = torch.zeros(3, T - 1 - mask_reft_len, H, W, device=device, dtype=dtype) + reft_vid = torch.concat([refert[0].to(dtype), zeros_tail], dim=1) # (3, T-1, H, W) + else: + mask_reft_len = 0 + reft_vid = torch.zeros(3, T - 1, H, W, device=device, dtype=dtype) + y_reft = pipe.vae.encode([reft_vid], device, tiled, tile_size, tile_stride)[0].to(dtype=dtype, device=device) # (16, lat_t-1, lat_h, lat_w) + msk_reft = self.animate2_get_i2v_mask(lat_t - 1, lat_h, lat_w, mask_len=mask_reft_len, device=device) + y_reft = torch.concat([msk_reft, y_reft]).to(dtype=dtype, device=device) + y = torch.concat([y_ref, y_reft], dim=1) # (20, lat_t, lat_h, lat_w) + + # Reference video -> condition_latents + ref_video = pipe.preprocess_video([f.resize((W, H)) for f in animate2_reference_video[:num_frames]]).to(device) # (1, C, num_frames, H, W) + condition_latents = pipe.vae.encode([ref_video[0].to(dtype)], device, tiled, tile_size, tile_stride).to(dtype=dtype, device=device) # (1, 16, lat_t_c, lat_h, lat_w) + # Reference video -> condition_y (mask + latent), mask_len = T + _, _, lat_t_c, lat_h_c, lat_w_c = condition_latents.shape + condition_y = condition_latents.clone()[0] + condition_msk_y = self.animate2_get_i2v_mask(lat_t_c, lat_h_c, lat_w_c, mask_len=ref_video.shape[2], device=device) + condition_y = torch.concat([condition_msk_y, condition_y]).to(dtype=dtype, device=device) # (20, lat_t_c, lat_h_c, lat_w_c) + + # sequence lengths + grid_sizes = torch.stack([torch.tensor([lat_t, lat_h // 2, lat_w // 2], dtype=torch.long)]) + grid_sizes_ref = torch.stack([torch.tensor([lat_t_c, lat_h_c // 2, lat_w_c // 2], dtype=torch.long)]) + seq_len = int(math.ceil(lat_t * lat_h * lat_w / 4)) + seq_len_ref = int(math.ceil(lat_t_c * lat_h * lat_w / 4)) + + return { + "y": y.unsqueeze(0), + "condition_latents": condition_latents, + "condition_y": condition_y, + "grid_sizes": grid_sizes, + "grid_sizes_ref": grid_sizes_ref, + "seq_len": seq_len, + "seq_len_ref": seq_len_ref, + "origin_len": num_frames, + "origin_area": [W, H], + } + + +class WanVideoUnit_Animate2RefKVCacheEmbedder(PipelineUnit): + def __init__(self): + super().__init__( + input_params=("animate2_reference_video", "condition_latents", "condition_y", "context_ref", "clip_fea_ref", "grid_sizes", "seq_len_ref", "animate2_offload_kv", "use_gradient_checkpointing", "use_gradient_checkpointing_offload", "use_unified_sequence_parallel"), + output_params=("animate2_k_cache", "animate2_v_cache"), + onload_model_names=("dit",) + ) + + def process(self, pipe: WanVideoPipeline, animate2_reference_video, condition_latents, condition_y, context_ref, clip_fea_ref, grid_sizes, seq_len_ref, animate2_offload_kv, use_gradient_checkpointing, use_gradient_checkpointing_offload, use_unified_sequence_parallel): + if animate2_reference_video is None or pipe.scheduler.training: + return {} + pipe.load_models_to_device(self.onload_model_names) + animate2_k_cache, animate2_v_cache = {}, {} + t = pipe.scheduler.timesteps[0] + timestep = t.unsqueeze(0).to(dtype=pipe.torch_dtype, device=pipe.device) + pipe.dit.forward_ref( + condition_latents, + grid_sizes=grid_sizes.to(pipe.device), + k_cache=animate2_k_cache, + v_cache=animate2_v_cache, + clip_fea_ref=clip_fea_ref, + y_ref=[condition_y], + context_ref=[context_ref[0]], + seq_len_ref=seq_len_ref, + t=timestep, + animate2_offload_kv=animate2_offload_kv, + use_unified_sequence_parallel=use_unified_sequence_parallel, + ) + return {"animate2_k_cache": animate2_k_cache, "animate2_v_cache": animate2_v_cache} + + +class WanVideoPostUnit_Animate2(PipelineUnit): + def __init__(self): + super().__init__(input_params=("latents", "animate2_reference_video")) + + def process(self, pipe: WanVideoPipeline, latents, animate2_reference_video): + if animate2_reference_video is None: + return {} + return {"latents": latents[:, :, 1:].to(pipe.torch_dtype)} + + class TeaCache: def __init__(self, num_inference_steps, rel_l1_thresh, model_id): self.num_inference_steps = num_inference_steps @@ -1349,6 +1528,15 @@ def model_fn_wan_video( use_gradient_checkpointing=use_gradient_checkpointing, use_gradient_checkpointing_offload=use_gradient_checkpointing_offload, ) + # Wan-Animate-2 + if isinstance(dit, WanAnimate2Transformer): + return model_fn_wananimate( + dit=dit, latents=latents, timestep=timestep, context=context, + clip_feature=clip_feature, y=y, use_unified_sequence_parallel=use_unified_sequence_parallel, + use_gradient_checkpointing=use_gradient_checkpointing, + use_gradient_checkpointing_offload=use_gradient_checkpointing_offload, + **kwargs, + ) # wan2.2 s2v if audio_embeds is not None: @@ -1715,3 +1903,70 @@ def custom_forward(*inputs): # make compatible with wan video x = torch.cat([origin_ref_latents, x], dim=2) return x + + +def model_fn_wananimate( + dit: WanAnimate2Transformer, + latents: torch.Tensor = None, + timestep: torch.Tensor = None, + context: torch.Tensor = None, + clip_feature: torch.Tensor = None, + y: torch.Tensor = None, + animate2_k_cache: dict = None, + animate2_v_cache: dict = None, + grid_sizes_ref: torch.Tensor = None, + origin_len: int = None, + origin_area: list = None, + seq_len: int = None, + positive: bool = True, + condition_latents: torch.Tensor = None, + clip_fea_ref: torch.Tensor = None, + condition_y: torch.Tensor = None, + context_ref: torch.Tensor = None, + seq_len_ref: int = None, + use_unified_sequence_parallel: bool = False, + animate2_log_scale: float = 0.0, + use_gradient_checkpointing_offload: bool = False, + use_gradient_checkpointing: bool = False, + **kwargs, +): + is_uncondtion = not positive + if animate2_k_cache is None or animate2_v_cache is None: + # training + out = dit.forward_origin( + x=[latents[0]], + clip_fea=clip_feature, + y=[y[0]], + context=[context[0]], + seq_len=seq_len, + x_ref=condition_latents, + clip_fea_ref=clip_fea_ref, + y_ref=[condition_y], + context_ref=[context_ref[0]], + seq_len_ref=seq_len_ref, + t=timestep, + origin_len=origin_len, + origin_area=origin_area, + log_scale=animate2_log_scale, + use_gradient_checkpointing=use_gradient_checkpointing, + use_gradient_checkpointing_offload=use_gradient_checkpointing_offload, + ) + else: + # inference + out = dit.forward_gen( + x=[latents[0]], + k_cache=animate2_k_cache, + v_cache=animate2_v_cache, + clip_fea=clip_feature, + y=[y[0]], + context=[context[0]], + seq_len=seq_len, + t=timestep, + grid_sizes_ref=grid_sizes_ref.to(latents.device), + origin_len=origin_len, + origin_area=origin_area, + is_uncondtion=is_uncondtion, + use_unified_sequence_parallel=use_unified_sequence_parallel, + log_scale=animate2_log_scale, + ) + return out[0].unsqueeze(0) diff --git a/diffsynth/utils/xfuser/__init__.py b/diffsynth/utils/xfuser/__init__.py index cf6025b7e..8a3a4f1f2 100644 --- a/diffsynth/utils/xfuser/__init__.py +++ b/diffsynth/utils/xfuser/__init__.py @@ -1 +1 @@ -from .xdit_context_parallel import usp_attn_forward, usp_dit_forward, usp_vace_forward, get_sequence_parallel_world_size, get_sequence_parallel_rank, get_sp_group, initialize_usp, get_current_chunk, gather_all_chunks +from .xdit_context_parallel import usp_attn_forward, usp_dit_forward, usp_vace_forward, get_sequence_parallel_world_size, get_sequence_parallel_rank, get_sp_group, initialize_usp, get_current_chunk, gather_all_chunks, all_to_all_4d, is_evenly_divisible diff --git a/diffsynth/utils/xfuser/xdit_context_parallel.py b/diffsynth/utils/xfuser/xdit_context_parallel.py index abf0f3fef..23b8508b9 100644 --- a/diffsynth/utils/xfuser/xdit_context_parallel.py +++ b/diffsynth/utils/xfuser/xdit_context_parallel.py @@ -2,6 +2,7 @@ from typing import Optional from einops import rearrange from yunchang.kernels import AttnType +from yunchang.comm.all_to_all import SeqAllToAll4D from xfuser.core.distributed import (get_sequence_parallel_rank, get_sequence_parallel_world_size, get_sp_group) @@ -204,3 +205,15 @@ def gather_all_chunks(x, seq_len=None, dim=1): slices[dim] = slice(0, seq_len) x = x[tuple(slices)] return x + + +def all_to_all_4d(x, scatter_dim, gather_dim): + world_size = get_sequence_parallel_world_size() + if world_size == 1: + return x + return SeqAllToAll4D.apply(get_sp_group().ulysses_group, x, scatter_dim, gather_dim) + + +def is_evenly_divisible(seq_len): + world_size = get_sequence_parallel_world_size() + return seq_len % world_size == 0 diff --git a/docs/en/Model_Details/Wan.md b/docs/en/Model_Details/Wan.md index ce5ca3a3c..632f007b8 100644 --- a/docs/en/Model_Details/Wan.md +++ b/docs/en/Model_Details/Wan.md @@ -91,6 +91,7 @@ graph LR; Wan-AI/Wan2.1-T2V-14B-->krea/krea-realtime-video; Wan-AI/Wan2.1-I2V-14B-720P-->ByteDance/Video-As-Prompt-Wan2.1-14B; Wan-AI/Wan2.1-T2V-14B-->Wan-AI/Wan2.2-Animate-14B; + Wan2.2-Series-->Wan-AI/Wan2.2-Animate-2-14B; Wan-AI/Wan2.1-T2V-14B-->Wan-AI/Wan2.2-S2V-14B; Wan2.2-Series-->Wan-AI/Wan2.2-T2V-A14B; Wan2.2-Series-->Wan-AI/Wan2.2-I2V-A14B; @@ -132,6 +133,8 @@ graph LR; |[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| |[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| |[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B: Distilled](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py)| |[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| |[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| |[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| @@ -184,6 +187,12 @@ Input parameters for `WanVideoPipeline` inference include: * `audio_sample_rate`: Audio sampling rate, default value is 16000. * `s2v_pose_video`: S2V model pose video. * `motion_video`: S2V model motion video. +* `animate2_reference_image`: Wan-Animate-2 reference image, providing the character identity. +* `animate2_reference_video`: Wan-Animate-2 driving video, providing the motion. +* `animate2_prompt_ref`: Wan-Animate-2 reference prompt for the driving video, describing its content. +* `animate2_refert_images`: Wan-Animate-2 reference frame images for temporal continuation, used in long video chunked generation. +* `animate2_offload_kv`: Whether Wan-Animate-2 offloads the reference video KV cache to memory, default value is `False`. +* `animate2_log_scale`: Wan-Animate-2 guidance log scale, default value is 0.0, recommended to set to -1.3 for the distilled model. * `height`: Video height, must be a multiple of 16. * `width`: Video width, must be a multiple of 16. * `num_frames`: Number of video frames, default value is 81, must be a multiple of 4 + 1. diff --git a/docs/zh/Model_Details/Wan.md b/docs/zh/Model_Details/Wan.md index afda86162..55fd0885f 100644 --- a/docs/zh/Model_Details/Wan.md +++ b/docs/zh/Model_Details/Wan.md @@ -92,6 +92,7 @@ graph LR; Wan-AI/Wan2.1-T2V-14B-->meituan-longcat/LongCat-Video; Wan-AI/Wan2.1-I2V-14B-720P-->ByteDance/Video-As-Prompt-Wan2.1-14B; Wan-AI/Wan2.1-T2V-14B-->Wan-AI/Wan2.2-Animate-14B; + Wan2.2-Series-->Wan-AI/Wan2.2-Animate-2-14B; Wan-AI/Wan2.1-T2V-14B-->Wan-AI/Wan2.2-S2V-14B; Wan2.2-Series-->Wan-AI/Wan2.2-T2V-A14B; Wan2.2-Series-->Wan-AI/Wan2.2-I2V-A14B; @@ -133,6 +134,8 @@ graph LR; |[Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py)| |[Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B)|`input_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py)| |[Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B)|`input_image`, `animate_pose_video`, `animate_face_video`, `animate_inpaint_video`, `animate_mask_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py)| +|[Wan-AI/Wan2.2-Animate-2-14B: Distilled](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B)|`animate2_reference_image`, `animate2_reference_video`, `animate2_prompt_ref`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py)| |[Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B)|`input_image`, `input_audio`, `audio_sample_rate`, `s2v_pose_video`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py)| |[PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B)|`vace_control_video`, `vace_reference_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py)| |[PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP)|`input_image`, `end_image`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py)| @@ -185,6 +188,12 @@ DeepSpeed ZeRO 3 训练:Wan 系列模型支持 DeepSpeed ZeRO 3 训练,将 * `audio_sample_rate`: 音频采样率,默认值为 16000。 * `s2v_pose_video`: S2V 模型的姿态视频。 * `motion_video`: S2V 模型的运动视频。 +* `animate2_reference_image`: Wan-Animate-2 模型的参考图像,提供角色身份。 +* `animate2_reference_video`: Wan-Animate-2 模型的驱动视频,提供动作。 +* `animate2_prompt_ref`: Wan-Animate-2 模型驱动视频的参考提示词,描述驱动视频中的内容。 +* `animate2_refert_images`: Wan-Animate-2 模型用于时序续接的参考帧图像,用于长视频分块生成。 +* `animate2_offload_kv`: Wan-Animate-2 模型是否将参考视频的 KV 缓存卸载到内存,默认值为 `False`。 +* `animate2_log_scale`: Wan-Animate-2 模型的引导对数缩放系数,默认值为 0.0,蒸馏模型建议设为 -1.3。 * `height`: 视频高度,需保证高度为 16 的倍数。 * `width`: 视频宽度,需保证宽度为 16 的倍数。 * `num_frames`: 视频帧数,默认值为 81,需保证为 4 的倍数 + 1。 diff --git a/examples/wanvideo/acceleration/Wan2.2-Animate-2-14B-usp.py b/examples/wanvideo/acceleration/Wan2.2-Animate-2-14B-usp.py new file mode 100644 index 000000000..8553a8a1b --- /dev/null +++ b/examples/wanvideo/acceleration/Wan2.2-Animate-2-14B-usp.py @@ -0,0 +1,117 @@ +import torch +import torch.distributed as dist +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + use_usp=True, + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) + +# Character animation: reference image (identity) + reference video (motion) -> animated video. +dataset_snapshot_download( + "DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="wanvideo/Wan2.2-Animate-2-14B/*" +) +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refvideo.mp4").raw_data() +# Example 1: single-clip generation +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +if dist.get_rank() == 0: + save_video(video, "video_Wan2.2-Animate-2-14B.mp4", fps=24, quality=5) + + +# Example 2: multi-clip long-video generation +def generate_long_video(pipe, reference_image, cond_images, clip_len, first_num=1, **kwargs): + assert clip_len > first_num, "clip_len must be greater than first_num" + + def zigzag_padding(array, target_len): + if len(array) == 1: + return [array[0]] * target_len + idx, flip, out = 0, False, [] + while len(out) < target_len: + out.append(array[idx]) + idx += -1 if flip else 1 + if idx == 0 or idx == len(array) - 1: + flip = not flip + return out[:target_len] + + real_len = len(cond_images) + if real_len == 0: + return [] + step = clip_len - first_num + # Precompute clip count so clips of `clip_len` stepping by `step` tile the (padded) driving video. + num_clips = 1 if real_len <= clip_len else (real_len - clip_len + step - 1) // step + 1 + target_len = clip_len + (num_clips - 1) * step + if real_len < target_len: + cond_images = zigzag_padding(cond_images, target_len) + + all_frames = [] + prev_tail = None + for i in range(num_clips): + start = i * step + seg_driving = cond_images[start:start + clip_len] + seg_out = pipe( + animate2_reference_image=reference_image, + animate2_reference_video=seg_driving, + animate2_refert_images=None if i == 0 else prev_tail, + num_frames=clip_len, + **kwargs, + ) + prev_tail = seg_out[-first_num:] + if i != 0: + seg_out = seg_out[first_num:] + all_frames.extend(seg_out) + return all_frames[:real_len] + + +clip_len = 81 +long_video = generate_long_video( + pipe, + reference_image=reference_image, + cond_images=reference_video, + clip_len=clip_len, + first_num=1, + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_offload_kv=True, + height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +if dist.get_rank() == 0: + save_video(long_video, "video_Wan2.2-Animate-2-14B-long.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py b/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py new file mode 100644 index 000000000..2ed119918 --- /dev/null +++ b/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py @@ -0,0 +1,115 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16_distillation.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) + +# Character animation: reference image (identity) + reference video (motion) -> animated video. +dataset_snapshot_download( + "DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="wanvideo/Wan2.2-Animate-2-14B-Distilled/*" +) +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refvideo.mp4").raw_data() + +# Example 1: single-clip generation +num_frames = 81 +# For distilled model, set animate2_log_scale to -1.3, num_inference_steps to 10, and cfg_scale to 1.0. +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + animate2_log_scale=-1.3, + num_frames=num_frames, height=1280, width=720, + num_inference_steps=10, cfg_scale=1.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B-Distilled.mp4", fps=24, quality=5) + + +# Example 2: multi-clip long-video generation +def generate_long_video(pipe, reference_image, cond_images, clip_len, first_num=1, **kwargs): + assert clip_len > first_num, "clip_len must be greater than first_num" + + def zigzag_padding(array, target_len): + if len(array) == 1: + return [array[0]] * target_len + idx, flip, out = 0, False, [] + while len(out) < target_len: + out.append(array[idx]) + idx += -1 if flip else 1 + if idx == 0 or idx == len(array) - 1: + flip = not flip + return out[:target_len] + + real_len = len(cond_images) + if real_len == 0: + return [] + step = clip_len - first_num + # Precompute clip count so clips of `clip_len` stepping by `step` tile the (padded) driving video. + num_clips = 1 if real_len <= clip_len else (real_len - clip_len + step - 1) // step + 1 + target_len = clip_len + (num_clips - 1) * step + if real_len < target_len: + cond_images = zigzag_padding(cond_images, target_len) + + all_frames = [] + prev_tail = None + for i in range(num_clips): + start = i * step + seg_driving = cond_images[start:start + clip_len] + seg_out = pipe( + animate2_reference_image=reference_image, + animate2_reference_video=seg_driving, + animate2_refert_images=None if i == 0 else prev_tail, + num_frames=clip_len, + **kwargs, + ) + prev_tail = seg_out[-first_num:] + if i != 0: + seg_out = seg_out[first_num:] + all_frames.extend(seg_out) + return all_frames[:real_len] + + +# For distilled model, set animate2_log_scale to -1.3, num_inference_steps to 10, and cfg_scale to 1.0. +long_video = generate_long_video( + pipe, + reference_image=reference_image, + cond_images=reference_video, + clip_len=81, + first_num=1, + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_offload_kv=True, + animate2_log_scale=-1.3, + height=1280, width=720, + num_inference_steps=10, cfg_scale=1.0, + seed=0, tiled=True, +) +save_video(long_video, "video_Wan2.2-Animate-2-14B-Distilled-long.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py b/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py new file mode 100644 index 000000000..b74e56da3 --- /dev/null +++ b/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py @@ -0,0 +1,113 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) + +# Character animation: reference image (identity) + reference video (motion) -> animated video. +dataset_snapshot_download( + "DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="wanvideo/Wan2.2-Animate-2-14B/*" +) +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refvideo.mp4").raw_data() +# Example 1: single-clip generation +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B.mp4", fps=24, quality=5) + + +# Example 2: multi-clip long-video generation +def generate_long_video(pipe, reference_image, cond_images, clip_len, first_num=1, **kwargs): + assert clip_len > first_num, "clip_len must be greater than first_num" + + def zigzag_padding(array, target_len): + if len(array) == 1: + return [array[0]] * target_len + idx, flip, out = 0, False, [] + while len(out) < target_len: + out.append(array[idx]) + idx += -1 if flip else 1 + if idx == 0 or idx == len(array) - 1: + flip = not flip + return out[:target_len] + + real_len = len(cond_images) + if real_len == 0: + return [] + step = clip_len - first_num + # Precompute clip count so clips of `clip_len` stepping by `step` tile the (padded) driving video. + num_clips = 1 if real_len <= clip_len else (real_len - clip_len + step - 1) // step + 1 + target_len = clip_len + (num_clips - 1) * step + if real_len < target_len: + cond_images = zigzag_padding(cond_images, target_len) + + all_frames = [] + prev_tail = None + for i in range(num_clips): + start = i * step + seg_driving = cond_images[start:start + clip_len] + seg_out = pipe( + animate2_reference_image=reference_image, + animate2_reference_video=seg_driving, + animate2_refert_images=None if i == 0 else prev_tail, + num_frames=clip_len, + **kwargs, + ) + prev_tail = seg_out[-first_num:] + if i != 0: + seg_out = seg_out[first_num:] + all_frames.extend(seg_out) + return all_frames[:real_len] + + +clip_len = 81 +long_video = generate_long_video( + pipe, + reference_image=reference_image, + cond_images=reference_video, + clip_len=clip_len, + first_num=1, + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_offload_kv=True, + height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(long_video, "video_Wan2.2-Animate-2-14B-long.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py b/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py new file mode 100644 index 000000000..0d4e5aa5d --- /dev/null +++ b/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py @@ -0,0 +1,117 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": "disk", + "offload_device": "disk", + "onload_dtype": torch.bfloat16, + "onload_device": "cpu", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16_distillation.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), + vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, +) + +# Character animation: reference image (identity) + reference video (motion) -> animated video. +dataset_snapshot_download( + "DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="wanvideo/Wan2.2-Animate-2-14B-Distilled/*" +) +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refvideo.mp4").raw_data() + +# Example 1: single-clip generation +num_frames = 81 +# For distilled model, set animate2_log_scale to -1.3, num_inference_steps to 10, and cfg_scale to 1.0. +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + animate2_log_scale=-1.3, + num_frames=num_frames, height=1280, width=720, + num_inference_steps=10, cfg_scale=1.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B-Distilled.mp4", fps=24, quality=5) + + +# Example 2: multi-clip long-video generation +def generate_long_video(pipe, reference_image, cond_images, clip_len, first_num=1, **kwargs): + assert clip_len > first_num, "clip_len must be greater than first_num" + + def zigzag_padding(array, target_len): + if len(array) == 1: + return [array[0]] * target_len + idx, flip, out = 0, False, [] + while len(out) < target_len: + out.append(array[idx]) + idx += -1 if flip else 1 + if idx == 0 or idx == len(array) - 1: + flip = not flip + return out[:target_len] + + real_len = len(cond_images) + if real_len == 0: + return [] + step = clip_len - first_num + # Precompute clip count so clips of `clip_len` stepping by `step` tile the (padded) driving video. + num_clips = 1 if real_len <= clip_len else (real_len - clip_len + step - 1) // step + 1 + target_len = clip_len + (num_clips - 1) * step + if real_len < target_len: + cond_images = zigzag_padding(cond_images, target_len) + + all_frames = [] + prev_tail = None + for i in range(num_clips): + start = i * step + seg_driving = cond_images[start:start + clip_len] + seg_out = pipe( + animate2_reference_image=reference_image, + animate2_reference_video=seg_driving, + animate2_refert_images=None if i == 0 else prev_tail, + num_frames=clip_len, + **kwargs, + ) + prev_tail = seg_out[-first_num:] + if i != 0: + seg_out = seg_out[first_num:] + all_frames.extend(seg_out) + return all_frames[:real_len] + + +clip_len = 81 +# For distilled model, set animate2_log_scale to -1.3, num_inference_steps to 10, and cfg_scale to 1.0. +long_video = generate_long_video( + pipe, + reference_image=reference_image, + cond_images=reference_video, + clip_len=clip_len, + first_num=1, + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_offload_kv=True, + animate2_log_scale=-1.3, + height=1280, width=720, + num_inference_steps=10, cfg_scale=1.0, + seed=0, tiled=True, +) +save_video(long_video, "video_Wan2.2-Animate-2-14B-Distilled-long.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py b/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py new file mode 100644 index 000000000..d89a51294 --- /dev/null +++ b/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py @@ -0,0 +1,114 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": "disk", + "offload_device": "disk", + "onload_dtype": torch.bfloat16, + "onload_device": "cpu", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), + vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, +) + +# Character animation: reference image (identity) + reference video (motion) -> animated video. +dataset_snapshot_download( + "DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="wanvideo/Wan2.2-Animate-2-14B/*" +) +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refvideo.mp4").raw_data() +# Example 1: single-clip generation +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B.mp4", fps=24, quality=5) + + +# Example 2: multi-clip long-video generation +def generate_long_video(pipe, reference_image, cond_images, clip_len, first_num=1, **kwargs): + assert clip_len > first_num, "clip_len must be greater than first_num" + + def zigzag_padding(array, target_len): + if len(array) == 1: + return [array[0]] * target_len + idx, flip, out = 0, False, [] + while len(out) < target_len: + out.append(array[idx]) + idx += -1 if flip else 1 + if idx == 0 or idx == len(array) - 1: + flip = not flip + return out[:target_len] + + real_len = len(cond_images) + if real_len == 0: + return [] + step = clip_len - first_num + # Precompute clip count so clips of `clip_len` stepping by `step` tile the (padded) driving video. + num_clips = 1 if real_len <= clip_len else (real_len - clip_len + step - 1) // step + 1 + target_len = clip_len + (num_clips - 1) * step + if real_len < target_len: + cond_images = zigzag_padding(cond_images, target_len) + + all_frames = [] + prev_tail = None + for i in range(num_clips): + start = i * step + seg_driving = cond_images[start:start + clip_len] + seg_out = pipe( + animate2_reference_image=reference_image, + animate2_reference_video=seg_driving, + animate2_refert_images=None if i == 0 else prev_tail, + num_frames=clip_len, + **kwargs, + ) + prev_tail = seg_out[-first_num:] + if i != 0: + seg_out = seg_out[first_num:] + all_frames.extend(seg_out) + return all_frames[:real_len] + + +clip_len = 81 +long_video = generate_long_video( + pipe, + reference_image=reference_image, + cond_images=reference_video, + clip_len=clip_len, + first_num=1, + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_offload_kv=True, + height=1280, width=720, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(long_video, "video_Wan2.2-Animate-2-14B-long.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh b/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh new file mode 100644 index 000000000..fdc450268 --- /dev/null +++ b/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh @@ -0,0 +1,35 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "wanvideo/Wan2.2-Animate-2-14B-Distilled/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch examples/wanvideo/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled \ + --dataset_metadata_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/metadata.json \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 1 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth,Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth,Wan-AI/Wan2.1-T2V-14B:Wan2.1_VAE.pth" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B-Distilled_full_splited_cache" \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:data_process" + +accelerate launch --config_file examples/wanvideo/model_training/full/accelerate_config_14B.yaml examples/wanvideo/model_training/train.py \ + --dataset_base_path models/train/Wan2.2-Animate-2-14B-Distilled_full_splited_cache \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 100 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:wan_animate_2/wan_animate_2_bf16_distillation.safetensors" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B-Distilled_full" \ + --trainable_models "dit.blocks.0.block.self_attn.q,dit.blocks.0.block.self_attn.k,dit.blocks.0.block.self_attn.v,dit.blocks.1.block.self_attn.q,dit.blocks.1.block.self_attn.k,dit.blocks.1.block.self_attn.v,dit.blocks.2.block.self_attn.q,dit.blocks.2.block.self_attn.k,dit.blocks.2.block.self_attn.v,dit.blocks.3.block.self_attn.q,dit.blocks.3.block.self_attn.k,dit.blocks.3.block.self_attn.v,dit.blocks.4.block.self_attn.q,dit.blocks.4.block.self_attn.k,dit.blocks.4.block.self_attn.v,dit.blocks.5.block.self_attn.q,dit.blocks.5.block.self_attn.k,dit.blocks.5.block.self_attn.v,dit.blocks.6.block.self_attn.q,dit.blocks.6.block.self_attn.k,dit.blocks.6.block.self_attn.v,dit.blocks.7.block.self_attn.q,dit.blocks.7.block.self_attn.k,dit.blocks.7.block.self_attn.v,dit.blocks.8.block.self_attn.q,dit.blocks.8.block.self_attn.k,dit.blocks.8.block.self_attn.v,dit.blocks.9.block.self_attn.q,dit.blocks.9.block.self_attn.k,dit.blocks.9.block.self_attn.v,dit.blocks.10.block.self_attn.q,dit.blocks.10.block.self_attn.k,dit.blocks.10.block.self_attn.v,dit.blocks.11.block.self_attn.q,dit.blocks.11.block.self_attn.k,dit.blocks.11.block.self_attn.v,dit.blocks.12.block.self_attn.q,dit.blocks.12.block.self_attn.k,dit.blocks.12.block.self_attn.v,dit.blocks.13.block.self_attn.q,dit.blocks.13.block.self_attn.k,dit.blocks.13.block.self_attn.v,dit.blocks.14.block.self_attn.q,dit.blocks.14.block.self_attn.k,dit.blocks.14.block.self_attn.v,dit.blocks.15.block.self_attn.q,dit.blocks.15.block.self_attn.k,dit.blocks.15.block.self_attn.v,dit.blocks.16.block.self_attn.q,dit.blocks.16.block.self_attn.k,dit.blocks.16.block.self_attn.v,dit.blocks.17.block.self_attn.q,dit.blocks.17.block.self_attn.k,dit.blocks.17.block.self_attn.v,dit.blocks.18.block.self_attn.q,dit.blocks.18.block.self_attn.k,dit.blocks.18.block.self_attn.v,dit.blocks.19.block.self_attn.q,dit.blocks.19.block.self_attn.k,dit.blocks.19.block.self_attn.v,dit.blocks.20.block.self_attn.q,dit.blocks.20.block.self_attn.k,dit.blocks.20.block.self_attn.v,dit.blocks.21.block.self_attn.q,dit.blocks.21.block.self_attn.k,dit.blocks.21.block.self_attn.v,dit.blocks.22.block.self_attn.q,dit.blocks.22.block.self_attn.k,dit.blocks.22.block.self_attn.v,dit.blocks.23.block.self_attn.q,dit.blocks.23.block.self_attn.k,dit.blocks.23.block.self_attn.v,dit.blocks.24.block.self_attn.q,dit.blocks.24.block.self_attn.k,dit.blocks.24.block.self_attn.v,dit.blocks.25.block.self_attn.q,dit.blocks.25.block.self_attn.k,dit.blocks.25.block.self_attn.v,dit.blocks.26.block.self_attn.q,dit.blocks.26.block.self_attn.k,dit.blocks.26.block.self_attn.v,dit.blocks.27.block.self_attn.q,dit.blocks.27.block.self_attn.k,dit.blocks.27.block.self_attn.v,dit.blocks.28.block.self_attn.q,dit.blocks.28.block.self_attn.k,dit.blocks.28.block.self_attn.v,dit.blocks.29.block.self_attn.q,dit.blocks.29.block.self_attn.k,dit.blocks.29.block.self_attn.v,dit.blocks.30.block.self_attn.q,dit.blocks.30.block.self_attn.k,dit.blocks.30.block.self_attn.v,dit.blocks.31.block.self_attn.q,dit.blocks.31.block.self_attn.k,dit.blocks.31.block.self_attn.v,dit.blocks.32.block.self_attn.q,dit.blocks.32.block.self_attn.k,dit.blocks.32.block.self_attn.v,dit.blocks.33.block.self_attn.q,dit.blocks.33.block.self_attn.k,dit.blocks.33.block.self_attn.v,dit.blocks.34.block.self_attn.q,dit.blocks.34.block.self_attn.k,dit.blocks.34.block.self_attn.v,dit.blocks.35.block.self_attn.q,dit.blocks.35.block.self_attn.k,dit.blocks.35.block.self_attn.v,dit.blocks.36.block.self_attn.q,dit.blocks.36.block.self_attn.k,dit.blocks.36.block.self_attn.v,dit.blocks.37.block.self_attn.q,dit.blocks.37.block.self_attn.k,dit.blocks.37.block.self_attn.v,dit.blocks.38.block.self_attn.q,dit.blocks.38.block.self_attn.k,dit.blocks.38.block.self_attn.v,dit.blocks.39.block.self_attn.q,dit.blocks.39.block.self_attn.k,dit.blocks.39.block.self_attn.v" \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:train" diff --git a/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh b/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh new file mode 100644 index 000000000..c4fdf46b1 --- /dev/null +++ b/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh @@ -0,0 +1,35 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "wanvideo/Wan2.2-Animate-2-14B/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch examples/wanvideo/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B \ + --dataset_metadata_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/metadata.json \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 1 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth,Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth,Wan-AI/Wan2.1-T2V-14B:Wan2.1_VAE.pth" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B_full_splited_cache" \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:data_process" + +accelerate launch --config_file examples/wanvideo/model_training/full/accelerate_config_14B.yaml examples/wanvideo/model_training/train.py \ + --dataset_base_path models/train/Wan2.2-Animate-2-14B_full_splited_cache \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 100 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:wan_animate_2/wan_animate_2_bf16.safetensors" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B_full" \ + --trainable_models "dit.blocks.0.block.self_attn.q,dit.blocks.0.block.self_attn.k,dit.blocks.0.block.self_attn.v,dit.blocks.1.block.self_attn.q,dit.blocks.1.block.self_attn.k,dit.blocks.1.block.self_attn.v,dit.blocks.2.block.self_attn.q,dit.blocks.2.block.self_attn.k,dit.blocks.2.block.self_attn.v,dit.blocks.3.block.self_attn.q,dit.blocks.3.block.self_attn.k,dit.blocks.3.block.self_attn.v,dit.blocks.4.block.self_attn.q,dit.blocks.4.block.self_attn.k,dit.blocks.4.block.self_attn.v,dit.blocks.5.block.self_attn.q,dit.blocks.5.block.self_attn.k,dit.blocks.5.block.self_attn.v,dit.blocks.6.block.self_attn.q,dit.blocks.6.block.self_attn.k,dit.blocks.6.block.self_attn.v,dit.blocks.7.block.self_attn.q,dit.blocks.7.block.self_attn.k,dit.blocks.7.block.self_attn.v,dit.blocks.8.block.self_attn.q,dit.blocks.8.block.self_attn.k,dit.blocks.8.block.self_attn.v,dit.blocks.9.block.self_attn.q,dit.blocks.9.block.self_attn.k,dit.blocks.9.block.self_attn.v,dit.blocks.10.block.self_attn.q,dit.blocks.10.block.self_attn.k,dit.blocks.10.block.self_attn.v,dit.blocks.11.block.self_attn.q,dit.blocks.11.block.self_attn.k,dit.blocks.11.block.self_attn.v,dit.blocks.12.block.self_attn.q,dit.blocks.12.block.self_attn.k,dit.blocks.12.block.self_attn.v,dit.blocks.13.block.self_attn.q,dit.blocks.13.block.self_attn.k,dit.blocks.13.block.self_attn.v,dit.blocks.14.block.self_attn.q,dit.blocks.14.block.self_attn.k,dit.blocks.14.block.self_attn.v,dit.blocks.15.block.self_attn.q,dit.blocks.15.block.self_attn.k,dit.blocks.15.block.self_attn.v,dit.blocks.16.block.self_attn.q,dit.blocks.16.block.self_attn.k,dit.blocks.16.block.self_attn.v,dit.blocks.17.block.self_attn.q,dit.blocks.17.block.self_attn.k,dit.blocks.17.block.self_attn.v,dit.blocks.18.block.self_attn.q,dit.blocks.18.block.self_attn.k,dit.blocks.18.block.self_attn.v,dit.blocks.19.block.self_attn.q,dit.blocks.19.block.self_attn.k,dit.blocks.19.block.self_attn.v,dit.blocks.20.block.self_attn.q,dit.blocks.20.block.self_attn.k,dit.blocks.20.block.self_attn.v,dit.blocks.21.block.self_attn.q,dit.blocks.21.block.self_attn.k,dit.blocks.21.block.self_attn.v,dit.blocks.22.block.self_attn.q,dit.blocks.22.block.self_attn.k,dit.blocks.22.block.self_attn.v,dit.blocks.23.block.self_attn.q,dit.blocks.23.block.self_attn.k,dit.blocks.23.block.self_attn.v,dit.blocks.24.block.self_attn.q,dit.blocks.24.block.self_attn.k,dit.blocks.24.block.self_attn.v,dit.blocks.25.block.self_attn.q,dit.blocks.25.block.self_attn.k,dit.blocks.25.block.self_attn.v,dit.blocks.26.block.self_attn.q,dit.blocks.26.block.self_attn.k,dit.blocks.26.block.self_attn.v,dit.blocks.27.block.self_attn.q,dit.blocks.27.block.self_attn.k,dit.blocks.27.block.self_attn.v,dit.blocks.28.block.self_attn.q,dit.blocks.28.block.self_attn.k,dit.blocks.28.block.self_attn.v,dit.blocks.29.block.self_attn.q,dit.blocks.29.block.self_attn.k,dit.blocks.29.block.self_attn.v,dit.blocks.30.block.self_attn.q,dit.blocks.30.block.self_attn.k,dit.blocks.30.block.self_attn.v,dit.blocks.31.block.self_attn.q,dit.blocks.31.block.self_attn.k,dit.blocks.31.block.self_attn.v,dit.blocks.32.block.self_attn.q,dit.blocks.32.block.self_attn.k,dit.blocks.32.block.self_attn.v,dit.blocks.33.block.self_attn.q,dit.blocks.33.block.self_attn.k,dit.blocks.33.block.self_attn.v,dit.blocks.34.block.self_attn.q,dit.blocks.34.block.self_attn.k,dit.blocks.34.block.self_attn.v,dit.blocks.35.block.self_attn.q,dit.blocks.35.block.self_attn.k,dit.blocks.35.block.self_attn.v,dit.blocks.36.block.self_attn.q,dit.blocks.36.block.self_attn.k,dit.blocks.36.block.self_attn.v,dit.blocks.37.block.self_attn.q,dit.blocks.37.block.self_attn.k,dit.blocks.37.block.self_attn.v,dit.blocks.38.block.self_attn.q,dit.blocks.38.block.self_attn.k,dit.blocks.38.block.self_attn.v,dit.blocks.39.block.self_attn.q,dit.blocks.39.block.self_attn.k,dit.blocks.39.block.self_attn.v" \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:train" diff --git a/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh b/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh new file mode 100644 index 000000000..481086f05 --- /dev/null +++ b/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh @@ -0,0 +1,40 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "wanvideo/Wan2.2-Animate-2-14B-Distilled/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch examples/wanvideo/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled \ + --dataset_metadata_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/metadata.json \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 1 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth,Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth,Wan-AI/Wan2.1-T2V-14B:Wan2.1_VAE.pth" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B-Distilled_lora_splited_cache" \ + --lora_base_model "dit" \ + --lora_target_modules "self_attn.q,self_attn.k,self_attn.v" \ + --lora_rank 32 \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:data_process" + +accelerate launch --config_file examples/wanvideo/model_training/full/accelerate_config_14B.yaml examples/wanvideo/model_training/train.py \ + --dataset_base_path models/train/Wan2.2-Animate-2-14B-Distilled_lora_splited_cache \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 100 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:wan_animate_2/wan_animate_2_bf16_distillation.safetensors" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B-Distilled_lora" \ + --lora_base_model "dit" \ + --lora_target_modules "self_attn.q,self_attn.k,self_attn.v" \ + --lora_rank 32 \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:train" diff --git a/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh b/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh new file mode 100644 index 000000000..2a812fcf2 --- /dev/null +++ b/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh @@ -0,0 +1,40 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "wanvideo/Wan2.2-Animate-2-14B/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch examples/wanvideo/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B \ + --dataset_metadata_path data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/metadata.json \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 1 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth,Wan-AI/Wan2.2-Animate-2-14B:videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth,Wan-AI/Wan2.1-T2V-14B:Wan2.1_VAE.pth" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B_lora_splited_cache" \ + --lora_base_model "dit" \ + --lora_target_modules "self_attn.q,self_attn.k,self_attn.v" \ + --lora_rank 32 \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:data_process" + +accelerate launch --config_file examples/wanvideo/model_training/full/accelerate_config_14B.yaml examples/wanvideo/model_training/train.py \ + --dataset_base_path models/train/Wan2.2-Animate-2-14B_lora_splited_cache \ + --data_file_keys "video,animate2_reference_image,animate2_reference_video" \ + --height 640 \ + --width 352 \ + --num_frames 81 \ + --dataset_repeat 100 \ + --model_id_with_origin_paths "Wan-AI/Wan2.2-Animate-2-14B:wan_animate_2/wan_animate_2_bf16.safetensors" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/Wan2.2-Animate-2-14B_lora" \ + --lora_base_model "dit" \ + --lora_target_modules "self_attn.q,self_attn.k,self_attn.v" \ + --lora_rank 32 \ + --extra_inputs "animate2_prompt_ref,animate2_reference_image,animate2_reference_video" \ + --use_gradient_checkpointing \ + --task "sft:train" diff --git a/examples/wanvideo/model_training/train.py b/examples/wanvideo/model_training/train.py index ce8fedcd5..45c941356 100644 --- a/examples/wanvideo/model_training/train.py +++ b/examples/wanvideo/model_training/train.py @@ -70,7 +70,7 @@ def parse_extra_inputs(self, data, extra_inputs, inputs_shared): inputs_shared["input_image"] = data["video"][0] elif extra_input == "end_image": inputs_shared["end_image"] = data["video"][-1] - elif extra_input == "reference_image" or extra_input == "vace_reference_image": + elif extra_input in ("reference_image", "vace_reference_image", "animate2_reference_image"): inputs_shared[extra_input] = data[extra_input][0] else: inputs_shared[extra_input] = data[extra_input] diff --git a/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py b/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py new file mode 100644 index 000000000..acb601ca7 --- /dev/null +++ b/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py @@ -0,0 +1,47 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.core import load_state_dict +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16_distillation.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) +state_dict = load_state_dict("models/train/Wan2.2-Animate-2-14B-Distilled_full/epoch-1.safetensors") +state_dict = {k.replace(".block.self_attn.", ".block.module.self_attn."): v for k, v in state_dict.items()} +pipe.dit.load_state_dict(state_dict, strict=False) + +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refvideo.mp4").raw_data() +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=640, width=352, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B-Distilled.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py b/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py new file mode 100644 index 000000000..2643b37c7 --- /dev/null +++ b/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py @@ -0,0 +1,47 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.core import load_state_dict +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) +state_dict = load_state_dict("models/train/Wan2.2-Animate-2-14B_full/epoch-1.safetensors") +state_dict = {k.replace(".block.self_attn.", ".block.module.self_attn."): v for k, v in state_dict.items()} +pipe.dit.load_state_dict(state_dict, strict=False) + +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refvideo.mp4").raw_data() +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=640, width=352, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py b/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py new file mode 100644 index 000000000..44c2b6a71 --- /dev/null +++ b/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py @@ -0,0 +1,44 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16_distillation.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) +pipe.load_lora(pipe.dit, "models/train/Wan2.2-Animate-2-14B-Distilled_lora/epoch-4.safetensors", alpha=1) + +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B-Distilled/refvideo.mp4").raw_data() +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=640, width=352, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B-Distilled.mp4", fps=24, quality=5) diff --git a/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py b/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py new file mode 100644 index 000000000..6e3e2a4a6 --- /dev/null +++ b/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py @@ -0,0 +1,44 @@ +import torch +from PIL import Image +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig + +vram_config = { + "offload_dtype": torch.bfloat16, + "offload_device": "cpu", + "onload_dtype": torch.bfloat16, + "onload_device": "cuda", + "preparing_dtype": torch.bfloat16, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = WanVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="wan_animate_2/wan_animate_2_bf16.safetensors", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_t5_umt5-xxl-enc-bf16.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth", **vram_config), + ModelConfig(model_id="Wan-AI/Wan2.1-T2V-14B", origin_file_pattern="Wan2.1_VAE.pth", **vram_config), + ], + tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.2-Animate-2-14B", origin_file_pattern="videomodel/Wan-AI/umt5-xxl/"), +) +pipe.load_lora(pipe.dit, "models/train/Wan2.2-Animate-2-14B_lora/epoch-4.safetensors", alpha=1) + +reference_image = Image.open("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refimage.jpg").convert("RGB") +reference_video = VideoData("data/diffsynth_example_dataset/wanvideo/Wan2.2-Animate-2-14B/refvideo.mp4").raw_data() +num_frames = 81 +video = pipe( + prompt="人物外观描述:一名长黑发女性,穿着白色半透明蕾丝长袖上衣,衣身带有花卉刺绣,下身搭配白色百褶短裙和黑色腰带,脚穿米白色厚底运动鞋。 背景描述:背景为现代室内空间,墙面和柜体以浅灰色为主,后方设有两扇深色落地窗或玻璃门,顶部安装长条形灯具,中央有一块浅色长方形台面。", + negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走", + animate2_prompt_ref="视频中的人在做动作,背景静止", + animate2_reference_image=reference_image, + animate2_reference_video=reference_video[:num_frames], + animate2_offload_kv=True, + num_frames=num_frames, height=640, width=352, + num_inference_steps=40, cfg_scale=3.0, + seed=0, tiled=True, +) +save_video(video, "video_Wan2.2-Animate-2-14B.mp4", fps=24, quality=5)