diff --git a/README.md b/README.md index 27dcb151..66b9fb3e 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ We believe that a well-developed open-source code framework can lower the thresh - **August 3, 2026** MiniMax-H3 open-sourced, welcome a new member to the video model family! Support includes text-to-video-audio generation, keyframe-guided generation, reference-driven generation, low VRAM inference, and NF4-quantized inference. For details, please refer to the [documentation](/docs/en/Model_Details/MiniMax-H3.md) and [example code](/examples/minimax_h3/). -- **July 28, 2026** LingBot-Video open-sourced, welcome a new member to the video model family! Support includes text-to-video, image-to-video and text-to-image generation, low VRAM inference, and LoRA / full training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/LingBot-Video.md) and [example code](/examples/lingbot_video/). Huge thanks to [NancyFyong](https://github.com/NancyFyong) for contributing the integration of this model! +- **July 28, 2026** LingBot-Video open-sourced, welcome a new member to the video model family! This release includes two variants, Dense-1.3B and MoE-30B-A3B (30B total parameters, ~3B active per token), both supporting text-to-video, image-to-video and text-to-image generation, low VRAM inference, and LoRA / full training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/LingBot-Video.md) and [example code](/examples/lingbot_video/). Huge thanks to [NancyFyong](https://github.com/NancyFyong) for contributing the integration of this model! - **July 21, 2026** We have open-sourced [DiffSynth-Studio Model Integration Skills](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills). This is a composable collection of Agent Skills that automates the entire workflow of integrating external diffusion models into DiffSynth-Studio, significantly improving the standardization and efficiency of model integration. Get started with the [example](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator/file/view/master/example.md?status=1)! @@ -1520,6 +1520,9 @@ Example code for LingBot-Video is available at: [/examples/lingbot_video/](/exam |[Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py)| |[Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py)| |[Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py)|-|-|-|-| +|[Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py)|-|-|-|-| diff --git a/README_zh.md b/README_zh.md index 90552413..91f7e6e9 100644 --- a/README_zh.md +++ b/README_zh.md @@ -38,7 +38,7 @@ DiffSynth 目前包括两个开源项目: - **2026年8月3日** MiniMax-H3 开源,欢迎加入视频生成模型家族!支持文生音视频、首尾帧引导生成、参考驱动生成、低显存推理以及 NF4 量化推理。详情请参考[文档](/docs/zh/Model_Details/MiniMax-H3.md)和[示例代码](/examples/minimax_h3/)。 -- **2026年7月28日** LingBot-Video 开源,欢迎加入视频生成模型家族!支持文生视频、图生视频、文生图推理、低显存推理以及 LoRA / 全量训练能力。详情请参考[文档](/docs/zh/Model_Details/LingBot-Video.md)和[示例代码](/examples/lingbot_video/)。特别感谢 [NancyFyong](https://github.com/NancyFyong) 贡献了本模型的接入代码! +- **2026年7月28日** LingBot-Video 开源,欢迎加入视频生成模型家族!本次开源包含 Dense-1.3B 和 MoE-30B-A3B 两个版本(MoE 版总参数量 30B、每个 token 激活约 3B),均支持文生视频、图生视频、文生图推理、低显存推理以及 LoRA / 全量训练能力。详情请参考[文档](/docs/zh/Model_Details/LingBot-Video.md)和[示例代码](/examples/lingbot_video/)。特别感谢 [NancyFyong](https://github.com/NancyFyong) 贡献了本模型的接入代码! - **2026年7月21日** 我们开源了 [DiffSynth-Studio Model Integration Skills](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills)。这是一套可组合的 Agent Skill 合集,将外部扩散模型接入 DiffSynth-Studio 的全流程自动化,大幅提升模型接入标准化程度与效率。从[使用示例](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator/file/view/master/example.md?status=1)开始体验吧! @@ -1520,6 +1520,9 @@ LingBot-Video 的示例代码位于:[/examples/lingbot_video/](/examples/lingb |[Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py)| |[Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py)| |[Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py)|-|-|-|-| +|[Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py)|[code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py)|[code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py)|-|-|-|-| diff --git a/diffsynth/configs/model_configs.py b/diffsynth/configs/model_configs.py index ff1737f0..bd3df6cf 100644 --- a/diffsynth/configs/model_configs.py +++ b/diffsynth/configs/model_configs.py @@ -1354,6 +1354,14 @@ "model_class": "diffsynth.models.lingbot_video_dit.LingBotVideoDiT", "state_dict_converter": "diffsynth.utils.state_dict_converters.lingbot_video_dit.LingBotVideoDiTStateDictConverter", }, + { + # Example: ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors") + "model_hash": "65b83aa625cd362ff5ff3409fb367a6f", + "model_name": "lingbot_video_dit", + "model_class": "diffsynth.models.lingbot_video_dit.LingBotVideoDiT", + "state_dict_converter": "diffsynth.utils.state_dict_converters.lingbot_video_dit.LingBotVideoDiTStateDictConverter", + "extra_kwargs": {'depth': 48, 'axes_lens': (4096, 512, 512), 'num_experts': 128, 'moe_intermediate_size': 768, 'n_group': 4, 'topk_group': 2, 'n_shared_experts': 1, 'routed_scaling_factor': 2.5}, + }, ] minimax_h3_series = [ diff --git a/diffsynth/configs/vram_management_module_maps.py b/diffsynth/configs/vram_management_module_maps.py index ecca0d9a..7a9231d6 100644 --- a/diffsynth/configs/vram_management_module_maps.py +++ b/diffsynth/configs/vram_management_module_maps.py @@ -451,7 +451,9 @@ "transformers.models.qwen3_vl.modeling_qwen3_vl.Qwen3VLVisionRotaryEmbedding": "diffsynth.core.vram.layers.AutoWrappedModule", }, "diffsynth.models.lingbot_video_dit.LingBotVideoDiT": { - "diffsynth.models.lingbot_video_dit.LingBotVideoBlock": "diffsynth.core.vram.layers.AutoWrappedModule", + "diffsynth.models.lingbot_video_dit.LingBotVideoBlock": "diffsynth.core.vram.layers.AutoWrappedNonRecurseModule", + "diffsynth.models.lingbot_video_dit.LingBotVideoRouter": "diffsynth.core.vram.layers.AutoWrappedNonRecurseModule", + "diffsynth.models.lingbot_video_dit.LingBotVideoGroupedExperts": "diffsynth.core.vram.layers.AutoWrappedNonRecurseModule", "diffsynth.models.lingbot_video_dit.LingBotVideoRMSNorm": "diffsynth.core.vram.layers.AutoWrappedModule", "torch.nn.Linear": "diffsynth.core.vram.layers.AutoWrappedLinear", }, diff --git a/diffsynth/models/lingbot_video_dit.py b/diffsynth/models/lingbot_video_dit.py index c98557fc..9fbe44df 100644 --- a/diffsynth/models/lingbot_video_dit.py +++ b/diffsynth/models/lingbot_video_dit.py @@ -212,7 +212,7 @@ def __init__(self, hidden_size, num_experts, top_k, score_func, norm_topk_prob, self.topk_group = topk_group self.route_scale = route_scale self.weight = nn.Parameter(torch.empty(num_experts, hidden_size)) - self.register_buffer("e_score_correction_bias", torch.zeros(num_experts), persistent=True) + self.e_score_correction_bias = nn.Parameter(torch.zeros(num_experts), requires_grad=False) def _group_limited_topk(self, scores_for_choice): seq_len = scores_for_choice.shape[0] @@ -228,12 +228,12 @@ def _group_limited_topk(self, scores_for_choice): def forward(self, tokens: torch.Tensor): with torch.amp.autocast(tokens.device.type, enabled=False): - logits = F.linear(tokens.float(), self.weight.float()) + logits = F.linear(tokens.float(), self.weight.to(device=tokens.device, dtype=torch.float32)) if self.score_func == "softmax": scores = F.softmax(logits, dim=-1) else: scores = logits.sigmoid() - scores_for_choice = scores + self.e_score_correction_bias.unsqueeze(0) + scores_for_choice = scores + self.e_score_correction_bias.to(device=scores.device, dtype=scores.dtype).unsqueeze(0) if self.n_group is not None and self.n_group > 1: top_indices = self._group_limited_topk(scores_for_choice) else: @@ -329,13 +329,14 @@ def _unpad_grouped_tokens(output, input_shape, permuted_indices): return unpermuted[:-1] def _run_grouped_experts(self, tokens, counts): - if not hasattr(torch, "_grouped_mm"): + if not hasattr(torch, "_grouped_mm") or tokens.device.type != "cuda": return self._run_experts_for_loop(tokens, counts) input_shape, padded_tokens, permuted_indices, aligned_counts = self._pad_grouped_tokens(tokens, counts) offsets = torch.cumsum(aligned_counts, dim=0, dtype=torch.int32) - h = F.silu(torch._grouped_mm(padded_tokens.bfloat16(), self.experts.w1.bfloat16().transpose(-2, -1), offs=offsets)) - h = h * torch._grouped_mm(padded_tokens.bfloat16(), self.experts.w3.bfloat16().transpose(-2, -1), offs=offsets) - out = torch._grouped_mm(h, self.experts.w2.bfloat16().transpose(-2, -1), offs=offsets).type_as(padded_tokens) + w1, w2, w3 = (w.to(device=tokens.device, dtype=torch.bfloat16) for w in (self.experts.w1, self.experts.w2, self.experts.w3)) + h = F.silu(torch._grouped_mm(padded_tokens.bfloat16(), w1.transpose(-2, -1), offs=offsets)) + h = h * torch._grouped_mm(padded_tokens.bfloat16(), w3.transpose(-2, -1), offs=offsets) + out = torch._grouped_mm(h, w2.transpose(-2, -1), offs=offsets).type_as(padded_tokens) return self._unpad_grouped_tokens(out, input_shape, permuted_indices) def _run_experts_for_loop(self, tokens, counts): @@ -345,9 +346,9 @@ def _run_experts_for_loop(self, tokens, counts): for expert_idx, expert_tokens in enumerate(splits): if expert_tokens.numel() == 0: continue - h = F.silu(expert_tokens @ self.experts.w1[expert_idx].transpose(-2, -1)) - h = h * (expert_tokens @ self.experts.w3[expert_idx].transpose(-2, -1)) - h = h @ self.experts.w2[expert_idx].transpose(-2, -1) + h = F.silu(expert_tokens @ self.experts.w1[expert_idx].to(device=expert_tokens.device, dtype=expert_tokens.dtype).transpose(-2, -1)) + h = h * (expert_tokens @ self.experts.w3[expert_idx].to(device=expert_tokens.device, dtype=expert_tokens.dtype).transpose(-2, -1)) + h = h @ self.experts.w2[expert_idx].to(device=expert_tokens.device, dtype=expert_tokens.dtype).transpose(-2, -1) outputs.append(h) if not outputs: return tokens.new_zeros(tokens.shape) @@ -420,7 +421,7 @@ def forward(self, x, temb6, rotary_emb, attention_mask=None, moe_padding_mask=No "LingBotVideoBlock expects token-level temb6 with shape (B*S, 6D); " f"got {tuple(temb6.shape)} for hidden states {tuple(x.shape)}." ) - mod = temb6.view(x.shape[0], x.shape[1], -1) + self.scale_shift_table.unsqueeze(0) + mod = temb6.view(x.shape[0], x.shape[1], -1) + self.scale_shift_table.to(dtype=temb6.dtype, device=temb6.device).unsqueeze(0) shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = mod.chunk(6, dim=-1) gate_msa, gate_mlp = gate_msa.tanh(), gate_mlp.tanh() scale_msa, scale_mlp = 1.0 + scale_msa, 1.0 + scale_mlp @@ -446,8 +447,6 @@ class LingBotVideoDiT(nn.Module): (`num_experts>0`, FFN = sparse MoE) variants from a single class. """ - _supports_gradient_checkpointing = True - _no_split_modules = ["LingBotVideoBlock"] _repeated_blocks = ["LingBotVideoBlock"] def __init__( diff --git a/docs/en/Model_Details/LingBot-Video.md b/docs/en/Model_Details/LingBot-Video.md index 61532103..0e6768ae 100644 --- a/docs/en/Model_Details/LingBot-Video.md +++ b/docs/en/Model_Details/LingBot-Video.md @@ -75,6 +75,9 @@ save_video(video, "video.mp4", fps=15, quality=10) |[Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py)| |[Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py)| |[Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py)|-|-|-|-| +|[Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py)|-|-|-|-| ## Model Inference diff --git a/docs/zh/Model_Details/LingBot-Video.md b/docs/zh/Model_Details/LingBot-Video.md index eca7dfb2..74c111f3 100644 --- a/docs/zh/Model_Details/LingBot-Video.md +++ b/docs/zh/Model_Details/LingBot-Video.md @@ -75,6 +75,9 @@ save_video(video, "video.mp4", fps=15, quality=10) |[Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py)| |[Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py)| |[Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py)|-|-|-|-| +|[Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py)| +|[Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py)|-|-|-|-| ## 模型推理 diff --git a/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py new file mode 100644 index 00000000..1264f521 --- /dev/null +++ b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py @@ -0,0 +1,32 @@ +import json +import torch +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) + +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2i/*", +) +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2i/t2i_example.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +frames = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt_image, + height=480, width=832, num_frames=1, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +frames[0].save("image_lingbot-video-moe-30b-a3b_t2i.png") diff --git a/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py new file mode 100644 index 00000000..c2d20f90 --- /dev/null +++ b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py @@ -0,0 +1,50 @@ +import torch +import json +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) + +# --- Text-to-video ------------------------------------------------------------------- +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2v/*", +) +# LingBot-Video is trained on structured-JSON captions, not free-form prose. This example +# runs on a released in-distribution caption; see the bottom for turning a brief idea into +# such a caption with the two-stage prompt rewriter. +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_t2v.mp4", fps=15, quality=10) + +# --- Video-to-video ------------------------------------------------------------------ +# denoising_strength < 1 keeps part of the input structure. +input_video = VideoData("video_lingbot-video-moe-30b-a3b_t2v.mp4", height=480, width=832) +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_video=input_video, denoising_strength=0.7, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=1, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_v2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py new file mode 100644 index 00000000..931630ad --- /dev/null +++ b/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py @@ -0,0 +1,39 @@ +import os +import json +import torch +from PIL import Image +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) + +# The condition first frame and its paired caption ship in the example dataset. +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*", +) +base = "data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v" +with open(os.path.join(base, "ti2v_example.json"), "r", encoding="utf-8") as f: + caption = json.load(f) +input_image = Image.open(os.path.join(base, "ti2v_first_frame.png")).convert("RGB") + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_image=input_image, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_ti2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py new file mode 100644 index 00000000..92cec75d --- /dev/null +++ b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py @@ -0,0 +1,44 @@ +import json +import torch +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": "disk", + "offload_device": "disk", + "onload_dtype": torch.float8_e4m3fn, + "onload_device": "cpu", + "preparing_dtype": torch.float8_e4m3fn, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors", **vram_config), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), + vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, +) + +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2i/*", +) +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2i/t2i_example.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +frames = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt_image, + height=480, width=832, num_frames=1, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +frames[0].save("image_lingbot-video-moe-30b-a3b_t2i_low_vram.png") diff --git a/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py new file mode 100644 index 00000000..652569c1 --- /dev/null +++ b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py @@ -0,0 +1,58 @@ +import torch +import json +from diffsynth.utils.data import save_video, VideoData +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": "disk", + "offload_device": "disk", + "onload_dtype": torch.float8_e4m3fn, + "onload_device": "cpu", + "preparing_dtype": torch.float8_e4m3fn, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors", **vram_config), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), + vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, +) + +# --- Text-to-video ------------------------------------------------------------------- +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2v/*", +) +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_t2v.mp4", fps=15, quality=10) + +# --- Video-to-video ------------------------------------------------------------------ +input_video = VideoData("video_lingbot-video-moe-30b-a3b_t2v.mp4", height=480, width=832) +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_video=input_video, denoising_strength=0.7, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=1, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_v2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py new file mode 100644 index 00000000..7b8de30d --- /dev/null +++ b/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py @@ -0,0 +1,51 @@ +import os +import json +import torch +from PIL import Image +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +vram_config = { + "offload_dtype": "disk", + "offload_device": "disk", + "onload_dtype": torch.float8_e4m3fn, + "onload_device": "cpu", + "preparing_dtype": torch.float8_e4m3fn, + "preparing_device": "cuda", + "computation_dtype": torch.bfloat16, + "computation_device": "cuda", +} + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors", **vram_config), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), + vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, +) + +# The condition first frame and its paired caption ship in the example dataset. +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*", +) +base = "data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v" +with open(os.path.join(base, "ti2v_example.json"), "r", encoding="utf-8") as f: + caption = json.load(f) +input_image = Image.open(os.path.join(base, "ti2v_first_frame.png")).convert("RGB") + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_image=input_image, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_ti2v_low_vram.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_training/full/accelerate_config_moe.yaml b/examples/lingbot_video/model_training/full/accelerate_config_moe.yaml new file mode 100644 index 00000000..3875a9da --- /dev/null +++ b/examples/lingbot_video/model_training/full/accelerate_config_moe.yaml @@ -0,0 +1,22 @@ +compute_environment: LOCAL_MACHINE +debug: false +deepspeed_config: + gradient_accumulation_steps: 1 + offload_optimizer_device: cpu + offload_param_device: cpu + zero3_init_flag: false + zero_stage: 2 +distributed_type: DEEPSPEED +downcast_bf16: 'no' +enable_cpu_affinity: false +machine_rank: 0 +main_training_function: main +mixed_precision: bf16 +num_machines: 1 +num_processes: 8 +rdzv_backend: static +same_network: true +tpu_env: [] +tpu_use_cluster: false +tpu_use_sudo: false +use_cpu: false diff --git a/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh b/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh new file mode 100644 index 00000000..0c242886 --- /dev/null +++ b/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh @@ -0,0 +1,19 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-moe-30b-a3b_t2v/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch --config_file examples/lingbot_video/model_training/full/accelerate_config_moe.yaml examples/lingbot_video/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v \ + --dataset_metadata_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/metadata.json \ + --data_file_keys "video" \ + --height 480 \ + --width 832 \ + --num_frames 81 \ + --dataset_repeat 50 \ + --model_id_with_origin_paths "Robbyant/lingbot-video-moe-30b-a3b:transformer/diffusion_pytorch_model*.safetensors,Qwen/Qwen3-VL-4B-Instruct:*.safetensors,Robbyant/lingbot-video-moe-30b-a3b:vae/diffusion_pytorch_model.safetensors" \ + --processor_path "Qwen/Qwen3-VL-4B-Instruct:" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/lingbot-video-moe-30b-a3b_t2v_full" \ + --trainable_models "dit" \ + --use_gradient_checkpointing \ + --use_gradient_checkpointing_offload diff --git a/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh b/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh new file mode 100644 index 00000000..aebed338 --- /dev/null +++ b/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh @@ -0,0 +1,20 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch --config_file examples/lingbot_video/model_training/full/accelerate_config_moe.yaml examples/lingbot_video/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v \ + --dataset_metadata_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v/metadata.json \ + --data_file_keys "video" \ + --height 480 \ + --width 832 \ + --num_frames 81 \ + --first_frame_as_condition \ + --dataset_repeat 50 \ + --model_id_with_origin_paths "Robbyant/lingbot-video-moe-30b-a3b:transformer/diffusion_pytorch_model*.safetensors,Qwen/Qwen3-VL-4B-Instruct:*.safetensors,Robbyant/lingbot-video-moe-30b-a3b:vae/diffusion_pytorch_model.safetensors" \ + --processor_path "Qwen/Qwen3-VL-4B-Instruct:" \ + --learning_rate 1e-5 \ + --num_epochs 2 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/lingbot-video-moe-30b-a3b_ti2v_full" \ + --trainable_models "dit" \ + --use_gradient_checkpointing \ + --use_gradient_checkpointing_offload diff --git a/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh b/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh new file mode 100644 index 00000000..4ce53a78 --- /dev/null +++ b/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh @@ -0,0 +1,20 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-moe-30b-a3b_t2v/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch --config_file examples/lingbot_video/model_training/full/accelerate_config_moe.yaml examples/lingbot_video/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v \ + --dataset_metadata_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/metadata.json \ + --data_file_keys "video" \ + --height 480 \ + --width 832 \ + --num_frames 81 \ + --dataset_repeat 50 \ + --model_id_with_origin_paths "Robbyant/lingbot-video-moe-30b-a3b:transformer/diffusion_pytorch_model*.safetensors,Qwen/Qwen3-VL-4B-Instruct:*.safetensors,Robbyant/lingbot-video-moe-30b-a3b:vae/diffusion_pytorch_model.safetensors" \ + --processor_path "Qwen/Qwen3-VL-4B-Instruct:" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/lingbot-video-moe-30b-a3b_t2v_lora" \ + --lora_base_model "dit" \ + --lora_target_modules "to_q,to_k,to_v,to_out" \ + --lora_rank 32 \ + --use_gradient_checkpointing diff --git a/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh b/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh new file mode 100644 index 00000000..7fd74180 --- /dev/null +++ b/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh @@ -0,0 +1,21 @@ +modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*" --local_dir ./data/diffsynth_example_dataset + +accelerate launch --config_file examples/lingbot_video/model_training/full/accelerate_config_moe.yaml examples/lingbot_video/model_training/train.py \ + --dataset_base_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v \ + --dataset_metadata_path data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v/metadata.json \ + --data_file_keys "video" \ + --height 480 \ + --width 832 \ + --num_frames 81 \ + --first_frame_as_condition \ + --dataset_repeat 50 \ + --model_id_with_origin_paths "Robbyant/lingbot-video-moe-30b-a3b:transformer/diffusion_pytorch_model*.safetensors,Qwen/Qwen3-VL-4B-Instruct:*.safetensors,Robbyant/lingbot-video-moe-30b-a3b:vae/diffusion_pytorch_model.safetensors" \ + --processor_path "Qwen/Qwen3-VL-4B-Instruct:" \ + --learning_rate 1e-4 \ + --num_epochs 5 \ + --remove_prefix_in_ckpt "pipe.dit." \ + --output_path "./models/train/lingbot-video-moe-30b-a3b_ti2v_lora" \ + --lora_base_model "dit" \ + --lora_target_modules "to_q,to_k,to_v,to_out" \ + --lora_rank 32 \ + --use_gradient_checkpointing diff --git a/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py b/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py new file mode 100644 index 00000000..906bb0dc --- /dev/null +++ b/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py @@ -0,0 +1,36 @@ +import torch +import json +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from diffsynth import load_state_dict +from modelscope import dataset_snapshot_download + + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) +state_dict = load_state_dict("models/train/lingbot-video-moe-30b-a3b_t2v_full/epoch-1.safetensors") +pipe.dit.load_state_dict(state_dict) +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2v/*", +) +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_t2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py b/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py new file mode 100644 index 00000000..4093ae5c --- /dev/null +++ b/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py @@ -0,0 +1,42 @@ +import os +import json +import torch +from PIL import Image +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from diffsynth import load_state_dict +from modelscope import dataset_snapshot_download + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) +state_dict = load_state_dict("models/train/lingbot-video-moe-30b-a3b_ti2v_full/epoch-1.safetensors") +pipe.dit.load_state_dict(state_dict) + +# The condition first frame and its paired caption ship in the example dataset. +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*", +) +base = "data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v" +with open(os.path.join(base, "ti2v_example.json"), "r", encoding="utf-8") as f: + caption = json.load(f) +input_image = Image.open(os.path.join(base, "ti2v_first_frame.png")).convert("RGB") + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_image=input_image, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_ti2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py b/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py new file mode 100644 index 00000000..bfa78a60 --- /dev/null +++ b/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py @@ -0,0 +1,34 @@ +import torch +import json +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) +pipe.load_lora(pipe.dit, "models/train/lingbot-video-moe-30b-a3b_t2v_lora/epoch-4.safetensors", alpha=1) +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_t2v/*", +) +with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f: + caption = json.load(f) + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_t2v.mp4", fps=15, quality=10) diff --git a/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py b/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py new file mode 100644 index 00000000..2e3237f2 --- /dev/null +++ b/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py @@ -0,0 +1,40 @@ +import os +import json +import torch +from PIL import Image +from diffsynth.utils.data import save_video +from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig +from modelscope import dataset_snapshot_download + +pipe = LingBotVideoPipeline.from_pretrained( + torch_dtype=torch.bfloat16, + device="cuda", + model_configs=[ + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), + ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors"), + ModelConfig(model_id="Robbyant/lingbot-video-moe-30b-a3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), + ], + processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), +) +pipe.load_lora(pipe.dit, "models/train/lingbot-video-moe-30b-a3b_ti2v_lora/epoch-4.safetensors", alpha=1) + +# The condition first frame and its paired caption ship in the example dataset. +dataset_snapshot_download( + dataset_id="DiffSynth-Studio/diffsynth_example_dataset", + local_dir="data/diffsynth_example_dataset", + allow_file_pattern="lingbot_video/lingbot-video-moe-30b-a3b_ti2v/*", +) +base = "data/diffsynth_example_dataset/lingbot_video/lingbot-video-moe-30b-a3b_ti2v" +with open(os.path.join(base, "ti2v_example.json"), "r", encoding="utf-8") as f: + caption = json.load(f) +input_image = Image.open(os.path.join(base, "ti2v_first_frame.png")).convert("RGB") + +video = pipe( + prompt=caption, + negative_prompt=pipe.default_negative_prompt, + input_image=input_image, + height=480, width=832, num_frames=81, + num_inference_steps=40, cfg_scale=3.0, + seed=0, +) +save_video(video, "video_lingbot-video-moe-30b-a3b_ti2v.mp4", fps=15, quality=10)