Wan2.2-Fun-5B-Control Question

[yuezhh]

Is there training and inference code provided for Wan2.2-Fun-5B-Control, which one is it, thanks！

[yuezhh]

v2v

[bubbliiiing]

Could you please take a look at this document when you have a moment? I’d like to see if the workflow works: https://github.com/aigc-apps/VideoX-Fun/blob/main/scripts/wan2.2_fun/README_TRAIN_CONTROL_zh-CN.md

[yuezhh]

export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-5B-Control"
export DATASET_NAME="datasets/"
export DATASET_META_NAME="datasets/datasets.json"
NCCL_DEBUG=INFO

accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_control_lora.py
--config_path="config/wan2.2/wan_civitai_5b.yaml"
--pretrained_model_name_or_path=$MODEL_NAME
--train_data_dir=$DATASET_NAME
--train_data_meta=$DATASET_META_NAME
--image_sample_size=512
--video_sample_size=512
--token_sample_size=512
--video_sample_stride=1
--video_sample_n_frames=81
--train_batch_size=1
--gradient_accumulation_steps=4
--dataloader_num_workers=0
--learning_rate=2e-05
--lr_warmup_steps=500
--lr_scheduler="constant_with_warmup"
--seed=42
--num_train_epochs=20
--checkpointing_steps=100
--output_dir="0602"
--gradient_checkpointing
--mixed_precision="bf16"
--adam_weight_decay=3e-2
--adam_epsilon=1e-10
--vae_mini_batch=1
--max_grad_norm=0.5
--uniform_sampling
--random_hw_adapt
--enable_bucket
--add_inpaint_info
--add_full_ref_image_in_self_attention
--training_with_video_token_length
--video_repeat=1
--train_mode="control_ref"
--control_ref_image="random"
--boundary_type="low"
--rank=32
--network_alpha=16
--target_name="q,k,v,ffn.0,ffn.2"
--use_peft_lora
--save_state
--low_vram
--resume_from_checkpoint="latest" \

我使用这个配置进行微调训练，但是loss整体起起伏伏的，用采样的结果跑fid、fvd指标也并没有明显的变好，想问下这是正常的吗，我该怎么让loss收敛呢，非常期待得到您的帮助！