Low GPU utilization and significant bubbles during training

[ecnuycxie]

Hi, @kcz358

Thanks for releasing the checkpoints and the repository!

I am attempting to train Qwen2.5-VL-3B using this repo. However, I’ve noticed that the GPU utilization is quite low, and there are significant pipeline bubbles during training.

Could you provide some guidance on how to optimize the training efficiency or suggest any specific configurations?

Thanks!

[kcz358]

Hi, may I ask what are the configuration you use? For example what is the packing length and do you use rmpad and liger kernel during training?