Some problems when using RICE-ViT

[zhangym0213]

Hello @anxiangsir 🤗

I'm Yiming and study in NTU. Recently I’ve been working with RICE-ViT and trying to reproduce baseline built on Qwen2.5-7B-Instruct. I ran into a couple of questions and would really appreciate your help:

About reproducing ViT-L-14-336px results

I used rice-vit-large-patch14-560 and modify the crop_size and shortest_edge in preprocessor_config to 336, attempting to match the ViT-L-14-336px setup. Is this the correct way to reproduce the 336px version? If not, where can I find the checkpoint specifically trained for ViT-L-14-336px?

Which MLCDVisionModel to use

I noticed that there are two version of MLCDVisionModel,

• one in LLaVA-NEXT,

• another in transformers

For RICE-ViT, I used the version from Transformers.
Is this the correct choice?

Thanks a lot for your time! 🙏

[anxiangsir]

Hello,

Regarding your questions:

1. Reproducing ViT-L-14-336px results:
   RICE-ViT is based on RoPE2d, which is compatible with variable resolutions. Although the checkpoint provided is 560px, it can accept 336px input directly. However, you will not be able to exactly reproduce the 336px results reported in the paper using this 560px RiceViT checkpoint; the performance might be slightly lower. The 336px results in the paper were obtained from a model specifically trained at 336px resolution. With the current model, you can only reproduce the 560px results.

2. Which MLCDVisionModel to use:
   Both implementations (in LLaVA-NEXT and Transformers) produce the same results, so you can use either one.

Hope this helps

[anxiangsir]

We will release the 336px RiceViT in the future, but not right now. We are currently busy working on the release of an even better ViT model.