Please add both tokenizer usage Qwen 1.5B and Qwen 7B. #189
Description
Hi, please add so we can select either 1.5B or 7B from the tokeniser, because as of now, in the tokeniser folder, only a single tokeniser file can stay.
{{ message }}
Hi, please add so we can select either 1.5B or 7B from the tokeniser, because as of now, in the tokeniser folder, only a single tokeniser file can stay.