another gem for the eco system

[washu]

https://github.com/washu/ruby_llm-tokenizer

facade over several tokenizers. lets you check token acccounting before you submit and deter,me if you need to truncate

[crmne]

That's pretty cool. Is there any chance to add Anthropic and Gemini tokenizers?

[washu]

if you can point me to thier tokenizers i can, when i went looking i couldnt find a tokenizer for anthorpic that was public. ill check for a gemini one today

[washu]

i found sentencepeice which should cover gemini so ill add support for it

[washu]

0.1.1 released, hass gemmi tokenization support. anthropic isnt possible atm as they dont detail what they use , known issue