mridulsaklani / My_Tokenizer Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

It is a small model of tokenizer also used by every AI GPT's model to perform tha task how to convert alphabets into a specific assigned token and encoding or decoding

2 stars 0 forks Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
venv		venv
README.md		README.md
main.py		main.py

Repository files navigation

My_Tokenizer

About

It is a small model of tokenizer also used by every AI GPT's model to perform tha task how to convert alphabets into a specific assigned token and encoding or decoding

encoding machine-learning ai tokenizer decoding artificial-intelligence tokenization gpt-4 gpt-models

Readme