ModuleNotFoundError: No module named 'transformers.tokenization_utils'

[llc-kc]

when using transformers 5.0.0rc0, there are some errors:

File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/handlers/chat_completions.py", line 35, in
from guidellm.mock_server.utils import (
File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/utils.py", line 17, in
from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer, TextInput
ModuleNotFoundError: No module named 'transformers.tokenization_utils'

File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/handlers/tokenizer.py", line 16, in
from transformers.tokenization_utils import PreTrainedTokenizer
ModuleNotFoundError: No module named 'transformers.tokenization_utils'

[sjmonson]

Hi, a contribution is welcome but the GuideLLM team will not attempt to fix this until transformers 5.0.0 is out of pre-release.