Skip to content

convert : fix bytes_to_unicode import for transformers >= 5.15 - #26217

Merged
ggerganov merged 1 commit into
ggml-org:masterfrom
SolshineCode:fix-bytes-to-unicode-import
Aug 4, 2026
Merged

convert : fix bytes_to_unicode import for transformers >= 5.15#26217
ggerganov merged 1 commit into
ggml-org:masterfrom
SolshineCode:fix-bytes-to-unicode-import

Conversation

@SolshineCode

@SolshineCode SolshineCode commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Overview

convert_hf_to_gguf.py fails with an ImportError on transformers >= 5.15 because bytes_to_unicode moved from transformers.models.gpt2.tokenization_gpt2 to transformers.convert_slow_tokenizer. Any model that goes through QwenModel.token_bytes_to_string hits it. This adds a try/except: old path first, new location as fallback, so both transformers ranges work.

Additional information

Hit while testing #26185 (the Kimi K3 tokenizer goes through this path). Verified on transformers 5.15.0.dev0: conversion completes and the resulting GGUF matched my HF reference outputs.

Requirements

@SolshineCode
SolshineCode requested a review from CISC as a code owner July 28, 2026 07:07
@ggml-gh-bot

This comment was marked as resolved.

@SolshineCode
SolshineCode force-pushed the fix-bytes-to-unicode-import branch from d0f3b93 to aad3720 Compare July 28, 2026 07:40
@SolshineCode

Copy link
Copy Markdown
Contributor Author

Rewrote the description and commit message per the guidelines.

@CISC CISC left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale is flawed, it wasn't moved, it was simply removed in huggingface/transformers#40936 as the function was already copied to this location in huggingface/transformers#30334 (4.54.1), meaning it is safe for us to use it from there.

Please make the same fix in chatglm.py.

Comment thread conversion/qwen.py Outdated
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.

Applies the same fix to chatglm.py.
@SolshineCode
SolshineCode force-pushed the fix-bytes-to-unicode-import branch from aad3720 to ef6113a Compare August 4, 2026 00:26
@SolshineCode

Copy link
Copy Markdown
Contributor Author

Thanks, you're right that it wasn't moved. It was removed in #40936 and had already been copied to convert_slow_tokenizer in #30334 (4.54.1). I've now corrected the commit message. I've also switched to importing directly from convert_slow_tokenizer, and applied the same fix to chatglm.py.

@CISC CISC added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 4, 2026
@ggerganov
ggerganov merged commit b06fbc9 into ggml-org:master Aug 4, 2026
5 checks passed
smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026
…rg#26217)

bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.

Applies the same fix to chatglm.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants