[RFC] autoparser unable to accommodate non-ASCII reasoning markers, e.g. ◁think▷

[ulidtko]

Hi @pwilkin and team!

In follow-up to #18675, let me open an issue after repeatedly failing to find prior discussion.

The attached patch is mostly to demonstrate an issue — it's short enough to tell exactly what is being "fixed". Below I'll describe textually anyway. I'm not convinced it's the best possible approach; sure enough, you'll find a better idea. I'm perfectly fine if the PR gets closed, if anyone more familiar finds a better way.

So, the issue reproduces when running this model: ggml-org/Kimi-VL-A3B-Thinking-2506-GGUF,

• which is a conversion & quantization of moonshotai/Kimi-VL-A3B-Thinking-2506,
• which is a "Thinking" fine-tune of moonshotai/Kimi-VL-A3B-Instruct,
• which is an instruction-following fine-tune of moonshotai/Moonlight-16B-A3B,
• which is a base model.

The surface-level symptom is, llama.cpp not parsing the reasoning stream properly:

Do notice the uncommon ◁think▷ marker; the closing one ◁/think▷ follows below in response. The characters are:

• ◁ U+25C1 WHITE LEFT-POINTING TRIANGLE
• ▷ U+25B7 WHITE RIGHT-POINTING TRIANGLE

These are a property of how the Thinking fine-tune was conducted, I believe. The multi-token sequences ◁think▷, ◁/think▷ are documented in the models readme's, too.

... With this patch, and a correction in chat template jinja, I'm getting reasoning_mode: TAG_BASED and proper parses in both llama-cli and llama-server:

However again, I'm clear that this is only a rough first cut, a PoC draft that worked; the added logic is too specific and not a good proper fix.

It is necessary, because segmentize_markers() cannot handle anything besides <foo> and [bar]:

llama.cpp/common/chat-auto-parser-helpers.cpp

Lines 274 to 275 in f36e5c3

	auto is_marker_opener = [](char c) -> bool { return c == '<' || c == '['; };
	auto is_marker_closer = [](char op, char c) -> bool { return (op == '<' && c == '>') || (op == '[' && c == ']'); };

(Plus a related change in PEG builder. This patch is made and tested against version b9747, but the same code lives in today's master ☝️)

The real question is:

• Can the autoparser be generalized to deduce this weirdness, too, from tokenizer.chat_template alone?
• I.e. without requiring code change / nasty hacks like this one.

Hence the RFC tag — Request For Comment.

---

Fixed & original chat templates

Original, distributed with model

{%- for message in messages -%}
  {%- if loop.first and messages[0]['role'] != 'system' -%}
    {{'<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>'}}
  {%- endif -%}
  {%- if message['role'] == 'system' -%}{{'<|im_system|>'}}{%- endif -%}
  {%- if message['role'] == 'user' -%}{{'<|im_user|>'}}{%- endif -%}
  {%- if message['role'] == 'assistant' -%}{{'<|im_assistant|>'}}{%- endif -%}
  {{- message['role'] -}}{{'<|im_middle|>'}}

  {%- if message['content'] is string -%}
    {{- message['content'] + '<|im_end|>' -}}
  {%- else -%}
    {%- for content in message['content'] -%}
      {%- if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}
        {{'<|media_start|>image<|media_content|><|media_pad|><|media_end|>'}}
      {%- else -%}
        {{content['text']}}
      {%- endif -%}
    {%- endfor -%}
    {{'<|im_end|>'}}
  {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}{{'<|im_assistant|>assistant<|im_middle|>'}}{%- endif -%}

Fixed, works as expected together with the PR's patch

{%- for message in messages -%}
  {%- if loop.first and messages[0]['role'] != 'system' -%}
    <|im_system|>system<|im_middle|>You are a helpful AI assistant called Kimi Dolores. You live in the house of Wizard Max. You have a gift of vision, use it wisely.<|im_end|>
  {%- endif -%}
  {%- if message['role'] == 'system' -%} {{'<|im_system|>'}} {%- endif -%}
  {%- if message['role'] == 'user' -%} {{'<|im_user|>'}} {%- endif -%}
  {%- if message['role'] == 'assistant' -%} {{'<|im_assistant|>'}} {%- endif -%}
  {{- message['role'] -}}{{'<|im_middle|>'}}

  {%- if message.get('reasoning_content') -%}
    ◁think▷{{ message['reasoning_content'] }}◁/think▷
  {%- endif -%}

  {%- if message['content'] is string -%}
    {{- message['content'] + '<|im_end|>' -}}
  {%- else -%}
    {%- for content in message['content'] -%}
      {%- if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}
        {{'<|media_start|>image<|media_content|><|media_pad|><|media_end|>'}}
      {%- else -%}
        {{content['text']}}
      {%- endif -%}
    {%- endfor -%}
    {{'<|im_end|>'}}
  {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{'<|im_assistant|>assistant<|im_middle|>'}}
{%- endif -%}

---

AI usage disclosure: Anthropic Sonnet 4.6 wrote this garbage of patch, with iterated verbal abuse direction provided by the operator (me). But hey, at least it was able to read the docs & follow the source — well done, good codebase. This PR description is 100% pure human writing; best regards.

[ggml-gh-bot]

Hi @ulidtko, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.