Fix DeepSeek4 crafted template by pwilkin · Pull Request #25414 · ggml-org/llama.cpp

pwilkin · 2026-07-07T17:45:32Z

Overview

It seems like I've been a bit lazy implementing the DS4 template. I used the standard behavior (retain last reasoning trace). However, the reference behavior is different. DS4 will retain all reasoning traces in one of two cases:

if drop_reasoning is set to false
in case the message history contains at least one tool call

I've patched the template and added a test to verify this.

Additional information

Note that under the current behavior llama.cpp defaults to setting drop_reasoning to true, so if you want it set to false you'll have to add --preserve-reasoning - not that it matters that much because for all agentic cases, DS4 will still force the retain of traces due to the presence of tool calls.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, a bit of GLM-5.2 and Claude to make the tests (funny thing: GLM-5.2 tripped on reading stuff that contained <think> and </think> :))

aldehir

If we want to start testing templates, probably best to move those to a separate test suite as this one is already pretty large.

IMO Python tests are more appropriate for the Jinja templates.

pwilkin · 2026-07-07T18:06:51Z

Yeah, I'll probably move to refactor test-chat.cpp in a followup PR.

Personally, I'm against Python tests because the C++ tests double down as a check on the template rendering / message passing, which Python won't do - I prefer the tests to run inside the same engine as the actual template processing.

tarruda · 2026-07-07T18:20:57Z

As I had explained in the reddit thread, this is not the only issue and I had to opt out of the autoparser in order to properly run DSV4. In particular, parallel tool calling is not working with the autoparser and while running on pi I was getting random stops with tool calls within the reasoning block.

Everything done to fix are contained in these two commits sent to fairydreaming's lightining indexer branch: fairydreaming#4

pwilkin · 2026-07-07T18:58:46Z

@tarruda yeah, working on the parser extension next :)

aldehir · 2026-07-07T19:07:18Z

@tarruda can you please include your response from Reddit for visibility purposes. Also because the link appears broken on my end.

pwilkin · 2026-07-07T19:12:32Z

This is basically pretty similar to @tarruda 's branch :) a few style differences.

@tarruda technically your approach to the template flag isn't in line with the reference: they require reasoning traces not if tools are declared, but if any tool calls are actually present (I know, practically it doesn't really matter too much, but just nitpicking :))

tarruda · 2026-07-08T02:45:19Z

@aldehir https://old.reddit.com/r/LocalLLaMA/comments/1une2il/i_merged_fixes_for_quantized_kv_cache_into_my/ovuun8r/

pwilkin · 2026-07-20T15:20:41Z

@ggml-org/maintainers need another approval, let's get this done so we can move on with the DS4 template fixes (I forgot this wasn't merged).

am17an · 2026-07-20T15:25:55Z

Fix the tests though https://github.com/ggml-org/llama.cpp/actions/runs/29754708141/job/88393960864?pr=25414

tarruda · 2026-07-20T15:26:13Z

@pwilkin Can you take a look at this one? https://github.com/ggml-org/llama.cpp/pull/25891/changes#diff-8dbbe2bce6d35f58f9ab0dc165bb2ae45c20923d4f144c35e77a0ae836d20e8d

It adds two things that are still missing:

"max reasoning" support
structured output instructions

tarruda · 2026-07-20T15:37:13Z

@pwilkin official encoding reference on these features I mentioned:

max effort reasoning: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py#L64-L68
structured output: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py#L49-L51

pwilkin · 2026-07-22T10:46:05Z

@ggml-org/maintainers need 1 more approval please.

tarruda · 2026-07-22T12:21:18Z

    ctx.set_val("preserve_thinking",         mk_val<value_bool>(enabled));
    ctx.set_val("clear_thinking",            mk_val<value_bool>(!enabled));
    ctx.set_val("truncate_history_thinking", mk_val<value_bool>(!enabled));
+    ctx.set_val("drop_reasoning",            mk_val<value_bool>(!enabled));


The variable in the template is named drop_thinking.

Also: drop_thinking defaults to false, so this seems to disable the --reasoning-preserve mechanics. IOW: you can force thinking to drop by explicitly passing the template kwargs drop_thinking=true, but this doesn't seem to be doing anything.

Ah crap :/

Yeah, the behavior is different than with other --reasoning-preserve models.

kallewoof · 2026-07-24T09:57:58Z

FWIW in pi.dev I have this, followed by the agent prematurely ending the turn, on current HEAD (0cea362):

 <｜DSML｜tool_calls>                                                                                                                                                                    
 <｜DSML｜invoke name="read">                                                                                                                                                            
 <｜DSML｜parameter name="path" string="true">/home/user/file</｜DSML｜parameter>                                                            
 <｜DSML｜parameter name="offset" string="false">7360</｜DSML｜parameter>                                                                                                                
 <｜DSML｜parameter name="limit" string="false">3</｜DSML｜parameter>                                                                                                                    
 </｜DSML｜invoke>                                                                                                                                                                       
 </｜DSML｜tool_calls>

pwilkin · 2026-07-24T10:01:49Z

@kallewoof yeah, might be another instance of model calling tools within reasoning block.

* chat: fix DS4 template to explicitly follow reference behavior * Support DeepSeekv4 flag (`drop_reasoning`). * fix: hook DS3.2 parser for DS4 as well * fix: add tool result reordering * fix: post-merge

tarruda · 2026-07-29T01:19:50Z

@kallewoof try to reproduce this bug on my branch (which is #25891 rebased on master) . I believe it has fixed all instances where this happens.

kallewoof · 2026-07-29T07:30:42Z

@tarruda Just gave it a shot. No <｜DSML stuff, and the output passes review. Why is this not a pull request again? Edit: Or why is the PR closed, rather.

pwilkin · 2026-07-29T08:26:46Z

@kallewoof it overlapped my fixes for the most part and with the tool calling in reasoning we were waiting for #25544 which just got merged.

tarruda · 2026-07-29T08:48:37Z

I can open a new PR if @pwilkin is fine with the changes in tarruda@4d561a3

tarruda · 2026-07-29T08:51:22Z

or someone could reopen #25891, I'm unable to do so

am17an · 2026-07-29T11:12:11Z

@tarruda can you also try #26269?

tarruda · 2026-07-29T11:14:58Z

@am17an in my branch I've added regression tests which reproduce the bug. If @pwilkin pulled those regressions into his branch and it passes, then it should be fixed

am17an · 2026-07-29T11:17:29Z

The test you posted is fixed, are there more?

am17an · 2026-07-29T11:18:42Z

Okay I see your branch. @pwilkin can you confirm if test-chat from that branch passes on your PR?

tarruda · 2026-07-29T11:19:47Z

@am17an there might be more cases that I haven't reproduced, but from my local testing using my branch, it hasn't reproduced that bug in a while.

pwilkin added 2 commits July 7, 2026 19:36

chat: fix DS4 template to explicitly follow reference behavior

481fa3f

Support DeepSeekv4 flag (drop_reasoning).

33751f0

pwilkin requested a review from CISC as a code owner July 7, 2026 17:45

pwilkin requested review from CISC, aldehir, am17an and ngxson and removed request for CISC July 7, 2026 17:45

github-actions Bot added testing Everything test related jinja parser Issues related to the jinja parser labels Jul 7, 2026

aldehir approved these changes Jul 7, 2026

View reviewed changes

fix: hook DS3.2 parser for DS4 as well

e4003fe

pwilkin requested a review from a team as a code owner July 7, 2026 19:06

fix: add tool result reordering

9c4fce1

Merge branch 'master' into ds4-template-fix

4bc25c1

am17an approved these changes Jul 20, 2026

View reviewed changes

fix: post-merge

6c61676

fairydreaming approved these changes Jul 22, 2026

View reviewed changes

pwilkin merged commit f534da2 into ggml-org:master Jul 22, 2026
22 of 25 checks passed

tarruda reviewed Jul 22, 2026

View reviewed changes

tarruda mentioned this pull request Jul 22, 2026

common/chat: fix DeepSeek V4 parsing/encoding #25891

Closed

Uh oh!

Conversation

pwilkin commented Jul 7, 2026

Overview

Additional information

Requirements

Uh oh!

aldehir left a comment

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

tarruda commented Jul 7, 2026

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

aldehir commented Jul 7, 2026

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

tarruda commented Jul 8, 2026

Uh oh!

pwilkin commented Jul 20, 2026

Uh oh!

am17an commented Jul 20, 2026

Uh oh!

tarruda commented Jul 20, 2026

Uh oh!

tarruda commented Jul 20, 2026

Uh oh!

pwilkin commented Jul 22, 2026

Uh oh!

Uh oh!

tarruda Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

tarruda Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

pwilkin Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kallewoof commented Jul 24, 2026

Uh oh!

pwilkin commented Jul 24, 2026

Uh oh!

tarruda commented Jul 29, 2026

Uh oh!

kallewoof commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Jul 29, 2026

Uh oh!

tarruda commented Jul 29, 2026

Uh oh!

tarruda commented Jul 29, 2026

Uh oh!

am17an commented Jul 29, 2026

Uh oh!

tarruda commented Jul 29, 2026

Uh oh!

am17an commented Jul 29, 2026

Uh oh!

am17an commented Jul 29, 2026

Uh oh!

tarruda commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kallewoof commented Jul 29, 2026 •

edited

Loading