Deepseek V4: split-mode tensor by S3kundes · Pull Request #25860 · ggml-org/llama.cpp

S3kundes · 2026-07-18T10:39:32Z

Overview

I have made Deepseek V4 flash (DeepSeek-V4-Flash-GGUF/ UD-Q8_K_XL) work with tensor-split tensor, on my 8x3090 setup.

Played with some books content processing and asking the LLM about the contents ( 2 questions 16k context first question, 40k second one) and the TG/PP greatly improved

prefill speed (~150–175% faster), generation speed (~24–29% faster)

Before (layer split mode):
1'st prompt:

PP: 283.86 tokens/s
TG: 28.34 t/s

2'nd prompt:

PP: 225.10 tokens/s
TG: 24.78 t/s

After ( tensor split mode)
1'st prompt:

PP: 713.49 tokens/s
TG: 35.14 t/s

2'nd prompt:

PP: 618.29 tokens/s
TG: 32.05 t/s

      --server
       -m /models/deepseek/unsloth-DeepSeek-V4-Flash-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-UD-Q8_K_XL-00001-of-00005.gguf
       --alias DeepSeek-V4-Flash-UD-Q8_K_XL
       --host 0.0.0.0
       --port 8080
       --metrics
       --temp 1.0
       --top-p 1.0
       --min-p 0.0
       --flash-attn on
       --ctx-size 100000
       --split-mode tensor
       --fit off
       --n-gpu-layers all
       --tensor-split 1,1,1,1,1,1,1,1
       --batch-size 4096
       --ubatch-size 256
       --parallel 1
       --verbose
       --log-timestamps
       --perf
       --jinja

Additional information

I have not made this to be merged as i realized it was butchered by the AI (debug code + i guess modified in the wrong places), but maybe someone who knows more about the project can do a proper version as this has good potential i think.

The downside to this is that the KV cache is still copied on all gpus and atm on 192gb vram i can only have 100k context because of that. ( i have tried to vibe code the 1m context part, to split the context on multiple gpus and not copy it, but it wrecked the TG to 5tps, so i just left that out)

@fairydreaming @ggerganov @am17an maybe one of you has time to look at this. ( sorry for tagging)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES (pi:codex)

am17an · 2026-07-18T11:10:10Z

I'm working on sm tensor for this model, and this PR is not an acceptable version.

S3kundes · 2026-07-18T12:47:03Z

I'm working on sm tensor for this model, and this PR is not an acceptable version.

i know it's not acceptable to be merged, i just wanted to share, as a Proof of concept / comparison vs layer split mode. I can close this if it's totally useless for you guys.

S3kundes · 2026-07-18T22:12:50Z

refactored with gpt-sol to remove the logging and free some more vram. performance stayed the same

am17an · 2026-07-19T08:18:39Z

On 4x 3090s

model	size	params	backend	ngl	sm	test	t/s
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	pp512	1366.30 ± 618.21
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	tg128	58.91 ± 4.95
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	pp512 @ d8192	1018.34 ± 462.22
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	tg128 @ d8192	53.09 ± 1.71
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	pp512	1111.64 ± 12.83
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	tg128	68.79 ± 0.94
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	pp512 @ d8192	1017.39 ± 14.40
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	tg128 @ d8192	64.99 ± 1.06

S3kundes · 2026-07-19T10:36:17Z

Did some tests as well (8x3090)

model	size	params	backend	ngl	n_batch	n_ubatch	sm	fa	ts	test	t/s
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	512.41 ± 5.75
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	30.52 ± 0.01
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	473.71 ± 4.39
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	29.30 ± 0.06
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	422.18 ± 3.23
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	30.58 ± 0.19
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	384.16 ± 3.53
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	29.37 ± 0.20
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	497.95 ± 4.70
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	30.56 ± 0.12
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	463.82 ± 2.47
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	29.32 ± 0.13
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	415.93 ± 5.20
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	30.61 ± 0.22
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	382.37 ± 4.34
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	layer	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	29.32 ± 0.09
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	697.56 ± 203.34
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	35.52 ± 5.04
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	686.09 ± 61.20
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	31.62 ± 1.07
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	1133.20 ± 7.00
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	37.22 ± 3.94
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	912.08 ± 247.21
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	2048	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	31.43 ± 1.04
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	729.90 ± 101.12
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	37.01 ± 4.06
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	677.65 ± 63.09
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	256	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	31.34 ± 1.09
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512	1125.82 ± 12.02
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128	36.93 ± 3.92
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	pp512 @ d8192	911.11 ± 247.09
deepseek4 ?B MXFP4 MoE	150.75 GiB	284.33 B	CUDA	-1	4096	512	tensor	1	1.00/1.00/1.00/1.00/1.00/1.00/1.00/1.00	tg128 @ d8192	31.25 ± 1.01

JohannesGaessler

For context: @am17an asked me for advice regarding -sm tensor for DS4 and I am commenting on this PR since I think it will be helpful for his purposes. I do not want you to feed these comments to a language model to try and get a working implementation, I will not review and possibly merge that since I very much expect it to be more work than doing the implementation myself.

JohannesGaessler · 2026-07-26T07:37:47Z

+        if (src_ss[0].axis == src_ss[1].axis && src_ss[0].axis >= GGML_BACKEND_SPLIT_AXIS_2 &&
+                src_ss[0].axis < GGML_MAX_DIMS) {
+            GGML_ASSERT(split_states_equal(src_ss[0], src_ss[1]));
+            return src_ss[0];
+        }


This is incorrect. The only valid axis that would be covered here is PARTIAL and there is no sensible way to combine two PARTIAL tensors via a matrix multiplication since (a + b) * (a + b) == a*a + 2*a*b + b*b.

JohannesGaessler · 2026-07-26T07:41:27Z

+        if (src_ss[0].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED) {
+            GGML_ASSERT(src_ss[1].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+            GGML_ASSERT(src_ss[2].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+            GGML_ASSERT(tensor->src[4] == nullptr || src_ss[4].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+            return {GGML_BACKEND_SPLIT_AXIS_MIRRORED, {0}, {1}, 1};
+        }


This is indicative of incorrect assignments in llama-model.cpp. It is in principle possible to just run everything as mirrored but for an expensive operation like FA that is going to massively gimp the performance.

JohannesGaessler · 2026-07-26T07:46:03Z

+            if (std::regex_match(tensor_name, pattern_kv_cache) ||
+                    std::regex_match(tensor_name, pattern_dsv4_state)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+            }


This is 100% the wrong way to do it and just a workaround in lieu of the correct logic.

JohannesGaessler · 2026-07-26T07:49:09Z

+            if (std::regex_match(tensor_name, pattern_attn_sinks)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_0, "attn_output_a.weight");
+            }
+            if (std::regex_match(tensor_name, pattern_attn_q_b_weight)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_1, "attn_output_a.weight");
+            }
+            if (std::regex_match(tensor_name, pattern_attn_out_a_weight)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_1, "attn_output_b.weight");
+            }
+            if (std::regex_match(tensor_name, pattern_attn_out_b_weight)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_0);
+            }
+            if (std::regex_match(tensor_name, pattern_ffn_up_shexp_weight) ||
+                    std::regex_match(tensor_name, pattern_ffn_gate_shexp_weight)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_1, "ffn_down_shexp.weight");
+            }
+            if (std::regex_match(tensor_name, pattern_ffn_down_shexp_weight)) {
+                return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_0, "ffn_down_shexp.weight");
+            }


I'd have to check against the compute graph (and also spin up the model with a debugger) but these may be correct. The bottom line is that attention heads should be split by GPU and combined once at the end. Mirroring them will result in duplicated KV cache and bad performance.

JohannesGaessler · 2026-07-26T07:51:08Z

+                if (ud->model->arch == LLM_ARCH_DEEPSEEK4) {
+                    return {1};
+                }


This is wrong unless the model does not use GQA at all.

JohannesGaessler · 2026-07-26T07:53:29Z

+                if (std::regex_match(tensor_name, pattern_attn_q_b_weight)) {
+                    GGML_ASSERT(segments.size() == 1);
+                    return {hparams.n_embd_head_k(il)};
+                }


This seems wrong to me but I don't know off the top of my head what the correct implementation would be.

split mode tensor working

a21950e

S3kundes requested review from CISC and JohannesGaessler as code owners July 18, 2026 10:39

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Jul 18, 2026

S3kundes marked this pull request as draft July 18, 2026 11:11

sol refactor and free some vram

3515faf

github-actions Bot added the testing Everything test related label Jul 18, 2026

JohannesGaessler reviewed Jul 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Deepseek V4: split-mode tensor#25860

Deepseek V4: split-mode tensor#25860
S3kundes wants to merge 2 commits into
ggml-org:masterfrom
S3kundes:dsv4-split-mode-tensor

S3kundes commented Jul 18, 2026 •

edited

Loading

Uh oh!

am17an commented Jul 18, 2026

Uh oh!

S3kundes commented Jul 18, 2026

Uh oh!

S3kundes commented Jul 18, 2026

Uh oh!

am17an commented Jul 19, 2026

Uh oh!

S3kundes commented Jul 19, 2026

Uh oh!

JohannesGaessler left a comment

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

JohannesGaessler Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

S3kundes commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

am17an commented Jul 18, 2026

Uh oh!

S3kundes commented Jul 18, 2026

Uh oh!

S3kundes commented Jul 18, 2026

Uh oh!

am17an commented Jul 19, 2026

Uh oh!

S3kundes commented Jul 19, 2026

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

S3kundes commented Jul 18, 2026 •

edited

Loading