File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -348,7 +348,9 @@ Set `model.draft_model` to enable speculative draft providers.
348348}
349349```
350350
351- Use Qwen3.5 MTP when the loaded model and llama.cpp build expose the required draft state.
351+ ### Multi-Token Prediction (MTP)
352+
353+ Use MTP when the loaded model and llama.cpp build expose the required draft state.
352354
353355``` json
354356{
@@ -384,33 +386,3 @@ MTP currently applies to text-only requests.
384386| ` min_tokens ` | Minimum prefix length that is worth saving. |
385387
386388The cache is versioned by model and context compatibility data and should be treated as ephemeral.
387-
388- ## Minimal Text-Only Config
389-
390- Use this as a starting point when multimodal support is not needed.
391-
392- ``` json
393- {
394- "server" : {
395- "host" : " 127.0.0.1" ,
396- "port" : 8000
397- },
398- "model" : {
399- "alias" : " local-model" ,
400- "path" : " /path/to/model.gguf" ,
401- "n_ctx" : 8192 ,
402- "n_seq_max" : 8 ,
403- "n_batch" : 512 ,
404- "n_ubatch" : 512 ,
405- "threads" : 8 ,
406- "threads_batch" : 8 ,
407- "kv_unified" : true
408- }
409- }
410- ```
411-
412- ## Notes
413-
414- This example is self-contained while the server design is still changing.
415- Keep checked-in ` config.json ` runnable and put machine-specific experiments in separate local config files.
416- Do not depend on ` llama_cpp_ext.py ` outside this example because it exposes experimental non-public llama.cpp APIs.
You can’t perform that action at this time.
0 commit comments