Skip to content

Commit fc7326c

Browse files
committed
docs: update server README
1 parent 4714849 commit fc7326c

1 file changed

Lines changed: 3 additions & 31 deletions

File tree

examples/server/README.md

Lines changed: 3 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -348,7 +348,9 @@ Set `model.draft_model` to enable speculative draft providers.
348348
}
349349
```
350350

351-
Use Qwen3.5 MTP when the loaded model and llama.cpp build expose the required draft state.
351+
### Multi-Token Prediction (MTP)
352+
353+
Use MTP when the loaded model and llama.cpp build expose the required draft state.
352354

353355
```json
354356
{
@@ -384,33 +386,3 @@ MTP currently applies to text-only requests.
384386
| `min_tokens` | Minimum prefix length that is worth saving. |
385387

386388
The cache is versioned by model and context compatibility data and should be treated as ephemeral.
387-
388-
## Minimal Text-Only Config
389-
390-
Use this as a starting point when multimodal support is not needed.
391-
392-
```json
393-
{
394-
"server": {
395-
"host": "127.0.0.1",
396-
"port": 8000
397-
},
398-
"model": {
399-
"alias": "local-model",
400-
"path": "/path/to/model.gguf",
401-
"n_ctx": 8192,
402-
"n_seq_max": 8,
403-
"n_batch": 512,
404-
"n_ubatch": 512,
405-
"threads": 8,
406-
"threads_batch": 8,
407-
"kv_unified": true
408-
}
409-
}
410-
```
411-
412-
## Notes
413-
414-
This example is self-contained while the server design is still changing.
415-
Keep checked-in `config.json` runnable and put machine-specific experiments in separate local config files.
416-
Do not depend on `llama_cpp_ext.py` outside this example because it exposes experimental non-public llama.cpp APIs.

0 commit comments

Comments
 (0)