feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend

[localai-bot]

Description

Serves Qwen3-TTS through the llama-cpp backend, so text-to-speech runs on the same accelerator matrix we already ship for text generation (CUDA, ROCm, SYCL, Vulkan, Metal, L4T) using upstream's own GGUF conversion.

llama.cpp merged Qwen3-TTS in ggml-org/llama.cpp#26254. The server-side plumbing lives in #26603, which is still a draft, so we carry its server hunks as patches/0002-add-server-task-type-tts.patch.

Why a patch instead of calling the mtmd API directly. Our grpc-server.cpp is an adapter over llama.cpp's shared server_context, which owns the llama_context and runs the slot scheduler on its own thread. A gRPC handler that drove the gen-audio loop itself would race that scheduler. Making TTS a slot-based SERVER_TASK_TYPE_TTS task is the only concurrency-safe integration, and it follows the existing 0001-add-server-task-type-score.patch precedent. Delete 0002 once #26603 merges upstream.

The existing qwen3-tts-cpp backend (qwentts.cpp) is untouched. This is a second, independent path, not a replacement.

What is included

• LLAMA_VERSION bump to 9de0fcf2b, and patches/0002 carrying the TTS server task
• TTS and TTSStream RPCs in grpc-server.cpp, both already declared in backend.proto but previously unimplemented
• disable-tts-task.sh, so turboquant and bonsai (which copy grpc-server.cpp into forks without our patches) still compile
• Gallery entries qwen3-tts-llamacpp and qwen3-tts-llamacpp-q4
• Docs under docs/content/features/text-to-audio.md

Verified end to end on CPU

Both paths return valid 24 kHz mono 16-bit WAV containing real speech, measured rather than eyeballed (peak 26314, RMS 4168, 76 per cent of frames voiced), in the reference clip's pitch register. Ten consecutive and interleaved requests all pass with different text on each. Gallery install verifies both checksums against freshly downloaded bytes, and a parakeet ASR round-trip read the synthesized audio back correctly.

Upstream bugs found along the way

Two are worth filing against #26603:

1. Only the first TTS request per process succeeds. TTS slots bypass the shared batch, so they skip the per-request KV hygiene, and the pipeline always decodes from position 0 into seq_id = slot.id. Request two overwrites request one and llama_decode fails. Reproduced against upstream's own POST /tts with -np 1 and no LocalAI code: request 1 returns 200, request 2 returns 500. It has gone unnoticed because llama-server defaults to four slots, so the first four requests land on different slots by LRU. Our fix is a one-line slot.prompt_clear() before set_input, carried in 0002.
2. A SRV_WRN call in the draft passes no variadic argument while the macro uses plain __VA_ARGS__, so it does not compile. Fixed in our copy to match the file's own "%s" idiom.

A third bug, a get_rows assert that aborted every request, turned out to be c8e03ce81, already fixed upstream. We briefly carried a 0003 patch masking non-codec tokens before establishing it was perturbing that graph-ordering bug rather than fixing a sampling one. It is deleted. Commits 5da3b2fc4 and b04de703a are that add-and-remove pair, left intact deliberately rather than rewritten; the second states the retraction.

Things reviewers should weigh

0002 widens server_n_outputs_max to return n_batch for any mmproj model, not just gen-audio ones. Reserved logits are n_vocab * n_outputs, so every vision GGUF gains roughly 65 MB at a 32k vocab to 311 MB at a 152k vocab under our DefaultPhysicalBatch of 512, reaching about 1.2 GB only on the Blackwell 2048 default. This is upstream's own design in the draft, carried faithfully, and it is the largest merge risk here because it costs users who have nothing to do with TTS. It has not been measured against a real vision GGUF: there was none on the test host and no disk headroom to fetch one. Worth measuring, and worth raising on the upstream draft, before or shortly after merge.

Generation occasionally does not stop on its own. The model normally ends an utterance with its end-of-speech token; sometimes it does not and the request runs to the 512-frame cap. max_frames is exposed as a request parameter for that, documented at 12.5 frames per second. The observed rate was 2 in 44 uncapped requests, too small a sample to state as a figure, so the docs keep it qualitative. Note that the repetition penalty upstream's draft configures is inert at this pin (penalty_last_n = -1 is clamped to 0); restoring it did not reduce the runaways, so the line is kept for parity with a comment saying so.

A streaming request that fails immediately now sends a header-only WAV before aborting, because the sample-rate reply goes out before the task outcome is known. That is a consequence of fixing streaming first-byte latency, which went from 30.48 s to 0.015 s.

Capability fixes that came out of this

VisionSupported() treated any non-empty mmproj as proof of image input, which would have advertised every Qwen3-TTS model as vision-capable. Fixing that surfaced a larger pre-existing bug: GuessUsecases had no FLAG_VISION branch and returned true unconditionally, and syncKnownUsecasesFromString wrote that back into KnownUsecaseStrings where the next load parsed it as an explicit bit. On master today, every loaded model reports vision and an image input modality on /v1/models/capabilities and /api/show. That is repaired here.

One gallery entry, nemotron-3-nano-omni-30b-a3b-reasoning-apex, relied on that blanket fallthrough and now declares its capabilities explicitly.

CI note

CI does exercise this branch: .github/workflows/backend_pr.yml runs the path-filtered backend image matrix on pull_request, and the tests-llama-cpp-grpc / tests-llama-cpp-grpc-transcription jobs build and run the backend. Those two caught two real regressions on the first push (see below), and both are green now. A LLAMA_VERSION bump still rebuilds every variant on every accelerator, so please merge when someone can watch the matrix and do not stack it under other master pushes that would supersede those runs.

Two regressions CI caught, now fixed in f48efa465

Both hit every ordinary llama-cpp model, and neither showed up locally because every test on this branch loaded a TTS model.

1. Null dereference on any non-TTS model. server_slot::tts_ctx::reset() called mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline is only allocated for models carrying a gen-audio mmproj, and upstream reads ctx->pipeline before null-checking. server_slot::reset() runs during slot initialization for every model, so any chat model segfaulted the backend the moment it loaded. Guarded on the is_supported() predicate already defined beside it. The missing null check inside mtmd_helper_gen_audio_* is upstream's, so the guard is carried as a third documented deviation in 0002.
2. Zero-valued repeat_penalty, unrelated to TTS. PredictOptions.Penalty is a bare proto float, so a caller naming no repetition penalty sends 0 rather than omitting the field. Since 9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat outright because it would divide logits by zero, turning every such request into "Failed to initialize samplers". 0 is now treated as unset. This is a pure pin-bump regression.

tests/e2e-backends now passes 6 of 6 locally, including the loads the model and predict specs that were red, and Qwen3-TTS still synthesises on both paths.

🤖 Generated with Claude Code