Skip to content

feat(mtmd): add Gemma4 audio SMT support#19

Open
muggle-stack wants to merge 1 commit into
spacemit-com:spacemit-mtmdfrom
muggle-stack:gemma4-audio-smt
Open

feat(mtmd): add Gemma4 audio SMT support#19
muggle-stack wants to merge 1 commit into
spacemit-com:spacemit-mtmdfrom
muggle-stack:gemma4-audio-smt

Conversation

@muggle-stack

Copy link
Copy Markdown

Overview

This PR adds Gemma4Audio support to the SMT audio path.

  • Adds Gemma4 dynamic-shape ONNX audio encoder routing through audio_model.encoder_model_path.
  • Wires Gemma4 audio prompts through the SMT CLI and llama-server audio request path.
  • Uses ORT CPU for the Gemma4 dynamic encoder session; the existing Qwen3-ASR split frontend/backend EP path is unchanged.
  • Keeps unsupported fixed-shape bucket selection out of the runtime path. The model directory should point encoder_model_path at the dynamic audio_encoder.q.onnx.

Validation

K3 clean build from a fresh synced source tree:

cmake -B build-dynamic-smt -S . \
  -DGGML_CPU_RISCV64_SPACEMIT=ON \
  -DGGML_RV_ZBA=ON \
  -DLLAMA_SERVER_SMT_VISION=ON \
  -DSPACEMIT_ORT_DIR=/root/gemma4-asr/spacemit-ort-native \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF
cmake --build build-dynamic-smt --target llama-server -j8

Runtime command:

build-dynamic-smt/bin/llama-server \
  -m /root/gemma4-asr/gemma4-models-final/gemma-4-E2B-it-Q4_0.gguf \
  --media-backend smt \
  --smt-config-dir /root/gemma4-asr/gemma4-models-final \
  --host 127.0.0.1 --port 18080 \
  -t 8 -c 4096 --warmup

Model config used the dynamic encoder only:

"encoder_model_path": "audio_encoder.q.onnx"

Performance

Measured on k3-2 with llama-server warmup enabled. The first audio request includes ORT CPU encoder session creation and encoder warmup; subsequent requests are the steady-state path.

Case Audio duration Elapsed RTF Result
First short EN ASR request 1.358s 10.481s 7.716 What is the date today?
ZH ASR, steady-state run 1 14.158s 14.745s 1.041 Completed
ZH ASR, steady-state run 2 14.158s 14.674s 1.036 Completed
ES -> EN translation 3.520s 3.711s 1.054 Excuse me, could you tell me how to get to Central Park?

Long ZH timing breakdown from server logs after warmup:

  • audio encode: ~2.61-2.67s / 356 audio tokens
  • audio prefill: ~5.34s / 356 audio tokens
  • decode: ~6.35s / 70 output tokens
  • total: ~14.66-14.73s

Notes

SpaceMIT EP was tested with the dynamic Gemma4 encoder and is not used for this path: EP 8-thread long-audio inference triggered a libspacemit_ep.so.2.0.5 SIGSEGV, and EP 1-thread returned std::bad_alloc during the audio request path. Keeping the dynamic Gemma4 encoder on ORT CPU avoids the crash while preserving the SMT server integration and existing Qwen3-ASR EP behavior.

- Route Gemma4Audio through a dynamic-shape ONNX encoder model
- Run the Gemma4 encoder session on ORT CPU while keeping the Qwen3-ASR split-encoder EP path unchanged
- Wire Gemma4 audio prompts through SMT CLI and server paths

Co-authored-by: codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant