feat(mtmd): add Gemma4 audio SMT support#19
Open
muggle-stack wants to merge 1 commit into
Open
Conversation
- Route Gemma4Audio through a dynamic-shape ONNX encoder model - Run the Gemma4 encoder session on ORT CPU while keeping the Qwen3-ASR split-encoder EP path unchanged - Wire Gemma4 audio prompts through SMT CLI and server paths Co-authored-by: codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR adds Gemma4Audio support to the SMT audio path.
audio_model.encoder_model_path.llama-serveraudio request path.encoder_model_pathat the dynamicaudio_encoder.q.onnx.Validation
K3 clean build from a fresh synced source tree:
cmake -B build-dynamic-smt -S . \ -DGGML_CPU_RISCV64_SPACEMIT=ON \ -DGGML_RV_ZBA=ON \ -DLLAMA_SERVER_SMT_VISION=ON \ -DSPACEMIT_ORT_DIR=/root/gemma4-asr/spacemit-ort-native \ -DCMAKE_BUILD_TYPE=Release \ -DLLAMA_BUILD_TESTS=OFF cmake --build build-dynamic-smt --target llama-server -j8Runtime command:
Model config used the dynamic encoder only:
Performance
Measured on
k3-2withllama-serverwarmup enabled. The first audio request includes ORT CPU encoder session creation and encoder warmup; subsequent requests are the steady-state path.What is the date today?Excuse me, could you tell me how to get to Central Park?Long ZH timing breakdown from server logs after warmup:
Notes
SpaceMIT EP was tested with the dynamic Gemma4 encoder and is not used for this path: EP 8-thread long-audio inference triggered a
libspacemit_ep.so.2.0.5SIGSEGV, and EP 1-thread returnedstd::bad_allocduring the audio request path. Keeping the dynamic Gemma4 encoder on ORT CPU avoids the crash while preserving the SMT server integration and existing Qwen3-ASR EP behavior.