Add descriptor-driven multimodal composition loading#38
Draft
jayden0701 wants to merge 13 commits into
Draft
Conversation
## Summary Brings the Quick.AI public API up to the v0.4.0 surface in a single commit. Models are now identified by a string model id (not a C enum): each model self-registers its descriptor at load time and the catalog is exposed via `getModelCatalogJson()` (C API) and `ModelCatalog` (Android AAR). The build discovers model directories generically, so additional models can be dropped in without editing build files or the public API. The tree and history contain no proprietary model sources or references; `git grep -i gauss` matches only the Qualcomm SDK `GAUSSIAN` constant. ## Change - API: string-id descriptor registry + catalog JSON; per-model self-registration via constructors; a lazy (Meyers-singleton) registry so cross-library registration survives static-init order. - API: generic multimodal composer and a vision-encoder capability, decoupled from any specific model. - QNN: set the HTP backend-ext-config before multi-model sub-model loads; gemma4-e2b-qnn (NATIVE/NPU) bring-up. - AAR: `ModelCatalog.selectableFamilies()` to hide embedding-only models in the Run/OpenAI and Chat family pickers. - Build: model build hooks (meson + ndk-build) auto-discover model directories instead of naming them, so proprietary models plug in cleanly. - Guards: allow-list `.gitignore` and a `pre-push` hook that block any non-allow-listed model source directory from reaching the public remote (allow-list = `src/models/qnn/gemma4-e2b-qnn`). Verified on device (S26 Ultra): the Chat and OpenAI tabs run qwen3-0.6b, gemma4-e2b-qnn (NPU), and function_gemma; the catalog lists no proprietary families. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…edding + JNI/Kotlin)
…ction UI - Add user-editable MODEL BASE PATH field (default: /sdcard/Download/aistudio-mobile/models/) replacing the previously hardcoded path, available in both OpenAI and Chat tabs - Replace MODEL NAME dropdown with read-only folder name display derived from the model descriptor; show error message when the expected folder is missing - Remove Quantization chip selector from the OpenAI tab (W4A32 used internally) - Change default model from Gemma4 LiteRT/GPU to Gemma4 Native/NPU (GEMMA4_E2B_QNN) - Pass modelBasePath through createEngine() to LiteRTLm and buildLoadRequest() - Add bordered card style to Chat tab's model selection section - Preserve modelBasePathText across theme rebuilds from both OpenAI and Chat tabs Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Introduce the first planning and catalog layer for pluggable multimodal model composition. This adds role-aware model descriptors, Android catalog decoding, placeholder LFM/SigLIP/JEPA component descriptors, and lightweight contract checks for the new catalog shape. Also document the repository orientation and the staged implementation plan so later sessions can continue Task-by-Task without relying on conversation context. Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
Signed-off-by: jrock-oh <jrock.oh@samsung.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
loadMultimodalCompositionJsonC API/JNI support for descriptor-driven multimodal compositions.