diff --git a/README.md b/README.md index 70dcb86b1b..948dac57dc 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ OpenHuman is three things most assistants aren't: **a brain** that builds a pers ### 🔬 The deep researcher & doer - **[SuperContext](https://tinyhumans.gitbook.io/openhuman/features/super-context)**: a research scout sweeps your memory and files before the model reads your first message. No cold starts. -- **Batteries included**: managed [web search](https://tinyhumans.gitbook.io/openhuman/features/native-tools/web-search), powered by [Exa](https://exa.ai), is included with your OpenHuman subscription and needs no API key; bring your own Exa key to search directly on your own Exa account and billing. Plus scraper, coder toolset, a real [browser](https://tinyhumans.gitbook.io/openhuman/features/native-tools/browser-and-computer), and [native voice](gitbooks/features/native-tools/voice.md) with in-process Whisper. [Model routing](https://tinyhumans.gitbook.io/openhuman/features/model-routing) picks the right LLM per workload on one subscription, with [local AI optional](https://tinyhumans.gitbook.io/openhuman/features/model-routing/local-ai). +- **Batteries included**: managed [web search](https://tinyhumans.gitbook.io/openhuman/features/native-tools/web-search), powered by [Exa](https://exa.ai), is included with your OpenHuman subscription and needs no API key; bring your own Exa key to search directly on your own Exa account and billing. Plus scraper, coder toolset, a real [browser](https://tinyhumans.gitbook.io/openhuman/features/native-tools/browser-and-computer), and [native voice](gitbooks/features/native-tools/voice.md) with in-process Whisper. [Model routing](https://tinyhumans.gitbook.io/openhuman/features/model-routing) picks the right LLM per workload on one subscription. That subscription is a default, not a lock-in: point any workload at [your own provider key or a fully local Ollama model](https://tinyhumans.gitbook.io/openhuman/features/model-routing/local-and-byok-models), and mix the three however you like. - **[Meeting agents](https://tinyhumans.gitbook.io/openhuman/features/mascot/meeting-agents)**: joins **Meet, Zoom, Teams, and Webex** with a face and a voice. It auto-joins from your calendar, streams a live transcript, answers by name, and files a summary with action items. - **[Image & video generation](https://tinyhumans.gitbook.io/openhuman/features/native-tools)**: Seedream/SeedEdit images and Seedance/Veo video, straight into your workspace on the same subscription. - **[17 messaging channels](https://tinyhumans.gitbook.io/openhuman/features/channels)**: Telegram, Discord, Slack, WhatsApp, Signal, iMessage… plus **native email** (IMAP IDLE + SMTP). Your agent reaches you where you already are. diff --git a/gitbooks/SUMMARY.md b/gitbooks/SUMMARY.md index 679a777aea..0e25f4339a 100644 --- a/gitbooks/SUMMARY.md +++ b/gitbooks/SUMMARY.md @@ -43,6 +43,7 @@ * [Themes & Theme Studio](features/theming.md) * [Smart Token Compression](features/token-compression.md) * [Automatic Model Routing](features/model-routing/README.md) + * [Local models & bring your own key](features/model-routing/local-and-byok-models.md) * [Local AI (optional)](features/model-routing/local-ai.md) * [Available Tools](features/native-tools/README.md) * [Web Search](features/native-tools/web-search.md) diff --git a/gitbooks/features/model-routing/README.md b/gitbooks/features/model-routing/README.md index fe2ccb1dae..1ff2130d7f 100644 --- a/gitbooks/features/model-routing/README.md +++ b/gitbooks/features/model-routing/README.md @@ -43,9 +43,11 @@ The router wraps several pre-created providers (Anthropic, OpenAI, Google, Groq, The exact mappings are configurable; the defaults ship sensible per-provider routes. -## One subscription +## One subscription, or your own -Routing happens behind a single OpenHuman subscription. You don't hold separate API keys for Anthropic, OpenAI, Google etc., the backend brokers access, and the router picks the right one per task. That's the "one subscription, many providers" promise from the README, made concrete. +Routing happens behind a single OpenHuman subscription by default. You don't hold separate API keys for Anthropic, OpenAI, Google etc., the backend brokers access, and the router picks the right one per task. That's the "one subscription, many providers" promise from the README, made concrete. + +The subscription is the default, not a requirement. The same router works against **your own provider key** or a **fully local model**, per workload, and you can mix all three. See [Local models & bring your own key](local-and-byok-models.md) for setup and for what each route supports for chat, vision, and embeddings. ## Overriding routes @@ -97,4 +99,5 @@ Routing isn't a UI dropdown. The agent loop itself emits hints based on what it' - [Smart Token Compression](../token-compression.md). what makes large reasoning calls affordable. - [Native Tools](../native-tools/README.md). different tool calls hint at different routes. +- [Local models & bring your own key](local-and-byok-models.md). run on your own key or fully on-device. - [Local AI (optional)](local-ai.md). lightweight chat hints can run on-device. diff --git a/gitbooks/features/model-routing/local-ai.md b/gitbooks/features/model-routing/local-ai.md index 59c95122b2..b57be04060 100644 --- a/gitbooks/features/model-routing/local-ai.md +++ b/gitbooks/features/model-routing/local-ai.md @@ -15,7 +15,7 @@ This is deliberate scoping. The previous design tried to put every modality on-d | Workload | Default model | Implementation | | ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -| **Memory embeddings** | `all-minilm:latest` | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | +| **Memory embeddings** | `bge-m3` | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | | **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` - source / topic / global summary builders for the Memory Tree. | | **Heartbeat loop** | small chat model | `src/openhuman/heartbeat/` - periodic background reflection. | | **Learning / reflection** | small chat model | `src/openhuman/learning/reflection.rs` - passes that consolidate what was learned. | @@ -31,7 +31,7 @@ Each of these is an explicit opt-in. Turning on local AI does not silently route | -------------- | ---------------------------------------------------------------------------------------------- | | **Chat** | Frontier reasoning quality unless `chat_provider` is explicitly set to a local provider. | | **Reasoning** | Stronger multi-step quality unless `reasoning_provider` is explicitly set to a local provider. | -| **Vision** | Same. | +| **Vision** | Same, unless `vision_provider` points at a local vision-capable model. See below. | | **STT** | Backend-proxied transcription (`src/openhuman/voice/cloud_transcribe.rs`). | | **TTS** | Hosted [text-to-speech](../native-tools/voice.md) under the hood (`reply_speech.rs`). | | **Web search** | Backend proxy (no API key on your machine). | @@ -102,10 +102,23 @@ Local AI is worth turning on if any of these are true: It is **not** worth turning on if you only have a few sources connected, the cloud path is faster and the privacy benefit is small. There is also a hardware cost: Ollama and a small Gemma model want a few GB of RAM and pull a few GB of weights. +## Local vision + +Vision is a separate capability from chat, and **most small local models cannot do it**. Ollama does not reject an image sent to a text-only model: it drops the image and answers from the prompt text, which produces a fluent description of something the model never saw. OpenHuman therefore resolves the vision model through a capability check and refuses to route a vision request at a chat-only model. + +What that means in practice: + +- `local_ai.vision_model_id` must name a vision-capable model. `moondream:1.8b-v2-q4_K_S` (~1.7 GB) is the smallest option; `gemma3:4b-it-qat` and `gemma4:e4b-it-q8_0` handle chat and vision with one set of weights. +- Gemma 3 is text-only at 270M and 1B, and multimodal from 4B up. **Gemma 3n is a different model and is text-only at every size**, so it is not usable for vision even though it is a capable chat model. +- Leaving `vision_model_id` empty is a valid "no local vision" setup. A vision request then returns a message naming the config key to set and the models to pull, rather than failing silently. +- If a configured vision model turns out to be chat-only, the core logs a warning and falls back to a vision-capable default instead of sending images to a model that would ignore them. + +The full per-model capability table lives in [Local models & bring your own key](local-and-byok-models.md). + ## What you'll need - [**Ollama**](https://ollama.com) installed and running locally, or [**LM Studio**](https://lmstudio.ai) with the local server enabled. -- Enough disk for the models (`gemma3:1b-it-qat` \~700 MB, `all-minilm:latest` \~23 MB). +- Enough disk for the models (`gemma3:1b-it-qat` \~1.0 GB, `bge-m3` \~1.2 GB, plus \~1.7 GB if you add Moondream for vision). - Enough RAM to keep the model resident (8 GB+ recommended, 16 GB+ ideal). OpenHuman handles the rest: lifecycle (`src/openhuman/inference/local/service/`), API clients, health checks, and graceful fallback to remote when the local provider disappears. @@ -119,6 +132,7 @@ OpenHuman handles the rest: lifecycle (`src/openhuman/inference/local/service/`) ## See also +- [Local models & bring your own key](local-and-byok-models.md). Per-model capability table and BYOK setup. - [Memory Tree](../obsidian-wiki/memory-tree.md). what local embeddings + summarization power. - [Automatic Model Routing](README.md). how lightweight chat hints prefer the local provider. - [Privacy & Security](../privacy-and-security.md). what moves on-device when you opt in. diff --git a/gitbooks/features/model-routing/local-and-byok-models.md b/gitbooks/features/model-routing/local-and-byok-models.md new file mode 100644 index 0000000000..f2c117e944 --- /dev/null +++ b/gitbooks/features/model-routing/local-and-byok-models.md @@ -0,0 +1,188 @@ +--- +description: >- + Three ways to power OpenHuman: the managed subscription, your own provider key + (BYOK), or fully local models via Ollama. What each one supports for chat, + vision, and embeddings, and how to configure it. +icon: sliders +--- + +# Local models & bring your own key + +The OpenHuman subscription is the **default**, not a requirement. Inference can come from any of three places, and you can mix them per workload: run embeddings locally, chat on your own Anthropic key, and leave vision on the managed route, all at once. + +This page covers how to set up the two self-owned options and, importantly, **what each one can actually do**. Not every local model can see images, and picking a chat-only model for vision work is the single most common way to end up with a setup that looks configured but quietly does the wrong thing. + +## The three routes at a glance + +| | **Managed (default)** | **BYOK cloud** | **Local (Ollama / LM Studio)** | +| --- | --- | --- | --- | +| **Chat & reasoning** | Included | Your key, your billing | Yes, quality scales with model size | +| **Vision** | Included | Your key, if the model supports images | Yes, but only with a vision-capable model | +| **Embeddings** | Included | Your key, if the provider serves embeddings | Yes, `bge-m3` recommended | +| **Speech to text** | Included | Not routed through BYOK | Local Whisper available | +| **Text to speech** | Included | Not routed through BYOK | Local Piper available | +| **Web search** | Included, no key needed | Bring your own Exa key | Not applicable | +| **Inference data leaves your machine** | Yes, to the OpenHuman backend | Yes, to your chosen provider | No | +| **API keys to manage** | None | One per provider | None | + +That last row is deliberately about **inference data only**. Sign-in, managed integration OAuth, billing, and hosted features such as meeting agents still use the OpenHuman backend even when inference is entirely yours, so running local models is not by itself a guarantee that nothing leaves the machine. If you want a hard guarantee that no inference leaves the machine, use [Privacy Mode](../privacy-mode.md), which enforces the local-only path in the Rust core rather than relying on configuration alone. + +## Route A: local models with Ollama + +### 1. Install Ollama and pull a model + +Install [Ollama](https://ollama.com), then pull what you need. Every model named on this page is pullable from the public Ollama library with no extra setup: + +```bash +ollama pull gemma3:1b-it-qat # small chat model +ollama pull bge-m3 # embeddings +ollama pull moondream:1.8b-v2-q4_K_S # vision, small +``` + +### 2. Know what each model supports + +This is the part that bites people. A model that only does text will still **accept** an image request on Ollama: it silently drops the image and answers from the prompt text alone, which reads as a confident but entirely invented description. OpenHuman guards against this by refusing to route a vision request at a chat-only model, but it is worth knowing which is which. + +| Model | Download | Chat | Vision | Embeddings | +| --- | --- | --- | --- | --- | +| `gemma3:270m-it-qat` | 0.2 GB | Yes | No | No | +| `gemma3:1b-it-qat` | 1.0 GB | Yes | No | No | +| `gemma3:4b-it-qat` | 4.0 GB | Yes | **Yes** | No | +| `gemma3n:e4b-it-q8_0` | 9.5 GB | Yes | No | No | +| `gemma4:e4b-it-q8_0` | 11.6 GB | Yes | **Yes** | No | +| `moondream:1.8b-v2-q4_K_S` | 1.7 GB | Minimal | **Yes** | No | +| `llava:7b` | 4.7 GB | Minimal | **Yes** | No | +| `bge-m3` | 1.2 GB | No | No | **Yes**, 1024 dim | +| `all-minilm:latest` | 0.05 GB | No | No | 384 dim, too small for Memory Tree | + +Two traps worth calling out: + +- **Gemma 3 is split by size.** The 270M and 1B builds are text-only. Vision starts at 4B. Picking `gemma3:1b-it-qat` for vision gets you a text-only model. +- **`gemma3n` is not `gemma3`.** Despite the name, Gemma 3n is a separate, text-only model on Ollama. It is a fine chat model and a bad vision model. + +For embeddings, prefer **`bge-m3`**. The Memory Tree stores vectors in a fixed 1024-dimension on-disk format, so a 384-dimension model such as `all-minilm` or a 768-dimension model such as `nomic-embed-text` will fail the dimension check at embed time. + +### 3. Point OpenHuman at it + +The quickest path is the desktop app: **Settings → AI & Skills → Local AI** exposes RAM tier presets that set every model ID for you and pull the weights. The tiers are: + +| Tier | Chat | Vision | Embeddings | Download | +| --- | --- | --- | --- | --- | +| 1 GB | `gemma3:270m-it-qat` | Disabled | `all-minilm:latest` (see note) | ~0.3 GB | +| 2-4 GB | `gemma3:1b-it-qat` | Disabled | `bge-m3` | ~2.3 GB | +| 4-8 GB | `gemma3:1b-it-qat` | `moondream:1.8b-v2-q4_K_S` | `all-minilm:latest` (see note) | ~2.8 GB | +| 8-16 GB | `gemma3:4b-it-qat` | `gemma3:4b-it-qat` | `bge-m3` | ~5.2 GB | +| 16 GB+ | `gemma4:e4b-it-q8_0` | `gemma4:e4b-it-q8_0` | `bge-m3` | ~12.8 GB | + +The two highest tiers use one multimodal model for both chat and vision, so you download a single set of weights rather than a chat model plus a separate vision sidecar. + +{% hint style="warning" %} +**The 1 GB and 4-8 GB tiers ship `all-minilm:latest`, which the Memory Tree cannot use.** It emits 384-dimension vectors and the Memory Tree's on-disk format is fixed at 1024, so memory embedding fails the dimension check at embed time. Those two tiers are usable for local chat and, on the 4-8 GB tier, vision, but if you want local Memory Tree embeddings set `embedding_model_id = "bge-m3"` explicitly after applying the preset, or pick the 2-4 GB tier or above. Aligning those presets is tracked as follow-up. +{% endhint %} + +To configure by hand, the keys live under `[local_ai]` in `config.toml`: + +```toml +[local_ai] +runtime_enabled = true +opt_in_confirmed = true +provider = "ollama" # or "lm_studio" +chat_model_id = "gemma3:4b-it-qat" +vision_model_id = "gemma3:4b-it-qat" # must be vision-capable +embedding_model_id = "bge-m3" +``` + +Leaving `vision_model_id` empty means "no local vision", which is a valid setup. A vision request then returns a message telling you what to set, rather than failing silently. + +### 4. Route workloads to it + +Turning local AI on does not move everything on-device. You choose per workload with a provider string of the form `ollama:`: + +```toml +chat_provider = "ollama:gemma3:4b-it-qat" +vision_provider = "ollama:gemma3:4b-it-qat" +embeddings_provider = "ollama:bge-m3" +``` + +The full set of workload fields is `chat_provider`, `reasoning_provider`, `agentic_provider`, `coding_provider`, `vision_provider`, `memory_provider`, `embeddings_provider`, `heartbeat_provider`, `learning_provider`, and `subconscious_provider`. Any field left unset, blank, or set to `cloud` stays on the default route. + +#### Attaching images in chat needs one more flag + +`vision_provider` routes the **vision workload** — image summaries and the OCR/description path. It does not by itself let you attach an image to a chat or agent turn. + +A turn only rehydrates image attachments when the resolved chat model is known to accept them, and for a local model that knowledge comes from the per-model registry, not from `vision_provider`. Set the model's **vision** flag in Settings → AI (the custom-model dialog), which records it in `model_registry`: + +```toml +[[model_registry]] +id = "gemma3:4b-it-qat" +provider = "ollama" +vision = true +``` + +Without that flag the images are stripped before dispatch and the model answers from the text alone — fluently, and with no indication that it never saw the picture. + +See [Local AI (optional)](local-ai.md) for the deeper runtime detail, LM Studio setup, and troubleshooting. + +## Route B: bring your own key + +BYOK keeps the routing, memory, tools, and agent harness exactly as they are, and swaps out who serves the tokens. Your key, your account, your billing, no OpenHuman inference charges. + +### 1. Add the provider + +Add your key in the desktop app under the LLM settings, which stores it in the OS keyring rather than in plain config. OpenHuman ships presets for these slugs, so you do not need to supply an endpoint: + +`openai`, `anthropic`, `google`, `openrouter`, `orcarouter`, `groq`, `mistral`, `deepseek`, `together`, `fireworks`, `cerebras`, `xai`, `moonshot`, `gmi`, `huggingface`, `nvidia`, `zai`, `minimax`, `stepfun`, `kilocode`, `deepinfra`, `novita`, `venice`, `vercel-ai-gateway`, `sumopod`, `modelscope` + +Anything else that speaks the OpenAI-compatible API works too: register it with your own slug and endpoint, and it routes the same way. + +### 2. Route workloads to it + +Provider strings follow `:`, using the same workload fields as the local route: + +```toml +chat_provider = "anthropic:claude-sonnet-4" +reasoning_provider = "openai:gpt-5.1" +coding_provider = "deepseek:deepseek-coder" +vision_provider = "openai:gpt-5.1" +``` + +To make one provider the default for everything that is not pinned, set `primary_cloud` to its slug. Every workload left on `cloud` then resolves to that provider instead of the OpenHuman backend. + +### 3. Check the model supports the workload + +BYOK inherits your provider's capabilities, not OpenHuman's. Before pinning `vision_provider`, confirm the model you named accepts image input, and before pinning `embeddings_provider`, confirm the provider serves an embeddings endpoint. Not every chat provider does. + +## Mixing routes + +The workload fields are independent, so a common privacy-conscious setup keeps recurring background work on-device and reserves a strong cloud model for the turns that need it: + +```toml +# On-device: everything that runs constantly over personal data +embeddings_provider = "ollama:bge-m3" +memory_provider = "ollama:gemma3:1b-it-qat" +heartbeat_provider = "ollama:gemma3:1b-it-qat" +subconscious_provider = "ollama:gemma3:1b-it-qat" + +# Your own key: the turns where quality matters +chat_provider = "anthropic:claude-sonnet-4" +reasoning_provider = "anthropic:claude-sonnet-4" +``` + +## Troubleshooting + +**"no local vision model is configured"** means `local_ai.vision_model_id` is empty. Set it to a vision-capable model and pull it, or point `vision_provider` at a cloud model instead. + +**"local vision model ... is not available"** means the model is configured but not pulled. Run the `ollama pull` command in the message. + +**Vision answers look plausible but describe the wrong image.** You are almost certainly on a chat-only model. Check `vision_model_id` against the capability table above. Current builds refuse this routing and fall back to a vision-capable model, so this points at an older build or a provider outside the local path. + +**A model you selected keeps reverting.** Local chat model IDs are checked against a supported list, and an unrecognized ID falls back to the default. Use one of the IDs from the tier table. + +**Embeddings fail with a dimension error.** The Memory Tree needs 1024-dimension vectors. Use `bge-m3`. + +## See also + +- [Local AI (optional)](local-ai.md). Runtime detail, LM Studio, and the opt-in flags. +- [Automatic Model Routing](README.md). How hints pick a model per task. +- [Privacy Mode](../privacy-mode.md). Enforcing local-only inference in the core. +- [Privacy & Security](../privacy-and-security.md). What moves on-device when you opt in. diff --git a/src/openhuman/inference/local/install_piper.rs b/src/openhuman/inference/local/install_piper.rs index 64914dfe21..50acf383fa 100644 --- a/src/openhuman/inference/local/install_piper.rs +++ b/src/openhuman/inference/local/install_piper.rs @@ -619,14 +619,69 @@ mod tests { use super::*; use crate::openhuman::inference::local::voice_install_common::reset_status; + /// Point [`paths::shared_root_dir`] at a test's own `TempDir`. + /// + /// `shared_root_dir` only honours `config.workspace_dir` when + /// `OPENHUMAN_WORKSPACE` is set; without it every write below lands in the + /// developer's real `~/.openhuman/bin/piper` and the cleanup deletes their + /// installed Piper (CodeRabbit, #5253). Setting the variable for the + /// duration keeps writes *and* cleanup inside the `TempDir`, so the + /// `TempDir`'s own `Drop` is the cleanup and it runs on unwind too: a + /// failing assertion can no longer leave a stub binary behind for the next + /// test to trip over. + /// + /// Callers must already hold [`shared_install_lock`] — this mutates + /// process-wide environment state. + /// + /// `#[cfg(unix)]` because its only consumer is the unix-only permissions + /// test; unconditional would be dead code on Windows, where clippy runs + /// with `-D warnings`. + #[cfg(unix)] + struct SharedRootOverride { + previous: Option, + } + + #[cfg(unix)] + impl SharedRootOverride { + fn set(root: &std::path::Path) -> Self { + let previous = std::env::var_os("OPENHUMAN_WORKSPACE"); + std::env::set_var("OPENHUMAN_WORKSPACE", root); + Self { previous } + } + } + + #[cfg(unix)] + impl Drop for SharedRootOverride { + fn drop(&mut self) { + match self.previous.as_ref() { + Some(previous) => std::env::set_var("OPENHUMAN_WORKSPACE", previous), + None => std::env::remove_var("OPENHUMAN_WORKSPACE"), + } + } + } + #[cfg(unix)] #[test] fn non_executable_workspace_binary_is_skipped_so_path_can_win() { // #5045 review (Codex P2): when the chmod repair fails, returning the // 0644 workspace copy anyway pins resolution to a binary that cannot // launch and makes the PIPER_BIN/PATH fallback unreachable. + // + // This test must hold the module lock: it mutates OPENHUMAN_WORKSPACE, + // which is process-wide, and `reset_status`/install state is shared + // with every sibling install_piper / install_whisper / paths test. + // + // `workspace_piper_binary_candidates` resolves through + // `paths::shared_root_dir`, which ignores `config.workspace_dir` unless + // OPENHUMAN_WORKSPACE is set and otherwise returns the real + // `~/.openhuman/bin/piper`. `SharedRootOverride` sets it to this test's + // TempDir so the stub written below, and its cleanup, stay inside the + // TempDir instead of touching a developer's installed Piper. use std::os::unix::fs::PermissionsExt; + let _g = shared_install_lock(); let (_dir, config) = temp_config(); + let _root = SharedRootOverride::set(&config.workspace_dir); + wipe_shared_install_dir(&config); let candidates = paths::workspace_piper_binary_candidates(&config); let candidate = candidates.first().expect("at least one candidate").clone(); std::fs::create_dir_all(candidate.parent().unwrap()).unwrap(); @@ -644,6 +699,10 @@ mod tests { Some(candidate.as_path()), "an executable workspace binary is still preferred" ); + + // No tail cleanup: it would only run when every assertion above passed. + // `_dir` (TempDir) and `_root` (SharedRootOverride) clean up on drop, + // which happens on the panic path too. } fn temp_config() -> (tempfile::TempDir, Config) { diff --git a/src/openhuman/inference/local/service/vision_embed.rs b/src/openhuman/inference/local/service/vision_embed.rs index 7a27c9d7a3..7d14691931 100644 --- a/src/openhuman/inference/local/service/vision_embed.rs +++ b/src/openhuman/inference/local/service/vision_embed.rs @@ -52,9 +52,75 @@ impl LocalAiService { ); } self.bootstrap(config).await; - let vision_model = model_ids::effective_vision_model_id(config); - self.ensure_ollama_model_available(config, &vision_model, "vision") - .await?; + + // Resolve through `resolve_vision_model_choice` rather than + // `effective_vision_model_id`: the latter returns an empty string when + // no vision model is configured, which used to be handed straight to + // `ensure_ollama_model_available` and became a nameless `POST + // /api/pull` retried three times before failing opaquely (#5146). + // The resolver guarantees a non-empty, vision-capable id or a message + // that says what to configure. + // NOTE: this Err arm is defence in depth, not a reachable branch here. + // `resolve_vision_model_choice` fails only when no vision model is + // configured, and that same condition makes `vision_mode_for_config` + // report `Disabled` (a blank `vision_model_id` cannot match any + // vision-enabled preset, so the tier resolves to `Custom` -> Disabled), + // which returns above. The empty case is covered directly in + // `model_ids::tests::resolve_vision_model_id_errors_when_unconfigured`. + let choice = match model_ids::resolve_vision_model_choice(config) { + Ok(choice) => choice, + Err(error) => { + self.status.lock().vision_state = "missing".to_string(); + tracing::warn!( + target: "local_ai::vision", + %error, + "[local_ai:vision] no vision-capable model resolved; refusing request" + ); + return Err(error); + } + }; + let vision_model = choice.model; + // A capability substitution means we are about to talk about a model the + // user never named. Carry that into the error text so "pull moondream" + // cannot read as a non-sequitur to someone who configured gemma3. + let substitution_note = choice + .replaced + .as_deref() + .map(|configured| { + format!( + " Your configured `{configured}` cannot accept images, so OpenHuman \ + selected `{vision_model}` instead; set `local_ai.vision_model_id` to \ + a vision-capable model to choose your own." + ) + }) + .unwrap_or_default(); + tracing::debug!( + target: "local_ai::vision", + model = %vision_model, + substituted_for = ?choice.replaced, + "[local_ai:vision] resolved vision-capable model" + ); + + // A model that is configured but not pulled (and cannot be pulled) + // must also read as a vision problem, not a generic pull failure. + if let Err(error) = self + .ensure_ollama_model_available(config, &vision_model, "vision") + .await + { + self.status.lock().vision_state = "missing".to_string(); + tracing::warn!( + target: "local_ai::vision", + model = %vision_model, + substituted_for = ?choice.replaced, + %error, + "[local_ai:vision] vision model unavailable" + ); + return Err(format!( + "local vision model `{vision_model}` is not available: {error}. \ + Pull it with `ollama pull {vision_model}`, or route the vision \ + workload to a cloud provider with `vision_provider`.{substitution_note}" + )); + } let images: Vec = image_refs .iter() @@ -339,4 +405,229 @@ mod tests { .unwrap_err(); assert!(err.contains("local ai is disabled")); } + + // ── #5146 §Part 1: which model a vision request actually reaches ──────── + // + // These drive the real `vision_prompt` path against a mock Ollama server. + // `ready_service` marks the status "ready", which makes `bootstrap` return + // early, so no process launch or network beyond the mock is involved. + + /// Mock Ollama exposing `/api/tags` with `installed` present, and an + /// `/api/generate` that echoes back the `model` field it was sent. The + /// echo is what lets a test assert *which* model the request targeted. + fn mock_ollama_echoing_requested_model(installed: &'static str) -> Router { + use axum::routing::get; + Router::new() + .route( + "/api/tags", + get(move || async move { + Json(json!({ + "models": [ + { "name": installed, "modified_at": "", "size": 0u64, "digest": "a" } + ] + })) + }), + ) + .route( + "/api/generate", + post(|Json(body): Json| async move { + Json(json!({ + "response": body["model"].as_str().unwrap_or(""), + "done": true + })) + }), + ) + } + + /// A configured, genuinely vision-capable model must reach Ollama unchanged. + /// + /// Before #5146 the `MVP_ALLOWED_VISION_MODELS = &[""]` allowlist rewrote + /// this to the empty string, so the request went out with `model: ""`. + #[tokio::test] + async fn vision_prompt_sends_the_configured_vision_capable_model() { + let _guard = crate::openhuman::inference::inference_test_guard(); + + let base = spawn_mock(mock_ollama_echoing_requested_model("llava:7b")).await; + unsafe { + std::env::set_var("OPENHUMAN_OLLAMA_BASE_URL", &base); + } + + let mut config = enabled_config(); + config.local_ai.vision_model_id = "llava:7b".to_string(); + let service = ready_service(&config); + + let result = service + .vision_prompt( + &config, + "describe", + &["data:image/png;base64,QUJD".to_string()], + None, + ) + .await; + + unsafe { + std::env::remove_var("OPENHUMAN_OLLAMA_BASE_URL"); + } + + assert_eq!( + result.expect("vision prompt should succeed"), + "llava:7b", + "the configured vision model must reach Ollama unchanged" + ); + } + + /// A chat-only model configured for vision must never be the model that + /// receives the images. + /// + /// Ollama accepts an `images` array against a text-only model, discards it, + /// and answers from the prompt alone, so passing `gemma3n` through would + /// return a fabricated description rather than an error. + #[tokio::test] + async fn vision_prompt_never_routes_images_at_a_chat_only_model() { + let _guard = crate::openhuman::inference::inference_test_guard(); + + let base = spawn_mock(mock_ollama_echoing_requested_model( + "moondream:1.8b-v2-q4_K_S", + )) + .await; + unsafe { + std::env::set_var("OPENHUMAN_OLLAMA_BASE_URL", &base); + } + + let mut config = enabled_config(); + // Text-only on Ollama, despite sharing a prefix with multimodal gemma3. + config.local_ai.vision_model_id = "gemma3n:e4b-it-q8_0".to_string(); + let service = ready_service(&config); + + let result = service + .vision_prompt( + &config, + "describe", + &["data:image/png;base64,QUJD".to_string()], + None, + ) + .await; + + unsafe { + std::env::remove_var("OPENHUMAN_OLLAMA_BASE_URL"); + } + + let model_used = result.expect("vision prompt should succeed"); + assert_ne!( + model_used, "gemma3n:e4b-it-q8_0", + "images must never be sent to a chat-only model" + ); + assert_eq!(model_used, "moondream:1.8b-v2-q4_K_S"); + } + + /// A configured-but-unpullable vision model must report a vision problem + /// naming the model and the `ollama pull` that fixes it. + #[tokio::test] + async fn vision_prompt_reports_an_unavailable_vision_model() { + use axum::routing::get; + let _guard = crate::openhuman::inference::inference_test_guard(); + + // Empty tag list, and a pull that refuses: nothing to fall back to. + let app = Router::new() + .route("/api/tags", get(|| async { Json(json!({ "models": [] })) })) + .route( + "/api/pull", + post(|| async { + ( + axum::http::StatusCode::INTERNAL_SERVER_ERROR, + "pull refused", + ) + }), + ); + let base = spawn_mock(app).await; + unsafe { + std::env::set_var("OPENHUMAN_OLLAMA_BASE_URL", &base); + } + + let mut config = enabled_config(); + config.local_ai.vision_model_id = "llava:7b".to_string(); + let service = ready_service(&config); + + let err = service + .vision_prompt( + &config, + "describe", + &["data:image/png;base64,QUJD".to_string()], + None, + ) + .await + .expect_err("an unpullable vision model must fail"); + + unsafe { + std::env::remove_var("OPENHUMAN_OLLAMA_BASE_URL"); + } + + assert!( + err.contains("llava:7b"), + "error should name the model: {err}" + ); + assert!( + err.contains("ollama pull"), + "error should say how to install it: {err}" + ); + assert_eq!(service.status.lock().vision_state, "missing"); + } + + /// greptile #5253: when the capability guard swaps a chat-only model for a + /// vision-capable default, an unavailable-model error must say so. Without + /// this the user is told to `ollama pull moondream:…` having configured + /// `gemma3n:…`, with nothing connecting the two. + #[tokio::test] + async fn unavailable_error_explains_a_capability_substitution() { + use axum::routing::get; + let _guard = crate::openhuman::inference::inference_test_guard(); + + let app = Router::new() + .route("/api/tags", get(|| async { Json(json!({ "models": [] })) })) + .route( + "/api/pull", + post(|| async { + ( + axum::http::StatusCode::INTERNAL_SERVER_ERROR, + "pull refused", + ) + }), + ); + let base = spawn_mock(app).await; + unsafe { + std::env::set_var("OPENHUMAN_OLLAMA_BASE_URL", &base); + } + + let mut config = enabled_config(); + // Chat-only: the guard substitutes the vision-capable default. + config.local_ai.vision_model_id = "gemma3n:e4b-it-q8_0".to_string(); + let service = ready_service(&config); + + let err = service + .vision_prompt( + &config, + "describe", + &["data:image/png;base64,QUJD".to_string()], + None, + ) + .await + .expect_err("an unpullable substituted model must fail"); + + unsafe { + std::env::remove_var("OPENHUMAN_OLLAMA_BASE_URL"); + } + + assert!( + err.contains("gemma3n:e4b-it-q8_0"), + "error must name the model the user actually configured: {err}" + ); + assert!( + err.contains("cannot accept images"), + "error must explain why it was replaced: {err}" + ); + assert!( + err.contains(crate::openhuman::inference::model_ids::DEFAULT_OLLAMA_VISION_MODEL), + "error must name the substitute it is asking the user to pull: {err}" + ); + } } diff --git a/src/openhuman/inference/mod.rs b/src/openhuman/inference/mod.rs index 8c26e14eee..9f4d3f5e4b 100644 --- a/src/openhuman/inference/mod.rs +++ b/src/openhuman/inference/mod.rs @@ -36,6 +36,7 @@ mod schemas; pub mod sentiment; pub mod temperature; pub mod types; +pub mod vision_models; pub mod voice; pub use ops as rpc; diff --git a/src/openhuman/inference/model_ids.rs b/src/openhuman/inference/model_ids.rs index bc88534d50..0bfccb0cf9 100644 --- a/src/openhuman/inference/model_ids.rs +++ b/src/openhuman/inference/model_ids.rs @@ -11,9 +11,19 @@ use crate::openhuman::config::Config; use crate::openhuman::inference::local::provider::{provider_from_config, LocalAiProvider}; +use crate::openhuman::inference::vision_models::{self, VISION_MODEL_SUGGESTIONS}; pub(crate) const DEFAULT_OLLAMA_MODEL: &str = "gemma3:1b-it-qat"; -pub(crate) const DEFAULT_OLLAMA_VISION_MODEL: &str = ""; + +/// Default local vision model. Must name a genuinely vision-capable model: +/// it is the fallback whenever a configured vision id turns out to be +/// chat-only, so a chat model here would defeat the whole guard (#5146). +/// +/// Moondream is the smallest vision model that is pullable with no extra +/// setup (~1.7 GB across model + projector layers), which keeps the fallback +/// affordable on the low-RAM tiers where vision is most likely to be enabled +/// on demand. +pub(crate) const DEFAULT_OLLAMA_VISION_MODEL: &str = "moondream:1.8b-v2-q4_K_S"; pub(crate) const DEFAULT_LOW_VISION_MODEL: &str = "moondream:1.8b-v2-q4_K_S"; pub(crate) const DEFAULT_OLLAMA_EMBED_MODEL: &str = "bge-m3"; @@ -23,16 +33,31 @@ pub(crate) const DEFAULT_OLLAMA_EMBED_MODEL: &str = "bge-m3"; /// Every id here must be pullable from the public Ollama library as written — /// an entry that does not resolve makes the allowlist silently redirect the /// user back to the default, or leaves them with a model that `ollama pull` -/// cannot fetch (GH #5055). `gemma4:*` was such an entry: there is no `gemma4` -/// namespace on Ollama. The intended model is Gemma 3n, published as -/// `gemma3n:e4b-it-q8_0`. -const MVP_ALLOWED_CHAT_MODELS: &[&str] = &["gemma3:1b-it-qat", "gemma3n:e4b-it-q8_0"]; +/// cannot fetch (GH #5055). +/// +/// This list must also cover every `chat_model_id` in +/// [`crate::openhuman::inference::presets`]: a preset whose model is missing +/// here is silently downgraded to `MVP_DEFAULT_CHAT_MODEL`, so the user picks +/// a tier and quietly gets the 1B model. +/// `preset_chat_models_are_allowlisted_and_resolve_unchanged` pins that +/// invariant. +/// +/// Verified against the live registry (#5146 §1.3): +/// `GET https://registry.ollama.ai/v2/library//manifests/` returns +/// `200` for all five entries. Note that `gemma4` **does** now exist on the +/// Ollama library (it did not when #5055 removed it) and is multimodal at +/// every size, which is why the 16 GB+ tier can use one model for chat and +/// vision. `gemma3n:e4b-it-q8_0` stays allowlisted for back-compat with users +/// who already pulled it under the previous default. +const MVP_ALLOWED_CHAT_MODELS: &[&str] = &[ + "gemma3:270m-it-qat", + "gemma3:1b-it-qat", + "gemma3:4b-it-qat", + "gemma4:e4b-it-q8_0", + "gemma3n:e4b-it-q8_0", +]; const MVP_DEFAULT_CHAT_MODEL: &str = "gemma3:1b-it-qat"; -/// Vision models allowed in MVP — only disabled (empty string) since the -/// 2–4 GB tier has no vision model. -const MVP_ALLOWED_VISION_MODELS: &[&str] = &[""]; - /// Embedding models allowed in MVP (2–4 GB tier uses all-minilm). // bge-m3 (1024-dim, 8192-token context) is the canonical local embedder // for memory tree's fixed on-disk format. all-minilm (384-dim) is kept @@ -55,18 +80,30 @@ fn enforce_mvp_chat_allowlist(resolved: &str) -> String { MVP_DEFAULT_CHAT_MODEL.to_string() } -fn enforce_mvp_vision_allowlist(resolved: &str) -> String { - let lower = resolved.to_ascii_lowercase(); - for allowed in MVP_ALLOWED_VISION_MODELS { - if lower == allowed.to_ascii_lowercase() { - return resolved.to_string(); - } +/// Guarantee a vision request never reaches a chat-only model. +/// +/// This replaces the previous `MVP_ALLOWED_VISION_MODELS = &[""]` allowlist, +/// which matched only the empty string and therefore rewrote *every* +/// configured vision model to `""` — including genuinely vision-capable ones. +/// Callers then sent an empty model name to Ollama, which is not a clean +/// failure: `ensure_ollama_model_available` tried to `POST /api/pull` a +/// nameless model and retried three times before surfacing an opaque error +/// (#5146 §Part 1). +/// +/// The tier restriction that allowlist was standing in for is enforced +/// upstream by [`crate::openhuman::inference::presets::vision_mode_for_config`], +/// which reports `VisionMode::Disabled` for the tiers that ship no vision +/// model. What is left for this function is the capability question alone. +fn enforce_vision_capability(resolved: &str) -> String { + if vision_models::is_vision_capable(resolved) { + return resolved.to_string(); } tracing::warn!( resolved, - "[local_ai] vision model not in MVP allowlist, disabling vision" + fallback = DEFAULT_OLLAMA_VISION_MODEL, + "[local_ai] configured vision model is chat-only, falling back to a vision-capable default" ); - String::new() + DEFAULT_OLLAMA_VISION_MODEL.to_string() } fn enforce_mvp_embedding_allowlist(resolved: &str) -> String { @@ -134,6 +171,14 @@ fn raw_chat_model_id(config: &Config) -> String { raw.to_string() } +/// Resolve the vision model for status / reporting surfaces. +/// +/// An empty return means "vision is not configured" and is a legitimate +/// state (the low tiers ship no vision model). A non-empty return is +/// **always** a vision-capable id. Call [`resolve_vision_model_choice`] instead +/// when about to issue an actual vision request — it turns the +/// not-configured case into an actionable error rather than an empty string +/// that downstream code would send to Ollama verbatim. pub(crate) fn effective_vision_model_id(config: &Config) -> String { let raw = config.local_ai.vision_model_id.trim(); if raw.is_empty() { @@ -145,7 +190,54 @@ pub(crate) fn effective_vision_model_id(config: &Config) -> String { } else { raw }; - enforce_mvp_vision_allowlist(resolved) + enforce_vision_capability(resolved) +} + +/// The vision model a request will actually use, plus what it displaced. +pub(crate) struct VisionModelChoice { + /// The vision-capable model id to send to Ollama. + pub(crate) model: String, + /// The configured id that was swapped out because it is chat-only. + /// + /// `Some` means the user asked for one model and is getting another, which + /// every downstream *error* must say out loud: a bare "`moondream:...` is + /// not available, pull it" is actively misleading when the user configured + /// `gemma3:1b-it-qat` and never mentioned moondream (greptile, #5253). + pub(crate) replaced: Option, +} + +/// Resolve the vision model for a real vision request, reporting any +/// capability substitution. +/// +/// Never returns an empty id: when no vision model is configured the caller +/// gets a message naming what to set and which models to pull, instead of +/// silently shipping an empty model name to Ollama (#5146 §Part 1). +pub(crate) fn resolve_vision_model_choice(config: &Config) -> Result { + let resolved = effective_vision_model_id(config); + if resolved.trim().is_empty() { + let suggestions = VISION_MODEL_SUGGESTIONS.join("`, `"); + tracing::warn!("[local_ai] vision request with no vision model configured"); + return Err(format!( + "no local vision model is configured. Set `local_ai.vision_model_id` to a \ + vision-capable model (for example `{suggestions}`) and pull it with \ + `ollama pull `, or route the vision workload to a cloud provider \ + with `vision_provider`." + )); + } + + // Report only a *capability* substitution. An alias rewrite (`moondream` -> + // the pinned tag) resolves to a different string but is the same model the + // user asked for, so it is not something they need to be told about. + let configured = config.local_ai.vision_model_id.trim(); + let replaced = (!configured.is_empty() + && !vision_models::is_vision_capable(configured) + && !resolved.eq_ignore_ascii_case(configured)) + .then(|| configured.to_string()); + + Ok(VisionModelChoice { + model: resolved, + replaced, + }) } pub(crate) fn effective_embedding_model_id(config: &Config) -> String { @@ -272,32 +364,53 @@ mod tests { #[test] fn chat_model_rejects_non_mvp_models() { let mut config = test_config(); - // All models outside the local allowlist are rejected. - config.local_ai.chat_model_id = "gemma3:4b-it-qat".to_string(); - assert_eq!(effective_chat_model_id(&config), MVP_DEFAULT_CHAT_MODEL); - - config.local_ai.chat_model_id = "gemma3:270m-it-qat".to_string(); - assert_eq!(effective_chat_model_id(&config), MVP_DEFAULT_CHAT_MODEL); // Bare `gemma3n:e4b` is a real Ollama tag but is NOT the allowlisted // quantization, so it still redirects to the default. config.local_ai.chat_model_id = "gemma3n:e4b".to_string(); assert_eq!(effective_chat_model_id(&config), MVP_DEFAULT_CHAT_MODEL); - // The retired `gemma4:*` namespace does not exist on Ollama at all. - config.local_ai.chat_model_id = "gemma4:e4b-it-q8_0".to_string(); + // Arbitrary non-preset models stay rejected. + config.local_ai.chat_model_id = "llama3.1:8b".to_string(); + assert_eq!(effective_chat_model_id(&config), MVP_DEFAULT_CHAT_MODEL); + + config.local_ai.chat_model_id = "totally-made-up-model:v0".to_string(); assert_eq!(effective_chat_model_id(&config), MVP_DEFAULT_CHAT_MODEL); } - /// GH #5055: every allowlisted chat model must be a real, pullable Ollama - /// id. `gemma4:*` shipped for a while and could never be pulled. + /// #5146 §1.3: the allowlist must cover every preset chat model. + /// + /// `gemma3:270m-it-qat` (1 GB tier) and `gemma3:4b-it-qat` (8-16 GB tier) + /// were previously absent, so applying either preset resolved straight + /// back to the 1B default — the user picked a tier and silently got a + /// different model than the one the preset advertised. #[test] - fn mvp_chat_allowlist_has_no_unpullable_namespaces() { - for model in MVP_ALLOWED_CHAT_MODELS { - assert!( - !model.starts_with("gemma4:"), - "`{model}` is not a real Ollama model — there is no gemma4 namespace" + fn preset_chat_models_are_allowlisted_and_resolve_unchanged() { + let mut config = test_config(); + for preset in crate::openhuman::inference::presets::all_presets() { + config.local_ai.chat_model_id = preset.chat_model_id.to_string(); + assert_eq!( + effective_chat_model_id(&config), + preset.chat_model_id, + "preset {:?} chat model `{}` is not allowlisted and was redirected", + preset.tier, + preset.chat_model_id ); + } + } + + /// GH #5055 / #5146 §1.3: every allowlisted chat model must be a real, + /// fully-qualified Ollama id. + /// + /// The #5055 form of this test asserted "no entry may start with + /// `gemma4:`", because no `gemma4` namespace existed at the time. Gemma 4 + /// has since been published and `gemma4:e4b-it-q8_0` resolves against + /// `registry.ollama.ai`, so that assertion was pinning an expired fact. + /// The durable invariant is the `:` shape plus the + /// preset cross-check above. + #[test] + fn mvp_chat_allowlist_entries_are_fully_qualified() { + for model in MVP_ALLOWED_CHAT_MODELS { assert!( model.contains(':'), "`{model}` must be a fully-qualified `:` id" @@ -308,15 +421,119 @@ mod tests { #[test] fn vision_model_normalizes_legacy_moondream_values() { let mut config = test_config(); + + // Empty stays empty: "vision not configured" is a real state. config.local_ai.vision_model_id = String::new(); assert_eq!(effective_vision_model_id(&config), ""); - // Moondream is not in the MVP vision allowlist (only "" is allowed), - // so it gets redirected to "" (vision disabled). + // Legacy shorthands normalize to the pinned Moondream build. Before + // #5146 these resolved to "" (vision silently disabled) because the + // vision allowlist contained only the empty string. config.local_ai.vision_model_id = "moondream".to_string(); - assert_eq!(effective_vision_model_id(&config), ""); + assert_eq!(effective_vision_model_id(&config), DEFAULT_LOW_VISION_MODEL); config.local_ai.vision_model_id = "moondream:1.8b".to_string(); - assert_eq!(effective_vision_model_id(&config), ""); + assert_eq!(effective_vision_model_id(&config), DEFAULT_LOW_VISION_MODEL); + } + + /// #5146 §Part 1: a genuinely vision-capable model must survive resolution + /// unchanged. The previous `MVP_ALLOWED_VISION_MODELS = &[""]` allowlist + /// rewrote every one of these to `""`. + #[test] + fn vision_capable_models_pass_through_unchanged() { + let mut config = test_config(); + for model in ["llava:7b", "gemma3:4b-it-qat", "gemma4:e4b-it-q8_0"] { + config.local_ai.vision_model_id = model.to_string(); + assert_eq!(effective_vision_model_id(&config), model); + } + } + + /// #5146 §Part 1: a chat-only model must never be returned as the vision + /// model. Ollama silently drops the `images` array for such a model, so + /// passing it through would produce a fabricated description instead of an + /// error. + #[test] + fn chat_only_vision_model_falls_back_to_a_vision_capable_default() { + let mut config = test_config(); + for chat_only in ["gemma3n:e4b-it-q8_0", "gemma3:1b-it-qat", "llama3.1:8b"] { + config.local_ai.vision_model_id = chat_only.to_string(); + let resolved = effective_vision_model_id(&config); + assert_eq!(resolved, DEFAULT_OLLAMA_VISION_MODEL); + assert!(vision_models::is_vision_capable(&resolved)); + } + } + + /// The default must itself be vision-capable — it is the fallback the + /// guard above lands on, so a chat-only default would defeat the guard. + #[test] + fn default_vision_model_is_vision_capable() { + assert!(!DEFAULT_OLLAMA_VISION_MODEL.is_empty()); + assert!(vision_models::is_vision_capable( + DEFAULT_OLLAMA_VISION_MODEL + )); + assert!(vision_models::is_vision_capable(DEFAULT_LOW_VISION_MODEL)); + } + + /// #5146 §Part 1: an unconfigured vision model must produce an actionable + /// error, not an empty model id that downstream code sends to Ollama. + #[test] + fn resolve_vision_model_id_errors_when_unconfigured() { + let mut config = test_config(); + config.local_ai.vision_model_id = String::new(); + + let err = resolve_vision_model_choice(&config) + .err() + .expect("expected a vision error"); + assert!( + err.contains("vision_model_id"), + "error should name the config key to set: {err}" + ); + assert!( + err.contains("ollama pull"), + "error should say how to install a model: {err}" + ); + // Whitespace-only is the same "not configured" state. + config.local_ai.vision_model_id = " ".to_string(); + assert!(resolve_vision_model_choice(&config).is_err()); + } + + #[test] + fn resolve_vision_model_id_returns_a_vision_capable_model_when_configured() { + let mut config = test_config(); + config.local_ai.vision_model_id = "llava:7b".to_string(); + assert_eq!( + resolve_vision_model_choice(&config).unwrap().model, + "llava:7b" + ); + + // Even a chat-only configured id resolves to something that can see. + config.local_ai.vision_model_id = "gemma3n:e4b-it-q8_0".to_string(); + let resolved = resolve_vision_model_choice(&config).unwrap().model; + assert!(vision_models::is_vision_capable(&resolved)); + } + + #[test] + fn resolve_vision_model_choice_reports_only_capability_substitutions() { + let mut config = test_config(); + + // A vision-capable id is used as-is, with nothing to report. + config.local_ai.vision_model_id = "llava:7b".to_string(); + let choice = resolve_vision_model_choice(&config).unwrap(); + assert_eq!(choice.model, "llava:7b"); + assert_eq!(choice.replaced, None); + + // A chat-only id is replaced, and the configured id is reported so the + // caller can explain the swap instead of naming a model out of nowhere. + config.local_ai.vision_model_id = "gemma3n:e4b-it-q8_0".to_string(); + let choice = resolve_vision_model_choice(&config).unwrap(); + assert!(vision_models::is_vision_capable(&choice.model)); + assert_eq!(choice.replaced.as_deref(), Some("gemma3n:e4b-it-q8_0")); + + // An alias rewrite resolves to a different string but is the same model + // the user asked for, so it must not be reported as a substitution. + config.local_ai.vision_model_id = "moondream".to_string(); + let choice = resolve_vision_model_choice(&config).unwrap(); + assert!(vision_models::is_vision_capable(&choice.model)); + assert_eq!(choice.replaced, None); } #[test] diff --git a/src/openhuman/inference/presets.rs b/src/openhuman/inference/presets.rs index a322a70d8a..9d489af572 100644 --- a/src/openhuman/inference/presets.rs +++ b/src/openhuman/inference/presets.rs @@ -157,27 +157,47 @@ pub fn all_presets() -> Vec { label: "8-16 GB", description: "Balanced Gemma multimodal preset with bundled vision support.", chat_model_id: "gemma3:4b-it-qat", + // Gemma 3 is multimodal from 4B upward, so one model covers chat + // and vision here. (The 270M and 1B builds are text-only.) vision_model_id: "gemma3:4b-it-qat", - embedding_model_id: "nomic-embed-text:latest", + // bge-m3 (1024 dims). nomic-embed-text is 768 dims and fails the + // memory tree's post-call dimension validator; the embedding + // allowlist already rewrote it to bge-m3 at resolution time, so + // naming bge-m3 here only makes the preset honest about what is + // actually pulled. + embedding_model_id: "bge-m3", quantization: "qat", vision_mode: VisionMode::Bundled, supports_screen_summary: true, target_ram_gb: 8, min_ram_gb: 8, - approx_download_gb: 4.3, + // gemma3:4b-it-qat (4.0 GB) + bge-m3 (1.2 GB) + approx_download_gb: 5.2, }, ModelPreset { tier: ModelTier::Ram16PlusGb, label: "16 GB+", - description: "Best local quality with Gemma 3n on higher-end devices.", - // GH #5055: was `gemma4:e4b`, which does not exist on the Ollama - // library — the tier shipped a model no `ollama pull` could fetch. - // Gemma 3n is the real publication of the "e4b" (effective-4B) - // variant, and `e4b-it-q8_0` (9.5 GB) is what `approx_download_gb` - // below was sized against. - chat_model_id: "gemma3n:e4b-it-q8_0", - vision_model_id: "gemma3n:e4b-it-q8_0", - embedding_model_id: "nomic-embed-text:latest", + description: "Best local quality with Gemma 4 on higher-end devices.", + // GH #5055 moved this tier off `gemma4:e4b` because no `gemma4` + // namespace existed on the Ollama library at the time, and landed + // on `gemma3n:e4b-it-q8_0`. Two things have changed (#5146 §1.3): + // + // 1. Gemma 4 has since been published, and `gemma4:e4b-it-q8_0` + // resolves (11.6 GB, 128K context). + // 2. Gemma 3n is **text-only** on Ollama, so using it as the + // `vision_model_id` of a `Bundled` vision tier pointed every + // vision request at a model with no vision encoder. Ollama + // accepts the `images` array against such a model, discards + // it, and answers from the prompt text — a hallucinated + // description rather than an error. + // + // Gemma 4 is multimodal at every published size, so this tier is + // back to one model serving both chat and vision, matching how the + // 8-16 GB tier uses `gemma3:4b-it-qat`. + chat_model_id: "gemma4:e4b-it-q8_0", + vision_model_id: "gemma4:e4b-it-q8_0", + // bge-m3 (1024 dims) — see the 8-16 GB tier note above. + embedding_model_id: "bge-m3", // The other tiers ship QAT builds; this one is q8_0. The field is a // display label (`effective_quantization`) and does not take part in // resolving the model tag, but it should still match what is pulled. @@ -186,7 +206,8 @@ pub fn all_presets() -> Vec { supports_screen_summary: true, target_ram_gb: 16, min_ram_gb: 16, - approx_download_gb: 9.9, + // gemma4:e4b-it-q8_0 (11.6 GB) + bge-m3 (1.2 GB) + approx_download_gb: 12.8, }, ] } @@ -432,35 +453,95 @@ mod tests { assert_eq!(vision_mode_for_config(&config), VisionMode::Bundled); } - /// GH #5055: every preset must name a model that actually exists on the - /// Ollama library. The 16 GB+ tier shipped `gemma4:e4b` for chat and vision - /// — there is no `gemma4` namespace, so the tier offered a model that no - /// `ollama pull` could ever fetch. Guard the whole table, not just that row. + /// GH #5055 / #5146 §1.3: every preset must name a model that actually + /// exists on the Ollama library and is fully qualified, so `ollama pull` + /// can fetch it and the allowlist does not silently redirect the user. + /// + /// The original #5055 form of this test asserted the narrower fact "no id + /// may start with `gemma4:`", because no `gemma4` namespace existed then. + /// Gemma 4 has since been published (`gemma4:e4b-it-q8_0` resolves against + /// `registry.ollama.ai`), so that assertion encoded a fact that expired. + /// The durable invariant is the shape check below plus the + /// `preset_chat_models_are_allowlisted_and_resolve_unchanged` cross-check + /// in `model_ids`. #[test] - fn presets_reference_no_unpullable_gemma4_namespace() { + fn preset_model_ids_are_fully_qualified() { for preset in all_presets() { for (field, id) in [ ("chat", preset.chat_model_id), ("vision", preset.vision_model_id), ("embedding", preset.embedding_model_id), ] { + if id.is_empty() { + // Only `vision` is legitimately empty (vision disabled). + assert_eq!( + field, "vision", + "preset {:?} {field} model must not be empty", + preset.tier + ); + continue; + } + // `bge-m3` is deliberately exempt rather than retagged to + // `bge-m3:latest` (greptile, #5253). The bare id is the + // canonical spelling across the embedding stack: it is the + // entry in `model_ids::MVP_ALLOWED_EMBEDDING_MODELS`, the + // `embeddings::catalog` id, and what `normalize_embed_model_id` + // collapses `bge-m3:latest` *to*. Retagging the preset alone + // would be silently rewritten back by + // `enforce_mvp_embedding_allowlist`; retagging all of them is a + // cross-cutting rename that must also migrate configs already + // persisting `bge-m3` - disproportionate to a cosmetic tag. assert!( - !id.starts_with("gemma4:"), - "preset {:?} {field} model `{id}` is not a real Ollama model \ - — there is no gemma4 namespace", + id.contains(':') || id == "bge-m3", + "preset {:?} {field} model `{id}` must be a fully-qualified \ + `:` id", preset.tier ); } } } - /// The 16 GB+ tier's chat model must be the allowlisted Gemma 3n build, so - /// selecting the preset does not immediately get redirected back to the - /// default by `enforce_mvp_chat_allowlist`. + /// #5146 §Part 1: a preset that declares a vision mode must name a model + /// that can actually accept images. + /// + /// The 16 GB+ tier previously used `gemma3n:e4b-it-q8_0` as its + /// `vision_model_id`. Gemma 3n is text-only on Ollama, and Ollama does not + /// reject an `images` array sent to a text-only model — it drops the + /// images and answers from the prompt alone, so the user got a fluent, + /// fabricated description of an image the model never saw. #[test] - fn high_tier_preset_uses_the_allowlisted_gemma3n_build() { + fn preset_vision_models_are_vision_capable() { + use crate::openhuman::inference::vision_models::is_vision_capable; + + for preset in all_presets() { + match preset.vision_mode { + VisionMode::Disabled => assert!( + preset.vision_model_id.is_empty(), + "preset {:?} disables vision but names `{}`", + preset.tier, + preset.vision_model_id + ), + VisionMode::Ondemand | VisionMode::Bundled => assert!( + is_vision_capable(preset.vision_model_id), + "preset {:?} routes vision at `{}`, which is not vision-capable", + preset.tier, + preset.vision_model_id + ), + } + } + } + + /// The 16 GB+ tier must name an allowlisted chat model, so selecting the + /// preset is not immediately redirected back to the default by + /// `enforce_mvp_chat_allowlist`, and that model must be multimodal so the + /// tier's `Bundled` vision mode is real rather than nominal. + #[test] + fn high_tier_preset_uses_one_multimodal_build_for_chat_and_vision() { + use crate::openhuman::inference::vision_models::is_vision_capable; + let preset = preset_for_tier(ModelTier::Ram16PlusGb).expect("16 GB+ preset"); - assert_eq!(preset.chat_model_id, "gemma3n:e4b-it-q8_0"); - assert_eq!(preset.vision_model_id, "gemma3n:e4b-it-q8_0"); + assert_eq!(preset.chat_model_id, "gemma4:e4b-it-q8_0"); + assert_eq!(preset.vision_model_id, preset.chat_model_id); + assert!(is_vision_capable(preset.vision_model_id)); } } diff --git a/src/openhuman/inference/vision_models.rs b/src/openhuman/inference/vision_models.rs new file mode 100644 index 0000000000..7a583729a7 --- /dev/null +++ b/src/openhuman/inference/vision_models.rs @@ -0,0 +1,194 @@ +//! Which local model IDs can actually accept image input. +//! +//! Routing a vision request at a chat-only model is **not** a loud failure on +//! Ollama. `POST /api/generate` accepts the `images` array against any model, +//! silently discards it when the model has no vision encoder, and answers from +//! the prompt text alone. The caller gets back a confident, entirely +//! hallucinated description rather than an error (#5146, §Part 1). +//! +//! Every vision path therefore resolves its model through this registry first, +//! so a vision tool-call either reaches a genuinely vision-capable model or +//! surfaces a clear error. See [`crate::openhuman::inference::model_ids`]. +//! +//! The families below were verified against the live Ollama library registry +//! (`GET https://registry.ollama.ai/v2/library//manifests/` plus the +//! published capability badges on `ollama.com/library/`). + +/// Model families where every published tag accepts image input. +/// +/// Kept as whole-family entries (the segment before `:`) rather than loose +/// substrings so a near-miss like `gemma3n` can never match a `gemma3` rule. +const VISION_CAPABLE_FAMILIES: &[&str] = &[ + "moondream", + "llava", + "llava-llama3", + "llava-phi3", + "bakllava", + "llama3.2-vision", + "llama4", + "minicpm-v", + "granite3.2-vision", + "qwen2-vl", + "qwen2.5vl", + "mistral-small3.1", + "mistral-small3.2", + // Gemma 4 is multimodal at every published size, including the `e2b` / + // `e4b` edge builds. Contrast `gemma3n` below. + "gemma4", +]; + +/// Families that look vision-capable by name but are text-only. +/// +/// `gemma3n` is the load-bearing entry: it is a *separate* model from +/// `gemma3`, shares its prefix, and ships **text input only** on Ollama. It +/// was the 16 GB+ preset's vision model before #5146. +const TEXT_ONLY_FAMILIES: &[&str] = &["gemma3n"]; + +/// Substrings that identify a repackaged upstream vision model, e.g. +/// `hf.co/user/llava-v1.6-mistral-7b` or a locally re-tagged `my-moondream`. +/// Only consulted after the exact-family rules above. +const VISION_MARKERS: &[&str] = &["llava", "moondream", "bakllava", "vision"]; + +/// Vision models suggested to the user when none is configured. Every entry is +/// pullable from the Ollama library with no extra setup. +pub(crate) const VISION_MODEL_SUGGESTIONS: &[&str] = + &["moondream:1.8b-v2-q4_K_S", "llava:7b", "gemma3:4b-it-qat"]; + +/// Gemma 3 is split by size: `270m` and `1b` are text-only, while `4b`, `12b` +/// and `27b` are multimodal. `gemma3:latest` resolves to the 4B build. +fn gemma3_tag_is_multimodal(tag: &str) -> bool { + if tag.is_empty() || tag == "latest" { + return true; + } + !(tag.starts_with("270m") || tag.starts_with("1b")) +} + +/// Returns `true` when `model_id` names a model that can accept image input. +/// +/// Errs toward `false`: an unknown id is treated as chat-only so the caller +/// reports "no vision model available" instead of shipping images to a model +/// that will quietly ignore them. +pub(crate) fn is_vision_capable(model_id: &str) -> bool { + let normalized = model_id.trim().to_ascii_lowercase(); + if normalized.is_empty() { + return false; + } + + let (family, tag) = match normalized.split_once(':') { + Some((family, tag)) => (family, tag), + None => (normalized.as_str(), ""), + }; + + if TEXT_ONLY_FAMILIES.contains(&family) { + return false; + } + if family == "gemma3" { + return gemma3_tag_is_multimodal(tag); + } + if VISION_CAPABLE_FAMILIES.contains(&family) { + return true; + } + + VISION_MARKERS + .iter() + .any(|marker| normalized.contains(marker)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn dedicated_vision_families_are_capable() { + assert!(is_vision_capable("moondream:1.8b-v2-q4_K_S")); + assert!(is_vision_capable("moondream")); + assert!(is_vision_capable("llava:7b")); + assert!(is_vision_capable("llava:13b")); + assert!(is_vision_capable("bakllava:latest")); + assert!(is_vision_capable("llama3.2-vision:11b")); + assert!(is_vision_capable("minicpm-v:8b")); + assert!(is_vision_capable("qwen2.5vl:7b")); + } + + #[test] + fn gemma3_is_multimodal_only_at_4b_and_above() { + // 270m / 1b ship without a vision encoder. + assert!(!is_vision_capable("gemma3:270m-it-qat")); + assert!(!is_vision_capable("gemma3:1b-it-qat")); + assert!(!is_vision_capable("gemma3:1b")); + + assert!(is_vision_capable("gemma3:4b-it-qat")); + assert!(is_vision_capable("gemma3:12b-it-qat")); + assert!(is_vision_capable("gemma3:27b")); + // `latest` is the 4B multimodal build. + assert!(is_vision_capable("gemma3:latest")); + assert!(is_vision_capable("gemma3")); + } + + #[test] + fn gemma3n_is_never_treated_as_vision_capable() { + // Regression guard for #5146: `gemma3n` shares a prefix with `gemma3` + // but is text-only, and it was previously wired in as the 16 GB+ + // tier's vision model. + assert!(!is_vision_capable("gemma3n:e4b-it-q8_0")); + assert!(!is_vision_capable("gemma3n:e2b")); + assert!(!is_vision_capable("gemma3n")); + assert!(!is_vision_capable("GEMMA3N:E4B-IT-Q8_0")); + } + + #[test] + fn gemma4_is_multimodal_at_every_size() { + assert!(is_vision_capable("gemma4:e4b-it-q8_0")); + assert!(is_vision_capable("gemma4:e2b-it-qat")); + assert!(is_vision_capable("gemma4:12b")); + assert!(is_vision_capable("gemma4")); + } + + #[test] + fn chat_only_models_are_rejected() { + assert!(!is_vision_capable("llama3.1:8b")); + assert!(!is_vision_capable("qwen2.5:14b")); + assert!(!is_vision_capable("deepseek-r1:7b")); + assert!(!is_vision_capable("phi4:latest")); + } + + #[test] + fn embedding_models_are_rejected() { + assert!(!is_vision_capable("bge-m3")); + assert!(!is_vision_capable("all-minilm:latest")); + assert!(!is_vision_capable("nomic-embed-text:latest")); + } + + #[test] + fn empty_and_whitespace_are_rejected() { + assert!(!is_vision_capable("")); + assert!(!is_vision_capable(" ")); + } + + #[test] + fn repackaged_upstream_vision_models_are_detected() { + assert!(is_vision_capable("hf.co/user/llava-v1.6-mistral-7b")); + assert!(is_vision_capable("my-moondream:custom")); + assert!(is_vision_capable("someone/llama3.2-vision-abliterated")); + } + + #[test] + fn detection_is_case_insensitive() { + assert!(is_vision_capable("LLaVA:7B")); + assert!(is_vision_capable("MoonDream")); + assert!(!is_vision_capable("Gemma3:1B-IT-QAT")); + } + + #[test] + fn every_suggested_vision_model_is_vision_capable() { + // The suggestions are quoted verbatim in the user-facing error from + // `model_ids::resolve_vision_model_choice`; a chat-only entry here would + // send users to a model that silently ignores their image. + for suggestion in VISION_MODEL_SUGGESTIONS { + assert!( + is_vision_capable(suggestion), + "suggested vision model `{suggestion}` is not vision-capable" + ); + } + } +}