fix(inference): route vision at vision-capable models and refresh local model defaults (#5146)

README.md

@@ -81,7 +81,7 @@ OpenHuman is three things most assistants aren't: **a brain** that builds a pers
 ### 🔬 The deep researcher & doer
 
 - **[SuperContext](https://tinyhumans.gitbook.io/openhuman/features/super-context)**: a research scout sweeps your memory and files before the model reads your first message. No cold starts.
-- **Batteries included**: managed [web search](https://tinyhumans.gitbook.io/openhuman/features/native-tools/web-search), powered by [Exa](https://exa.ai), is included with your OpenHuman subscription and needs no API key; bring your own Exa key to search directly on your own Exa account and billing. Plus scraper, coder toolset, a real [browser](https://tinyhumans.gitbook.io/openhuman/features/native-tools/browser-and-computer), and [native voice](gitbooks/features/native-tools/voice.md) with in-process Whisper. [Model routing](https://tinyhumans.gitbook.io/openhuman/features/model-routing) picks the right LLM per workload on one subscription, with [local AI optional](https://tinyhumans.gitbook.io/openhuman/features/model-routing/local-ai).
+- **Batteries included**: managed [web search](https://tinyhumans.gitbook.io/openhuman/features/native-tools/web-search), powered by [Exa](https://exa.ai), is included with your OpenHuman subscription and needs no API key; bring your own Exa key to search directly on your own Exa account and billing. Plus scraper, coder toolset, a real [browser](https://tinyhumans.gitbook.io/openhuman/features/native-tools/browser-and-computer), and [native voice](gitbooks/features/native-tools/voice.md) with in-process Whisper. [Model routing](https://tinyhumans.gitbook.io/openhuman/features/model-routing) picks the right LLM per workload on one subscription. That subscription is a default, not a lock-in: point any workload at [your own provider key or a fully local Ollama model](https://tinyhumans.gitbook.io/openhuman/features/model-routing/local-and-byok-models), and mix the three however you like.
 - **[Meeting agents](https://tinyhumans.gitbook.io/openhuman/features/mascot/meeting-agents)**: joins **Meet, Zoom, Teams, and Webex** with a face and a voice. It auto-joins from your calendar, streams a live transcript, answers by name, and files a summary with action items.
 - **[Image & video generation](https://tinyhumans.gitbook.io/openhuman/features/native-tools)**: Seedream/SeedEdit images and Seedance/Veo video, straight into your workspace on the same subscription.
 - **[17 messaging channels](https://tinyhumans.gitbook.io/openhuman/features/channels)**: Telegram, Discord, Slack, WhatsApp, Signal, iMessage… plus **native email** (IMAP IDLE + SMTP). Your agent reaches you where you already are.

gitbooks/SUMMARY.md

@@ -43,6 +43,7 @@
 * [Themes & Theme Studio](features/theming.md)
 * [Smart Token Compression](features/token-compression.md)
 * [Automatic Model Routing](features/model-routing/README.md)
+  * [Local models & bring your own key](features/model-routing/local-and-byok-models.md)
   * [Local AI (optional)](features/model-routing/local-ai.md)
 * [Available Tools](features/native-tools/README.md)
   * [Web Search](features/native-tools/web-search.md)

gitbooks/features/model-routing/README.md

@@ -43,9 +43,11 @@ The router wraps several pre-created providers (Anthropic, OpenAI, Google, Groq,
 
 The exact mappings are configurable; the defaults ship sensible per-provider routes.
 
-## One subscription
+## One subscription, or your own
 
-Routing happens behind a single OpenHuman subscription. You don't hold separate API keys for Anthropic, OpenAI, Google etc., the backend brokers access, and the router picks the right one per task. That's the "one subscription, many providers" promise from the README, made concrete.
+Routing happens behind a single OpenHuman subscription by default. You don't hold separate API keys for Anthropic, OpenAI, Google etc., the backend brokers access, and the router picks the right one per task. That's the "one subscription, many providers" promise from the README, made concrete.
+
+The subscription is the default, not a requirement. The same router works against **your own provider key** or a **fully local model**, per workload, and you can mix all three. See [Local models & bring your own key](local-and-byok-models.md) for setup and for what each route supports for chat, vision, and embeddings.
 
 ## Overriding routes
 
@@ -97,4 +99,5 @@ Routing isn't a UI dropdown. The agent loop itself emits hints based on what it'
 
 - [Smart Token Compression](../token-compression.md). what makes large reasoning calls affordable.
 - [Native Tools](../native-tools/README.md). different tool calls hint at different routes.
+- [Local models & bring your own key](local-and-byok-models.md). run on your own key or fully on-device.
 - [Local AI (optional)](local-ai.md). lightweight chat hints can run on-device.

gitbooks/features/model-routing/local-ai.md

@@ -15,7 +15,7 @@ This is deliberate scoping. The previous design tried to put every modality on-d
 
 | Workload                  | Default model                     | Implementation                                                                                                          |
 | ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
-| **Memory embeddings**     | `all-minilm:latest`               | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search.    |
+| **Memory embeddings**     | `bge-m3`                          | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search.    |
 | **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` - source / topic / global summary builders for the Memory Tree.                  |
 | **Heartbeat loop**        | small chat model                  | `src/openhuman/heartbeat/` - periodic background reflection.                                                            |
 | **Learning / reflection** | small chat model                  | `src/openhuman/learning/reflection.rs` - passes that consolidate what was learned.                                      |
@@ -31,7 +31,7 @@ Each of these is an explicit opt-in. Turning on local AI does not silently route
 | -------------- | ---------------------------------------------------------------------------------------------- |
 | **Chat**       | Frontier reasoning quality unless `chat_provider` is explicitly set to a local provider.       |
 | **Reasoning**  | Stronger multi-step quality unless `reasoning_provider` is explicitly set to a local provider. |
-| **Vision**     | Same.                                                                                          |
+| **Vision**     | Same, unless `vision_provider` points at a local vision-capable model. See below.              |
 | **STT**        | Backend-proxied transcription (`src/openhuman/voice/cloud_transcribe.rs`).                     |
 | **TTS**        | Hosted [text-to-speech](../native-tools/voice.md) under the hood (`reply_speech.rs`).          |
 | **Web search** | Backend proxy (no API key on your machine).                                                    |
@@ -102,10 +102,23 @@ Local AI is worth turning on if any of these are true:
 
 It is **not** worth turning on if you only have a few sources connected, the cloud path is faster and the privacy benefit is small. There is also a hardware cost: Ollama and a small Gemma model want a few GB of RAM and pull a few GB of weights.
 
+## Local vision
+
+Vision is a separate capability from chat, and **most small local models cannot do it**. Ollama does not reject an image sent to a text-only model: it drops the image and answers from the prompt text, which produces a fluent description of something the model never saw. OpenHuman therefore resolves the vision model through a capability check and refuses to route a vision request at a chat-only model.
+
+What that means in practice:
+
+- `local_ai.vision_model_id` must name a vision-capable model. `moondream:1.8b-v2-q4_K_S` (~1.7 GB) is the smallest option; `gemma3:4b-it-qat` and `gemma4:e4b-it-q8_0` handle chat and vision with one set of weights.
+- Gemma 3 is text-only at 270M and 1B, and multimodal from 4B up. **Gemma 3n is a different model and is text-only at every size**, so it is not usable for vision even though it is a capable chat model.
+- Leaving `vision_model_id` empty is a valid "no local vision" setup. A vision request then returns a message naming the config key to set and the models to pull, rather than failing silently.
+- If a configured vision model turns out to be chat-only, the core logs a warning and falls back to a vision-capable default instead of sending images to a model that would ignore them.
+
+The full per-model capability table lives in [Local models & bring your own key](local-and-byok-models.md).
+
 ## What you'll need
 
 - [**Ollama**](https://ollama.com) installed and running locally, or [**LM Studio**](https://lmstudio.ai) with the local server enabled.
-- Enough disk for the models (`gemma3:1b-it-qat` \~700 MB, `all-minilm:latest` \~23 MB).
+- Enough disk for the models (`gemma3:1b-it-qat` \~1.0 GB, `bge-m3` \~1.2 GB, plus \~1.7 GB if you add Moondream for vision).
 - Enough RAM to keep the model resident (8 GB+ recommended, 16 GB+ ideal).
 
 OpenHuman handles the rest: lifecycle (`src/openhuman/inference/local/service/`), API clients, health checks, and graceful fallback to remote when the local provider disappears.
@@ -119,6 +132,7 @@ OpenHuman handles the rest: lifecycle (`src/openhuman/inference/local/service/`)
 
 ## See also
 
+- [Local models & bring your own key](local-and-byok-models.md). Per-model capability table and BYOK setup.
 - [Memory Tree](../obsidian-wiki/memory-tree.md). what local embeddings + summarization power.
 - [Automatic Model Routing](README.md). how lightweight chat hints prefer the local provider.
 - [Privacy & Security](../privacy-and-security.md). what moves on-device when you opt in.