fix(inference): local model reliability — /api/tags fallback + real Ollama model ids - #5219
Conversation
…llama model ids Scope is narrower than issue tinyhumansai#5055 reads, because its two root-cause sub-issues already shipped and are on main: - tinyhumansai#5053 (discovery endpoint): `model_discovery_api` / `endpoint_is_openai_v1` in inference/local/provider.rs already gate discovery by endpoint *type* rather than "is it localhost", wired into diagnostics.rs with a regression test. Left as-is. - tinyhumansai#5017 (embedding verification): the setup probe already sends a real POST /v1/embeddings with the user's model and key and classifies seven outcomes (auth / unreachable / model-incompatible / dimension-mismatch / no-embeddings-API / no-model-loaded / generic), surfaced by EmbeddingsPanel.tsx and pinned by a mock-endpoint test that captures the auth header and request body. Left as-is. What was actually missing: 1. No 404 fallback. A `/v1/models` 404 now retries the host-rooted `/api/tags` exactly once, logging a warning, so a runtime that only speaks the Ollama listing still discovers its models. Three cases deliberately do NOT recover, because a permissive fallback would reintroduce tinyhumansai#5053: a non-404 status never triggers it (a 500 is a server fault, not a wrong-endpoint signal); an empty fallback catalog is not a recovery (LM Studio answers unknown paths with `200 {"error": …}` and zero models); and when the fallback fails the caller sees the original /v1/models error rather than a second, more confusing one. The new `ollama_tags_fallback_url` strips a trailing `/v1` so the probe can never become the malformed `/v1/api/tags`. 2. The resolved discovery URL is now logged at DEBUG, so a wrong base URL is diagnosable from app logs without reproducing against the runtime's own request log. 3. `gemma4` is not a real Ollama namespace. `MVP_ALLOWED_CHAT_MODELS` carried `gemma4:e4b-it-q8_0` and the 16 GB+ preset shipped `gemma4:e4b` for chat and vision, so that tier offered a model no `ollama pull` could fetch — the "stale or unavailable default" failure mode in the issue. The intended model is Gemma 3n, published as `gemma3n:e4b-it-q8_0` (9.5 GB, which is what the tier's approx_download_gb was sized against). Verified against the public library: gemma3:1b-it-qat, gemma3:4b-it-qat, moondream:1.8b-v2-q4_K_S, bge-m3 and gemma3n:e4b-it-q8_0 all resolve; gemma4:* does not. This also closes a latent second bug: the preset (`gemma4:e4b`) and the allowlist (`gemma4:e4b-it-q8_0`) disagreed, so selecting the tier would have been silently redirected to gemma3:1b-it-qat by enforce_mvp_chat_allowlist. They now agree. The tier's `quantization` label moves "qat" -> "q8_0" to match what is actually pulled. It is a display string (effective_quantization) and takes no part in resolving the model tag. Tests added: four for the fallback (recovers, original error preserved when the fallback also fails, empty catalog is not a recovery, non-404 never probes twice — asserted with a hit counter), one for the fallback URL derivation, and three guards that no allowlist entry or preset can name the unpullable gemma4 namespace again. The Ollama path is untouched: the fallback lives only in list_lm_studio_models, which a genuine Ollama config never reaches because diagnostics() takes the /api/tags branch first. Fixes tinyhumansai#5055
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughLM Studio model discovery now falls back from ChangesLM Studio discovery fallback
Gemma model refresh
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant LocalAiService
participant LMStudioModels
participant OllamaTagsFallback
participant OllamaTags
LocalAiService->>LMStudioModels: GET /v1/models
LMStudioModels-->>LocalAiService: 404 Not Found
LocalAiService->>OllamaTagsFallback: request fallback catalog
OllamaTagsFallback->>OllamaTags: GET /api/tags
OllamaTags-->>OllamaTagsFallback: model catalog
OllamaTagsFallback-->>LocalAiService: non-empty models or None
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
coderabbitai
Bot
added
bug
rust-core
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e499ce4407
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
| Filename | Overview |
|---|---|
| src/openhuman/inference/local/service/lm_studio.rs | Adds list_ollama_tags_fallback: a one-shot /api/tags probe triggered only on HTTP 404 from /v1/models. Error-envelope detection (LM Studio {"error":…}) correctly distinguishes fake-empty from real-empty catalogs; non-404 statuses are correctly blocked by the hit-counter test. Minor: empty-catalog recovery emits two consecutive INFO logs. |
| src/openhuman/inference/local/lm_studio.rs | Extracts shared host_root_of helper (strips trailing /v1) and adds ollama_tags_fallback_url; new unit tests cover the host-rooted URL derivation and the malformed /v1/api/tags guard. |
| src/openhuman/inference/local/ollama.rs | Adds error: Option<String> to OllamaTagsResponse with #[serde(default)]; backward-compatible and required to distinguish LM Studio's error-envelope {"error":…} from a genuinely empty Ollama catalog. |
| src/openhuman/inference/model_ids.rs | Replaces non-existent gemma4:e4b-it-q8_0 with gemma3n:e4b-it-q8_0 in MVP_ALLOWED_CHAT_MODELS; adds regression guard mvp_chat_allowlist_has_no_unpullable_namespaces. |
| src/openhuman/inference/presets.rs | Corrects the 16 GB+ tier: gemma4:e4b → gemma3n:e4b-it-q8_0 for both chat and vision IDs, and quantization label qat → q8_0; adds two regression tests. vision_mode: Bundled with a text-only model remains a known follow-up item (acknowledged by PR author). |
| src/openhuman/inference/local/service/ollama_admin_tests.rs | Adds five new integration tests for the fallback path: happy-path recovery, original-error preservation, error-envelope rejection, empty-catalog recovery, and non-404 no-probe guard (with atomic hit counter). |
| tests/raw_coverage/inference_local_admin_raw_coverage_e2e.rs | Mechanical gemma4:e4b-it-q8_0 → gemma3n:e4b-it-q8_0 rename throughout the e2e coverage harness to match the corrected model ID. |
| tests/raw_coverage/inference_provider_admin_round22_raw_coverage_e2e.rs | Same mechanical gemma4 → gemma3n rename in the round-22 coverage harness. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[list_lm_studio_models] --> B[GET /v1/models]
B --> C{HTTP status?}
C -->|2xx| D[Parse LmStudioModelsResponse]
D --> E[Return Ok models]
C -->|non-404 error| F[Return Err original status]
C -->|404| G[list_ollama_tags_fallback]
G --> H[GET /api/tags\nhost_root_of base + /api/tags]
H --> I{Fallback result?}
I -->|request error| J[Return None]
I -->|non-2xx| J
I -->|parse error| J
I -->|error envelope present| J
I -->|models list empty or non-empty| K[Return Some models]
K --> E
J --> F
Reviews (2): Last reviewed commit: "fix(inference): accept a valid empty Oll..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/inference/local/service/ollama_admin_tests.rs`:
- Around line 1322-1345: The test
v1_models_404_reports_the_original_error_when_fallback_also_fails does not
distinguish the primary /v1/models error from the fallback failure. Update the
mock router to make the fallback /api/tags response return a distinct error,
then strengthen the assertion to verify the surfaced error contains the original
/v1/models 404 details rather than the fallback error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bc653390-d3fa-453e-a317-544c4d3d2c0a
📒 Files selected for processing (7)
src/openhuman/inference/local/lm_studio.rssrc/openhuman/inference/local/service/lm_studio.rssrc/openhuman/inference/local/service/ollama_admin_tests.rssrc/openhuman/inference/model_ids.rssrc/openhuman/inference/presets.rstests/raw_coverage/inference_local_admin_raw_coverage_e2e.rstests/raw_coverage/inference_provider_admin_round22_raw_coverage_e2e.rs
… tests (tinyhumansai#5055) P2 (Codex) — the /api/tags fallback rejected an EMPTY catalog, but `{"models":[]}` is exactly what a fresh Ollama with nothing pulled returns. Treating it as a failure hid a reachable runtime behind the original 404, so availability/bootstrap reported a degraded error and the UI could not offer the model-download action. The real LM Studio signal is an error envelope, not emptiness: it answers unknown paths with `200 {"error": …}` and no models (tinyhumansai#5053). `OllamaTagsResponse` now carries an optional `error` field and the fallback branches on that, accepting an empty `models` array as a successful zero-model catalog. The field is `#[serde(default)]` and the type is only ever deserialized, so no construction sites change. Minor (CodeRabbit) — `v1_models_404_reports_the_original_error_when_fallback_also_fails` was not actually distinguishing the two errors: Axum's implicit fallback route also answers 404, so the assertion passed even if the implementation surfaced the /api/tags failure. The mock now returns a distinct 503 for /api/tags and the test asserts the 404 survives AND the 503 does not leak. Splits the old empty-catalog test into the two cases it was conflating: an error envelope is still not a recovery, an empty catalog now is.
|
@coderabbitai review |
✅ Action performedReview finished.
|
1 participant
Summary
main. I verified both in the code rather than assuming, and left them alone. Details in Problem below./api/tagsfallback on a/v1/models404 — the piece of the discovery fix that was genuinely missing — with three deliberate non-recovery cases so a permissive fallback can't reintroduce LM Studio / OpenAI-compatible providers: app calls Ollama-style /api/tags instead of /v1/models, breaking model discovery for nvidia/nemotron-3-nano-omni and similar #5053.gemma4is not an Ollama namespace. The 16 GB+ preset shippedgemma4:e4bfor chat and vision, and the chat allowlist carriedgemma4:e4b-it-q8_0— a model tier noollama pullcould ever fetch. Corrected togemma3n:e4b-it-q8_0.Problem
#5055 lists five failure modes. Three of them were already fixed before this PR, so re-implementing them would have been duplicated work and churn on tested code. What I found on
main:#5053 (wrong discovery endpoint) — already fixed.
inference/local/provider.rshasmodel_discovery_api()andendpoint_is_openai_v1(), which gate discovery on endpoint type (does the path end in/v1?) rather than "is it localhost" — the exact conflation that caused the bug. It is wired intoollama_admin/diagnostics.rsand pinned by a regression test whose mock serves only/v1/models.#5017 (embedding verification) — already fixed.
embeddings/rpc.rssends a realPOST /v1/embeddingscarrying the user's model name and API key, andclassify_embed_probesplits the outcome into seven distinct causes:EMBEDDINGS_AUTH_FAILED(401/403),_ENDPOINT_UNREACHABLE(network/DNS),_MODEL_INCOMPATIBLE(a chat model in the embeddings field — the reporter's exact case),_DIMENSION_MISMATCH,_ENDPOINT_NO_API(404/405),_NO_MODEL_LOADED, and a generic fallback.EmbeddingsPanel.tsxsurfaces the backend message for every code, andconformant_custom_endpoint_verifies_and_sends_expected_requestcaptures the auth header and request body against a mock endpoint to prove the request shape.That maps onto the issue's requested
NetworkUnreachable / AuthFailed / ModelNotFound / DimensionMismatch / UnexpectedResponseset, so scope items 2 and 4 needed no code.What was actually still broken:
/v1/models→ 404 → try/api/tagsonce with a warning. That did not exist in any form.MVP_ALLOWED_CHAT_MODELScontainedgemma4:e4b-it-q8_0and the 16 GB+ preset usedgemma4:e4bfor bothchat_model_idandvision_model_id. There is nogemma4namespace on the Ollama library — this is precisely the issue's failure mode 3 ("defaults may be stale or unavailable, causing silent fallback or no-model errors"), and it had a second-order effect described below.Solution
1. One-shot
/api/tagsfallback (local/service/lm_studio.rs)On a
/v1/models404 only,list_ollama_tags_fallbackretries the host-rooted/api/tagsonce and WARNs on entry (taking this path means the base URL and provider type disagree, so the user should still fix their config even though discovery recovered).Three cases deliberately do not recover, because a permissive fallback would recreate the bug this is meant to fix:
/api/tagscatalog200 {"error": …}and zero models (#5053). Accepting that would look like "discovered 0 models" instead of a real error./v1/modelserror is surfacedNew
ollama_tags_fallback_url()(local/lm_studio.rs) strips a trailing/v1via a sharedhost_root_of()helper, so the probe can never become the malformed/v1/api/tagsthat LM Studio logs asUnexpected endpoint or method.Discovery is still chosen by provider type first. This is a recovery path, not a probe order.
2. DEBUG log of the resolved discovery URL
discovery_url+api = "openai_v1_models"on the outbound GET, and the equivalent on the fallback.3.
gemma4→gemma3n(model_ids.rs,presets.rs)Verified every local default against the public Ollama library before changing anything:
gemma3:1b-it-qatgemma3:4b-it-qatmoondream:1.8b-v2-q4_K_Sbge-m3DEFAULT_OLLAMA_EMBED_MODELgemma3n:e4b-it-q8_0gemma4:e4b,gemma4:e4b-it-q8_0"e4b" is Gemma 3n's Effective 4B designation, and 9.5 GB +
nomic-embed-textmatches the tier's existingapprox_download_gb: 9.9— sogemma3n:e4b-it-q8_0is what the tier was sized against.Latent second bug closed: the preset (
gemma4:e4b) and the allowlist (gemma4:e4b-it-q8_0) named different ids, so selecting that tier would have been silently redirected togemma3:1b-it-qatbyenforce_mvp_chat_allowlist. They now agree. (The tier is not MVP-gated today, so this bit only on a lifted ceiling — but it was still wrong.)quantizationon that tier moves"qat"→"q8_0"to match what is actually pulled. I checked first: it is a display string consumed byeffective_quantizationand takes no part in resolving the model tag, and no test asserts on it.Not addressed (deliberate)
click(issue item 4) is outside this PR's dispatched scope and untouched — it needs its own change.vision_model_idon the 16 GB+ tier: Ollama lists gemma3n variants as text-input, sovision_mode: Bundledmay be non-functional there. It was already non-functional with a model that did not exist, so I corrected the namespace and did not re-architect the vision choice on a guess. Flagged under Related as a follow-up for someone who can test it.Submission Checklist
gemma4namespace returning.diff-cover) meet the gate enforced by.github/workflows/ci-lite.yml. Runpnpm test:coverageandpnpm test:rustlocally; PRs below 80% on changed lines will not merge. — every changed Rust path has a new or updated test. Not run locally (see Validation Blocked); CI is the authority on the number.docs/TEST-COVERAGE-MATRIX.mdreflect this change (orN/A: behaviour-only change) —N/A: behaviour-only change— no feature leaf added, removed or renamed; this fixes existing local-inference behaviour.## Relatedaxummock on127.0.0.1. The Ollama library lookups used to verify model ids were research during authoring, not test-time calls.docs/RELEASE-MANUAL-SMOKE.md) —N/A: no release-cut smoke step changes— local-model setup is not in the release-cut checklist, and the default (managed/cloud) path is untouched.Closes #NNNin the## RelatedsectionImpact
list_lm_studio_models, which a genuine Ollama config never reaches —diagnostics()routes it to the/api/tagsbranch first. Confirmed by tracing every caller./v1/modelsbut serves/api/tagsnow discovers models instead of failing. Users on the 16 GB+ preset get a model that can actually be pulled; anyone who had literally selectedgemma4:*will be redirected to the default by the existing allowlist, which is the pre-existing behaviour for an unknown id.gemma4:*in a stored config was already non-functional.Related
main, not re-implemented here): LM Studio / OpenAI-compatible providers: app calls Ollama-style /api/tags instead of /v1/models, breaking model discovery for nvidia/nemotron-3-nano-omni and similar #5053, Custom OpenAI-compatible embedding endpoint fails verification even with valid endpoint and key #5017N/A— no matrix row corresponds to local model discovery today.click(Fix local model reliability: validate local model behavior in Tiny Agents and correct provider configuration #5055 item 4) — out of this PR's scope, still open.gemma4find, the 16 GB+ preset should be confirmed to actually pull before Fix local model reliability: validate local model behavior in Tiny Agents and correct provider configuration #5055 is closed.vision_mode: Bundledwith a text-input gemma3n build may be a no-op; needs someone who can run it.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
N/A— tracked in GitHub (Fix local model reliability: validate local model behavior in Tiny Agents and correct provider configuration #5055), not Linear.Commit & Branch
fix/local-model-reliabilitye499ce440(branched from9a484aa50)Validation Run
pnpm --filter openhuman-app format:check—N/A— no frontend files changed in this PR.pnpm typecheck—N/A— no TypeScript changed in this PR.GGML_NATIVE=OFF cargo check -p openhuman --testscompleted clean twice (exit 0, only three pre-existing warnings:address_book,Memory,alice_phoenix_thread), so everything compiles including the new tests; the test run was stopped before it produced results.cargo check -p openhuman --testsclean, as above.cargo fmtnot re-run after the final edit — see Validation Blocked.N/A— no files underapp/src-tauri/changed.Validation Blocked
command:cargo test -p openhuman --lib,cargo fmt, and any furthercargo checkerror:not run — the build machine was overloaded and further cargo compiles were stopped mid-run by an operator instruction; no compile has been run since.impact:The six new tests compile but have never been executed — CI is the first thing to actually run them. Two late edits (thequantization: "q8_0"label and its comment) landed after the last successfulcargo check, so they are not type-checked either; both are string literals in an existing struct field with no test asserting on them. If CI shows a formatting failure,cargo fmtonpresets.rsis the likely fix.Behavior Changes
/v1/models404 falls back once to/api/tags; the 16 GB+ local preset and the chat allowlist name a real, pullable Gemma 3n build.Parity Contract
model_discovery_api) before any fallback; the Ollama/api/tagsbranch is unchanged; the embeddings verification path is untouched;enforce_mvp_chat_allowlistsemantics are unchanged.non_404_status_does_not_trigger_the_fallbackasserts with a hit counter that a 500 never probes/api/tags;empty_api_tags_fallback_is_not_treated_as_recoveryandv1_models_404_reports_the_original_error_when_fallback_also_failspin the two non-recovery cases;presets_reference_no_unpullable_gemma4_namespaceandmvp_chat_allowlist_has_no_unpullable_namespacesprevent the bad namespace returning.Duplicate / Superseded PR Handling
N/A— no duplicates. Note LM Studio / OpenAI-compatible providers: app calls Ollama-style /api/tags instead of /v1/models, breaking model discovery for nvidia/nemotron-3-nano-omni and similar #5053 and Custom OpenAI-compatible embedding endpoint fails verification even with valid endpoint and key #5017 were fixed by earlier, already-merged PRs; this PR does not supersede them, it completes the remainder.Summary by CodeRabbit
New Features
Bug Fixes
Tests