refactor(inference): route local AI through TinyAgents RPC - #5240
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (13)
📝 WalkthroughWalkthroughLocal chat, generation, and embedding paths now use TinyAgents-backed model interfaces. Provider-specific LM Studio and Ollama HTTP models and calls are removed, while tool-message fixtures and Ollama message traits are updated for the new interfaces. ChangesLocal inference migration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant InferenceService
participant model_rpc
participant TinyAgentsModel
InferenceService->>model_rpc: Build messages and invoke
model_rpc->>TinyAgentsModel: Send ModelRequest
TinyAgentsModel-->>model_rpc: Return reply and usage
model_rpc-->>InferenceService: Return ModelRpcOutcome
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10107e57eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fd6950d16
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2fd6950 to
96a28ec
Compare
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
coderabbitai
Bot
added
feature
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
src/openhuman/agent/message_convert.rs (1)
114-118: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueCorrect default — converted tool history should not be trusted verbatim.
The fallback branch at Line 113 wraps arbitrary unparsed content, so
falseis the right conservative value here.Nit: no test pins this.
seeded_native_tool_round_recovers_structure_and_round_trips(Lines 445-450) already destructures the convertedToolMessage, soassert!(!tm.trusted_verbatim);there would lock in the security intent cheaply.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/agent/message_convert.rs` around lines 114 - 118, Keep trusted_verbatim set to false in the fallback ToolMessage conversion for unparsed content. Update seeded_native_tool_round_recovers_structure_and_round_trips to assert that the destructured converted ToolMessage has trusted_verbatim disabled.src/openhuman/inference/local/service/vision_embed.rs (1)
308-313: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winHelper is covered, but the rewritten
embedRPC path still has no asserting test.
embed_against_mock_returns_vectors_with_dimensions(Lines 253-282) discards its result withlet _ = result;, so the switch toOllamaEmbeddingModelis exercised without verifying vector count, dimensions, or error mapping. Worth tightening that mock test to assert on the returnedLocalAiEmbeddingResultnow that the wire handling moved to the harness.Nit: the
bge-m3case really pinsDEFAULT_OLLAMA_DIMENSIONSrather than a bge-m3-specific rule; asserting against the constant would keep the intent clear if TinyAgents ever changes its default.As per coding guidelines, "Add tests for new or changed behavior; untested code is incomplete."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/inference/local/service/vision_embed.rs` around lines 308 - 313, Strengthen embed_against_mock_returns_vectors_with_dimensions by asserting the returned LocalAiEmbeddingResult: verify the expected vector count, embedding dimensions, and error mapping from the mock RPC response. Keep the existing OllamaEmbeddingModel harness coverage, and update the bge-m3 assertion in embedding_dimensions_match_supported_legacy_models to use DEFAULT_OLLAMA_DIMENSIONS rather than duplicating its numeric value.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/inference/local/service/model_rpc.rs`:
- Around line 62-76: Add focused unit tests for the RPC boundary decision logic
in local_model and invoke. Use Config-only cases to cover local_model provider
branch selection, test invoke’s allow_empty gate, and verify zero input/output
usage values become None while positive values are preserved in ModelRpcOutcome.
- Around line 21-33: Add verbose, grep-friendly diagnostics around the new local
RPC boundary, updating local_model and invoke to log entry and exit, the
selected LocalAiProvider branch, resolved endpoints, external model calls, and
failure paths. Use redact_ollama_base_url for Ollama endpoints and equivalent
safe handling for LM Studio, ensuring API keys, secrets, and full PII are never
logged; preserve the existing error propagation and model-selection behavior.
In `@src/openhuman/inference/local/service/vision_embed.rs`:
- Around line 17-27: Update embedding_dimensions so unknown or user-managed
model IDs are not assigned DEFAULT_OLLAMA_DIMENSIONS; either add explicit
mappings for supported models or derive the dimension from the embedding
response. Ensure the resolved model ID and dimension are emitted at debug level,
using effective_embedding_model_id and the existing embedding flow.
---
Nitpick comments:
In `@src/openhuman/agent/message_convert.rs`:
- Around line 114-118: Keep trusted_verbatim set to false in the fallback
ToolMessage conversion for unparsed content. Update
seeded_native_tool_round_recovers_structure_and_round_trips to assert that the
destructured converted ToolMessage has trusted_verbatim disabled.
In `@src/openhuman/inference/local/service/vision_embed.rs`:
- Around line 308-313: Strengthen
embed_against_mock_returns_vectors_with_dimensions by asserting the returned
LocalAiEmbeddingResult: verify the expected vector count, embedding dimensions,
and error mapping from the mock RPC response. Keep the existing
OllamaEmbeddingModel harness coverage, and update the bge-m3 assertion in
embedding_dimensions_match_supported_legacy_models to use
DEFAULT_OLLAMA_DIMENSIONS rather than duplicating its numeric value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d64cad9a-9847-461b-a066-c59d59b3aebd
📒 Files selected for processing (9)
src/openhuman/agent/message_convert.rssrc/openhuman/inference/local/lm_studio.rssrc/openhuman/inference/local/ollama.rssrc/openhuman/inference/local/service/lm_studio.rssrc/openhuman/inference/local/service/mod.rssrc/openhuman/inference/local/service/model_rpc.rssrc/openhuman/inference/local/service/public_infer.rssrc/openhuman/inference/local/service/vision_embed.rsvendor/tinyagents
💤 Files with no reviewable changes (2)
- src/openhuman/inference/local/ollama.rs
- src/openhuman/inference/local/lm_studio.rs
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 426b2bafe4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4a62158dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9431c45df8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a2632037d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
d748869 to
a9a110d
Compare
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
senamakel has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
1 participant
Summary
ChatModel.OllamaEmbeddingModel.Problem
OpenHuman maintained parallel Ollama and LM Studio inference clients despite TinyAgents already owning the provider-neutral model and embedding transports. This duplicated authentication, endpoint normalization, reasoning parsing, usage handling, and error behavior while keeping local inference coupled to the application core.
Solution
OpenHuman now resolves configuration and delegates out-of-process inference over local HTTP to TinyAgents. Lifecycle and model-catalog administration remain in OpenHuman for now. Depends on tinyhumansai/tinyagents#79.
Submission Checklist
Impact
Desktop/core local inference keeps its existing RPC surface and configuration. Local chat and embeddings now share TinyAgents transport behavior. No migration is required.
Related
AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
inference-rpc-cutover10107e57ebef1a34ba98229df2958c080bbf5a90Validation Run
cargo test local_runtime_presets_normalize_endpoint_and_model --libin TinyAgentscargo fmt --all;GGML_NATIVE=OFF cargo check --manifest-path Cargo.tomlValidation Blocked
command:OpenHuman focused inference test binary final linkerror:no compiler error; stopped after prolonged resource-heavy linking once fullcargo checkpassedimpact:CI will run the authoritative changed-file Rust tests and coverage gateBehavior Changes
Parity Contract
Duplicate / Superseded PR Handling
Summary by CodeRabbit
New Features
Improvements