feat(inference): Phase 3 P3-B — crate-native turn path + delete compatible*.rs [WIP]#4784
Merged
senamakel merged 74 commits intoJul 13, 2026
Conversation
…e path (P3-B step 1)
The crate-native managed model (OpenHumanBackendModel) delegates to a crate
OpenAiModel whose parse_response leaves the `openhuman.{billing,usage}` envelope
only on ModelResponse.raw — it has no field for charged USD — so the seam's
usage_info_from_response (which reads the `openhuman_usage_meta` key ProviderModel
writes) recovered $0 charged + dropped backend cached tokens. This is a latent
underbilling bug already live for one-shot managed callers (memory/chat.rs,
meeting/checkpoint summaries) and would extend to every interactive turn once the
turn path goes crate-native.
- model.rs: add merge_openhuman_usage_meta — the symmetric writer for
usage_info_from_response, stashing charged-USD + context window under
OPENHUMAN_USAGE_META_KEY on the response raw (no-op when both are zero).
- openhuman_backend_model.rs: project_managed_usage reads the openhuman envelope
off raw, re-projects charged-USD + context window into openhuman_usage_meta, and
backfills crate Usage.cache_read_tokens from the (authoritative) envelope count
when the standard parse produced none. Applied on the invoke path.
- Streaming: documented parity gap — the crate SSE parser sets raw:None on the
terminal Completed, so streaming managed turns fall back to the catalog cost
estimate (token counts survive); restoring authoritative charged USD for
streaming needs the crate to preserve the final chunk raw (tracked upstream).
- Tests: project_managed_usage recovers charged/cached/window; no-op without envelope.
Part of Phase 3 P3-B (turn-path crate-native cutover, tinyhumansai#4249). Local full builds
OOM on the shared box; verification defers to CI.
Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (85)
📝 WalkthroughWalkthroughThis PR migrates inference and agent execution toward TinyAgents crate-native chat models, adds provider compatibility adapters, defers production provider construction, updates channel and triage routing, and revises streaming, tool-call, usage, and error handling across consumers and tests. ChangesCrate-native inference foundation
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant ChannelRuntime
participant TurnModelSource
participant ChatModel
participant ProviderAdapter
ChannelRuntime->>TurnModelSource: build turn models
TurnModelSource->>ChatModel: create crate-native model
ChatModel->>ProviderAdapter: invoke or stream ModelRequest
ProviderAdapter-->>ChatModel: response and usage
ChatModel-->>ChannelRuntime: text, tool calls, and metadata
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
Comment |
…native turn builder (P3-B step 2 foundation) The per-(role, explicit-model) analogue of create_chat_model_with_model_id, for the crate-native TurnModelSource to build the primary + each workload-tier route without a host Provider. Managed -> OpenHumanBackendModel pinned to the model (backend resolves the tier from request.model); local/cloud -> crate builders; bespoke (claude-code) -> ProviderModel over the resolved Provider. Respects the test-provider override. Additive (allow(dead_code)) — wired into TurnModelSource::build in the next increment. Part of Phase 3 P3-B (tinyhumansai#4249). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…e opt-in (P3-B step 2) TurnModelSource gains an optional crate_native (role, Arc<Config>): when set, build()/build_summarizer() construct the tiered TurnModels crate-natively via build_turn_models_crate (per-tier factory::create_turn_chat_model) instead of wrapping one host Provider in ProviderModels. The source still holds the resolved provider, used as (a) the escape-hatch for sub-agent inheritance / rhai and (b) a graceful fallback if crate-native construction fails — so producers can flip to new_crate_native one at a time without regressing behavior. build_turn_models_crate derives the harness-visible caps without a Provider: native_tools/supports_vision from the primary ModelProfile (managed backend, profile None, falls back to true / oh_tier_supports_vision(model)); provider_id from the resolved slug (managed for the backend); error_slot empty (crate-native surfaces TinyAgentsError directly — Sentry suppression is unaffected since both skips_sentry cases are host-loop-raised). Unbuildable tiers are skipped, not fatal. No producer flipped yet (new_crate_native is allow(dead_code)); the build/summarizer dispatch + fallback land first so the next increment can flip producers behind a verified path. cargo check --lib clean. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…3-B step 3, producer 1/7)
First producer cutover: the sub-agent extract summarizer now builds its
TurnModelSource via new_crate_native("summarization", cfg) when config loads, so
build_summarizer constructs a crate-native ChatModel (managed -> OpenHumanBackendModel)
instead of a ProviderModel. The resolved provider is retained for the escape-hatch
+ crate-native-failure fallback; a config-load failure degrades to the parent
provider on the Provider path (unchanged). Lowest-blast-radius producer (internal
extraction, no interactive Sentry path). new_crate_native is now live (allow(dead_code)
removed). cargo check --lib clean. Draft PR tinyhumansai#4784.
Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…serving tinyhumansai#1257 (P3-B step 3, producer 2/7) The triage turn now builds crate-native models for the remote/managed arm via TurnModelSource::new_crate_native_from_string, preserving the tinyhumansai#1257 "triage never goes local" invariant: build_remote_provider's effective provider string (openhuman when force-managed) rides through a new primary override so build() resolves the managed backend rather than re-deriving from the role (which could go local). - factory::create_turn_chat_model_from_string(role, provider_string, config, model, temp): managed when the string is openhuman/empty/cloud (-> OpenHumanBackendModel pinned to model); else delegates to create_turn_chat_model. Respects test override. - CrateNativeSource gains primary_override: Option<String>; new_crate_native_from_string; build_turn_models_crate uses it for the primary + summarizer (routes unchanged). - ResolvedProvider gains crate_native: Option<(effective_string, Arc<Config>)> — Some for the remote/managed arm, None for the local arm (kept on the Provider path so it doesn't silently route to the managed backend and defeat the local fallback). - Evaluator flips on resolved.crate_native; resolved provider retained as escape-hatch + crate-native-failure fallback. cargo check --lib clean. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…sai#55 prompt-guided tools) Pulls in tinyagents#55 (prompt-guided/text-mode tool calling for non-native models) + the v1.9.0 release. Unblocks flipping LOCAL runtimes to crate-native on the turn path: they force native_tools=false and now get prompt-guided tools from the crate instead of losing tool calls. Host lib compiles clean against 1.9.0. Part of Phase 3 P3-B (tinyhumansai#4249). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…age e2e (P3-B fix) The triage flip added ResolvedProvider.crate_native; cargo check --lib doesn't compile tests/ integration targets, so the 4 mock ResolvedProvider constructors in tests/raw_coverage/inference_agent_raw_coverage_e2e.rs (globbed into the raw_coverage_all target) were missed and broke the full-suite CI on the pin-bump push. Added crate_native: None (mocks stay on the Provider path). Verified via cargo check --test raw_coverage_all. Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…r (P3-B) The extract flip (producer 1/7) re-resolved "summarization" from config via create_turn_chat_model, bypassing the extract's already-resolved provider — incl. a test-injected mock the raw_coverage e2e observes (extraction_prompts()). Full suite went red: integrations_text_mode_handoffs_oversized_result_and_extracts_from_cache "got 0 prompts". Production stays managed either way, but a producer that injects/ resolves a SPECIFIC provider must keep it on the Provider path (or pass an effective string via new_crate_native_from_string, like triage) — plain new_crate_native role-resolves and drops the injected provider. Extract now stays TurnModelSource::new(extract_provider). new_crate_native (plain) is allow(dead_code) until the session-builder flip. Verified: the exact failing test now passes (cargo test --test raw_coverage_all --features e2e-test-support). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…P3-B, producer 3) The production main-turn agent now builds crate-native ChatModels from (provider_role, config) via create_turn_chat_model instead of a ProviderModel- wrapped provider (managed -> OpenHumanBackendModel, local -> crate OpenAiModel w/ prompt-guided tools, cloud -> crate OpenAiModel native). - setters.rs: AgentBuilder::crate_native_provider(provider, role, Arc<Config>) — new_crate_native source; the plain .provider() setter (Provider path) stays for tests that inject a mock they observe. - factory.rs:1144: .provider(provider) -> .crate_native_provider(provider, provider_role, Arc::new(config.clone())). - EXACT metadata parity: build_turn_models_crate now takes provider_id/native_tools/ supports_vision as params, derived in TurnModelSource::build from the RETAINED provider (self.provider.telemetry_provider_id/supports_native_tools/supports_vision) — the SAME source build_turn_models uses — so these fields are byte-identical to the Provider path (fixes the supports_vision divergence); only the MODELS go crate-native. Sub-agents inherit the retained provider via the escape-hatch (Provider path, unaffected). Crate-native-build failure falls back to the Provider path. Safe for tests: e2e/raw_coverage tests build agents via Agent::builder().provider(mock) directly (Provider path) or TurnModelSource::new(provider); NONE drive the production factory, so the flip is not reachable from them. Live per-provider validation deferred to the dedicated tinyagents run. Compiles clean (lib + raw_coverage_all). Draft PR tinyhumansai#4784. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…rness_e2e (P3-B) The session-builder flip (2528fa0) broke ~14 tests/agent_harness_e2e.rs turn tests: they drive the production factory (factory.rs:1144) via the harness/SSE flow and inject a ScriptedProvider NOT through test_provider_override, so create_turn_chat_model's override guard doesn't catch it — the crate-native re-resolution builds a real managed model (no backend) -> no SSE events -> timeout. Reverted factory.rs:1144 to .provider(provider). Kept (allow(dead_code)): the metadata-parity refactor, crate_native_provider setter, new_crate_native. Restores tinyhumansai#4784 to green (triage flip + infrastructure intact). Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…2e mock (P3-B, producer 3) Adds SSE streaming to the agent_harness_e2e scripted upstream mock (serves text/event-stream data: choices[].delta chunks + [DONE] when stream:true, matching the crate OpenAiModel::stream parser; both old + crate paths parse real SSE), which was the blocker: the crate streaming path needs real SSE but the mock only returned JSON. Re-flips factory.rs:1144 to .crate_native_provider so the main interactive turn builds crate-native models (managed->OpenHumanBackendModel etc.) with metadata parity from the retained provider. Verifying at the end per the full-cutover plan. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
ChannelRuntimeContext gains config: Option<Arc<Config>> (Some in production via
start_channels; None in tests keeps the Provider path). The channel turn builds
crate-native models via new_crate_native_from_string("chat", route.provider, config)
when config is present; the cached provider is retained for the escape-hatch +
fallback + metadata parity. cargo check --lib clean (all 24 ChannelRuntimeContext
sites carry the new field).
Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
Pins tinyagents#56 (OpenAiModel::stream falls back to parsing a non-SSE JSON body as one completion) so the crate-native turn cutover works against OpenHuman's JSON-only e2e mocks. Pinned to the branch commit (the vendored [patch] path build resolves it directly; no crates.io release needed). Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
…roducer 5) Both sub-agent graph paths (default + custom) build a crate-native subagent_source: a workload Hint -> that role; any other spec (Inherit/Exact/override) reuses the parent's role via TurnModelSource::crate_native_role() when the parent is crate-native. Resolved provider retained for fallback + metadata parity; falls back to the Provider path when no role is derivable / config load fails. Completes the turn-producer cutover (session, channels, triage, sub-agent all crate-native). cargo check --lib clean. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
coderabbitai
Bot
added
agent
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bdad55cf70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| // A concrete non-managed string equals the role's resolution (triage only | ||
| // honours a BYOK **cloud** route as-is), so the role-based builder matches. | ||
| create_turn_chat_model_with_native_tools(role, config, model, temperature, native_tool_calling) |
There was a problem hiding this comment.
When new_crate_native_from_string is used with a concrete non-managed provider string, this branch discards that string and rebuilds from provider_for_role(role, config) instead. That breaks callers whose effective provider intentionally differs from the role default: for example build_local_provider_with_config passes ollama:<local_ai.chat_model_id> for the triage local-fallback arm, but the actual turn is built from the subconscious route again, so cloud budget/unhealthy fallback can hit the same remote provider instead of the configured local model.
Useful? React with 👍 / 👎.
| tinyagents::harness::model::ModelRequest::new(vec![ | ||
| tinyagents::harness::message::Message::user(message.to_string()), | ||
| ]) | ||
| .with_model(default_model.clone()) |
There was a problem hiding this comment.
For openhuman.agent_chat_simple with a BYOK/local chat_provider such as ollama:llama3 or openai:gpt-4o, create_chat_model_with_model_id returns that concrete resolved_model, but this request overrides the model with default_model (often a managed tier like reasoning-v1). Since ModelRequest.model overrides the model baked into the crate OpenAI client, these calls can send an invalid managed-tier id to the user's selected provider instead of the configured provider model.
Useful? React with 👍 / 👎.
11 tasks
oxoxDev
added a commit
to oxoxDev/openhuman
that referenced
this pull request
Jul 13, 2026
…sai#4436) The S2 egress emit was wired only into the legacy Provider path (create_chat_provider_from_string). Post-tinyhumansai#4784 real agent turns build via the crate-native ChatModel/turn path (create_chat_model_with_model_id and the create_turn_chat_model* family), whose managed / local-runtime / BYOK-cloud sub-paths return early WITHOUT touching the Provider path — so the default managed chat turn disclosed no egress (confirmed by an inference-probe smoke on staging firing zero events). Emit at the three universal construction chokepoints shared by every ChatModel and turn entry, guaranteeing exactly one descriptor per construction: - managed backend -> resolve_managed_backend (external); the sole funnel for the Provider AND ChatModel/turn managed builds. emit_inference_egress now skips PROVIDER_OPENHUMAN so the Provider path doesn't double-emit. - local runtime -> try_create_local_runtime_chat_model_from_string, past the non-local return + gates (disclosed as NON-external, no pending event). - BYOK cloud slug -> try_create_cloud_slug_chat_model_from_string_with_native_tools, past the managed/bespoke/OpenhumanJwt returns + gates (external). The Provider path's top-level emit still covers its direct callers and the bespoke fallthrough; the two paths are mutually exclusive per construction.
12 tasks
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal (issue #4249, Phase 3 P3-B)
Route the agent turn path off the host
OpenAiCompatibleProvider(
compatible*.rs, aBox<dyn Provider>wire client) onto the vendoredtinyagentscrate's nativeChatModels, then deletecompatible*.rs(~430KB).create_chat_model_with_model_idalready returns crate-native models (managed →OpenHumanBackendModel, local/cloud → crateOpenAiModel); the turn path is thelast big consumer still on the
Providerpath.Landed so far
managed model left the
openhuman.{billing,usage}envelope only onModelResponse.raw, so the seam'susage_info_from_responserecovered $0charged + dropped backend cached tokens — a latent underbilling bug already
live for one-shot managed callers (
memory/chat.rs, meeting/checkpointsummaries).
OpenHumanBackendModel::project_managed_usagenow re-projectscharged-USD + context window into the
openhuman_usage_metashape the costbridge reads, and backfills crate
Usage.cache_read_tokens. Verified:cargo check --libclean; unit tests added.Planned increments (each pushed for CI)
TurnModelSource/build_turn_models(per-tier models, capsderived without a
Provider).sub-agent runner → channels → root session builder).
OpenAiCompatibleProvider::newsites + one-shotcreate_chat_providercallers.compatible*.rs; mark ready.Known parity notes (deferred to the dedicated live-test run)
raw: Noneon the terminalchunk, so streaming managed turns fall back to the catalog cost estimate
(token counts survive; authoritative charged-USD needs an upstream crate change
to preserve the final chunk raw). Non-streaming
invokeis fully correct.TinyAgentsError, so theDomainEvent::AgentErrorkindtag softens for provider failures. Sentrysuppression is unaffected (both
skips_sentrycases —MaxIterationsExceeded,EmptyProviderResponse— are raised in the host turnloop, not via the model). A lightweight
ProviderError→AgentErrorclassifiercan restore kind tags if wanted.
https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
Summary by CodeRabbit
New Features
Bug Fixes