feat(voice): ElevenLabs Voice Agents — core RPC + realtime session + local-agent relay handler (#5399) - #5407
Merged
M3gA-Mind merged 9 commits intoAug 5, 2026
Conversation
…humansai#5399) Adds openhuman.voice_agent_signed_url — a no-param RPC the desktop client calls to obtain a short-lived signed WebSocket URL for a realtime voice session (ElevenLabs Agents). Proxies the backend /voice-agent/get-signed-url following the reply_speech auth pattern (session token -> BackendOAuthClient -> authed_json, flatten_authed_error so a lapsed-session 401 classifies as SESSION_EXPIRED and skips Sentry). - new voice/realtime.rs (gated by the voice feature): mint fn + pure, unit-tested response parser (wrapped envelope / bare object / missing url) - handler + controller registered in the voice schema registry
…#5399) Additive, default-OFF plumbing for the realtime ElevenLabs Agents voice mode. Nothing changes until VITE_VOICE_MODE is set and the user flips the toggle; the classic turn-based path is untouched. - config: VOICE_MODE_FLAG_ENABLED (VITE_VOICE_MODE) gates the Settings toggle - mascotSlice: persisted voiceMode ('classic' | 'realtime') + setVoiceMode reducer + selectVoiceMode selector + rehydrate guard (falls back to classic) - voiceAgentApi.fetchVoiceAgentSignedUrl -> the voice.agent_signed_url core RPC - VoicePanel: a "Realtime voice (beta)" toggle rendered only behind the flag - i18n: voice.mode.* keys added to all 14 locales - tests: slice reducer/selector/rehydrate + api client mapping (7 green)
…inyhumansai#5399) Completes the realtime relay round trip. The backend Custom-LLM bridge emits voice:harness { correlationId, messages } down the socket; the core now runs the FULL local orchestrator (the user's tools/memory/MCP, same brain as chat + meet) on the turn and streams the reply back up as voice:harness:delta / :done / :error. This is what keeps a cloud ElevenLabs voice session backed by the desktop-local brain. - voice/realtime_harness.rs: extract_prompt (pure, unit-tested) + run the orchestrator via Agent::from_config_for_agent_with_profile + run_single with a 90s timeout, ExternalChannel turn-origin (audit-gated tools), and a spoken directive; emit the reply over the SocketManager - platform/socket/event_handlers.rs: voice:harness arm spawns the handler (voice-gated; no-op with a warning when the voice feature is off)
…humansai#5399) Adds the client half of the realtime voice mode, behind the existing flag + voiceMode toggle. Classic path untouched. - useRealtimeVoiceSession: @elevenlabs/react useConversation over WebSocket (so per-audio-event character alignment is available for lip-sync). start() fetches a signed URL via the core RPC and opens the session with the mascot voice override; exposes state/isSpeaking/mode/error + start/stop. - RealtimeVoiceControls: self-contained ConversationProvider + start/stop control with connecting/listening/speaking status, rendered on the Human tab only when VOICE_MODE_FLAG_ENABLED && voiceMode === 'realtime'. - HumanPage: additive overlay behind the flag; the classic push-to-talk pipeline is unchanged. - i18n: voice.mode.{start,stop,connecting,listening,speaking} in all 14 locales. - dep: @elevenlabs/react; tests: hook lifecycle (signed-url + startSession args, default voice, fetch-error, stop, SDK onError) — 5 green. Follow-up: drive the Rive mascot mouth from onAudioAlignment (visemesFromAlignment).
Contributor
📝 WalkthroughWalkthroughThe change adds feature-gated realtime voice mode. Rust exposes signed voice-agent sessions and a streaming harness. React adds mode selection, session controls, persistence, localization, and tests. ChangesRealtime voice mode
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant VoicePanel
participant HumanPage
participant RealtimeVoiceControls
participant useRealtimeVoiceSession
participant voiceAgentApi
VoicePanel->>HumanPage: persist realtime voice mode
HumanPage->>RealtimeVoiceControls: render when realtime mode is enabled
RealtimeVoiceControls->>useRealtimeVoiceSession: start or stop session
useRealtimeVoiceSession->>voiceAgentApi: fetch signed agent URL
voiceAgentApi-->>useRealtimeVoiceSession: return signed URL and agent ID
useRealtimeVoiceSession-->>RealtimeVoiceControls: update connection and speaking state
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
) The voice:harness relay arm added a #[cfg(feature="voice")] block to platform/socket/event_handlers.rs, which already carries tests — so the rust-feature-gate-smoke lane's gated-test guard (tinyhumansai#5022) flagged the changed file set. Add the file to the EXPECTED allowlist; its tests carry no ungated assertion, so the scoped cargo-test filter needs no change.
YellowSnnowmann
marked this pull request as ready for review
August 5, 2026 12:24
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
YellowSnnowmann has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae43ee3683
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
coderabbitai
Bot
added
agent
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-lite.yml:
- Line 503: Extend both scoped cargo test filters in the CI workflow to include
openhuman::platform::socket::event_handlers::, alongside the existing
event_handlers.rs entry, so its #[cfg(test)] module runs in this lane.
In `@app/src/features/human/HumanPage.tsx`:
- Around line 39-40: Update RealtimeVoiceControlsInner’s useRealtimeVoiceSession
lifecycle to clean up on unmount by ending the conversation session and
releasing the microphone. Guard the asynchronous start flow so an in-flight
start cannot open a session after the component has unmounted, using the hook’s
existing conversation and microphone controls.
In `@app/src/features/human/voice/useRealtimeVoiceSession.ts`:
- Around line 45-62: Update the useRealtimeVoiceSession start flow around start
to add grep-friendly lifecycle diagnostics for entry, signed-URL request
success, SDK session-start success, and failures, using only stable error
categories and excluding signedUrl and raw error text. Add a privacy-safe
diagnostic for the stop lifecycle as well, and retain the existing state/error
handling behavior.
In `@app/src/lib/i18n/ar.ts`:
- Line 2377: Update the Arabic translation for the `voice.mode.realtime` key to
use the repository’s established term for “realtime,” such as `الصوت في الوقت
الفعلي`, instead of `الصوت الفوري`.
In `@app/src/lib/i18n/fr.ts`:
- Line 2509: Update the French translation for voice.mode.speaking to use a
state label rather than the imperative “Parle”, matching the existing
overlay.companion.speaking wording “En train de parler…” or another equivalent
noun/state phrase.
In `@app/src/lib/i18n/pl.ts`:
- Around line 2472-2473: Update the Polish translations for voice.mode.listening
and voice.mode.speaking to use the existing first-person status forms “Słucham…”
and “Mówię…”, matching the established realtime status labels.
In `@app/src/store/mascotSlice.ts`:
- Line 349: Add voiceMode to the mascot persistence whitelist in mascotSlice so
the selected realtime or classic mode is persisted across restarts. Update the
existing whitelist only; preserve all other persistence behavior.
In `@src/openhuman/platform/socket/event_handlers.rs`:
- Around line 411-417: Add concurrency protection to handle_voice_harness_turn
using a static Semaphore and an in-flight set keyed by correlation_id. Reject or
skip duplicate correlation IDs and acquire a semaphore permit before creating
the full agent turn, ensuring the permit remains held through completion or
error and is dropped when the function exits.
In `@src/openhuman/voice/realtime_harness.rs`:
- Around line 89-114: Update run_agent_turn and its voice transcript-resume flow
to use a stable voice-session identifier, such as the correlation/session key,
instead of the shared "orchestrator" agent_definition_name when recovering
history. Ensure each turn resumes only its own voice conversation and cannot
inherit or expose another conversation’s transcript; alternatively, explicitly
pass prior messages to the newly created Agent.
In `@src/openhuman/voice/realtime.rs`:
- Around line 53-59: Update the realtime request flow in the voice agent path so
BackendOAuthClient::new is only used after validating that
effective_backend_api_url(&config.api_url) resolves to HTTPS, and reject any
non-HTTPS backend URL before the token-bearing authed_json call. Also ensure the
credentialed request does not follow HTTP downgrade redirects, while keeping the
existing 120-second timeout behavior intact.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a0523f02-6ad9-4c1c-8b1d-eb5bc7fb1ec8
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (33)
.github/workflows/ci-lite.ymlapp/package.jsonapp/src/components/settings/panels/VoicePanel.tsxapp/src/features/human/HumanPage.tsxapp/src/features/human/RealtimeVoiceControls.tsxapp/src/features/human/voice/useRealtimeVoiceSession.test.tsapp/src/features/human/voice/useRealtimeVoiceSession.tsapp/src/lib/i18n/ar.tsapp/src/lib/i18n/bn.tsapp/src/lib/i18n/de.tsapp/src/lib/i18n/en.tsapp/src/lib/i18n/es.tsapp/src/lib/i18n/fr.tsapp/src/lib/i18n/hi.tsapp/src/lib/i18n/id.tsapp/src/lib/i18n/it.tsapp/src/lib/i18n/ko.tsapp/src/lib/i18n/pl.tsapp/src/lib/i18n/pt.tsapp/src/lib/i18n/ru.tsapp/src/lib/i18n/zh-CN.tsapp/src/services/api/voiceAgentApi.test.tsapp/src/services/api/voiceAgentApi.tsapp/src/store/mascotSlice.tsapp/src/store/mascotSlice.voiceMode.test.tsapp/src/utils/config.tssrc/openhuman/platform/socket/event_handlers.rssrc/openhuman/voice/mod.rssrc/openhuman/voice/realtime.rssrc/openhuman/voice/realtime_harness.rssrc/openhuman/voice/schemas/handlers.rssrc/openhuman/voice/schemas/handlers/transcribe_tts.rssrc/openhuman/voice/schemas/registry.rs
…fixes (tinyhumansai#5399) Address review on the realtime voice-agent PR: Rust core: - Bind the session to the signed-in user: mint carries a userToken the renderer echoes back as the ElevenLabs userId; the backend relay verifies it instead of trusting a raw id. Parse userToken (defaults empty vs older backend). - Enforce HTTPS for the credentialed mint (CWE-319): reject a non-loopback http:// backend so the session token never travels in cleartext. - Preserve multi-turn context and stop cross-conversation history bleed: seed the orchestrator from the relayed OpenAI messages and scope the voice transcript namespace to "voice" so a fresh-per-turn agent can't resume the chat orchestrator's transcript by name. - Bound concurrent relay turns: a process-wide semaphore (3) plus per- correlationId in-flight dedup, so a retry/burst can't spawn unbounded 90s orchestrator sessions. Frontend: - Pass userId to startSession; map userToken through the RPC client. - Unmount teardown ends a live session (releases WS + mic); privacy-safe lifecycle diagnostics (categories only, never signedUrl or raw error text). - Persist mascot voiceMode so realtime survives a restart. i18n: clearer Arabic "realtime", French/Polish speaking/listening state labels. CI: run the event_handlers.rs test module in the gated-test lane.
There was a problem hiding this comment.
YellowSnnowmann has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/voice/realtime.rs`:
- Around line 47-53: Update the host extraction in the loopback validation flow
to parse the URL authority structurally rather than splitting on the first
colon, so bracketed IPv6 addresses such as [::1]:5005 are preserved. Ensure the
existing is_loopback check recognizes the documented IPv6 loopback backend and
add coverage for the http://[::1]:5005 case.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 028d5504-d926-4545-beac-22b38be24e9e
📒 Files selected for processing (11)
.github/workflows/ci-lite.ymlapp/src/features/human/voice/useRealtimeVoiceSession.test.tsapp/src/features/human/voice/useRealtimeVoiceSession.tsapp/src/lib/i18n/ar.tsapp/src/lib/i18n/fr.tsapp/src/lib/i18n/pl.tsapp/src/services/api/voiceAgentApi.test.tsapp/src/services/api/voiceAgentApi.tsapp/src/store/index.tssrc/openhuman/voice/realtime.rssrc/openhuman/voice/realtime_harness.rs
🚧 Files skipped from review as they are similar to previous changes (7)
- app/src/lib/i18n/fr.ts
- app/src/services/api/voiceAgentApi.test.ts
- app/src/lib/i18n/ar.ts
- .github/workflows/ci-lite.yml
- app/src/features/human/voice/useRealtimeVoiceSession.test.ts
- app/src/lib/i18n/pl.ts
- app/src/features/human/voice/useRealtimeVoiceSession.ts
…inyhumansai#5399) VoicePanel reads VOICE_MODE_FLAG_ENABLED (added with the realtime toggle), but the global vitest config mock in test/setup.ts never mirrored it, so any test rendering VoicePanel's tree (VoicePanel, Skills intelligence tabs) failed the changed-files coverage lane with a missing-mock-export error. Add it, defaulting false to match production.
There was a problem hiding this comment.
YellowSnnowmann has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
…inyhumansai#5399) - realtime.rs: parse the http:// authority structurally in ensure_secure_backend_url so a bracketed IPv6 loopback ([::1]:5005) is recognised. The prior first-`:` split turned it into "[" and wrongly rejected the documented loopback backend. Adds IPv6 loopback + non-loopback-IPv6 tests (CodeRabbit review). - Add tests covering the realtime voice UI to clear the 80% diff-coverage gate (was 72%): - RealtimeVoiceControls.test.tsx: full render/state/wiring (was 0%) - useRealtimeVoiceSession.test.ts: onDisconnect -> idle - HumanPage.realtimeMode.test.tsx: realtime overlay render gate - VoicePanel.realtimeMode.test.tsx: realtime-mode toggle dispatch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
YellowSnnowmann has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Fixes the Frontend Checks format:check failure — RealtimeVoiceControls.test.tsx and VoicePanel.realtimeMode.test.tsx were not Prettier-formatted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
YellowSnnowmann has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
M3gA-Mind
approved these changes
Aug 5, 2026
17 tasks
graycyrus
added a commit
to graycyrus/openhuman
that referenced
this pull request
Aug 6, 2026
Upstream tinyhumansai#5407 added ElevenLabs realtime voice agents to the Human page while this branch was replacing that page with the chat's voice stage. The feature is ported rather than dropped: the gate (build flag + persisted `realtime` mode) now lives on ChatMascotStage, which IS the voice surface, and replaces the turn-based mic when active. - mascotSlice / store persist: union of both sides — upstream's `voiceMode` alongside this branch's `chatMascotExpanded` / `speakReplies`. - HumanPage.realtimeMode.test.tsx targeted the deleted page; replaced by ChatMascotStage.realtimeMode.test.tsx, which pins the same contract plus the classic-mic fallback.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
voice.agent_signed_urlRPC (mints the session URL via the backend) + the inboundvoice:harnesssocket handler that runs the local orchestrator agent for a relayed realtime turn and streams the reply back.voiceModeflag/slice/toggle + a realtime@elevenlabs/reactWebSocket session (useRealtimeVoiceSession) + a flag-gatedRealtimeVoiceControlson the Human tab.Problem
#5399 wants realtime voice chat via ElevenLabs Agents, but the assistant's brain (tools/memory/MCP) is desktop-local. Handing the conversation to ElevenLabs' own LLM would drop all of that. This wires ElevenLabs as the realtime voice I/O while keeping the local agent as the LLM (via the backend Custom-LLM relay in the paired backend PR).
Solution
openhuman.voice_agent_signed_url(voice/realtime.rs) proxies the backend/voice-agent/get-signed-urlusing thereply_speechauth pattern; pure response parser is unit-tested.voice/realtime_harness.rs+ avoice:harnessarm inplatform/socket/event_handlers.rs): runsAgent::from_config_for_agent_with_profile("orchestrator")on the relayed turn (ExternalChannel origin, 90s timeout, spoken-output directive) and emitsvoice:harness:delta/done/errorback over the socket.extract_promptis pure + unit-tested.mascotSlice.voiceMode(persisted, rehydrate-guarded) +VOICE_MODE_FLAG_ENABLED;useRealtimeVoiceSession(WebSocket, mascot voice override, so per-audio-event alignment is available for lip-sync);RealtimeVoiceControls(self-containedConversationProvider) rendered on the Human tab only when the flag + realtime mode are on.Submission Checklist
extract_prompt; frontend: voiceMode slice/selector/rehydrate,voiceAgentApimapping,useRealtimeVoiceSessionlifecycle (signed-url + startSession args, default voice, fetch-error, stop, SDK onError).@elevenlabs/reactsession, the orchestrator agent run, socket emit) are contract-level and cannot be unit-executed without a live agent + socket (see Notes). Rancargo test voice::realtime voice::realtime_harnessandviteston the changed frontend files.Closes #NNN— epic Replace current voice chat with ElevenLabs Voice Agents #5399 spans this PR + the backend PR and still has follow-ups (e2e validation, mascot viseme wiring, old-path removal), so it is referenced, not closed. See Related.Impact
voicedomain (gated). Zero change with the flag off. New dep@elevenlabs/react. Pairs with backend PR tinyhumansai/backend#1215 (signed-URL mint + Custom-LLM relay).Related
onAudioAlignment(reusevisemesFromAlignment); multi-turn in-session context; then validate end-to-end and remove the classic Human-tab path.AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
Validation Run
pnpm --filter openhuman-app format:check— ran (prettier clean; pre-push auto-fixes folded in).pnpm typecheck— clean.viteston the voiceMode/api/session specs (17 green) +cargo test voice::realtime voice::realtime_harness(7 green).cargo fmtapplied; core compiles clean (cargo test --lib).src-tauriclippy clean.Validation Blocked
command:end-to-end realtime sessionerror:requires a provisioned ElevenLabs agent + Custom-LLM URL + live desktop socketimpact:the live session, the orchestrator run, and the socket round-trip are unit-tested to contract but not exercised end-to-end hereBehavior Changes
VITE_VOICE_MODE+ the toggle on, a realtime start/stop control appears.Parity Contract
VOICE_MODE_FLAG_ENABLED && voiceMode === 'realtime'; the voice feature gate covers the core additions.Duplicate / Superseded PR Handling
Summary by CodeRabbit
New Features
Localization