feat(voice): stream tool call status events#1929
feat(voice): stream tool call status events#1929rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 6321578 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🚩 Realtime path does not emit tool_reply_updated events
The pipeline path in _pipelineReplyTaskImpl emits tool_reply_updated events (lines 3096–3122 in agent_activity.ts), but the realtime path in _realtimeGenerationTaskImpl (around line 3617) only emits FunctionToolsExecuted without corresponding tool_reply_updated events. This means consumers listening for ToolExecutionUpdated with tool_reply_updated will not receive events for realtime model tool replies. This may be intentional since the realtime model handles tool replies differently (auto-generation), but it creates an asymmetry between the two paths that consumers should be aware of.
(Refers to lines 3616-3628)
Was this helpful? React with 👍 or 👎 to provide feedback.
| toolResponseTask.result.finally(() => { | ||
| const status = speechHandle.interrupted | ||
| ? 'interrupted' | ||
| : speechHandle.chatItems.length > initialChatItemCount | ||
| ? 'completed' | ||
| : 'skipped'; | ||
| this.agentSession.emit( | ||
| AgentSessionEventTypes.ToolExecutionUpdated, | ||
| createToolExecutionUpdatedEvent({ | ||
| type: 'tool_reply_updated', | ||
| updateIds, | ||
| status, | ||
| speechId: speechHandle.id, | ||
| }), | ||
| ); | ||
| this.onPipelineReplyDone(); | ||
| }); |
There was a problem hiding this comment.
🔴 Agent cleanup step skipped when an event listener throws, leaving the agent stuck
A critical post-reply cleanup call is placed after an event emission (this.agentSession.emit(...) at agents/src/voice/agent_activity.ts:3112) inside a .finally() block, so if any user-registered listener throws, the cleanup never runs.
Impact: The agent can get stuck in a non-listening state, preventing it from responding to further user input.
Mechanism: emit() can prevent onPipelineReplyDone() from executing
Previously, the .finally() callback was simply () => this.onPipelineReplyDone() (see the unchanged pattern at agents/src/voice/agent_activity.ts:1062 and agents/src/voice/agent_activity.ts:2054). The new code at lines 3106–3122 calls this.agentSession.emit() first, then this.onPipelineReplyDone(). Since AgentSession extends Node.js EventEmitter, emit() propagates listener exceptions synchronously. If any listener for ToolExecutionUpdated throws, onPipelineReplyDone() at line 3121 is never reached.
onPipelineReplyDone() (defined at line 2109) transitions the agent state back to 'listening' and notifies audioRecognition.onEndOfAgentSpeech(). Without it, the agent remains in 'thinking' or 'speaking' and won't process new user turns.
| toolResponseTask.result.finally(() => { | |
| const status = speechHandle.interrupted | |
| ? 'interrupted' | |
| : speechHandle.chatItems.length > initialChatItemCount | |
| ? 'completed' | |
| : 'skipped'; | |
| this.agentSession.emit( | |
| AgentSessionEventTypes.ToolExecutionUpdated, | |
| createToolExecutionUpdatedEvent({ | |
| type: 'tool_reply_updated', | |
| updateIds, | |
| status, | |
| speechId: speechHandle.id, | |
| }), | |
| ); | |
| this.onPipelineReplyDone(); | |
| }); | |
| toolResponseTask.result.finally(() => { | |
| try { | |
| const status = speechHandle.interrupted | |
| ? 'interrupted' | |
| : speechHandle.chatItems.length > initialChatItemCount | |
| ? 'completed' | |
| : 'skipped'; | |
| this.agentSession.emit( | |
| AgentSessionEventTypes.ToolExecutionUpdated, | |
| createToolExecutionUpdatedEvent({ | |
| type: 'tool_reply_updated', | |
| updateIds, | |
| status, | |
| speechId: speechHandle.id, | |
| }), | |
| ); | |
| } finally { | |
| this.onPipelineReplyDone(); | |
| } | |
| }); |
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
tool_execution_updatedvoice events for tool call start/update/end and tool reply lifecycle statusRunContext.update(), and generated tool repliesTesting
pnpm test -- agents/src/voice/generation_tools.test.ts agents/src/voice/remote_session.test.tspnpm build:agentspnpm --filter @livekit/agents lint(passes with existing warnings)Notes
pnpm api:check --filter=@livekit/agentscurrently fails before API comparison due to API Extractor not supporting the repo's generatedexport * as ___syntax inagents/dist/index.d.ts.Ported from livekit/agents#6100
Original PR description
Adds a
tool_status_updatedsession event that streams the lifecycle of each function tool call, so frontends and remote sessions can render tool progress (especially for async/long-running tools that usectx.update()).The event (
ToolStatusUpdatedEvent) carries one of three updates:ToolCallStarted— a tool call was dispatched.ToolCallUpdated— a progress update (running) fromctx.update(), or the call's single terminal entry (done/error/cancelled).ToolReplyUpdated— the deferred reply that voices buffered updates (scheduledthencompleted/interrupted/skipped).SessionHostforwards the event to remote sessions over the newAgentSessionEvent.ToolStatusUpdatedproto (livekit/protocol#1625), and the new types are exported fromlivekit.agents. Theasync_tool_agentexample shows publishing the stream to a frontend.Depends on livekit/protocol#1625 and livekit/python-sdks#716 for the generated bindings.