Agent stops responding if interrupted during a tool call with self.session.say

[aumeshm]

Bug Description

If the agent has a self.session.say inside of a tool call, and you interrupt right before it. The agent will be stuck and unable to respond. All events are blocked and only until the participant leaves will the agent continue, but it will run into an error because the AgentSession has been terminated.

I found the culprit PR: 6439f60

The agent keeps waiting for user silence but the event is never emitted. I don't know if it's because it's detected differently within the context of a tool call.

Expected Behavior

If the tool call has a session.say, the agent should be able to gracefully handle interruptions in a tool call without manually checking https://docs.livekit.io/agents/logic/tools/#interruptions.

Reproduction Steps

1. 
2.
3.
...
- Sample code snippet, or a GitHub Gist link -

Operating System

macOS

Models Used

No response

Package Versions

livekit-agents[cartesia,deepgram,openai,rime,silero,turn-detector]~=1.3 (1.3.10 at the time)

Session/Room/Call IDs

No response

Proposed Solution

Investigate why waiting for user silence locks out the agent from responding and why this happens specifically when trying to do self.session.say inside a tool call.

Additional Context

No response

Screenshots and Recordings

No response

[longcw]

does this happen randomly or can be reproducible consistently? can you share an example of the function tool and your agent session definition?

[arxliu-commure]

@longcw @aumeshm Hey, I've been looking into a similar issue for a few weeks now and last night found a workaround.

Issue is very hard to reproduce because it appears to be a very slim bug on the order of tenths of a second. But it is reproducible if you know the cause, which I think I've found.

It has nothing to do w/ session.say directly, although it's easier to trigger w/ session.say because it generates a 2nd speech handle in the function tool which causes the lock earlier.

As far as I can tell, if you have any function tool where you invoke run_ctx.disallow_interruptions(), it can crash. I just had a crash on prod this morning w/o my hotfix, but every function tool w/ my hotfix appears to be fine.

The issue is if a user begins speaking in the gap between the LLM node returning a function tool and the agent actually invoking the function, the user_state gets set to 'listening.' This gap is less than 1 second, but is wide enough that a user who talks indecisively or repeatedly interrupts the agent can hit w/ pretty high consistency. It generally happens with us with angry callers who are trying to get the agent to shut up or take a quick action during a step where a function tool is about to be invoked.

Once this happens, the function begins execution (because we have already processed the function tool response from the LLM) and enters the function_tool, which is protected from interruption by run_ctx.disallow_interruptions(). Therefore, we cannot interrupt the function_tool. The user's state is locked to listening and never changes. If you attempt to run another SpeechHandle during this time, it will never play out because the user is still 'speaking.' Everything locks because the user is still 'speaking' and is never reset to 'listening,' even though the user has stopped talking.

Calling SpeechHandle.interrupt does nothing. Only run_ctx.session.interrupt(force=True) manages to clear out the session context. But this comes at the cost of nuking your function_tool from the chat context, which means the next fragment of user speech will be interpreted on it's own. You can do run_ctx.clear_user_turn() to dump the fragment but then your agent is stuck sitting in silent, although at the very least it's not locked.

Here is my agent session:

session = AgentSession[SessionInfo](userdata=SessionInfo(), use_tts_aligned_transcript=True, resume_false_interruption=True)

await session.start(
            room=ctx.room,
            agent=initial_agent,
            room_options=room_io.RoomOptions(
                audio_input=room_io.AudioInputOptions(
                    noise_cancellation=noise_cancellation.BVCTelephony(),
                ),
            )
            # room_input_options=RoomInputOptions(
                # LiveKit Cloud enhanced noise cancellation
                # - If self-hosting, omit this parameter
                # - For telephony applications, use `BVCTelephony` for best results
            #     noise_cancellation=noise_cancellation.BVCTelephony(), 
            # ),
        )

I use the following models for a subagent

super().__init__(
            instructions=instructions,
            stt=deepgram.STT(keyterms=provider_id_keywords),
            llm=google.LLM(
                model="gemini-2.5-flash",
                max_output_tokens=4096,
                thinking_config={
                    "thinking_budget": 512
                }
            ),
            tts=cartesia.TTS(
                model="sonic-3",
                voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
            ),
            vad=ctx.proc.userdata["vad"],
            turn_detection=MultilingualModel(),
            chat_ctx=chat_ctx
        )

And here is a function tool with the fix that hasn't crashed yet today:

@function_tool
    async def find_provider(self, run_ctx: RunContext, provider_fname: str = "", provider_lname: str = "", provider_specialty: str = "") -> str:
        user_state = run_ctx.session.user_state
        logging.info(f"agent_state on entry: {run_ctx.session.agent_state}, user_state: {run_ctx.session.user_state}")
        if user_state == 'speaking':
            logging.warning("Aborting function tool because user interrupted prior to critical work")
            return "You were interrupted by the user before you could execute your function tool. " \
            "You must interpret the next turn from the user in the context of the prior user turn. " \
            "Combine the two turns and then decide on the next course of action. " \
            "Because no critical work has occured, you can re-invoke the previous function tool if " \
            "you are in doubt of the next step"
        run_ctx.disallow_interruptions()

        async def _speak_status_update(delay: float = 0.5):
            waiting_handle = None
            try: 
                await asyncio.sleep(delay)
                waiting_handle = run_ctx.session.say("One moment please.", allow_interruptions=False)
                logging.debug(f"one_moment_please_handle: {waiting_handle.id}, interruption_allowed: {waiting_handle.allow_interruptions}")
                logging.info("Waiting for 'One moment please' playout")
                await waiting_handle
            except asyncio.CancelledError:
                logging.info("'One moment please' interrupted by parent function tool.")
                if waiting_handle and not waiting_handle.interrupted:
                    # Interrupting the handle isn't sufficiently strong, for some reason.
                    # Uou have to abort the session. Leaving this in as a reminder for devs that
                    # the below line of code does not work
                    # waiting_handle.interrupt(force=True)
                    logging.debug(f"Cleaning up speech handle {waiting_handle.id}")
                    logging.debug(f"run_ctx.speech_handle: {run_ctx.speech_handle.id} {run_ctx.speech_handle.allow_interruptions} {run_ctx.speech_handle.interrupted} run_ctx.session.current_speech: {run_ctx.session.current_speech.id} {run_ctx.session.current_speech.allow_interruptions} {run_ctx.session.current_speech.interrupted}")
                    if (run_ctx.session.user_state == 'speaking'):
                        # This is a hard abort. The entire stack will abort and the user will be left in a hanging state.
                        # We are relying on the user inactivity timer to restart the session. This will cause user
                        # frustration but currently we don't have another solution and this guarantees the call 
                        # will resolve somehow.
                        logging.debug(f"activity speech handle: {run_ctx.session._activity.current_speech.id}")
                        logging.error(f"agent state: {run_ctx.session.agent_state}, user state: {run_ctx.session.user_state}")
                        run_ctx.session.emit('custom_event', 'find_provider')
                        run_ctx.session.interrupt(force=True)
                raise 

    # Do function logic here

And any other function tool that doesn't have this header has a chance of locking up.

If you want to reproduce, you have to snipe that thin window between the turn ending, which triggers the LLM to trigger the function tool response, but then start talking before it actually invokes the function. I have never been able to reproduce it locally on the LiveKit Cloud Playground, I have to be on telephony. We use Twilio.

I had best success reproducing the issue by slowly spelling my response to the LLM, which is obviously ridiculous but it causes more turn breakdowns which increases the odds of you hitting the gap.

[aumeshm]

@arxliu-commure For us, the issue was once we were in the function tool before the session.say. Here's some psuedocode that shows the example:

@function_tool
async def add_call_reason_and_tranfer(self, call_reason: CallReason):

1. business logic to determine call reason # If you interrupt during this phase, that is where you get issues
2. self.session.say(..., allow_interruptions=False) # If you interrupt while the agent is already speaking, there are no issues.
3. transfer to agent or to live team

Was able to replicate by replacing business logic with async delay and speaking during that delay

[chenghao-mou]

@aumeshm thanks for reporting the issue!

I can confirm the behavior and have identified the likely cause: audio frames were not passed to VAD once the uninterruptible speech is created. Originally, this was designed to skip STT during agent speech, but VAD was most likely skipped unintentionally.

[aumeshm]

@chenghao-mou thanks, I looked at the PR and it makes sense.

Are you able to repro the error if allow_interruptions=True? I believe you shouldn't, since you were always pushing to VAD in that scenario; however, I remember running into that once or twice in prod.

[chenghao-mou]

@chenghao-mou thanks, I looked at the PR and it makes sense.

Are you able to repro the error if allow_interruptions=True? I believe you shouldn't, since you were always pushing to VAD in that scenario; however, I remember running into that once or twice in prod.

There will be rare cases where the VAD model itself fails, and we have seen it before.