Task Groups to Keep Connection When Providers Don't Change

[tech-Mitra-coach]

Feature Type

I cannot use LiveKit without it

Feature Description

Currently Agent/AgentTask initialize a new connection between each tasks. This is very problematic when it comes to things like Diarization when the provider has already recognized and matched the speech of the different users in the stream. So when new tasks initialize the connection then that data is essentially gone and thus becomes unusable.

This was acknowledge here on Issue 4159

The ask is to keep the same connection alive if the providers are not changing between different AgentTask

Workarounds / Alternatives

No response

Additional Context

No response

[longcw]

I think it makes sense if both agents use the providers defined in the AgentSession, so we know they are not changed.

[longcw]

another blocker is that the Agent.stt_node can be customized by the user, even the stt is not specified in the agent. probably we need an explicit flag in Agent to indicate it should use the previous connection.