feat(maya): add Maya Research TTS plugin - #6899
Conversation
|
|
Streaming speech for ten Indian languages and Indian English, every voice speaking all eleven. One websocket carries a whole conversation, and each sentence is sent as the LLM writes it, so the agent starts speaking before the reply is finished. Interrupting it stops generation at the server rather than only muting playback, which keeps a reused connection from carrying a turn nobody is listening to into the next one. Voice, language and model are passed through as given rather than checked against a fixed list, so values Maya adds later work without a release here. Leaving the language unset lets Maya detect it per sentence, which is what code-switched Hinglish needs.
868553f to
adda8f5
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| async def _recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| nonlocal turn_closed | ||
| while True: | ||
| msg = await ws.receive(timeout=self._conn_options.timeout) |
There was a problem hiding this comment.
🔴 Spoken replies fail when the assistant's first sentence takes more than ten seconds to arrive
The reply's audio-listening loop starts counting its timeout immediately (ws.receive(timeout=...) at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:411) instead of waiting until the first piece of text has actually been sent, so a slow-to-start reply is aborted as a timeout.
Impact: If the language model takes longer than the connect timeout (10s by default) to produce its first sentence, the whole spoken reply errors out and is retried instead of being spoken.
Missing input-sent gate before the receive loop
_recv_task is created together with _input_task/_sentence_stream_task at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:448-452 and immediately blocks in await ws.receive(timeout=self._conn_options.timeout). Nothing has been sent to Maya yet, so the server has nothing to answer with; if no sentence is emitted by the tokenizer within conn_options.timeout, asyncio.TimeoutError is raised and converted into APITimeoutError at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:461-462.
Both sibling websocket plugins guard against exactly this: cartesia waits on input_sent_event before entering the loop (livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/tts.py:462-464) and rime does the same (livekit-plugins/livekit-plugins-rime/livekit/plugins/rime/tts.py:560-563), setting the event right after the first frame is sent.
Prompt for agents
In livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py, SynthesizeStream._recv_task enters `ws.receive(timeout=self._conn_options.timeout)` as soon as the three tasks are started, before any text frame has been sent to Maya. The receive timeout therefore measures the time until the LLM/tokenizer emits its first sentence rather than the server's response latency, and a slow first sentence aborts the turn with APITimeoutError. Follow the pattern used by the cartesia and rime plugins: introduce an asyncio.Event that _sentence_stream_task sets right after it sends its first frame, have _recv_task await that event before the receive loop, and make sure the event is also set in the finally block so the receive task can never be left waiting forever.
Was this helpful? React with 👍 or 👎 to provide feedback.
| async for ev in sent_tokenizer_stream: | ||
| self._mark_started() | ||
| await ws.send_str(_text_frame(context_id, ev.token, cont=True)) | ||
|
|
||
| await ws.send_str(_text_frame(context_id, "", cont=False)) |
There was a problem hiding this comment.
🟡 A turn with no text stalls for the full timeout and then reports a failure
When no sentence is produced for a turn, an empty closing frame is still sent (_text_frame(context_id, "", cont=False) at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:397) for a turn the server never started, so nothing ever comes back and the turn hangs until it times out.
Impact: An empty reply blocks the speech pipeline for the timeout period and then surfaces an error, repeated once per retry attempt.
No early-exit for zero sentences
Maya only opens a context when it receives text; the closing empty continue: false frame for a context that was never opened produces no end (the test double models exactly this at tests/test_plugin_maya_tts.py:131-147, where open_turns is only populated when text is non-empty). _recv_task therefore keeps waiting and eventually raises asyncio.TimeoutError, converted to APITimeoutError at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:461-462, and the base class retries up to max_retry (3) times.
The rime plugin handles this explicitly: if no sentence was emitted it sets an empty_input flag, calls output_emitter.end_input() and returns without sending a flush, and the receive task exits immediately (livekit-plugins/livekit-plugins-rime/livekit/plugins/rime/tts.py:546-563).
Prompt for agents
In livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py, SynthesizeStream._sentence_stream_task always sends the empty `continue: false` closing frame, even when the tokenizer produced zero sentences (e.g. the stream was ended without any text, or only whitespace was pushed). Maya only opens a context once it receives text, so the closing frame for an unopened context yields no `end` message and _recv_task blocks until the receive timeout, failing the turn and retrying. Mirror the rime plugin's approach: count the sentences actually sent; if none were sent, skip the closing frame, signal the receive task to finish (end the emitter input and return) so the run completes cleanly instead of timing out.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # closes the turn and makes Maya emit its single `end`. | ||
| async for ev in sent_tokenizer_stream: | ||
| self._mark_started() | ||
| await ws.send_str(_text_frame(context_id, ev.token, cont=True)) |
There was a problem hiding this comment.
🟡 Consecutive sentences are sent glued together with no separator
Each sentence is sent with its surrounding whitespace stripped (_text_frame(context_id, ev.token, cont=True) at livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/tts.py:395), so the server sees the end of one sentence run straight into the start of the next.
Impact: Multi-sentence replies can be pronounced with words merged across sentence boundaries.
Tokenizer strips whitespace; siblings re-add a space
The default tokenize.blingfire.SentenceTokenizer emits stripped tokens (retain_format=False strips each span, see livekit-agents/livekit/agents/tokenize/blingfire.py:28-31), so consecutive frames concatenate as ...forest.He walked... on the server side. Cartesia (livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/tts.py:438), rime (livekit-plugins/livekit-plugins-rime/livekit/plugins/rime/tts.py:549), murf and asyncai all append a trailing space to ev.token for this reason.
| await ws.send_str(_text_frame(context_id, ev.token, cont=True)) | |
| await ws.send_str(_text_frame(context_id, ev.token + " ", cont=True)) |
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
livekit-plugins-maya(livekit-plugins/livekit-plugins-maya/), a WebSocket TTS provider covering ten Indian languages plus Indian English, with every voice speaking all eleven. Connections come fromutils.ConnectionPool, so one socket carries a whole conversation and the handshake is paid once rather than per utterance._connect_wssends{"type": "start", "v2": true, ...}and waits for themetadatareply before returning the socket — turns sent before that are rejected rather than served in the older frame shape, so the pool must never hand out a socket that hasn't completed it. A rejected handshake raisesAPIErrorinstead of yielding one that would fail every turn.SynthesizeStreammints acontext_id, sends each tokenized sentence under it withcontinue: true, and closes with an emptycontinue: falseframe, yielding exactly oneend. Sentences are not gated on the previous one's audio, so a multi-sentence reply streams continuously.cancelbefore the socket returns to the pool. Without it the server keeps generating and the next borrower of that connection receives audio from a dead turn.errorframe, so values it adds later work without a release here.ChunkedStreamruns over the same websocket rather than Maya's HTTP endpoint, so the plugin has one transport and one auth path.mayaoptional dependency, and an entry in the cross-providerSYNTHESIZE_TTSsuite.version.pytracks thelivekit-agentsversion (1.6.10), matching the sibling plugins so thelivekit-plugins-maya>=1.6.10pin resolves.Example usage
The API key is read from
MAYA_API_KEY.Test plan
uv run pytest tests/test_plugin_maya_tts.py— 25 tests against a local server speaking the v2 protocol, covering the handshake contents, the turn model, connection reuse across turns, cancel-on-interrupt, no cancel for a turn that finished, a rejected handshake, and error framesuv run ruff check && uv run ruff format --checkuv run mypy livekit-plugins/livekit-plugins-maya/livekit/plugins/maya/Happy to adjust anything to match house style.