Skip to content

Fix: SautiSpeaker main-thread stall during synthesis + CTS leak (B5/B6) - #4

Open
BernardMasika wants to merge 1 commit into
mainfrom
fix/speaker-async-cts
Open

Fix: SautiSpeaker main-thread stall during synthesis + CTS leak (B5/B6)#4
BernardMasika wants to merge 1 commit into
mainfrom
fix/speaker-async-cts

Conversation

@BernardMasika

Copy link
Copy Markdown

Fix: SautiSpeaker main-thread stall during synthesis + CTS leak (B5/B6)

Symptom

Calling Speak()/SpeakAsync() on a SautiSpeaker freezes the whole application for the full synthesis time — hundreds of ms to several seconds per line, longer on Quest-class hardware. In VR that is a comfort violation: every spoken line drops frames and causes judder. Additionally, rapid re-entrant Speak calls (e.g. dialogue interruptions) leak one CancellationTokenSource per superseded call.

Root cause

Two related issues in SautiSpeaker, the designer-facing component:

  1. B5 — main-thread inference. KokoroTtsRunner.SynthesizeAsync deliberately runs ONNX inference synchronously on the calling thread and returns Task.FromResult — its own header says "For Unity main-thread callers who need true async, Task.Run-wrap externally." SautiSpeaker.SpeakAsync awaited it directly on the Unity main thread, so the documented wrap never happened anywhere and every synth stalled rendering.

  2. B6 — CTS leak. Re-entrant SpeakAsync cancels the prior linked CancellationTokenSource but never disposes it. A linked CTS registers a callback on the caller's external token; until disposed, that registration keeps the CTS alive for the external token's lifetime.

Fix

All changes are inside SautiSpeaker; KokoroTtsRunner is untouched and its sync-on-calling-thread contract still holds for code-only users.

  • Inference now hops to a thread-pool worker via Task.Run. The await continuation returns to the Unity main thread (standard SynchronizationContext behaviour), so OnPcmReady/OnAudioReady/OnSpeakError and AudioClip creation/playback still fire on the main thread — no observable API change for consumers.
  • A per-speaker SemaphoreSlim(1,1) gate serialises synthesis. This preserves the previous implicit guarantee (main-thread calls could never overlap) now that calls run on workers — the runner documents that InferenceSession is not concurrent-safe.
  • Profile swaps and OnDestroy no longer dispose the runner inline; disposal waits on the same gate (DisposeRunnerWhenIdleAsync), so the native ONNX session can never be disposed while a worker thread is inside Run() — that would be a native crash, not a catchable exception.
  • voiceId and Time.frameCount (main-thread-only API) are captured before the worker hop; a mid-synth Profile swap can no longer mix the old runner with the new profile's voice id.
  • Re-entrant SpeakAsync now disposes the superseded CTS after cancelling it.

Verification

  • Editor A/B on a real Kokoro model (Unity 6000.0.70f1, consumer project), synthesising the same sentence while counting EditorApplication.update ticks: sync baseline (old behaviour — runner invoked directly on the main thread): 11.7 s synth, 0 ticks (main thread frozen for the entire synthesis); new SpeakAsync path: 8.2 s synth, 83 ticks (main thread responsive throughout), OnPcmReady fired on the main thread, 154 200 PCM samples produced.
  • New EditMode regression test SpeakAsync_ReEntry_DisposesSupersededCts pins the B6 contract (superseded CTS throws ObjectDisposedException on .Token).
  • Full Sauti EditMode suite green in the consumer project: 73 passed / 0 failed / 0 skipped.

Backward compatibility

  • Public API unchanged. Events still fire on the main thread. Speak/SpeakAsync semantics (latest call wins, prior call cancelled) unchanged.
  • One behavioural difference: back-to-back Speak calls no longer interleave their start on the main thread — the superseded synth finishes its inference on the worker (its result is discarded) before the next one starts, because inference is serialised per speaker. Net effect for designers is identical: the last call's audio plays.
  • Related known ceiling (not addressed here, tracked as W4): synthesis is still whole-utterance, so first-audio latency equals full synth time; this PR removes the freeze, not the latency. A sentence-chunking queue on the speaker would be the follow-up.

B5: SpeakAsync awaited KokoroTtsRunner.SynthesizeAsync directly on the
Unity main thread; the runner deliberately runs ONNX inference
synchronously on the calling thread, so every Speak() froze rendering
for the full synth time (seconds on Quest — a VR comfort violation).
Inference now hops to a thread-pool worker via Task.Run; events and
AudioClip playback still fire on the main thread. A per-speaker
SemaphoreSlim(1,1) gate serialises synthesis (InferenceSession is not
concurrent-safe) and Profile-swap/OnDestroy disposal now waits on the
same gate so the native session can never be disposed mid-inference.
voiceId and Time.frameCount are captured before the hop. The runner
itself is untouched — its documented sync contract still holds.

B6: re-entrant SpeakAsync cancelled the superseded linked CTS but never
disposed it; the registration on the caller's external token kept it
alive for that token's lifetime. Re-entry now disposes it. Regression
test pins the contract via the error path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant