Fix: SautiSpeaker main-thread stall during synthesis + CTS leak (B5/B6) - #4
Open
BernardMasika wants to merge 1 commit into
Open
Fix: SautiSpeaker main-thread stall during synthesis + CTS leak (B5/B6)#4BernardMasika wants to merge 1 commit into
BernardMasika wants to merge 1 commit into
Conversation
B5: SpeakAsync awaited KokoroTtsRunner.SynthesizeAsync directly on the Unity main thread; the runner deliberately runs ONNX inference synchronously on the calling thread, so every Speak() froze rendering for the full synth time (seconds on Quest — a VR comfort violation). Inference now hops to a thread-pool worker via Task.Run; events and AudioClip playback still fire on the main thread. A per-speaker SemaphoreSlim(1,1) gate serialises synthesis (InferenceSession is not concurrent-safe) and Profile-swap/OnDestroy disposal now waits on the same gate so the native session can never be disposed mid-inference. voiceId and Time.frameCount are captured before the hop. The runner itself is untouched — its documented sync contract still holds. B6: re-entrant SpeakAsync cancelled the superseded linked CTS but never disposed it; the registration on the caller's external token kept it alive for that token's lifetime. Re-entry now disposes it. Regression test pins the contract via the error path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix:
SautiSpeakermain-thread stall during synthesis + CTS leak (B5/B6)Symptom
Calling
Speak()/SpeakAsync()on aSautiSpeakerfreezes the whole application for the full synthesis time — hundreds of ms to several seconds per line, longer on Quest-class hardware. In VR that is a comfort violation: every spoken line drops frames and causes judder. Additionally, rapid re-entrantSpeakcalls (e.g. dialogue interruptions) leak oneCancellationTokenSourceper superseded call.Root cause
Two related issues in
SautiSpeaker, the designer-facing component:B5 — main-thread inference.
KokoroTtsRunner.SynthesizeAsyncdeliberately runs ONNX inference synchronously on the calling thread and returnsTask.FromResult— its own header says "For Unity main-thread callers who need true async, Task.Run-wrap externally."SautiSpeaker.SpeakAsyncawaited it directly on the Unity main thread, so the documented wrap never happened anywhere and every synth stalled rendering.B6 — CTS leak. Re-entrant
SpeakAsynccancels the prior linkedCancellationTokenSourcebut never disposes it. A linked CTS registers a callback on the caller's external token; until disposed, that registration keeps the CTS alive for the external token's lifetime.Fix
All changes are inside
SautiSpeaker;KokoroTtsRunneris untouched and its sync-on-calling-thread contract still holds for code-only users.Task.Run. Theawaitcontinuation returns to the Unity main thread (standardSynchronizationContextbehaviour), soOnPcmReady/OnAudioReady/OnSpeakErrorandAudioClipcreation/playback still fire on the main thread — no observable API change for consumers.SemaphoreSlim(1,1)gate serialises synthesis. This preserves the previous implicit guarantee (main-thread calls could never overlap) now that calls run on workers — the runner documents thatInferenceSessionis not concurrent-safe.Profileswaps andOnDestroyno longer dispose the runner inline; disposal waits on the same gate (DisposeRunnerWhenIdleAsync), so the native ONNX session can never be disposed while a worker thread is insideRun()— that would be a native crash, not a catchable exception.voiceIdandTime.frameCount(main-thread-only API) are captured before the worker hop; a mid-synthProfileswap can no longer mix the old runner with the new profile's voice id.SpeakAsyncnow disposes the superseded CTS after cancelling it.Verification
EditorApplication.updateticks: sync baseline (old behaviour — runner invoked directly on the main thread): 11.7 s synth, 0 ticks (main thread frozen for the entire synthesis); newSpeakAsyncpath: 8.2 s synth, 83 ticks (main thread responsive throughout),OnPcmReadyfired on the main thread, 154 200 PCM samples produced.SpeakAsync_ReEntry_DisposesSupersededCtspins the B6 contract (superseded CTS throwsObjectDisposedExceptionon.Token).Backward compatibility
Speak/SpeakAsyncsemantics (latest call wins, prior call cancelled) unchanged.Speakcalls no longer interleave their start on the main thread — the superseded synth finishes its inference on the worker (its result is discarded) before the next one starts, because inference is serialised per speaker. Net effect for designers is identical: the last call's audio plays.