fix(engines): restart capture on mid-session input-device config change (WhisperKit, Apple Speech, SpeechAnalyzer) - #226
Open
initcore0 wants to merge 1 commit into
Open
Conversation
…Kit/AppleSpeech/SpeechAnalyzer PR #224 gave ParakeetStreamingEngine an AVAudioEngineConfigurationChange observer that rebuilds capture onto the same feed stream when the input device disconnects/switches/renegotiates mid-dictation. The other three streaming engines had the same hole: AVAudioEngine stops rendering on the change, the session keeps showing "Listening..." and every word after the glitch is silently lost. Port the Parakeet pattern (selector-based observer to stay @Sendable-warning-free, makeRoutedEngine/startCapture split, restart-then-loud-error, CaptureConfigChangePolicy gates) to: - AppleSpeechEngine: rebuild engine + tap appending to the SAME SFSpeechAudioBufferRecognitionRequest, so the recognizer sees one continuous audio stream. - SpeechAnalyzerStreamingEngine: rebuild engine + tap yielding into the SAME raw-buffer AsyncStream. The analyzer-format converter is now ADAPTIVE (rebuilt when the incoming buffer format changes) — a converter pinned to the start-time mic format would drop every buffer from the replacement device, leaving the transcript dead even after a successful capture restart. - WhisperKitStreamingEngine: the mic belongs to WhisperKit's AudioProcessor, so the rebuild lives on WhisperKitStreamHandle (restartCapture mirrors the internal setupEngine via public API and feeds processBuffer, appending to the same decode buffer the realtime loop polls); the engine arms the observer on the processor's engine object at first state diff and re-arms on the replacement after a restart. Restart failure (e.g. no input device left) surfaces through the session error path via CaptureConfigChangePolicy.restartFailedMessage instead of eating speech silently. Each engine's restart path still needs a live-mic device-change pass (unplug/switch mid-dictation); the policy gates are pinned by CaptureConfigChangePolicyTests. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #224 fixed the mid-dictation word-loss dropout for Parakeet by observing
AVAudioEngineConfigurationChangeand rebuilding capture onto the same feed stream. The same hole existed in the three other streaming engines — each owns a capture tap and none observed the notification, so a mid-session input-device disconnect/switch/format renegotiation left the session showing "Listening…" while capturing nothing.This ports the Parakeet pattern (selector-based observer,
makeRoutedEngine/startCapturesplit, restart-then-loud-error, sharedCaptureConfigChangePolicygates) to all three:SFSpeechAudioBufferRecognitionRequest; the recognizer sees one continuous audio stream and keeps everything already heard.AsyncStream. The analyzer-format converter is now adaptive (rebuilt when the incoming buffer format changes): a converter pinned to the start-time mic format would silently drop every buffer from the replacement device — capture restarted, transcript still dead.AudioProcessor, so the rebuild lives onWhisperKitStreamHandle.restartCapture(mirrors the internalsetupEnginevia public API — native-format tap → resample to 16 kHz mono →processBuffer— appending to the same decode buffer the realtime loop polls). The engine arms the observer on the processor's engine object at first state diff and re-arms on the replacement engine after a restart.Restart failure (e.g. no input device left after the change) surfaces loudly through the session error path via
CaptureConfigChangePolicy.restartFailedMessage.Testing
swift test— 1904 tests green; the shared gates stay pinned byCaptureConfigChangePolicyTests(header updated to cover all four engines)../build.sh— clean; no new warnings in the touched files (the adaptive converter is@unchecked Sendablewith the sequential-consumption invariant documented).swift testbecause the recovery hangs off a realAVAudioEnginenotification.🤖 Generated with Claude Code