fix(parakeet): restart capture on mid-session input-device configuration change - #224
Merged
Merged
Conversation
…ion change Reported: mid-dictation, Parakeet stops producing words (~a minute in), then "comes back" later — with the gap permanently missing from the transcript. Investigation found NO regression in the decode path: FluidAudio has been pinned at 0.15.5 since the original integration (#163), the recent Parakeet changes (#218, #222) only touch callback snapshots and the final-transcript artifact strip, and a real-model harness run (parakeet-unified-320ms over 2.5 min of continuous fixture speech) sustained RTFx 5.5 with zero dropouts and no transcript gaps. The decoder does not go deaf on its own. What CAN produce exactly this symptom is a mid-session AVAudioEngineConfigurationChange: the input device disconnects, switches, or renegotiates its format, AVAudioEngine stops rendering, and — because the streaming engines never observe the notification — the session keeps showing "Listening…" while capturing NOTHING. Every word until the user gives up and re-dictates is silently lost. AudioRecorder has handled precisely this since the AirPods-disconnect hang ("the session keeps showing 'recording' while capturing nothing"); the streaming engines were never given the observer, and Parakeet only became the default (and long dictations routine) recently — which is why it reads as a new bug. Unlike the recorder's fail-only handling, Parakeet can recover seamlessly: the decode session and feed stream survive the mic teardown, so the engine rebuilds a routed AVAudioEngine + tap onto the SAME continuation and the dictation continues — everything already captured stays, and only the glitch itself is lost. If the rebuild fails (no input device left), the session fails LOUDLY through the error path instead of eating speech. The observer is selector-based because the block API takes a hard @sendable closure and every capture the restart needs is non-Sendable. The decision gates live in core (CaptureConfigChangePolicy) so swift test pins them: restart only when the session generation matches and no stop ran; stale/superseded notifications touch nothing. Route/format setup is extracted into makeRoutedEngine()/startCapture() shared by session start and restart. swift test: 1904 passing (4 new). ./build.sh green, no new warnings. Caveat (same as #222): the restart path only exercises against a real device change with a live mic — worth a live pass (unplug/replug the pinned mic mid-dictation) before the next release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 29, 2026
Merged
initcore0
added a commit
that referenced
this pull request
Jul 30, 2026
…Kit/AppleSpeech/SpeechAnalyzer (#226) PR #224 gave ParakeetStreamingEngine an AVAudioEngineConfigurationChange observer that rebuilds capture onto the same feed stream when the input device disconnects/switches/renegotiates mid-dictation. The other three streaming engines had the same hole: AVAudioEngine stops rendering on the change, the session keeps showing "Listening..." and every word after the glitch is silently lost. Port the Parakeet pattern (selector-based observer to stay @Sendable-warning-free, makeRoutedEngine/startCapture split, restart-then-loud-error, CaptureConfigChangePolicy gates) to: - AppleSpeechEngine: rebuild engine + tap appending to the SAME SFSpeechAudioBufferRecognitionRequest, so the recognizer sees one continuous audio stream. - SpeechAnalyzerStreamingEngine: rebuild engine + tap yielding into the SAME raw-buffer AsyncStream. The analyzer-format converter is now ADAPTIVE (rebuilt when the incoming buffer format changes) — a converter pinned to the start-time mic format would drop every buffer from the replacement device, leaving the transcript dead even after a successful capture restart. - WhisperKitStreamingEngine: the mic belongs to WhisperKit's AudioProcessor, so the rebuild lives on WhisperKitStreamHandle (restartCapture mirrors the internal setupEngine via public API and feeds processBuffer, appending to the same decode buffer the realtime loop polls); the engine arms the observer on the processor's engine object at first state diff and re-arms on the replacement after a restart. Restart failure (e.g. no input device left) surfaces through the session error path via CaptureConfigChangePolicy.restartFailedMessage instead of eating speech silently. Each engine's restart path still needs a live-mic device-change pass (unplug/switch mid-dictation); the policy gates are pinned by CaptureConfigChangePolicyTests. Co-authored-by: Claude <noreply@anthropic.com>
initcore0
added a commit
that referenced
this pull request
Jul 30, 2026
…227) The v1.0.12 regression (mic dead on every dictation, fine on v1.0.11): PR #224's AVAudioEngineConfigurationChange handler restarted capture on EVERY notification. The notification also fires for changes the handler's own teardown+rebuild causes — on this hardware a capture start itself posts one — so the first notification at session start entered a self-sustaining teardown→rebuild→notification loop (~4 restarts/second, 65 restarts observed in one 15s session). Capture never lived long enough to feed the decoder; every session timed out with no speech. Fix — the restart is no longer reflexive: - A notification schedules a settle-check (0.3s, storm-coalescing) instead of restarting inline. - The check rebuilds only when the engine genuinely STOPPED (AVAudioEngine.isRunning == false — the real device-loss case the observer exists for). A notification that leaves the engine running is a transient renegotiation and is ignored; restarting a running engine is exactly what fed the loop. - Rebuilds are budgeted (3 per capture session). A device that still stops capture after that fails the session loudly (gaveUpMessage) instead of looping silently. Gates live in core CaptureConfigChangePolicy (settleDelay, budget, the action matrix) and are pinned by CaptureConfigChangePolicyTests. Verified live against the released artifacts on the reporting machine: v1.0.12 reproduces the loop with both the pinned Shure MV7+ (35 restarts /10s) and a BlackHole fixture harness (65 restarts/15s, empty transcript); this build transcribes the fixture correctly across consecutive sessions with ZERO capture restarts. swift test green (1908). Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported symptom
Mid-dictation with Parakeet, the app stops producing words (~a minute in), then "comes back" later — with the words spoken during the gap permanently missing.
Investigation: no regression in the decode path
"You"strip — neither is in the mid-session flow.StreamingUnifiedAsrManagerwith the app's exact 320ms[70,2,2]config over 2.5 minutes of continuous fixture speech, fed in the app's 1024-frame buffers) sustained RTFx 5.5 with zero dropout windows — even transcript growth in every 10s bucket, all 60 repetitions of the fixture phrase present in the final. The decoder does not go deaf on its own.Root cause
A mid-session
AVAudioEngineConfigurationChange(input device disconnected/switched/format renegotiated — USB mic glitch, device pinning fallback, another app grabbing the device) stops AVAudioEngine's rendering. The streaming engines never observe the notification, so the session keeps showing "Listening…" while capturing nothing — every word until the user re-dictates is silently lost, and there is no error.AudioRecorderhas handled exactly this since the AirPods-disconnect hang (its comment: "the session keeps showing 'recording' while capturing nothing"); the streaming engines were never given the observer. Parakeet only recently became the default engine, and long dictations only recently became routine (same reason #222's WhisperKit 30s truncation surfaced late) — which is why this reads as a new bug.Fix
AVAudioEngine+ tap onto the same continuation and the dictation continues. Everything already captured stays; only the glitch itself is lost. If the rebuild fails (no input device left), the session fails loudly through the error path.CaptureConfigChangePolicy) soswift testpins them: restart only when the session generation matches and no stop ran; stale/superseded notifications touch nothing.makeRoutedEngine()/startCapture(), shared by session start and restart (start-path behavior and error messages unchanged).@Sendableclosure and every capture the restart needs is non-Sendable — this keeps the file at zero warnings.Testing
swift test: 1904 passing (4 new inCaptureConfigChangePolicyTests)../build.sh(WhisperKit + Parakeet): green, no new warnings vs baseline.Follow-ups (not in this PR)
WhisperKitStreamingEngine/AppleSpeechEngine/SpeechAnalyzerStreamingEnginehave the same missing observer.stopAppleSpeech's 2.0s stuck-session fallback can clobber a slow Parakeet finalize (feed-drain +finish()> 2s) with a stale partial and drop the genuine final — a second, independent path to "missing tail words".🤖 Generated with Claude Code