Skip to content

fix(engines): restart capture on mid-session input-device config change (WhisperKit, Apple Speech, SpeechAnalyzer) - #226

Open
initcore0 wants to merge 1 commit into
mainfrom
claude/practical-volhard-3d5eba
Open

fix(engines): restart capture on mid-session input-device config change (WhisperKit, Apple Speech, SpeechAnalyzer)#226
initcore0 wants to merge 1 commit into
mainfrom
claude/practical-volhard-3d5eba

Conversation

@initcore0

Copy link
Copy Markdown
Owner

Summary

PR #224 fixed the mid-dictation word-loss dropout for Parakeet by observing AVAudioEngineConfigurationChange and rebuilding capture onto the same feed stream. The same hole existed in the three other streaming engines — each owns a capture tap and none observed the notification, so a mid-session input-device disconnect/switch/format renegotiation left the session showing "Listening…" while capturing nothing.

This ports the Parakeet pattern (selector-based observer, makeRoutedEngine/startCapture split, restart-then-loud-error, shared CaptureConfigChangePolicy gates) to all three:

  • AppleSpeechEngine — rebuilds the engine + tap appending to the same SFSpeechAudioBufferRecognitionRequest; the recognizer sees one continuous audio stream and keeps everything already heard.
  • SpeechAnalyzerStreamingEngine — rebuilds the engine + tap yielding into the same raw-buffer AsyncStream. The analyzer-format converter is now adaptive (rebuilt when the incoming buffer format changes): a converter pinned to the start-time mic format would silently drop every buffer from the replacement device — capture restarted, transcript still dead.
  • WhisperKitStreamingEngine — the mic belongs to WhisperKit's AudioProcessor, so the rebuild lives on WhisperKitStreamHandle.restartCapture (mirrors the internal setupEngine via public API — native-format tap → resample to 16 kHz mono → processBuffer — appending to the same decode buffer the realtime loop polls). The engine arms the observer on the processor's engine object at first state diff and re-arms on the replacement engine after a restart.

Restart failure (e.g. no input device left after the change) surfaces loudly through the session error path via CaptureConfigChangePolicy.restartFailedMessage.

Testing

  • swift test — 1904 tests green; the shared gates stay pinned by CaptureConfigChangePolicyTests (header updated to cover all four engines).
  • ./build.sh — clean; no new warnings in the touched files (the adaptive converter is @unchecked Sendable with the sequential-consumption invariant documented).
  • ⚠️ Still needed: a live-mic device-change pass per engine (unplug/switch the input mid-dictation on WhisperKit, Apple Speech, and SpeechAnalyzer streaming) — same manual pass Parakeet's restart got. The wiring can't be driven from fixture audio in swift test because the recovery hangs off a real AVAudioEngine notification.

🤖 Generated with Claude Code

…Kit/AppleSpeech/SpeechAnalyzer

PR #224 gave ParakeetStreamingEngine an AVAudioEngineConfigurationChange
observer that rebuilds capture onto the same feed stream when the input
device disconnects/switches/renegotiates mid-dictation. The other three
streaming engines had the same hole: AVAudioEngine stops rendering on the
change, the session keeps showing "Listening..." and every word after the
glitch is silently lost.

Port the Parakeet pattern (selector-based observer to stay
@Sendable-warning-free, makeRoutedEngine/startCapture split,
restart-then-loud-error, CaptureConfigChangePolicy gates) to:

- AppleSpeechEngine: rebuild engine + tap appending to the SAME
  SFSpeechAudioBufferRecognitionRequest, so the recognizer sees one
  continuous audio stream.
- SpeechAnalyzerStreamingEngine: rebuild engine + tap yielding into the
  SAME raw-buffer AsyncStream. The analyzer-format converter is now
  ADAPTIVE (rebuilt when the incoming buffer format changes) — a
  converter pinned to the start-time mic format would drop every buffer
  from the replacement device, leaving the transcript dead even after a
  successful capture restart.
- WhisperKitStreamingEngine: the mic belongs to WhisperKit's
  AudioProcessor, so the rebuild lives on WhisperKitStreamHandle
  (restartCapture mirrors the internal setupEngine via public API and
  feeds processBuffer, appending to the same decode buffer the realtime
  loop polls); the engine arms the observer on the processor's engine
  object at first state diff and re-arms on the replacement after a
  restart.

Restart failure (e.g. no input device left) surfaces through the session
error path via CaptureConfigChangePolicy.restartFailedMessage instead of
eating speech silently.

Each engine's restart path still needs a live-mic device-change pass
(unplug/switch mid-dictation); the policy gates are pinned by
CaptureConfigChangePolicyTests.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant