Skip to content

fix(parakeet): restart capture on mid-session input-device configuration change - #224

Merged
initcore0 merged 1 commit into
mainfrom
claude/parakeet-dictation-dropout-83e3c9
Jul 29, 2026
Merged

fix(parakeet): restart capture on mid-session input-device configuration change#224
initcore0 merged 1 commit into
mainfrom
claude/parakeet-dictation-dropout-83e3c9

Conversation

@initcore0

Copy link
Copy Markdown
Owner

Reported symptom

Mid-dictation with Parakeet, the app stops producing words (~a minute in), then "comes back" later — with the words spoken during the gap permanently missing.

Investigation: no regression in the decode path

  • FluidAudio is pinned at 0.15.5 since the original integration (feat(engine): Parakeet realtime engine — full integration (MAK-46) #163) — the decoder hasn't changed.
  • The recent Parakeet-adjacent changes (#218, #222) only touch stop-time callback snapshots and the final-transcript "You" strip — neither is in the mid-session flow.
  • A real-model harness (drives StreamingUnifiedAsrManager with the app's exact 320ms [70,2,2] config over 2.5 minutes of continuous fixture speech, fed in the app's 1024-frame buffers) sustained RTFx 5.5 with zero dropout windows — even transcript growth in every 10s bucket, all 60 repetitions of the fixture phrase present in the final. The decoder does not go deaf on its own.

Root cause

A mid-session AVAudioEngineConfigurationChange (input device disconnected/switched/format renegotiated — USB mic glitch, device pinning fallback, another app grabbing the device) stops AVAudioEngine's rendering. The streaming engines never observe the notification, so the session keeps showing "Listening…" while capturing nothing — every word until the user re-dictates is silently lost, and there is no error.

AudioRecorder has handled exactly this since the AirPods-disconnect hang (its comment: "the session keeps showing 'recording' while capturing nothing"); the streaming engines were never given the observer. Parakeet only recently became the default engine, and long dictations only recently became routine (same reason #222's WhisperKit 30s truncation surfaced late) — which is why this reads as a new bug.

Fix

  • Seamless restart, not fail-only: the decode session and feed stream survive the mic teardown, so on the notification the engine rebuilds a routed AVAudioEngine + tap onto the same continuation and the dictation continues. Everything already captured stays; only the glitch itself is lost. If the rebuild fails (no input device left), the session fails loudly through the error path.
  • Decision gates extracted to core (CaptureConfigChangePolicy) so swift test pins them: restart only when the session generation matches and no stop ran; stale/superseded notifications touch nothing.
  • Route/format setup extracted into makeRoutedEngine() / startCapture(), shared by session start and restart (start-path behavior and error messages unchanged).
  • Observer is selector-based on purpose: the block API takes a hard @Sendable closure and every capture the restart needs is non-Sendable — this keeps the file at zero warnings.

Testing

  • swift test: 1904 passing (4 new in CaptureConfigChangePolicyTests).
  • ./build.sh (WhisperKit + Parakeet): green, no new warnings vs baseline.
  • Real-model streaming harness: 2.5 min continuous audio, no dropouts (evidence for "no decode regression" above).

⚠️ Same caveat as #222: the restart path only exercises against a real device change — worth a live pass (unplug/replug the pinned mic mid-dictation, or trigger a format change) before the next release.

Follow-ups (not in this PR)

  • WhisperKitStreamingEngine / AppleSpeechEngine / SpeechAnalyzerStreamingEngine have the same missing observer.
  • stopAppleSpeech's 2.0s stuck-session fallback can clobber a slow Parakeet finalize (feed-drain + finish() > 2s) with a stale partial and drop the genuine final — a second, independent path to "missing tail words".

🤖 Generated with Claude Code

…ion change

Reported: mid-dictation, Parakeet stops producing words (~a minute in), then
"comes back" later — with the gap permanently missing from the transcript.

Investigation found NO regression in the decode path: FluidAudio has been
pinned at 0.15.5 since the original integration (#163), the recent Parakeet
changes (#218, #222) only touch callback snapshots and the final-transcript
artifact strip, and a real-model harness run (parakeet-unified-320ms over
2.5 min of continuous fixture speech) sustained RTFx 5.5 with zero dropouts
and no transcript gaps. The decoder does not go deaf on its own.

What CAN produce exactly this symptom is a mid-session
AVAudioEngineConfigurationChange: the input device disconnects, switches, or
renegotiates its format, AVAudioEngine stops rendering, and — because the
streaming engines never observe the notification — the session keeps showing
"Listening…" while capturing NOTHING. Every word until the user gives up and
re-dictates is silently lost. AudioRecorder has handled precisely this since
the AirPods-disconnect hang ("the session keeps showing 'recording' while
capturing nothing"); the streaming engines were never given the observer, and
Parakeet only became the default (and long dictations routine) recently —
which is why it reads as a new bug.

Unlike the recorder's fail-only handling, Parakeet can recover seamlessly:
the decode session and feed stream survive the mic teardown, so the engine
rebuilds a routed AVAudioEngine + tap onto the SAME continuation and the
dictation continues — everything already captured stays, and only the glitch
itself is lost. If the rebuild fails (no input device left), the session
fails LOUDLY through the error path instead of eating speech. The observer
is selector-based because the block API takes a hard @sendable closure and
every capture the restart needs is non-Sendable.

The decision gates live in core (CaptureConfigChangePolicy) so swift test
pins them: restart only when the session generation matches and no stop ran;
stale/superseded notifications touch nothing. Route/format setup is extracted
into makeRoutedEngine()/startCapture() shared by session start and restart.

swift test: 1904 passing (4 new). ./build.sh green, no new warnings.
Caveat (same as #222): the restart path only exercises against a real device
change with a live mic — worth a live pass (unplug/replug the pinned mic
mid-dictation) before the next release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@initcore0
initcore0 merged commit f58bed2 into main Jul 29, 2026
10 checks passed
@initcore0
initcore0 deleted the claude/parakeet-dictation-dropout-83e3c9 branch July 29, 2026 01:35
initcore0 added a commit that referenced this pull request Jul 30, 2026
…Kit/AppleSpeech/SpeechAnalyzer (#226)

PR #224 gave ParakeetStreamingEngine an AVAudioEngineConfigurationChange
observer that rebuilds capture onto the same feed stream when the input
device disconnects/switches/renegotiates mid-dictation. The other three
streaming engines had the same hole: AVAudioEngine stops rendering on the
change, the session keeps showing "Listening..." and every word after the
glitch is silently lost.

Port the Parakeet pattern (selector-based observer to stay
@Sendable-warning-free, makeRoutedEngine/startCapture split,
restart-then-loud-error, CaptureConfigChangePolicy gates) to:

- AppleSpeechEngine: rebuild engine + tap appending to the SAME
  SFSpeechAudioBufferRecognitionRequest, so the recognizer sees one
  continuous audio stream.
- SpeechAnalyzerStreamingEngine: rebuild engine + tap yielding into the
  SAME raw-buffer AsyncStream. The analyzer-format converter is now
  ADAPTIVE (rebuilt when the incoming buffer format changes) — a
  converter pinned to the start-time mic format would drop every buffer
  from the replacement device, leaving the transcript dead even after a
  successful capture restart.
- WhisperKitStreamingEngine: the mic belongs to WhisperKit's
  AudioProcessor, so the rebuild lives on WhisperKitStreamHandle
  (restartCapture mirrors the internal setupEngine via public API and
  feeds processBuffer, appending to the same decode buffer the realtime
  loop polls); the engine arms the observer on the processor's engine
  object at first state diff and re-arms on the replacement after a
  restart.

Restart failure (e.g. no input device left) surfaces through the session
error path via CaptureConfigChangePolicy.restartFailedMessage instead of
eating speech silently.

Each engine's restart path still needs a live-mic device-change pass
(unplug/switch mid-dictation); the policy gates are pinned by
CaptureConfigChangePolicyTests.

Co-authored-by: Claude <noreply@anthropic.com>
initcore0 added a commit that referenced this pull request Jul 30, 2026
…227)

The v1.0.12 regression (mic dead on every dictation, fine on v1.0.11):
PR #224's AVAudioEngineConfigurationChange handler restarted capture on
EVERY notification. The notification also fires for changes the handler's
own teardown+rebuild causes — on this hardware a capture start itself
posts one — so the first notification at session start entered a
self-sustaining teardown→rebuild→notification loop (~4 restarts/second,
65 restarts observed in one 15s session). Capture never lived long enough
to feed the decoder; every session timed out with no speech.

Fix — the restart is no longer reflexive:

- A notification schedules a settle-check (0.3s, storm-coalescing) instead
  of restarting inline.
- The check rebuilds only when the engine genuinely STOPPED
  (AVAudioEngine.isRunning == false — the real device-loss case the
  observer exists for). A notification that leaves the engine running is a
  transient renegotiation and is ignored; restarting a running engine is
  exactly what fed the loop.
- Rebuilds are budgeted (3 per capture session). A device that still stops
  capture after that fails the session loudly (gaveUpMessage) instead of
  looping silently.

Gates live in core CaptureConfigChangePolicy (settleDelay, budget, the
action matrix) and are pinned by CaptureConfigChangePolicyTests.

Verified live against the released artifacts on the reporting machine:
v1.0.12 reproduces the loop with both the pinned Shure MV7+ (35 restarts
/10s) and a BlackHole fixture harness (65 restarts/15s, empty transcript);
this build transcribes the fixture correctly across consecutive sessions
with ZERO capture restarts. swift test green (1908).

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant