feat(translate): on-device text translation via Apple Translation framework (macOS 15+) - #223
Merged
Merged
Conversation
…mework (macOS 15+) Replaces the retired dual-runtime approach (PR #219): one ASR runtime, translate the TEXT per session on-device. When "Translate to English" is on, the spoken language is non-English (or auto), and the active engine is ASR-only (Parakeet / Apple Speech / SpeechAnalyzer), the session's FINAL transcript is translated as text before it is pasted — no audio tap, no second inference stack, and the fallback on any failure or timeout is always the ORIGINAL transcript, never lost text. - TextTranslationPolicy (core, unit-tested): pure arm/offer decisions, capability-driven (exact complement of the engine-level translate path, contract-tested across allEngineIDs). effectiveTranslateToEnglish / outputLanguageForCleaning grow text-path-aware variants so refine prompts and the RefineOutputGuard expected script see English coming. - AppleTranslationProvider (app side): TranslationSession is only obtainable through SwiftUI's .translationTask, so the provider keeps a persistent invisible anchor — a 1x1 alpha-0 offscreen NSWindow hosting a view with the modifier — and serves a MainActor request queue from the session inside the action. Nil source language (auto-detect), per-request 12s watchdog, prepareTranslation surfaces the asset-download flow. Gated #if canImport(Translation) + #available(macOS 15) — macOS 14 builds and runs with the feature simply unavailable. - Dictation path: completeFinalText translates the final when the policy arms (skipped mid-refine — the tail is a spoken instruction); liveChunks sessions fall back to a single final paste so the original language is never typed first. - Stream overlay: StreamOverlayCoordinator now injects the provider as the server's existing Translator seam (any target language), plus a small Translation section in the overlay pane (toggle + language; macOS 14 shows an availability note). New server test pins the nil-fallback (failed translation keeps the original caption). - Offer gates: menu bar + Dictation pane now share ONE predicate (AppState.translationOffered) — offered when the engine translates natively OR the text path exists; macOS 14 + ASR-only keeps today's dimmed/absent behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review finding: only prepareTranslation success reset lastError, so after one failure every later "Kept original text — …" status could cite a stale cause (e.g. blaming missing language assets long after they were installed) — misleading precisely during the manual live-test pass this PR is waiting on. A successful translate now clears it, so the status always names the current failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tchet PR #223's AppState additions (+71 net) tripped the LOC ratchet. Offset by moving out three self-contained pieces, per the ratchet's own playbook: - ModelDownloader (+ the URL.createDirectories helper) → its own file; a URLSession delegate class with no AppState state that lived at the bottom of AppState.swift only for historical reasons. - FluidAudioModelsLocator + SettingsBlobLoaders — the two extractions originally proven on the retired dual-runtime branch (PR #219), ported: pure filesystem/UserDefaults lookups AppState now forwards to. Tracked AppState surface: 7163 → 7052 LOC; budget lowered 7092 → 7052 in the same PR to lock in the win. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ork hangs on nil-source and not-installed pairs Live-testing the feature (ru→en on Parakeet) produced "Translating…" then the original Russian pasted, every time. A standalone harness against the exact hidden-anchor setup isolated two framework behaviors on macOS 26: 1. A nil-source TranslationSession.Configuration is unusable in practice: prepareTranslation fails with unableToIdentifyLanguage and translate() then HANGS past any deadline. The provider now always resolves an explicit source — the session's language setting, or NLLanguageRecognizer over the text when it's "auto" — before the framework sees the request. 2. A merely `supported` (not installed) pair also hangs: the asset-download consent can't present from the invisible anchor window. (ru→en assets were not installed on the test machine; es→en, which IS installed, translated in 2.5s through the same hidden anchor.) translate() now pre-checks LanguageAvailability and fails FAST on such pairs — no 12s watchdog stall — while kicking off a one-time visible consent flow: the anchor becomes a small centered panel for the duration of prepareTranslation so the system download sheet has somewhere to present, then goes invisible again. Once the download lands, requests translate headless (harness-proven). Same-language requests (detected source == target) now return the text unchanged instead of failing. AppState passes the session language as the source hint; ratchet budget re-locked at the new lower total. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…plicit Download button, no surprise consent window
User feedback from the live test: the mid-dictation consent window was
confusing ("weird window"), and after approving it there was no way to see
whether the download had landed or whether translation was ready at all.
- New TranslationAssetStatusView (Dictation pane, under the translate
toggle, and Stream Overlay pane's translation section): shows the pair's
concrete state — downloaded ✓ / needs download (with a Download… button) /
downloading (polls every 3s and flips to ✓ when the pack lands) /
unsupported / needs macOS 15. "Auto" language shows an explanatory note
instead (no concrete pair to check).
- Dictations no longer pop the consent sheet themselves: a not-downloaded
pair fails fast and the status names the fix ("Russian → English isn't
downloaded — Settings → Dictation has the download"). The Settings row is
now the single place downloads start, so the sheet only ever appears right
after the user clicks Download.
- Status text uses language display names, not BCP-47 tags.
Verified on the machine that hit the bug: ru→en assets (whose download the
user approved mid-test) now read installed, and the harness translates
Russian through the invisible anchor in ~1s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the approach of the closed PR #219 (dual-runtime Parakeet + Whisper translate). That design ran a second full ASR stack over audio the fast engine had already transcribed — duplicated inference, English-only, and drop-oldest/drain-timeout behavior that could silently lose dictated text. This PR keeps one ASR runtime and translates the text per session/utterance on-device, with a single hard rule: the fallback is always "leave the text untranslated", NEVER "lose text".
Design
Hidden-anchor
TranslationSession(OpenWhisp/Services/AppleTranslationProvider.swift). Apple gives no direct initializer forTranslationSession; the only way to obtain one is SwiftUI's.translationTask(configuration:)modifier. The provider keeps a persistent invisible anchor — a 1×1, alpha-0, borderless, offscreenNSWindowhosting a SwiftUI view that carries the modifier — alive for the app's lifetime. Publishing a newTranslationSession.Configuration(re)triggers the modifier; the action holds the session and drains a MainActor request queue until the next configuration change. The configuration always uses a nil source language (the framework auto-detects the dictated language); only the target is pinned, and a request for a different target swaps the configuration so a fresh session serves it.prepareTranslation()runs once per session to surface the language-asset download flow; every request also carries a 12 s watchdog, so a missing asset or framework stall resolves to nil (caller keeps the original text) instead of hanging a dictation.Dictation path (final text, no audio tap). When "Translate to English" is on, the language is non-English (or auto), and the active engine is ASR-only (
Parakeet,Apple Speech,SpeechAnalyzer),completeFinalTexttranslates the session's final transcript before the normal delivery/paste flow — identical for every streaming engine because it needs no audio tee. The arm decision is a pure core policy,TextTranslationPolicy.shouldTranslateFinal(mirroring how PR #219 structuredDualRuntimeTranslationPolicy), capability-driven only — deliberately the exact complement of the engine-level translate predicate, so no session ever translates twice or falls in the gap (contract-tested acrossallEngineIDs). Details:liveChunkssessions fall back to a single final paste while armed, so the original language is never typed into the document first.effectiveTranslateToEnglish/outputLanguageForCleaninggrow text-path-aware derivations, so refine prompts and theRefineOutputGuardexpected script correctly anticipate English output.Stream overlay.
StreamOverlayServeralready had an injectedTranslatorclosure seam honoringconfig.translationEnabled+config.targetLanguage— but nothing supplied it.StreamOverlayCoordinator.startServer()now injects the provider (any target language, not just English), and the overlay pane gets a small Translation section (toggle + target-language picker) so the seam is actually reachable. Overlay internals (reducer, server) are untouched beyond the constructor argument; PR #219's two-trackcaptionTrackUI is left for a follow-up.Offer gates. Menu bar and Dictation pane now read ONE predicate (
AppState.translationOffered→TextTranslationPolicy.translationOffered): "Translate to English" is offered when the engine translates natively OR the text path exists (macOS 15+). A past bug was exactly these two surfaces disagreeing.macOS availability
Everything framework-facing is gated
#if canImport(Translation)+if #available(macOS 15.0, *). The app's deployment target stays macOS 14 (build.shis unchanged): macOS 14 compiles and runs with the feature simply unavailable — ASR-only engines keep today's dimmed menu row / absent pane toggle, and sessions never arm the text path.Tested vs. needs a manual pass
Unit-tested (
swift test, 1914 green):TextTranslationPolicyTests— arm/offer/derivation rules, incl. the exactly-one-path-arms contract per engine and macOS-14 equivalence with the oldLanguageResolverbehavior.StreamOverlayServerTests.testFailedTranslationKeepsOriginalCaption— a nil-returning translator degrades to the original caption line (the never-lose-text rule at the seam now wired live).Needs your manual pass on macOS 15+ (the framework can't run under
swift test):prepareTranslationfires from an invisible window — confirm the consent sheet actually appears; if it doesn't, the request times out safely and text passes through untranslated)..translationTaskfires from it on a menu-bar app.Notes
mainincluding the merged overlay-retirement PR fix(overlay): resumed speech replays old subtitles, settings edits kill capture, dictation pill on stream #221; the StreamOverlay area may still conflict slightly with any further open work there (StreamOverlayCoordinator/StreamOverlayPanetouches are intentionally minimal).🤖 Generated with Claude Code