Skip to content

feat(translate): on-device text translation via Apple Translation framework (macOS 15+) - #223

Merged
initcore0 merged 5 commits into
mainfrom
claude/apple-translation-text-path
Jul 31, 2026
Merged

feat(translate): on-device text translation via Apple Translation framework (macOS 15+)#223
initcore0 merged 5 commits into
mainfrom
claude/apple-translation-text-path

Conversation

@initcore0

Copy link
Copy Markdown
Owner

Replaces the approach of the closed PR #219 (dual-runtime Parakeet + Whisper translate). That design ran a second full ASR stack over audio the fast engine had already transcribed — duplicated inference, English-only, and drop-oldest/drain-timeout behavior that could silently lose dictated text. This PR keeps one ASR runtime and translates the text per session/utterance on-device, with a single hard rule: the fallback is always "leave the text untranslated", NEVER "lose text".

Design

Hidden-anchor TranslationSession (OpenWhisp/Services/AppleTranslationProvider.swift). Apple gives no direct initializer for TranslationSession; the only way to obtain one is SwiftUI's .translationTask(configuration:) modifier. The provider keeps a persistent invisible anchor — a 1×1, alpha-0, borderless, offscreen NSWindow hosting a SwiftUI view that carries the modifier — alive for the app's lifetime. Publishing a new TranslationSession.Configuration (re)triggers the modifier; the action holds the session and drains a MainActor request queue until the next configuration change. The configuration always uses a nil source language (the framework auto-detects the dictated language); only the target is pinned, and a request for a different target swaps the configuration so a fresh session serves it. prepareTranslation() runs once per session to surface the language-asset download flow; every request also carries a 12 s watchdog, so a missing asset or framework stall resolves to nil (caller keeps the original text) instead of hanging a dictation.

Dictation path (final text, no audio tap). When "Translate to English" is on, the language is non-English (or auto), and the active engine is ASR-only (Parakeet, Apple Speech, SpeechAnalyzer), completeFinalText translates the session's final transcript before the normal delivery/paste flow — identical for every streaming engine because it needs no audio tee. The arm decision is a pure core policy, TextTranslationPolicy.shouldTranslateFinal (mirroring how PR #219 structured DualRuntimeTranslationPolicy), capability-driven only — deliberately the exact complement of the engine-level translate predicate, so no session ever translates twice or falls in the gap (contract-tested across allEngineIDs). Details:

  • Mid-refine finals skip translation (the tail is a spoken instruction and the content-prefix subtraction must keep matching byte-for-byte).
  • liveChunks sessions fall back to a single final paste while armed, so the original language is never typed into the document first.
  • effectiveTranslateToEnglish / outputLanguageForCleaning grow text-path-aware derivations, so refine prompts and the RefineOutputGuard expected script correctly anticipate English output.

Stream overlay. StreamOverlayServer already had an injected Translator closure seam honoring config.translationEnabled + config.targetLanguage — but nothing supplied it. StreamOverlayCoordinator.startServer() now injects the provider (any target language, not just English), and the overlay pane gets a small Translation section (toggle + target-language picker) so the seam is actually reachable. Overlay internals (reducer, server) are untouched beyond the constructor argument; PR #219's two-track captionTrack UI is left for a follow-up.

Offer gates. Menu bar and Dictation pane now read ONE predicate (AppState.translationOfferedTextTranslationPolicy.translationOffered): "Translate to English" is offered when the engine translates natively OR the text path exists (macOS 15+). A past bug was exactly these two surfaces disagreeing.

macOS availability

Everything framework-facing is gated #if canImport(Translation) + if #available(macOS 15.0, *). The app's deployment target stays macOS 14 (build.sh is unchanged): macOS 14 compiles and runs with the feature simply unavailable — ASR-only engines keep today's dimmed menu row / absent pane toggle, and sessions never arm the text path.

Tested vs. needs a manual pass

Unit-tested (swift test, 1914 green):

  • TextTranslationPolicyTests — arm/offer/derivation rules, incl. the exactly-one-path-arms contract per engine and macOS-14 equivalence with the old LanguageResolver behavior.
  • StreamOverlayServerTests.testFailedTranslationKeepsOriginalCaption — a nil-returning translator degrades to the original caption line (the never-lose-text rule at the seam now wired live).
  • Existing overlay translator-seam and capability-matrix tests unchanged and green.

Needs your manual pass on macOS 15+ (the framework can't run under swift test):

  • Live dictation with Parakeet / Apple Speech / SpeechAnalyzer + Translate to English on a non-English language — verify the pasted final is English and that a translator failure pastes the original transcript.
  • The language-asset download prompt on first use (prepareTranslation fires from an invisible window — confirm the consent sheet actually appears; if it doesn't, the request times out safely and text passes through untranslated).
  • Overlay translated subtitles end-to-end in OBS/browser (enable in Settings → Stream Overlay → Translation).
  • Confirmation that the hidden 1×1 anchor window never becomes visible/clickable and that .translationTask fires from it on a menu-bar app.

Notes

🤖 Generated with Claude Code

initcore0 and others added 5 commits July 26, 2026 10:55
…mework (macOS 15+)

Replaces the retired dual-runtime approach (PR #219): one ASR runtime,
translate the TEXT per session on-device. When "Translate to English" is
on, the spoken language is non-English (or auto), and the active engine is
ASR-only (Parakeet / Apple Speech / SpeechAnalyzer), the session's FINAL
transcript is translated as text before it is pasted — no audio tap, no
second inference stack, and the fallback on any failure or timeout is
always the ORIGINAL transcript, never lost text.

- TextTranslationPolicy (core, unit-tested): pure arm/offer decisions,
  capability-driven (exact complement of the engine-level translate path,
  contract-tested across allEngineIDs). effectiveTranslateToEnglish /
  outputLanguageForCleaning grow text-path-aware variants so refine
  prompts and the RefineOutputGuard expected script see English coming.
- AppleTranslationProvider (app side): TranslationSession is only
  obtainable through SwiftUI's .translationTask, so the provider keeps a
  persistent invisible anchor — a 1x1 alpha-0 offscreen NSWindow hosting a
  view with the modifier — and serves a MainActor request queue from the
  session inside the action. Nil source language (auto-detect), per-request
  12s watchdog, prepareTranslation surfaces the asset-download flow.
  Gated #if canImport(Translation) + #available(macOS 15) — macOS 14
  builds and runs with the feature simply unavailable.
- Dictation path: completeFinalText translates the final when the policy
  arms (skipped mid-refine — the tail is a spoken instruction); liveChunks
  sessions fall back to a single final paste so the original language is
  never typed first.
- Stream overlay: StreamOverlayCoordinator now injects the provider as the
  server's existing Translator seam (any target language), plus a small
  Translation section in the overlay pane (toggle + language; macOS 14
  shows an availability note). New server test pins the nil-fallback
  (failed translation keeps the original caption).
- Offer gates: menu bar + Dictation pane now share ONE predicate
  (AppState.translationOffered) — offered when the engine translates
  natively OR the text path exists; macOS 14 + ASR-only keeps today's
  dimmed/absent behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review finding: only prepareTranslation success reset lastError, so after
one failure every later "Kept original text — …" status could cite a stale
cause (e.g. blaming missing language assets long after they were installed)
— misleading precisely during the manual live-test pass this PR is waiting
on. A successful translate now clears it, so the status always names the
current failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tchet

PR #223's AppState additions (+71 net) tripped the LOC ratchet. Offset by
moving out three self-contained pieces, per the ratchet's own playbook:

- ModelDownloader (+ the URL.createDirectories helper) → its own file; a
  URLSession delegate class with no AppState state that lived at the bottom
  of AppState.swift only for historical reasons.
- FluidAudioModelsLocator + SettingsBlobLoaders — the two extractions
  originally proven on the retired dual-runtime branch (PR #219), ported:
  pure filesystem/UserDefaults lookups AppState now forwards to.

Tracked AppState surface: 7163 → 7052 LOC; budget lowered 7092 → 7052 in
the same PR to lock in the win.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ork hangs on nil-source and not-installed pairs

Live-testing the feature (ru→en on Parakeet) produced "Translating…" then
the original Russian pasted, every time. A standalone harness against the
exact hidden-anchor setup isolated two framework behaviors on macOS 26:

1. A nil-source TranslationSession.Configuration is unusable in practice:
   prepareTranslation fails with unableToIdentifyLanguage and translate()
   then HANGS past any deadline. The provider now always resolves an
   explicit source — the session's language setting, or NLLanguageRecognizer
   over the text when it's "auto" — before the framework sees the request.

2. A merely `supported` (not installed) pair also hangs: the asset-download
   consent can't present from the invisible anchor window. (ru→en assets
   were not installed on the test machine; es→en, which IS installed,
   translated in 2.5s through the same hidden anchor.) translate() now
   pre-checks LanguageAvailability and fails FAST on such pairs — no 12s
   watchdog stall — while kicking off a one-time visible consent flow: the
   anchor becomes a small centered panel for the duration of
   prepareTranslation so the system download sheet has somewhere to present,
   then goes invisible again. Once the download lands, requests translate
   headless (harness-proven).

Same-language requests (detected source == target) now return the text
unchanged instead of failing. AppState passes the session language as the
source hint; ratchet budget re-locked at the new lower total.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…plicit Download button, no surprise consent window

User feedback from the live test: the mid-dictation consent window was
confusing ("weird window"), and after approving it there was no way to see
whether the download had landed or whether translation was ready at all.

- New TranslationAssetStatusView (Dictation pane, under the translate
  toggle, and Stream Overlay pane's translation section): shows the pair's
  concrete state — downloaded ✓ / needs download (with a Download… button) /
  downloading (polls every 3s and flips to ✓ when the pack lands) /
  unsupported / needs macOS 15. "Auto" language shows an explanatory note
  instead (no concrete pair to check).
- Dictations no longer pop the consent sheet themselves: a not-downloaded
  pair fails fast and the status names the fix ("Russian → English isn't
  downloaded — Settings → Dictation has the download"). The Settings row is
  now the single place downloads start, so the sheet only ever appears right
  after the user clicks Download.
- Status text uses language display names, not BCP-47 tags.

Verified on the machine that hit the bug: ru→en assets (whose download the
user approved mid-test) now read installed, and the harness translates
Russian through the invisible anchor in ~1s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@initcore0
initcore0 merged commit 1d69725 into main Jul 31, 2026
10 checks passed
@initcore0
initcore0 deleted the claude/apple-translation-text-path branch July 31, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant