diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 03bf51df..be89c565 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -129,6 +129,25 @@ jobs: mkdir -p resources/bin cp scripts/dictation-hotkey/dictation-hotkey resources/bin/dictation-hotkey chmod +x resources/bin/dictation-hotkey + # Compile the Scribe system-wide overlay helper (AX read/measure + squiggle window) + # and stage it into resources/bin. Self-contained, deployment target pinned. If it ever + # fails to ship, Scribe simply falls back to the select-and-hotkey flow (the overlay + # service no-ops when the binary is absent), so it can't break a release. + - name: Build Scribe overlay helper + run: | + bash scripts/build-scribe-overlay.sh + mkdir -p resources/bin + cp scripts/scribe-overlay/scribe-overlay resources/bin/scribe-overlay + chmod +x resources/bin/scribe-overlay + # Stage the Parakeet STT runtime (sherpa-onnx CLI + ONNX model) into + # resources/bin/parakeet. Additive: with SHERPA_ONNX_URL / PARAKEET_MODEL_URL + # unset this is a no-op and transcription stays on whisper, so it can't break a + # release. Set those vars once the exact sherpa-onnx build + Parakeet export are pinned. + - name: Stage Parakeet STT runtime (optional) + env: + SHERPA_ONNX_URL: ${{ vars.SHERPA_ONNX_URL }} + PARAKEET_MODEL_URL: ${{ vars.PARAKEET_MODEL_URL }} + run: bash scripts/fetch-parakeet.sh # Private pro source into pro/ → __OFFGRID_PRO__ true. The checkout action # injects the token as an http.extraheader (never in a clone URL) and scrubs # it afterwards — safer than embedding ${TOKEN} in a git URL. diff --git a/.gitignore b/.gitignore index b60be922..1ac0c18f 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,9 @@ electron/accessibility/ocr # them. Only the heavy optional runtimes we don't ship are excluded. resources/bin/coreml-sd/ resources/bin/mflux/ +# Parakeet STT runtime — CI-staged via scripts/fetch-parakeet.sh (like mflux/coreml-sd), +# not committed. Devs stage it locally with the same script. +resources/bin/parakeet/ # Gateway probe scratch output .gateway-probe/ # TS build cache @@ -50,3 +53,7 @@ scripts/dictation-hotkey/dictation-hotkey # Local demo profile — synthetic-data run target for `npm run demo` (never real userData) .demo-profile/ + +# local PR-campaign target list (not for commit) +pr-targets.local.md +scripts/scribe-overlay/scribe-overlay diff --git a/CLAUDE.md b/CLAUDE.md index a29c5759..f808d720 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This is **Off Grid AI Desktop** — an Electron (macOS) desktop app. The product ## Design — DESKTOP-FIRST, Off Grid brand -Full design doc: **`docs/DESIGN.md`**. The essentials, which OVERRIDE any mobile-first or monochrome assumptions: +Inherit the shared Off Grid design philosophy from **`../brand/DESIGN_PHILOSOPHY.md`** (the source of truth — brutalist/terminal, Menlo mono, emerald accent, tokens in `@offgrid/design`). Platform specifics: **`docs/DESIGN.md`** (brand canon) and **`docs/DESIGN_PHILOSOPHY.md`** (whole-app north star, motion, desktop-first density). The essentials below OVERRIDE any mobile-first or monochrome assumptions: - **Desktop-first.** Wide canvas: multi-column layouts, dense lists/tables, side panels, detail screens, hover affordances. Never design mobile-first or for narrow viewports. (The mobile app is a separate product with its own guide.) - **Typeface: Menlo** (monospace) everywhere — terminal/brutalist. @@ -122,3 +122,14 @@ If the answer to 1 is "no", say so and write the simple version. If "yes", build ## Copy & content standards Any change to UI strings, docs, essays, or marketing copy follows the brand voice (`mobile/docs/brand_tone_voice.md`). Easy-to-miss rules: proof-first ("15-30 tok/s", not "fast"); privacy as mechanism ("runs in your Mac's RAM, nothing leaves the device", not "we value privacy"); no em dashes (use " - "), no curly quotes, no exclamation marks; banned words (revolutionary, seamless, empower, leverage, robust, comprehensive, crucial, delve, tapestry, testament, foster, showcase, enhance) and AI-slop phrases ("serves as", "stands as", "it's not X, it's Y") — say it plainly. No emojis in UI. + + +## Multi-agent operating model (how we build here) + +Substantial work is executed by a fleet of parallel subagents orchestrated by the main session — not one linear thread. The standard: + +- **Parallel workers, 3 at a time.** Decompose work into worktree-isolated subagents that run concurrently in a rolling window of ~3, each on a DISJOINT file-set so they never merge-conflict. As each lands: review against the engineering standards, merge, run a **local production build gate** (typecheck + tests do NOT catch build/route errors — build before deploy), deploy, verify, then launch the next from the backlog. One agent owns nav/shared-file changes per round; the others avoid them. +- **The gap agent.** Any gap, regression, or "not fully done" is logged to the repo's gaps doc (`docs/GAPS_BACKLOG.md`). A standing gap agent is woken whenever there are gaps: it picks them up, closes them, and marks them resolved with evidence. Gaps are surfaced honestly, never hidden. +- **The QA / platform-integration + docs sweep agent.** After every 3 agent completions, run a sweep agent that (a) verifies the whole platform integrates and works end-to-end (run the integration harness + exercise real cross-service/-surface flows), (b) surfaces any new gaps into the gaps doc, and (c) writes/updates USER-FACING documentation live — how to use / what to do / why / when, per surface — so docs stay current with the build. +- **Merge gate (every merge, non-negotiable):** SOLID + pure logic isolated (unit-testable, zero-IO) separated from I/O; thin handlers; REAL tests exercising real behavior (mocks sparingly); typecheck clean; tests pass; a clean local production build; verify UI by screenshot (vision) and integration by the harness. Nothing is "done" until VERIFIED live, not merely merged. +- **Honesty bar:** report status as a gate (code / wired / verified), never inflate "done." A premature "complete" is a defect. diff --git a/docs/GAPS_BACKLOG.md b/docs/GAPS_BACKLOG.md new file mode 100644 index 00000000..fe4f7e8d --- /dev/null +++ b/docs/GAPS_BACKLOG.md @@ -0,0 +1,106 @@ +# Gaps backlog + +Honest running list of gaps, regressions, and "not fully done" items. Each entry: what, why it +matters, where, and status. Close with evidence; don't hide gaps. (Convention from `CLAUDE.md` +multi-agent operating model.) + +Status legend: **OPEN** · **IN PROGRESS** · **RESOLVED** (with evidence) · **DECISION NEEDED** + +--- + +## Scribe (writing companion) + +### Design-system compliance (audited against `../brand/DESIGN_PHILOSOPHY.md`, 2026-07-06) + +- **G-1 · Hardcoded dark hex → broken in light mode** — RESOLVED. All four hex-hardcoded components + converted to theme-aware Tailwind (`neutral-*`/`emerald` → `--og-*` tokens): `ReviewPanel.tsx`, + `RewriteToolbar.tsx`, `LearnedStyle.tsx`, `StyleGuides.tsx`. typecheck clean, 554 tests pass. + Remaining inline hex is only in `AssistedTextarea.tsx` (squiggle overlay positioning + the + category hues, which is the G-2 decision) — not a light/dark bug. Evidence still owed: light-mode + screenshots of the settings panel + rewrite toolbar (folded into G-10). + +- **G-2 · Color-coded issue categories** — RESOLVED (user decision: brand-align). Dropped the + rainbow for the two brand colors: **semantic error red** for correctness (spelling/grammar/ + punctuation) and **emerald** for suggestions (style/clarity/tone/word-choice). Applied in all three + places: overlay binary `CATEGORY_COLOR`, `AssistedTextarea` `UNDERLINE`, `ReviewPanel` + `CATEGORY_COLOR`. + +- **G-3 · a11y on icon-only buttons** — RESOLVED (DOM buttons). Added `aria-label` (+`title`) to the + settings-close X and the disabled-apps remove X. The overlay card's gear/× are native AppKit chips + (Swift `NSView`), not DOM — out of scope for web a11y; AppKit handles their accessibility. + +- **G-4 · `prefers-reduced-motion` not honored** — RESOLVED. Added the standard global rule in + `src/renderer/src/assets/main.css` collapsing all transitions/animations to ~instant under + reduced-motion (was only disabling the shooting-star). App-wide, not just Scribe. + +- **G-5 · Toggle uses `rounded-full`** — LOW / likely-accept. `WritingSettings` toggle is a pill. + Anti-pattern list says avoid pill shapes, but a toggle switch is a conventional control, not a + pill *button*. Flagging for awareness; probably keep. + +### Spelling false-positives (proactive audit, 2026-07-06 — the class the user keeps hitting) + +- **G-11 · Contractions flagged** — RESOLVED. `I'll`, `I've`, `don't`, `it's`, `o'clock` were + flagged (tokenize as one non-dictionary word). `isContraction()` accepts known head + standard + clitic. Tested. (User-reported.) +- **G-12 · URLs / emails / @mentions / #tags / `code` flagged** — RESOLVED. Their letter-runs + (getoffgridai, wednesday, handles) flooded chat/email with red squiggles. `protectedRanges()` + skips any token inside such a span. Benefits the overlay too (same engine). Tested. +- **G-13 · Hyphenated compounds flagged** — RESOLVED. `well-known`, `on-device`, `end-to-end` + accepted when every part is a valid word. Tested. + +### Functional (from on-device testing, 2026-07-06) + +- **G-6 · Slack live squiggles — not visually confirmed** — OPEN. Root cause fixed + PROVEN: the + overlay now sees Slack's field (`[scribe-overlay] focused Slack (com.tinyspeck.slackmacgap)` in + the log) after enabling `AXManualAccessibility` per app. NOT yet confirmed visually that squiggles + draw on misspelled Slack text after the ~1.2s typing-pause (blind automation kept hitting the wrong + window; risk of sending a real message). Evidence to close: user types a misspelling in Slack, + pauses, sees a squiggle; or a controlled capture. + +- **G-7 · Live-underline pause behavior needs real-use tuning** — OPEN. In Chromium/Electron the + selection-trick moves the caret, so measurement is gated to ≥1.2s idle + collapsed caret. The + threshold is a guess; confirm it feels right (not too laggy, no cursor flicker) in real typing and + tune `selectionIdle`. + +- **G-8 · No live squiggles while actively typing in Slack/browser** — BY DESIGN, documented. Only + native Cocoa apps get truly-live underlines (non-invasive bounds). Revisit if a non-invasive + per-range bounds method is found (AX text markers returned whole-element bounds in testing). + +- **G-9 · provit E2E never got a clean green run** — OPEN (findings written up, handed off). + NativeMacActor works; the acceptance journey failed on foreground contention (other automations + stealing front), not a Scribe bug. Full writeup + ranked provit fixes + repro steps in + **`SCRIBE_PROVIT_FINDINGS.md`**. Ball is in the provit agent's court (re-focus appTarget per + capture; capture-by-windowID / serialize native-mac runs; fix the journey's red-vs-emerald copy). + +### Verification debt (mine — built but not visually confirmed by me) + +- **G-10 · New Scribe screen + busy card look** — OPEN. Redesign + compact busy card are typechecked + and code-reviewed but I did not screenshot the final look. Evidence to close: light + dark screenshots. + +## Editor & rich text (2026-07-07) + +- **G-14 · Markdown preview looked unstyled** — RESOLVED. It rendered but Tailwind Preflight strips + heading/bold/list styles; added scoped `.scribe-preview` prose CSS. +- **G-15 · Real rich-text mode (TipTap)** — DONE (needs on-device visual check). Rich text is now a + single WYSIWYG (TipTap/ProseMirror), no preview split; live squiggles as PM decorations, click-to- + fix; own toolbar. Compiles + production-bundles. NOT visually verified by me. +- **G-16 · Rich-mode side-panel "apply fix" is inert** — OPEN (minor). The Review panel's one-click + apply (onApply) edits plain text; in rich mode the doc is HTML, so applying from the side panel + won't reflect. Rich mode has its own inline click-to-fix instead. Wire the review panel's apply to + the TipTap doc later, or hide per-issue apply buttons in rich mode. +- **G-17 · Rich-mode squiggle hover card** — RESOLVED. Extracted a single shared `IssuePopover` + (message + fixes + Remove + Add-to-dictionary/Ignore) consumed by BOTH the plain-text editor and + the TipTap editor; rich mode now hovers the real card resolved to the ProseMirror range, not a dead + native `title=` tooltip. (User-reported: "I liked the card better, this is useless".) +- **G-18 · Rephrase pill / squiggles in the browser address bar** — RESOLVED. The `AXSearchField` + subrole alone missed Chromium's omnibox (Chrome/Arc/Edge/Brave expose a plain AXTextField). Added + `isAddressOrSearchField` (subrole + descriptive metadata: role-description/description/placeholder/ + title matching address/search/url markers) gating BOTH the squiggle and pill paths; `isRephrasableProse` + also rejects selections whose tokens are bare URLs/domains/emails. Overlay recompiles clean. + (User-reported.) +- **G-19 · Whole-feature SOLID/DRY audit** — RESOLVED. 3 parallel auditors across engine / main-wiring + / renderer, then 3 fixers. Collapsed every duplicated source of truth: category→color (was 4 copies), + button/chip primitives (6 copies), issue-card renderer (2), countSyllables (2), severity map (2), + extractJson (2), model-call seam (3 bypasses), REWRITE_ACTIONS/tone list/hover-close/escapeRegExp. + Fixed a real bug (rich-mode minimap/jump on stale text). ~400 dup lines removed, +tests. Gate: tsc + web+node clean, 617 tests, production build OK. See commit `ad88bf1` (pro) + `b3b0200` (overlay). diff --git a/docs/SCRIBE_PROVIT_FINDINGS.md b/docs/SCRIBE_PROVIT_FINDINGS.md new file mode 100644 index 00000000..58539621 --- /dev/null +++ b/docs/SCRIBE_PROVIT_FINDINGS.md @@ -0,0 +1,55 @@ +# Scribe overlay × provit — E2E findings & handoff + +Notes from running provit's `NativeMacActor` against the Scribe system-wide overlay (2026-07-05/06). +For the provit maintainer/agent. Provit lives at `../provit`; the acceptance journey is +`../provit/journeys/scribe-overlay.json`, the actor `../provit/src/mac/nativeMacActor.ts`, design +`../provit/docs/NATIVE_MAC_ACTOR.md`. + +## What worked +- **NativeMacActor primitives pass.** `session() → "native-mac"`, `windowSize()` returns points, + full-screen `screenshot()` produces a real PNG. Permissions (Screen Recording + Accessibility on + the terminal) were sufficient — the same-process CGEvent helper is covered by the terminal grant. +- **The vision loop ran** — the gateway brain (`https://ai.getoffgridai.co/v1`, `qwen3-vl-8b`) judged + each `observe` step and returned structured pass/fail with a reason. The plumbing is sound. + +## Why the run never went green (not a Scribe bug) +Every `observe` failed because the **captured frame showed the wrong app** — Gmail/Finder/iTerm +instead of TextEdit. Root cause: **foreground contention**. `NativeMacActor` captures the whole +screen, and other processes kept stealing the foreground mid-run: +1. The actor focuses `appTarget` **once** at `launchApp`, then never re-asserts it; focus drifts. +2. ~20 concurrent `web-todomvc` provit runs (separate session) repeatedly threw a Chromium window to + the front. Full-screen capture has no defense against that. +3. My own screenshot automation + the user working in parallel also pulled focus. + +So the failures are an environment/harness issue, not the overlay. (The overlay itself was verified +by hand: emerald/red squiggles land accurately, the hover card applies fixes, Rephrase works — see +`GAPS_BACKLOG.md` and the desktop commits.) + +## Recommended provit changes (ranked) +1. **Re-assert `frontmost(appTarget)` before every capture**, not just at `launchApp`. Cheap, and it + fixes the dominant failure mode. (Its own gotcha doc says "if a click lands in the wrong app, + focus shifted — re-run"; make it proactive instead of a manual retry.) +2. **Serialize native-mac runs behind a foreground lock.** A `surface: "native-mac"` run needs + exclusive display foreground — it must refuse/queue while any web/Electron automation is active + on the same display. Better: capture the **target window by `windowID`** (`CGWindowListCreateImage` + for that window) instead of the whole screen, so another app grabbing front doesn't corrupt the + frame. That's the real fix; #1 is the quick one. +3. **Fix the journey copy.** `journeys/scribe-overlay.json` step-1 `expect` says "emerald wavy + underlines under the misspelled words." The overlay now uses **brand color-coding**: red for + correctness (spelling/grammar/punctuation), **emerald for suggestions** (style/clarity/tone). + So spelling underlines are RED. Update the expectation to "red wavy underlines under the misspelled + words; emerald under the passive-voice/style sentence." + +## To reproduce a clean run +Quit other automations, then (from `../provit`): +``` +PROVIT_VISION=gateway PROVIT_PROVIDER=gateway \ + PROVIT_ORACLE_URL="https://ai.getoffgridai.co/v1" PROVIT_ORACLE_KEY="" \ + PROVIT_ORACLE_MODEL="qwythos-9b" PROVIT_VISION_MODEL="qwen3-vl-8b" \ + PROVIT_MAC_TARGET="com.apple.TextEdit" \ + node --experimental-strip-types --experimental-sqlite \ + src/ios/visionReplay.ts journeys/scribe-overlay.json recordings/scribe +``` +Setup first: run the desktop app (`OFFGRID_PRO=1`, overlay on) and `open -a TextEdit` a doc seeded +with `please recieve teh alot of wierd notes.` + a passive-voice line. Keep the machine's foreground +free for the duration (single-owner display). diff --git a/docs/SCRIBE_SYSTEMWIDE_OVERLAY.md b/docs/SCRIBE_SYSTEMWIDE_OVERLAY.md new file mode 100644 index 00000000..06cd3f55 --- /dev/null +++ b/docs/SCRIBE_SYSTEMWIDE_OVERLAY.md @@ -0,0 +1,191 @@ +# Scribe system-wide inline overlay — implementation plan + +Grammarly-style **inline squiggles in any app**, no hotkey, no browser extension. This doc is +the design for review before code. It builds on the working writing engine (see +`WRITING_ASSISTANT_PLAN.md`) and the dictation AX/paste infra. + +## What we proved (AX probe, on-device, 2026-07-05) + +`scripts/scribe-ax-probe/` measured, for the focused text field in the frontmost app, +whether macOS gives us a per-word bounding rectangle (the thing we need to draw a squiggle). + +| App | Type | `AXBoundsForRange` (direct) | Selection trick | Verdict | +|---|---|---|---|---| +| Notes | native Cocoa | ✅ 12/12 | — | direct | +| iTerm2 | native | ✅ 12/12 | — | direct (neg. Y = 2nd monitor, normalizable) | +| Slack | Electron | ✗ 0/12 | ✅ 12/12 | selection trick + `AXManualAccessibility` | +| Brave | browser web field | ✗ 0/12 | ✅ 12/12 | selection trick | + +**Conclusion: a single native overlay covers native + Electron + browser web fields.** No +browser extension. Canvas-rendered surfaces (Google Docs) are the expected gap → hotkey fallback. + +Reference (Apache-2.0, build-our-own, do not vendor): `PhilipSchmid/textwarden` +(`Sources/Accessibility/TextMonitor.swift`, `Sources/Positioning/Strategies/ChromiumStrategy.swift`), +`Automattic/harper`. + +## The three AX techniques (proven) + +1. **Enable the AX tree.** Chromium/Electron keep accessibility off until they detect a screen + reader. + - Electron desktop (Slack/Teams/VSCode): `AXUIElementSetAttributeValue(app, "AXManualAccessibility", true)` — succeeds, no side effects. + - Browsers (Chrome/Brave): reject that (`-25205`); they use `AXEnhancedUserInterface` (VoiceOver's switch, can disturb window positioning → set, use, restore). Note: in testing the selection trick returned real bounds even when both enables were rejected, i.e. the tree was already warm — **cold-start handling is an open item (below).** +2. **Measure bounds.** `AXBoundsForRange` returns garbage in Chromium. Instead: save cursor → + `kAXSelectedTextRangeAttribute = range` → wait ~45 ms → read `AXSelectedTextMarkerRange` → + `AXBoundsForTextMarkerRange` → poll ≤10× rejecting **stale** (unchanged vs last) and + **whole-line** (width > 80% of field) rects → restore cursor. Native apps skip all this and + use `AXBoundsForRange` directly. +3. **Apply a fix.** Set `kAXSelectedTextRangeAttribute` to the issue span, then either set + `kAXSelectedTextAttribute` (where supported) or synthesize paste (reuse dictation's + `text-injection` paste-back). Restore cursor. + +## Architecture — mechanism (Swift) vs policy (our engine) + +Clean split so no writing logic leaks into the native binary and the engine stays the single +source of truth: + +``` +┌ Swift AX overlay service (mechanism only, generic) ──────────────┐ +│ - AXObserver: focus / value-changed / scroll / resize / move │ +│ - read focused text + element frame │ +│ - measure bounds for REQUESTED ranges (direct → selection trick) │ +│ - transparent, click-through NSWindow: draws squiggle underlines │ +│ - hit-test hover/click on a squiggle → emit event │ +│ - apply replacement over a span (AX set / paste) │ +│ ↕ JSON lines over stdio │ +└──────────────────────────────────────────────────────────────────┘ + ↑ text, focus, bounds, hover/click ↓ issues (spans), fix +┌ Pro main (policy) ───────────────────────────────────────────────┐ +│ - runs the PURE rules engine (instant) + LLM on demand │ +│ - maps text → issues; asks service to measure only ISSUE spans │ +│ - owns WritingSettings (per-app enable, pause, checks) │ +└──────────────────────────────────────────────────────────────────┘ + ↓ show correction card at (x,y) +┌ Pro renderer ────────────────────────────────────────────────────┐ +│ - correction card = reuse existing IssuePopover (React) │ +│ in a small non-activating panel window positioned by coords │ +└──────────────────────────────────────────────────────────────────┘ +``` + +**Why Swift draws the squiggles (not an Electron full-screen overlay):** a transparent +click-through Electron window floating over *other* apps is notoriously unreliable +(focus-stealing, multi-monitor, click-through-except-regions, repaint cost). TextWarden proves the +native NSWindow path works. Squiggles are cheap CoreGraphics lines. **The interactive correction +card** stays our React `IssuePopover` in a small on-demand panel (reuse + streaming rewrites), +shown only when a squiggle is engaged — one small window, not a full-screen overlay. + +**Rejected alternatives:** (a) Electron full-screen click-through overlay — reliability risk above. +(b) Native Swift correction card — loses React reuse + streaming; only fall back to this if the +panel-window positioning proves unreliable over other apps. + +## IPC contract (Swift ⟷ Electron, JSON lines over stdio) + +Service → main: +- `{type:"focus", app, bundleId, role, elementFrame:{x,y,w,h}, hasText:bool}` +- `{type:"text", text, charCount, selection:{loc,len}}` (debounced on value-changed) +- `{type:"bounds", ranges:[{loc,len,rect:{x,y,w,h}|null}]}` (reply to measure) +- `{type:"hover", span:{loc,len}, at:{x,y}}` / `{type:"click", span, at}` +- `{type:"scroll"}` / `{type:"blur"}` + +Main → service: +- `{cmd:"measure", ranges:[{loc,len}]}` — measure only issue spans (not every word) +- `{cmd:"draw", underlines:[{rect,color}]}` — repaint overlay +- `{cmd:"apply", span:{loc,len}, replacement}` — perform the edit +- `{cmd:"enable", bundleId}` / `{cmd:"clear"}` + +## Coordinate mapping + +AX rects: points (= DIP, so Retina is 1:1 with Electron), **top-left origin, global** across all +displays (hence iTerm's negative Y on a second monitor). For the Swift NSWindow overlay, convert to +Cocoa bottom-left: `cocoaY = totalScreenHeight - axY - axHeight`, choosing the `NSScreen` that +contains the rect. For the Electron card panel, `screen.getDisplayNearestPoint` + DIP coords map +directly. Unit-test the transform with fixtures (incl. negative-Y multi-monitor). + +## Per-app strategy registry + +`AppStrategy` keyed by bundle id (mirrors TextWarden's ContentParsers), each declaring: +enable-attr (`manual` | `enhanced` | `none`), bounds-method (`direct` | `selection`), index space +(`utf16` | `grapheme`), selection offset (newline handling), and `visualUnderlines: bool` +(off for terminals / unreliable apps → card-only). Defaults: native → direct; Chromium bundle +prefixes → selection. One place; the service and settings both read it. + +## Cold-start AX enable (open item) + +In testing the selection trick worked even when both enable attrs were rejected (tree already +warm). Unknown: a freshly-launched browser with cold AX. Plan: on first focus in a Chromium app, +attempt enable → poll `AXValue`/bounds for up to ~1.5 s before first measure; if still cold, show +the squiggles on the next value-changed tick. **Verify with the quit-and-relaunch probe run +before P2.** + +## Open-core placement + +- **Swift service = generic mechanism, no writing logic** → `scripts/scribe-overlay/` (core), + built + staged to `resources/bin` by `release.yml`, exactly like `scripts/dictation-hotkey`. + Carries only AX + rect drawing, so it leaks no pro source. (Open decision: if we'd rather keep + even the binary out of core, move source to `pro/native/` + a pro CI build step.) +- **All policy is pro:** engine use, issue→underline mapping, correction card, settings, activation + live in `pro/main/writing/overlay/` + `pro/renderer`. Gated by `proEnabled()`; `OFFGRID_PRO=0` + disables. Degrades gracefully to the existing select-and-hotkey flow if the binary is absent. +- Entitlements/permissions: reuse dictation's Accessibility grant + `text-injection`. + +## Phases + +- **P1 — Native apps, direct bounds.** Swift service (focus observer + `AXBoundsForRange`) + + NSWindow overlay drawing squiggles + coordinate transform. Prove pixel-accurate squiggles over + **Notes/Mail**, correct on scroll/resize/window-move. Correction card panel shows + applies a fix. + *Accept:* squiggles track the text in Notes through scroll/resize; apply-fix works. +- **P2 — Electron + browser, selection trick.** Add enable-attr + selection-based measurement + (cached, typing-pause gated, whole-line/stale rejection) + per-app registry + cold-start handling. + Prove **Slack + a browser web field**. *Accept:* squiggles in Slack/Brave, no cursor disruption + while typing, ≥ the probe's hit rate. +- **P3 — Interactions + settings.** Hover card (reuse IssuePopover) with apply / add-to-dictionary, + per-app enable + pause honored, learning-loop feedback wired. *Accept:* full loop in ≥3 apps + across all three buckets; per-app toggle + pause respected. +- Canvas apps (Google Docs): documented hotkey fallback, not in scope. + +## Performance + +Measure only **issue** spans (few) not every word. Cache bounds by (text hash, element frame, +attributed-string hash); invalidate on value/scroll/resize. Selection measurement only during +typing pauses (~400 ms idle) to avoid cursor interference. AX messaging timeout ~1 s. Hard cap on +issues drawn (reuse engine's `maxIssues`). + +## Testing + +- **Pure/unit (vitest, no Electron):** coordinate transform (incl. negative-Y multi-monitor), + per-app strategy selection, issue→underline mapping, whole-line/stale rejection predicate. +- **Swift unit:** range→bounds parsing, selection-trick state machine (mirror TextWarden's + `SlackStrategyValidationTests`), cursor save/restore. +- **Manual matrix (can't be E2E'd):** Notes, Mail, Slack, Brave, VSCode — squiggle accuracy, + scroll tracking, apply-fix, no-typing-disruption. Screenshots per app in the PR. + +## Build status (v1 — 2026-07-05) + +Implemented + green (compile / 3× tsc / 547 unit tests). **Needs on-device visual verification** +(marked ⚠) — I can't drive the GUI. + +- ✅ **Swift binary** (`scripts/scribe-overlay/main.swift`): `--demo` visual mode (VERIFIED in Notes + by the user) + IPC mode (mechanism only). Direct bounds for native, selection-trick for + Chromium/Electron/browser (cursor saved/restored), per-category colors, viewport culling, cap, + bounds cache + typing-pause gating. +- ✅ **Pro overlay service** (`pro/main/writing/overlay/`): spawns the binary, runs `checkSync`, + gates on enabled + systemWide + `inlineOverlay` + per-app rules + built-in terminal/secure-field + denylist, pushes spans, restarts on crash, no-ops when the binary is absent. Pure `issuesToSpans` + (tested). +- ✅ **Hover-to-fix** ⚠: global-monitor dwell → native menu (engine's fixes + Add-to-dictionary / + Ignore) → AX apply → auto re-check. Add/Ignore round-trip to the engine (teach / session-ignore). +- ✅ **Settings**: `surfaces.inlineOverlay` (default on) + toggle in Scribe settings + sanitizer. +- ✅ **CI**: `scripts/build-scribe-overlay.sh` (target pinned, minos logged) + `release.yml` step → + `resources/bin/scribe-overlay`. Additive: absent binary → hotkey fallback. +- ⚠ **To verify on-device**: squiggle accuracy in Slack + a browser (selection trick, scroll lag), + hover-menu placement + AX apply per app, no cursor disruption while typing, multi-monitor + coordinates. Then screenshots into the PR. +- **Not yet**: AXObserver-driven refresh (still 15fps poll — fine but not optimal), scroll re-measure + for selection apps, branded React card (native menu for v1), Google Docs (canvas — hotkey only). + +## Risks / open questions + +1. Cold-start browser AX (verify with quit-relaunch probe). 2. Card panel positioning over other +apps (fallback: native card). 3. Per-app quirks are the real cost ("98% of effort is macOS +integration" — TextWarden). 4. `AXEnhancedUserInterface` window side effect on browsers (set → +use → restore; measure impact). 5. Selection-trick cursor flicker if a measure escapes the +typing-pause gate. diff --git a/docs/WRITING_ASSISTANT_DESKTOP_IMPLEMENTATION.md b/docs/WRITING_ASSISTANT_DESKTOP_IMPLEMENTATION.md new file mode 100644 index 00000000..b808aca2 --- /dev/null +++ b/docs/WRITING_ASSISTANT_DESKTOP_IMPLEMENTATION.md @@ -0,0 +1,250 @@ +# Writing Assistant — Desktop implementation plan + +Scope: **desktop only** (this repo). Engine is built pure/portable so mobile lifts it later, but no mobile work here. Companion doc: `WRITING_ASSISTANT_PLAN.md` (why/what). This doc is the how. + +Guiding rules (from CLAUDE.md): engine depends on abstractions (injected LLM + store); reusable engine is Electron-free; pro feature code lives in `pro/`, core carries only the inert shell; reuse existing components before building; verify with `tsc` + tests each phase; PR evidence (screenshots/video) per surface. + +--- + +## 0. Architecture at a glance + +``` +packages/writing/ (PURE TS — no electron, no react) ← the portable engine + types.ts Issue, WritingSettings, RewriteRequest, engine + port interfaces + segment.ts sentence + token segmentation + spell/ SymSpell (symmetric-delete) + frequency wordlist asset + rules/ rule DSL + starter catalog (write-good/proselint-style) + postag.ts lightweight POS (lexicon + suffix heuristics) — optional v1 + proofread.ts RuleProofreader (sync) + LlmProofreader (via CompletionClient) + rewrite.ts LlmRewriter (tone/rephrase/translate/report) via CompletionClient + merge.ts dedup + rank + severity across rule/llm sources + ports.ts CompletionClient, ContextStore interfaces (injected) + index.ts WritingService facade + │ imports NOTHING from the app + ▼ +pro/main/writing/ (desktop main — the integration) + completion.ts CompletionClient impl over core llm / gateway + context.ts ContextStore impl over crm resolve/rag/preferences/observations + service.ts wires WritingService with the two adapters + settings persistence + ipc.ts writing:* IPC handlers + controller.ts WritingController (system-wide: hotkey→read→engine→overlay→apply) + read-selection.ts / apply-edit.ts native read + range-replace + ▼ +pro/renderer/ (desktop UI) + components/writing/writingApi.ts proInvoke/proOn wrapper (writing:*) + components/writing/AssistedTextarea textarea + squiggle overlay + popover + components/writing/RewriteToolbar selection actions (tone/rephrase/translate) + screens/WritingOverlay.tsx system-wide suggestions panel + settings/WritingSettings.tsx settings section +``` + +The two ports are the whole SOLID story: engine never imports Electron; desktop supplies `CompletionClient` (the model) and `ContextStore` (the shared brain). Mobile later supplies its own two adapters, zero engine change. + +**Placement decision:** build the engine as `packages/writing/` in this repo now (mirrors the existing local `packages/design/`), with the hard invariant of **zero Electron/React imports** so it lifts to `shared/@offgrid/writing` unchanged when mobile arrives. Flagged in §8. + +--- + +## Phase 0 — The engine (`packages/writing/`), pure + tested + +No app wiring. Builds and is tested standalone before anything imports it. + +**0.1 Contracts** (`types.ts`, `ports.ts`) +```ts +interface Issue { + span: [start: number, end: number]; + category: 'spelling'|'grammar'|'punctuation'|'style'|'clarity'|'tone'|'word-choice'; + severity: 'error'|'warning'|'suggestion'; + message: string; + replacements: string[]; + source: 'rules'|'llm'; + ruleId?: string; +} +interface CompletionClient { // injected — the model + json(prompt: string, schema: object, opts?: {maxTokens?: number}): Promise; + stream(prompt: string, onToken: (t: string) => void, opts?: {maxTokens?: number}): Promise; +} +interface ContextStore { // injected — the shared brain + knownTerms(): Promise>; // entity names+aliases → never-flag + styleContext(text: string): Promise;// RAG voice + learned-prefs doc + recordAccepted(edit: AcceptedEdit): Promise; + recordFeedback(f: EditFeedback): Promise; + teachTerm(term: string): Promise; +} +``` + +**0.2 Segmentation** (`segment.ts`) — sentence split + tokenizer with char offsets (spans must map back to the original string exactly). Pure, unit-tested on tricky cases (abbreviations, URLs, emoji). + +**0.3 SymSpell speller** (`spell/`) — build from scratch (algorithm public, §1a of the plan doc): +- precompute delete-index from a frequency-ranked wordlist asset (ship a compact EN list); +- lookup: query-deletes → candidates → verify Damerau-Levenshtein ≤2 → rank by frequency; +- respect `customDictionary` + `ContextStore.knownTerms()` (never flag those). +- Unit tests: known misspellings → expected top suggestion; known words → no flag; custom term → no flag. + +**0.4 Rule engine** (`rules/`) — our own compact DSL (JSON/TS), not LanguageTool XML. Each rule: matcher (literal | regex | POS | phrase-map) → Issue with message + replacements. Starter catalog = the write-good/proselint set: +- passive voice, weasel words, adverb-weakeners, wordy phrases (`in order to`→`to`), repeated words, clichés, sentence-start `so`/`there is`, double spaces, punctuation basics. +- Each rule is a pure function `(tokens, text) => Issue[]`; catalog is data. Unit-test each rule with a positive + negative case (regression guard per CLAUDE.md). + +**0.5 POS tagger** (`postag.ts`) — lexicon lookup + suffix heuristics (`-ly`→RB, `-ing`→VBG) + a few Brill-style fixups. Only needed by grammar rules that require word class; v1 can ship the regex/word-list rules without it and add POS-dependent rules later. Decision in §8. + +**0.6 LLM proofread + rewrite** (`proofread.ts`, `rewrite.ts`) — CoEdIT-style instruction prompts: +- `LlmProofreader`: `CompletionClient.json` with a strict schema returning `{issues:[{span,category,message,replacements}]}` over a bounded chunk; for heavy grammar/GEC the rule engine can't do. +- `LlmRewriter`: `CompletionClient.stream` with templated instructions built from `WritingSettings` (tone/audience/formality/domain) + `ContextStore.styleContext()` prepended. Actions: rewrite, tone-shift, shorten, expand, simplify, translate, report. +- Prompt templates guarded by a regression test that reads the source (per CLAUDE.md `extract-prompt.test.ts` pattern). + +**0.7 Merge + facade** (`merge.ts`, `index.ts`) — dedup overlapping spans (rules win on spelling, llm on grammar), rank by severity+position. `WritingService` exposes `checkSync(text)` (rules only, instant), `checkFull(text)` (rules+llm), `rewrite(req, onToken)`, `report(text)`. + +**Deliverable:** `packages/writing` builds; `npm test` green; zero app imports. Nothing wired yet. + +--- + +## Phase 1 — Main-process integration (`pro/main/writing/`) + +**1.0 (first task) Verify the model transport.** Confirm `@offgrid/core/main/llm` supports (a) grammar/JSON-constrained output and (b) token streaming. If yes, `completion.ts` wraps `llm.chat`. If not, wrap the gateway directly: `callLlamaJson` (JSON) + `proxyToLlama`/SSE (stream) from `src/main/model-server.ts`, with `chat_template_kwargs:{enable_thinking:false}` + `response_format` for the JSON path. This unblocks 0.6's assumptions. + +**1.1 `completion.ts`** — `CompletionClient` impl (json + stream) over the chosen transport. + +**1.2 `context.ts`** — `ContextStore` impl over the shared brain (all seams already exist): +- `knownTerms()` ← entity names+aliases (`pro/main/crm/resolve.ts`). +- `styleContext(text)` ← `ragService.searchProject(...)` (`src/main/rag/index.ts`) + `getPreferenceDoc()` (`pro/main/crm/preferences.ts`). +- `recordAccepted()` ← `recordObservation({surface:'WritingAssistant', ...})` (`pro/main/crm/observations.ts`). +- `recordFeedback()` ← `recordFeedback()`; `teachTerm()` ← `addAlias()`. + +**1.3 `service.ts`** — instantiate `WritingService(completion, context)`; load/persist `WritingSettings` via core settings store. + +**1.4 `ipc.ts`** — `writing:*` handlers (mirror `pro/main/dictation/ipc.ts`): +`writing:get-settings` / `set-settings` / `check` (sync rules) / `check-full` / `rewrite` (streams via `writing:rewrite:token` events) / `report` / `apply` / `teach-term` / `feedback` / `invoke-systemwide`. + +**1.5** Register `setupWritingIpc()` + `writingService.start()` in `pro/main/services.ts` (next to dictation). + +**Tests:** `context.ts` against a temp SQLite + LanceDB (integration, per CLAUDE.md real-collaborator preference); `completion.ts` transport smoke test. + +--- + +## Phase 2 — In-app UI (`pro/renderer/`), the everyday surface + +**2.1 `writingApi.ts`** — proInvoke/proOn wrapper for `writing:*` (mirror `voiceApi.ts`). + +**2.2 `AssistedTextarea` (the one hard component, built once, reused).** Reuse check: no existing overlay/decoration component exists → net-new, justified. Technique: a `pointer-events-none` mirror `
` absolutely positioned over the `