feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433
Open
dishit-wednesday wants to merge 91 commits into
Open
feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433dishit-wednesday wants to merge 91 commits into
dishit-wednesday wants to merge 91 commits into
Conversation
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…, stop, native log) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…option (core) whisperService.transcribeFile gains offset/duration so a recording can be transcribed in chunks, and streams segments live via onNewSegments (cumulative result + per-segment t0/t1). loadModel takes useGpu/useFlashAttn/useCoreML and passes them to initWhisper (useCoreMLIos on iOS). Bumps the pro submodule to the chunked/resumable transcription experience. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…header Replaces the Settings bottom tab with a Memory tab. MemoryTabScreen renders the Pro recorder when it is registered (pro.activate) and a paywall otherwise, read reactively so unlocking Pro swaps it in without a restart - the tab shell stays in core, the recorder injects from the submodule. Settings becomes a pushed RootStack screen reached from a gear in the Home header. Updates nav types and the onboarding step that pointed at the old SettingsTab, and the SettingsScreen nav-prop type (no longer a tab). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Adds a map-reduce summarizer (transcriptSummarizer) for text too large for the context window: split into context-sized chunks, summarize each, then combine (recursing if needed) - so the whole transcript is read instead of truncated to the tail like contextCompaction does. A Summarize action on a document/transcript attachment chip streams progress (part i/N, combining) into an assistant message, then the final summary. MediaAttachment gains recordingId + transcript range metadata, and a one-shot chatAttachmentInbox lets the Pro recorder hand a transcript to the composer across the navigation boundary. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summarizing an oversized transcript used to show a static "part i of N" counter for minutes and only stream on the final combine pass, and the Summarize button was clipped invisible inside the 60x60 attachment chip. - generateWithMaxTokens gains an optional onToken callback. - transcriptSummarizer streams each map part as it is written (new 'combining' phase marks the final pass), so text flows from part 1, not part 3. - useSummarizeAttachment renders the live work (parts stack while streaming, then the combined answer streams over the top) on a 50ms flush so the JS thread is not saturated by per-token deep store updates. - Document/transcript chips get a larger 168x76 layout with a filled, visible Summarize button and a spinner+label busy state. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…er.rn 0.5.5) Hoist the segment-callback user_data to the dispatch-block scope so live segment streaming on the iOS file-transcribe path does not deref a dangling pointer; add an NSLog marker to verify the patch is compiled into a build. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…d project Thread per-chunk metadata (recordingId/startMs/eventTitle) through chunking, the SQLite store, and retrieval; add indexText() to index a raw transcript; seed a 'Recordings' project via ensureProject so recordings can be asked about. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Move the exact timestamp out of the system prompt (which busted llama.rn's prefix cache every turn) and onto the latest user message; keep only the stable date in the system prompt so the ~800-token system+tools prefix is reused. Add tool-loop diagnostic logging. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… model Guard CoreML so it falls back to CPU when the per-model encoder asset is absent (was crashing on A12); force 'en' for English-only ggml-*.en models; add the small.en-tdrz model entry; re-enable iOS live segment streaming. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Used by transcribeChunked to log per-chunk memory footprint. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…tream fast On CPU-only low-RAM devices, reasoning/preamble burned the whole token budget before any summary streamed. Force the thinking channel off for internal generations (summarize, tool-selection), instruct the prompts to emit no preamble, strip any leaked control tokens, and cap each MAP chunk to ~1500 input tokens so each part prefills in under a minute and starts streaming sooner. Reduce/combine passes still use the full context budget. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Snapshot RAM before and after loadModel. On 4 GB iOS devices a large whisper model can push the app past the jetsam limit and the OS kills it mid-load, which presents as a crash; the before/after pair localizes a kill to load vs transcription. Fire-and-forget so it adds no await points to the load critical path. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The tab and paywall are about on-device recording/transcription, so call it Recorder. Tab label, testID (recorder-tab), and paywall title updated. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
loadModel gained an options param (CoreML/lang config); update the store tests to match the new call signature. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Turn the bare gating card into a feature list - always-on recording (Android), on-device transcription, summaries, calendar context - with a privacy line and the unlock CTA. Copy follows the brand voice (privacy-as-mechanism, no jargon). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Points the pro submodule at the locket recorder feature set: continuous recorder, on-device transcription, calendar matching, summaries, knowledge-base sync, and the recording-detail/list UI polish. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Update tests left stale by intended code changes, and add coverage for new
behavior. No production code changed; each assertion was verified against the
current implementation (the source of truth), not weakened to pass.
- nav/onboarding: Memory tab renamed to Recorder (recorder-tab), and Settings
moved off the tab bar to a Home-header route, so STEP_TAB_MAP.exploredSettings
is 'Settings' and the 5th tab label/testID is Recorder.
- whisper: loadModel now forwards {useGpu,useFlashAttn,useCoreMLIos}; .en models
force language 'en' (plain .bin auto-detects).
- rag: schema gained a metadata column (4th CREATE = ALTER migration; 4th insert
bind = metadata, null when absent).
- generationToolLoop: the precise time-of-day note now rides on the latest user
message (keeps the system+tools prefix cacheable), not the system prompt.
- whisperService.branches: reset RNFS exists/stat mocks per test - clearAllMocks
does not drain the mockResolvedValueOnce queue, so onces leaked between tests.
Added: CoreML-encoder-missing -> CPU fallback guard, and RAG metadata JSON
serialization on the write path.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
RNWhisper.mm's vadDetectSpeech/vadDetectSpeechFile decode the WAV to a malloc'd float PCM buffer and pass it to detectSpeech synchronously, but never free it. Over a long recording that leaks ~one slice of PCM per call and OOMs. Add free(data) after the synchronous detectSpeech consumes it, in both methods. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Add ungated controls to clear the cached/stored Pro license so the free -> activate flow can be re-tested on a release build (where the __DEV__ toggle is inert). The Keygen machine slot stays claimed (fingerprint persists), so re-pasting the key re-activates instantly. Marked TEMPORARY - re-gate behind __DEV__ or remove before shipping. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Point at the pro commits adding the recorder transcription/archive UI polish and the speech/noise/silence cleanup classifier. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The ChatInput render fn was 358 lines (limit 350) - the #431 lint blocker. Lift the send/stop/voice action button into a module-scope ActionButton component (derives colors/styles from the same hooks; behaviour identical to the previous inline ternary). Render fn now ~343 lines. 91/91 ChatInput tests pass, tsc clean. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The submodule URL still referenced the old wednesday-solutions/offgrid-pro repo. Point it at the canonical off-grid-ai/mobile-pro so CI and fresh clones fetch the pro recorder code from the right place. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…a, prompt fix, recorder baseline) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…p (ec24cac) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… default whisperStore deleteModel/deleteModelById now fall back to another on-disk model (mock listDownloadedModels); loadModel auto-enables CoreML when the encoder asset is present. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
When a dev-grammar completion fails and retries without the grammar, the streaming state carried over from the failed attempt, duplicating/garbling the retry's output (and re-emitting via onStream). Reset fullContent/fullReasoningContent/streamed* in generateResponse, and fullResponse/collectedToolCalls + the ToolCallTokenFilter buffer in generateWithToolsImpl. Matches the existing reset in generateWithMaxTokens. (Gitar code-review finding.) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…-transcribe off + recorder comment) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… delete deleteModel updated downloadedModelId (fallback) but left the deleted model in presentModelIds, so the models list kept showing a model whose file is gone. Recompute presentModelIds from disk, matching deleteModelById. (Gitar re-review finding.) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ropout) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ndoff doc Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…oisy audio whisper.cpp -mc 0: don't condition on previously-decoded text, which sent it into a repeat-the-same-token spiral on ambient clips. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…-gated-recorder # Conflicts: # .gitignore # android/app/src/main/assets/ggml-hexagon/libggml-htp-v69.so # android/app/src/main/assets/ggml-hexagon/libggml-htp-v73.so # android/app/src/main/assets/ggml-hexagon/libggml-htp-v75.so # android/app/src/main/assets/ggml-hexagon/libggml-htp-v79.so # android/app/src/main/assets/ggml-hexagon/libggml-htp-v81.so # pro # src/services/llmHelpers.ts # src/stores/whisperStore.ts
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summaries prefer a remote provider whenever active (offloads generation off-device to save battery/RAM); does not gate on a local model being loaded. Chat generation keeps its own local-first policy. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Forwards a short proper-noun/jargon list to whisper.cpp as its initial prompt so names are spelled the user's way. Pairs with the pro-side vocabulary setting. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…GC-time UAF Insight generation churns conversations (reset->close->recreate). Each finished generation leaves fbjni HybridData peers to be reclaimed on the GC 'HybridData Dest' thread; under churn that reclaim can fire mid-decode of the next conversation and dereference freed memory (observed SIGSEGV, poison 0x01). Drain the reference queue at the quiescent teardown point (gc + runFinalization + brief yield) so reclaims do not overlap live native work. Includes temporary DIAG-UAF lifecycle logging to confirm the fix on-device; strip before merge. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…, permissions, iOS recorder, VAD teardown fix) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Pure selectTextModelToLoad(models, budgetMB, { activeId, footprintMB }): active-if-fits -> largest-that-fits -> smallest. Footprint is injected (caller passes hardwareService.estimateModelRam) so weights + mmproj + overhead count the one canonical way. Unit-tested for every branch.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
loadTextModel({ textOnly: true }) skips loading the mmproj (saving its RAM) and sizes the residency budget on the gguf weights alone. Gated - chat still loads vision by default. Guards the post-load mmproj-clear so a text-only load never erases a model's persisted vision link.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…n prompt) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…s, auto-prune) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… redesing.md) android/temp.txt was a 1532-line AI planning transcript; redesing.md a stray design note with broken image refs. Neither belongs in the repo (flagged by review). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summaries prefer a remote provider, so a local-only stopGeneration left a remote completion running after Stop. abort() now also calls the active provider's stopGeneration. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ot path Removes the DIAG-UAF create/send/close Log.i lines and the convSeq/currentConvSeq bookkeeping added to diagnose the GC-time HybridData UAF. Keeps the mitigation itself (gc + runFinalization drain at close). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…prompt) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…nscription
cleanTranscription only kept text containing [a-z0-9], so a Hindi/Arabic/CJK
transcript (no ASCII letters) was wiped to '' - silent data loss for non-English
users. Match any-script letters/digits (\p{L}\p{N}) instead. Also: the single
shared Whisper context can't run two file transcriptions at once, so a second
overlapping request now throws WhisperBusyError instead of orphaning the first
job's cancel handle, and unloadModel cancels an in-flight file transcribe before
releasing the context (prevents a use-after-free SIGSEGV).
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…budget Expose the residency manager (the one owner of the RAM cap + load gate) from the services barrel so callers that auto-pick a model to load can budget against the SAME number the loader enforces (modelResidencyManager.getBudgetMB), instead of a divergent hardcoded fraction. Used by memory-aware auto-load callers. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The pro recorder/insights feature lives in the private pro/ submodule; its architecture, product-design handoffs, and transcript-quality R&D notes must not ship in the public core repo. Remove the 22 locket/recorder/insights/transcript planning + handoff docs from docs/plans/. Keeps only genuinely-core planning (backend-per-device, whisper-download-sync). Tree-only removal; a full history scrub, if needed, is a separate step at the public cutover. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
feat(core): summarizer streaming/dynamic-context + whisper model fallback + pro bump [WIP/divergent]
217a2a0 to
79f0ee8
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



TL;DR - Open-core host that wires in the private
pro/recorder submodule and shipsthe shared on-device transcription + retrieval engine it relies on. Recorder feature code
stays closed-source in
pro/; this PR is the public wiring + shared services. Pairs withoff-grid-ai/mobile-pro#11 (the submodule pointer references a commit on that pro branch, so
they land together).
flowchart TB subgraph pro["pro/ submodule (private, autolinked)"] rec["Recorder + AudioNormalizer"] end subgraph core["core (public host)"] slot["MemoryTabScreen<br/>paywall + registry slot"] ws["whisperService<br/>windowed transcribe + CoreML guard"] rag["RAG 'Recordings' KB<br/>metadata column + migration"] chat["chat: summarize attachment"] end slot -. "registered-screen slot" .-> rec rec -->|"16k mono WAV"| ws ws -->|"transcript + t0/t1 segments"| rag ws -->|"transcript"| chatWhat's in this PR
native autolink (
react-native.config.js); iOS background-audio mode..enlanguage forcing; iOS streaming-lifetime patch (whisper.rn+0.5.5).metadatacolumn + migration);summarize oversized transcript/document attachments.
.gitmodulesre-pointed to canonicaloff-grid-ai/mobile-pro;submodule pointer bumped to the merged pro commit.
How it works (expand for detail)
Transcription pipeline (whisper.cpp via whisper.rn)
transcribeFiledrives whisper.rn with an{offset, duration}window so long recordings transcribe in chunks and resume from acheckpoint (last finished chunk) instead of reloading the whole file. Live cumulative
segments surface through whisper.rn's
onNewSegments(nativenew_segment_callback),wired only when
nProcessors == 1(parallel decode cannot stream in order).(
transcribeFile-> ObjCtranscribeData) crashed becausenew_segment_callback'suser_datawas a stack struct freed before the callback fired. The patch hoists it so itoutlives the call; compiled into the iOS binary, so live segments work on both platforms.
ggml-<model>-encoder.mlmodelcbundle makes whisper.rn fail CoreML init, fall back toCPU, then crash at transcribe 0% on some devices (A12 / iPhone XS).
loadModelchecks theencoder asset exists and downgrades
useCoreMLIosto false when missing, so the toggle isa no-op instead of a crash.
.enmodels carry only English tokens; auto-detect(returns codes like "tg") or an explicit non-English pick makes whisper unstable on iOS
(crash at 0%, 762s for 13%, garbled).
transcribeFileforceslanguage: 'en'for anymodel whose catalogue
lang === 'en', falling back to the.en.binfilename convention.Retrieval (recordings knowledge base)
Transcripts index into a local RAG "Recordings" KB.
rag_chunksgains ametadata TEXTcolumn (recordingId, startMs, eventTitle, ...) via a guarded
ALTER TABLEmigration forolder installs; chunk metadata is JSON-serialized on write.
Integration / build
react-native.config.js(onlywhen the pro android/iOS sources are present), so closed-source code never leaks into the
public repo but builds when the submodule is checked out.
.gitmodulesre-pointed from the oldwednesday-solutions/offgrid-proto the canonicaloff-grid-ai/mobile-pro; submodule pointer bumped to the merged pro commit.Testing
npm run lint(0 errors),tsc --noEmit(clean), and the full jest suite pass; thepre-push hook ran them, no
--no-verifyused. Added CoreML-fallback + RAG-metadata coverage.Follow-ups (not blocking)
MemoryTabScreenlooks up the recorder by the concrete proscreen name; move it behind an abstract slot key so public never names a pro internal.
max-lines: deferred with a TODO; clearing it means moving modeldownload/management into its own module (~11 call sites across core + pro).
Reviewer's guide (commits by theme)
The branch is chronological; this groups the commits by area.
recorder/paywall) + move Settings to Home header; autolink pro native + pod install; iOS
background-audio mode.
segments + CoreML; CoreML asset guard +
.enforcing + tdrz; iOS streaming lifetimepatch; memory-snapshot logging;
whisperremote-server provider type.oversized attachments; stream summaries live.
off-grid-ai/mobile-pro.