feat: local-first Videoboom — on-device MLX pipeline + optional BYOK cloud, per-stage auto-config

[lBroth]

Replaces #1 and #2, which are closed in favour of this one. Everything in both is here: #1's head was an ancestor of this branch, and #2 has been rebased on top with its two conflicts resolved (see "Merge note" below). Nothing was dropped — verified with git rev-list, all previous branches had zero unique commits.

What this is

Videoboom becomes local-first. A resident Python MLX sidecar on 127.0.0.1 runs the heavy stages on-device (Wan 2.2 i2v, FLUX keyframes, Whisper STT, Qwen LLM, Gemma VLM, RIFE, Real-ESRGAN), with cloud kept as an optional bring-your-own-key path resolved per stage. 82 commits, 94 files.

The per-stage auto-config resolver decides local-vs-cloud per stage from what is actually available — models on disk, RAM, and whether the user supplied keys — and emits the decision as VB_* env into the render process. There is a hard RAM gate at 48 GB (measured peaks: bf16-relay 32.6 GB, FastWan-5B 54.8 GB at 121f, Q4 67.7 GB) that refuses rather than silently falling back to cloud.

The video path

Quality is Wan 2.2 I2V-A14B in bf16 through a vendored "relay" fork of the MLX diffusion loop that keeps only one expert of the dual-expert model resident at a time — that is what makes a 14B model fit in 48 GB (32.6 GB peak) where the quantized build did not (67.7 GB, swaps). Lightning 4-step LoRA with CFG off, at 832x480, then RIFE interpolation and Real-ESRGAN to 1080p plus a filmic grade.

Measured on an M5 Pro 48 GB: 130.0 s for a 37-frame clip (110.0 s denoise, 27.5 s/step), which is ~1.2-1.4 h per minute of finished video. First+last-frame conditioning is always on — each scene morphs toward the next scene's keyframe, so boundaries are seamless rather than chained hard cuts.

Merge note (#2)

renderLocalScene/renderClip had gained a new last argument on both branches independently, for two different things, so the cherry-pick collided there. Both are kept and they are not interchangeable:

• endImg — the next scene's keyframe, passed to the model as end_image so the scene morphs toward it. Only the sub-clip that ends a scene gets it.
• anchor — the still that extracted last frames are pulled back toward with matchLevels, to bound exposure/saturation drift along a chain. It is the scene's own keyframe, or the anchor of the CUT scene that started the run.

Resolved as (..., endImg?: string, anchor: string | null = null) with the driver computing both.

Fixes included

#2's eight render-chain fixes, all of which were silent in production — nothing crashed, the video just came out worse than the code was capable of. The two worst: setEnv was a merge rather than a per-op replace, so one click of "Re-render · Quality" injected VB_LOCAL_WAN_STEPS=20 and poisoned every later render in the process; and RIFE targeted a fixed 2x, taking the 14B's 16 fps to 32, where the 24 fps conform then dropped one frame in four at uneven phase — a repeating cadence break on the default model.

Plus one found while auditing the MLX paths: the Settings "Download" button provisioned the wrong video engine. download.py picks the engine off VB_LOCAL_VIDEO_MODEL and falls back to '5b', but that variable is only ever emitted into the render spawn env, never process.env, and localEnv() does not carry it. So Download fetched the retired FastWan-5B (~24 GB) and wrote .model-path-5b, while videoReady() looked for .model-path — the VIDEO stage never went ready, and re-clicking repeated the 24 GB download.

Verification

npm test passes: typecheck clean, 33/33 unit tests (21 existing plus 12 added by #2's test/render-quality.test.ts), and the no-cloud check.

Note for reviewers: test/scene-shot-routing.test.ts depends on .vbdata-test/, which is gitignored, so it fails in a fresh clone or a git worktree where that fixture is absent. It passes in a normal working copy. Worth fixing separately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NkT9zMsGYFw6sJ39Q9mjZZ

[lBroth]

Audit backlog — verified defects still open on this branch

A four-dimension audit of the sidecar, the render chain, the Electron main process and the env/contract surface produced 16 candidates; each was then re-checked adversarially against this branch. Four were rejected as stale or not real, three are fixed in f52a357, and the rest are below. Everything here has a concrete user-visible failure, not a code smell.

Two candidates were rejected specifically because #2's fixes already cover them: the setEnv env-poisoning bug and the swallowed RIFE failure. They were reported against the pre-#2 tree.

Medium

tiny-VAE monkeypatch is permanent, so Quality silently decodes with the tiny VAE — local/tiny_vae.py:97 exposes patch() with no unpatch(), and it rebinds a module global for the life of the process. Render one clip on Fast, switch Settings to Quality, render again: the user pays the full 40-step denoise (~38 min/clip measured) but the frames still come out of the 22 MB TAEHV approximation instead of the official Wan VAE — which is the entire point of the Quality tier. Fix: save the original on first patch and add unpatch(), called per request from run_i2v based on tiny_vae.

assemble() drops pending scenes, desyncing picture from song — src/engine/pipeline.ts:596 skips any scene that is not done/failed, but the audio is only truncated. Tick scenes #10-#12 of 30 in the editor and hit "Render selezionate": the assembled video plays the chorus footage over the intro. Fix: still-fill pending scenes that precede the last rendered index instead of skipping them.

refreshScene loses both morph targets and re-renders the neighbour for nothing — src/engine/backends/local/video.ts:234 and :238 call renderClip without endImg/anchor, so "Regenerate scene" drops the seamless boundary, and it additionally re-renders scene k-1 with byte-identical inputs (same keyframe, same prompt, same hardcoded seed 42) — minutes of 14B GPU time for an identical result. Fix: mirror the cloud backend and pass the on-disk next keyframe.

Morph target ignores keyframes already on disk — src/engine/backends/local/video.ts:156 reads kfPaths, which is populated only over the scenes being rendered this pass. Rendering a subset from the editor gives the last scene of the subset a hard visual jump into the next, even though that keyframe exists on disk. Fix: fall back to S.mediaExists(...) / keyframePath(pid, k + 1).

A failed upscale leaks tens of GB and says nothing — local/upscale.py:65 dumps every frame of the whole timeline as PNGs (~15-20 GB for a full song) with the cleanup outside a try/finally. If it runs out of disk or hits the watchdog, the user gets a soft 832x480 master with no message after an hour-plus render, and the frame dump stays on disk. Fix: try/finally: shutil.rmtree(...), chunk the extraction, and surface the failure as an event.

Low

Storyboard keyframes thrash the single-resident manager — local/manager.py:38 evicts every other heavy model on each get(), and since the shot-routing change the Fase A loop alternates between the cast-ref model and the plain one, so a 30-scene storyboard can reload a ~10 GB model on every alternation. Fix: partition the loop by refsForScene(p, s).length > 0 so each variant loads once.

test/scene-shot-routing.test.ts depends on a gitignored fixture — it reads .vbdata-test/, which .gitignore:35 excludes, so the suite is green in a normal working copy and red in a fresh clone, in CI, or in a git worktree. Fix: build the fixture in the test, or commit a small tracked one.

Rejected on inspection

The /i2v GPU-lock-past-deadline concern and the "upscale scratch is left in the project directory" report did not survive verification — the scratch is under $TMPDIR/videoboom, and on the default configuration the lock case does not produce the described cascade.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NkT9zMsGYFw6sJ39Q9mjZZ

[lBroth]

Whole-project review — 21 confirmed, 19 dismissed

A five-dimension review (cross-platform reality, docs vs code, security, unhappy-path UX, dead weight), each finding then re-checked adversarially against the code. Six verifier agents died on connection errors, so a handful of items below carry their reviewer's evidence but no independent second read — flagged where that is the case.

Two were fixed in a3d98b4: interrupted renders leaving a project permanently stuck with no Delete, and local/setup.sh provisioning the Q4 model the app cannot run. The rest is below, grouped by what it costs.

Ships wrong behaviour

Windows/Linux onboarding claims every stage is "Local · private" — renderer/Onboarding.tsx. A user finishes the wizard on Windows, pastes both keys, and the summary screen says Lyric timing / Story / Keyframes / Video / Face caption all run locally. Nothing local can run there: localCapabilities() requires darwin+arm64. The first render then refuses with a message contradicting what they were just shown.

Deleting a character leaves dangling ids in every scene — src/main/index.ts. character:delete scrubs the id from each project's cast, but scene JSONs keep it in scene.characters. assertCastExists inspects only p.cast and passes; refsForScene then resolves the id against media that is gone. A re-render silently stars a different person, or none.

Scene-editor save and image-replace swallow their errors — renderer/App.tsx. Both wrap setSceneKeyframe in try/finally with no catch. Its real, user-caused failures — 'image not found', 'could not import that image' (an HEIC the bundled ffmpeg cannot decode, a file on an unmounted volume) — vanish. The dialog closes as if it worked.

Security

The sidecar accepts jobs from anything on the machine — local/server.py. Fixed port 127.0.0.1:8765, no auth, no Origin check, no Host check. Any other process running as the user, and any web page in a browser that does not gate local-network requests, can drive it while the app is open. The handlers take caller-controlled filesystem paths. Reported as verified live. An Origin/Host reject plus a per-launch token in the spawn env would close it without changing the client contract.

Documentation that would send someone into a wall

AGENTS.md orders the deletion of the product. It is the file every contributor and agent is told to obey, and it still describes a local-only, no-keys app — instructing that src/engine/cloud/* be removed. That code is the shipped BYOK path.

Every doc still says Fast = FastWan-5B. The 5B engine was retired from the UI; both tiers are the 14B now, separated by localQuality. docs/LOCAL-MODELS.md, docs/MODELS.md, docs/ROADMAP.md and the renderer's type comments all disagree with settingsSchema.ts:28.

local/README.md documents the first BYOK-era prototype — 720p defaults, 40 steps, 81 frames, "everything else stays cloud". docs/LOCAL-MODELS.md:9 points at it as the current tunables page, so its numbers get used. Every video default in its table is wrong.

Docs say local i2v has no last-frame conditioning. First+last morph has been unconditionally on since d436ec9. Anyone reasoning about scene seams reasons from a design that was replaced.

The landing page promises "0 keys required to start" and never mentions the 48 GB floor — site/index.html. Every Apple Silicon Mac below 48 GB, which is most of them, fails the gate and is told to add an OpenRouter key.

docs/FOLDER-STRUCTURE.md omits src/engine/cloud/, src/engine/backends/, src/shared/, src/engine/cost.ts and five src/main/ modules.

local/requirements.txt says torch is convert-time only. It is a runtime dependency of the default render path (the TAEHV tiny-VAE runs on torch-MPS), 437 MB installed, and vendor-runtime.sh bundles it deliberately.

Size

Every installer carries all three platforms' ffprobe binaries — ffprobe-static ships darwin + linux + win32 in one package (335 MB on disk), electron-builder copies production deps wholesale, and asarUnpack forces the tree out unpacked. Roughly 200 MB of dead weight in every install, on every OS.

Notable dismissals

Checked and found not to be problems: the deny-by-default firewall does cover the paths that reach providers; renderer-supplied keys are contained; Linux safeStorage does not silently fall back to plaintext in a way that matters here; local/download.py's '5b' default is now unreachable from the app.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NkT9zMsGYFw6sJ39Q9mjZZ