Lip sync: animate the avatar's mouth while TTS audio plays

[noisy]

Depends on #19 (per-voice avatars).

Idea

While the daemon plays a TTS utterance, the current voice's avatar moves its mouth in sync with the audio. The dashboard comes alive — you can see which character is speaking, in the rhythm of real speech.

Library candidates (researched 2026-07-20)

• lipsync-engine — favorite. Browser-native, zero dependencies (~15 KB, MIT), renderer-agnostic: feed it audio (an <audio> element, a MediaStream, or streaming PCM via feedAudio()) and it emits real-time viseme events (15 Oculus/MPEG-4 shapes or a simple A–F set). Ships a sprite-sheet CanvasRenderer — a 1:1 fit for our avatars (add a mouth-shape strip per character).
• Rhubarb Lip Sync — offline CLI: WAV → a timeline of 6–9 mouth shapes designed for sprite-swap animation. The daemon has the TTS MP3 BEFORE playback, so it could compute the timeline server-side and ship it to the dashboard with the audio. More accurate than live spectral analysis, at the cost of a preprocessing step.
• wawa-lipsync — similar to lipsync-engine (Web Audio API → visemes) but aimed at 3D (Three.js / React Three Fiber). An option if we ever go 3D.
• Heavier / photorealistic (likely overkill): NVIDIA Audio2Face (open source, 3D + GPU), MuseTalk / LivePortrait (lip-syncing a real face, GPU).

Note: this was a one-day research pass — before implementing, look around again; something better or newer may exist.

Implementation notes

• Minimal plan: lipsync-engine in the dashboard, attached to the same audio element that plays speak utterances — zero daemon changes.
• Upgrade path: Rhubarb timelines computed in the daemon if live-analysis quality disappoints.
• Mouth sprites: a separate strip of shapes (A–F is enough to start) in the same HUD style as the avatars.

🤖 Generated with Claude Code