fix(db): stop SQLITE_BUSY 500s — WAL journal mode, drop redundant ingest writes, transcript retention

[miguelrisero]

What was actually wrong

The "internal error" on workspace creation was never a workspace-creation bug. It was a 500 from SQLITE_BUSY.

From vk.log:

POST /api/workspaces/start → 500
  error=Container(Sqlx(SqliteError { code: 5, message: "database is locked" }))
POST /api/workspaces/start → 500
  error=Workspace(Database(SqliteError { code: 5, message: "database is locked" }))

2,150 lock errors in the log; 1,218 on Jul 25 alone, up from 14 on Jul 22.

Root cause

journal_mode = DELETE gives SQLite one writer whose exclusive lock blocks every reader. Three background services write continuously (transcript ingest, CliActivityMonitor at 2s, registry reconcile + outbox poll at 30s, pr_monitor) against a database that had grown to 215 MB. Requests spent their entire busy timeout queued behind that lock and then failed.

The decisive evidence is that most slow statements are reads: of 14,071 slow statements, ~9,000 are SELECTs. Readers were not slow — they were blocked.

slow statement	count
UPDATE cli_native_files SET discovered_workspace_id …	3,643
SELECT id as "id!: …	3,351
SELECT id, workspace_id, session_id, …	2,997
SELECT f.id, f.claude_session_id, f.dir_path, …	1,402

Changes

1. WAL journal mode + 30s busy timeout (crates/db/src/lib.rs)

Readers stop blocking the writer, which addresses ~2/3 of the contention directly. Busy timeout goes from sqlx's 5s default to 30s so contention delays a request rather than failing it. synchronous=NORMAL pairs with WAL to drop an fsync per commit.

Upstream pinned delete in BloopAI#1882 after reverting BloopAI#1806, recording no reason — the PR body is just This reverts commit 25c6d0a7…, with no discussion and no matching issue. The plausible reason is portability: WAL needs an mmap-able -shm sidecar and fails on some network mounts. Rather than assume either way, this defaults to WAL, falls back to delete automatically if WAL cannot be established, and adds VIBE_KANBAN_SQLITE_JOURNAL_MODE to force either mode. All three connection sites now route through one helper.

2. Stop rewriting unchanged registrations (crates/db/src/models/cli_native_file.rs)

register() wrote unconditionally on every scan of every file — taking the single write lock just to restamp updated_at, which nothing reads and which a real import restamps anyway — then re-read the row it had just written. The top slow statement in the log. Now compares against the row find_latest_by_path already loaded and skips both round trips when nothing changed.

3. Transcript retention (prune_unreachable + a sweep in the ingest service)

Nothing ever reclaimed raw transcript records: 196 MB of a 215 MB database, accumulated in six days. cli_native_files has no foreign key to sessions, so deleting a session stranded its transcript forever — and the cascades that would have collected the rest never fire, because PRAGMA foreign_keys defaults to off on every connection. That is also why the sweep deletes children explicitly rather than relying on ON DELETE CASCADE.

A file is prunable only once it is past the window and unreachable: no claude_session_links row (so nothing can render it, and the unassigned-CLI adoption list has had the full window to claim it) and no outbox row belonging to a still-existing session (so no live feed can replay it). Sweeps are capped per run with one transaction per file, since holding the write lock is the problem retention exists to relieve.

Defaults to 14 days; VIBE_KANBAN_CLI_TRANSCRIPT_RETENTION_DAYS tunes it, 0 disables.

4. Surface contention instead of hiding it (crates/server/src/error.rs)

Every DB error mapped to ErrorInfo::internal, so losing a write race answered 500 / "An internal error occurred." Contention is retryable and the server is healthy, so it now answers 503 with a message that says so. Matching walks the error source chain rather than adding a case per variant, since the same sqlx error arrives wrapped in Container, Workspace or ScratchError depending on route.

Two details the tests pin down, both of which silently defeated an earlier attempt:

• sqlx implements StdError for Box<dyn DatabaseError>, so the chain element is that box, not the concrete SqliteError — downcasting to SqliteError never matches.
• sqlx reports SQLite's extended result code, so the primary code must be masked from the low byte. WAL raises SQLITE_BUSY_SNAPSHOT (517), not 5.

Deliberately not done

• No data deleted. Retention reclaims nothing today: every transcript file is ≤7 days old, so no sane window matches anything yet. It exists to stop unbounded growth going forward.
• The 47 orphan transcript files (86 MB) are left in place. They back the unassigned-CLI adoption list, so deleting them would discard sessions that can still be claimed. Retention will collect them once they age past the window.
• PRAGMA foreign_keys is left off. Turning it on globally could start failing writes that currently succeed against possibly-already-violating data — too broad to bundle into a contention fix. Worth its own change.

Testing

612 workspace tests pass; cargo clippy --workspace --all-targets clean.

New coverage:

• WAL and the 30s timeout are the pragmas SQLite actually settles on (journal_mode is a property of the file, so requesting WAL ≠ getting it), plus the delete fallback.
• An open read transaction does not block a writer under WAL — the behaviour the change exists for.
• register() does not write when stats are unchanged, does when they change, and records first workspace attribution exactly once.
• prune_unreachable removes unreachable transcripts past the window, and spares those inside it, those a session still owns, and those a live session can replay; sweeps respect the file limit.
• Contention is detected through each wrapper a real route uses, using a genuine SQLITE_BUSY provoked from a held BEGIN EXCLUSIVE — SqliteError cannot be constructed outside sqlx, and a hand-built chain would not have caught either of the two bugs above. A real primary-key violation confirms the masking does not over-match.