Skip to content

feat(playback): burst-aware budgets — fix 8-cell cache-starvation freezes - #49

Merged
tcconnally merged 1 commit into
mainfrom
fix/burst-aware-budgets
Jul 15, 2026
Merged

feat(playback): burst-aware budgets — fix 8-cell cache-starvation freezes#49
tcconnally merged 1 commit into
mainfrom
fix/burst-aware-budgets

Conversation

@tcconnally

Copy link
Copy Markdown
Owner

Problem

The first 8-cell production run buffered heavily: 78 paused-for-cache freezes / 543 freeze-seconds (~9 min) over a ~1h50m session, spread across ~15 different files. That signature is systemic delivery starvation — not one bad asset, and not decode (main-thread PERF stalls stayed <200ms all session; every prior 4-cell bench measured 0 drops / 0 stalls).

Root cause: since v10.3.0 dropped the resolution gate, 96.7% of plays go DIRECT (242 direct vs 8 transcode). Eight cells therefore pull raw high-bitrate files straight off greg simultaneously, each with a 60s / 1 GB readahead. The per-open fill-bursts starve the other cells' steady reads — 80% of freezes began within 8s of a stream-open. This is the "8-stream concurrency ceiling" flagged as the tranche-2 target in SOAK #3.

Fix

Two cell-count-aware budgets, both no-ops at ≤4 cells (the configuration measured clean):

  • scale_readahead_s — demuxer readahead 60s → 30s at 8 cells (and cache_secs follows), shrinking the per-open fill-burst so cells stop stepping on each other.
  • scale_bitrate_budget_mbps — direct-play bitrate cap 60 → ~33 Mbps at 8 cells, so the rare 40–60+ Mbps outliers transcode server-side (Emby throttles a transcode to ~realtime = smooth by construction) while the library bulk stays direct. An explicit HYPERWALL_MAX_DIRECT_BITRATE_MBPS still wins verbatim.

wall.py resolves the per-cell budget at grid setup, wires it into the transcode heuristic (urls.needs_transcode), and logs readahead + bitrate cap alongside the demuxer budget.

Tests

5 new tests (readahead scaling, graduated bitrate, env-override precedence, apply_cache_budget readahead scaling). Full suite green at v10.13.0 (version literal bumped in __init__.py + run_repo_guards.py::test_02).

Caveats before treating this as the complete fix

  • scale_bitrate_budget_mbps assumes link_mbps=800 — an assumption about the greg↔client path, not a measurement.
  • The cap is per-file, not aggregate: 8 × 33 ≈ 264 Mbps sustained. If greg's real bottleneck is disk throughput or a 1GbE link, this reduces starvation but may not fully clear it. Worth measuring the actual ceiling and tuning link_mbps accordingly.

Verify before trusting before/after numbers

Not yet built — the running artifact is still frozen v10.12.1. Rebuild (build.ps1, pwsh 7) and confirm the log's Runtime: banner reads v10.13.0 before A/B'ing; the stale-exe trap has caused false bug hunts here three times.

🤖 Generated with Claude Code

…ezes

The first 8-cell production run buffered ~9 min/2 hr (78 paused-for-cache
freezes, 543 freeze-seconds), spread across ~15 files — systemic delivery
starvation, not one bad asset and not decode (PERF stalls stayed <200ms).
Root cause: 96.7% of plays go DIRECT (242 vs 8 transcode) since v10.3.0
dropped the resolution gate, so 8 cells pull raw high-bitrate files off
greg at once, each with a 60s/1GB readahead. The fill-bursts starve each
other's steady reads — 80% of freezes began within 8s of a stream-open.

Two cell-count-aware budgets, both no-ops at <=4 cells (measured clean):
- scale_readahead_s: demuxer_readahead 60s->30s at 8 cells (+cache_secs
  follows) — shrinks the per-open fill-burst.
- scale_bitrate_budget_mbps: direct-play cap 60->~33 Mbps at 8 cells, so
  the rare 40-60+ Mbps outliers transcode (Emby throttles to ~realtime =
  smooth) while the library bulk stays direct. Explicit
  HYPERWALL_MAX_DIRECT_BITRATE_MBPS still wins verbatim.

wall.py wires the resolved per-cell budget into the transcode heuristic and
logs readahead + bitrate cap alongside the demuxer budget.

Tests: 5 new (scale_readahead, graduated bitrate, env-override precedence,
apply_cache_budget readahead scaling). Full suite green (v10.13.0).

Note: scale_bitrate_budget_mbps assumes link_mbps=800 and caps per-file,
not aggregate (8x33 ~= 264 Mbps sustained) — validate against greg's real
disk/link ceiling before treating this as the complete fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tcconnally
tcconnally merged commit fae11d8 into main Jul 15, 2026
2 checks passed
@tcconnally
tcconnally deleted the fix/burst-aware-budgets branch July 15, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant