Heap: evidence-driven second-pass optimization after incident collection

[Juliusolsson05]

Problem

The newly merged incident/heap collection work gives Agent Code much better crash evidence, but it does not by itself reduce the remaining heap pressure. We should use that evidence to drive the next optimization pass instead of guessing.

This is deliberately narrower than the broad perf sweep (#103) and complementary to the current post-#288 OOM tracker (#327). The goal is to turn the new heap.pressure, shared appRunId, Crashpad minidump correlation, and adaptive heap sampling from #363/#364 into concrete memory reductions.

Context from existing work

• main: bound long-running memory growth + heap-pressure watchdog #48 added the first main-process heap watchdog after a ~4 GiB V8 OOM and fixed several unbounded main-process maps.
• feat(perf): always-visible main-process heap + rss badge in header #76/feat(system-perf): expand popover with heap-space + stress signals + on-demand snapshot #86 added live main heap/RSS visibility, heap-space breakdown, stress signals, and manual heap snapshots.
• perf(worktree-activity): stream JSONL transcripts instead of readFile #87 replaced a whole-file transcript parser with streaming JSONL to remove 100-200 MB transient spikes.
• Main-process heap unbounded — full transcript + tool-output text retained in memory (681MB, 92% strings) #288 identified the major main-process retained-string leak: sub-agent JSONL entries and huge tool outputs retained in memory.
• SubAgentWatcher reads full JSONL files and retains parsed entries unbounded #310/fix(#288): bound CodexSubAgentTracker retained entries + clear on stop (Codex twin of #300) #317/fix(#288): intern per-session JSONL entry metadata (cwd/sessionId/role) #318/feat(#288): heap-snapshot analyzer + watchdog auto-summary #319/fix(#288): cap sub-agent entry body retention (88% of heap) + dominator --owners analysis #320/fix(#288): make sub-agent body truncation actually free memory (V8 slice trap + toolUseResult miss) #321/fix(#288): derive-and-drop sub-agent state (stop retaining raw entries — 85% of heap) #322 resolved the Main-process heap unbounded — full transcript + tool-output text retained in memory (681MB, 92% strings) #288 class: analyzer/dominator tooling, Codex tracker bounds, metadata interning, V8 sliced-string trap fix, toolUseResult coverage, and the final derive-and-drop accumulator for SubAgentWatcher.
• Heap: main-process OOM under heavy multi-agent orchestration — follow-up to #288 #327 remains open for a later hard main-process OOM under heavy multi-agent orchestration, probably from a secondary accumulator or orchestration retained output.
• Reduce orchestration worktree freeze pressure #355 reduced orchestration/worktree freeze pressure by adding backpressure and tail-bounded reads, but it was primarily event-loop starvation work, not a retained-heap audit.
• feat(incident): always-on App Run Journal (Phases 1–8 + run-identity unification) #363 merged the always-on App Run Journal and made heap pressure incidents durable/correlated by appRunId.
• fix(incident): correlate native crashes with Crashpad minidumps + adaptive heap sampling #364 followed a real V8 OOM where the watchdog missed a <5s climb; it added Crashpad minidump correlation and adaptive heap sampling.

Why now

Before #363/#364, a crash often left us manually timestamp-matching logs, snapshots, minidumps, debug bundles, and perf runs. Now a bad run should leave a single run id and clearer evidence. The next heap work should consume that evidence and answer:

1. What retained owner still grows with heavy orchestration?
2. Is the remaining pressure in the Electron main process, the renderer, or both?
3. Which long-lived structures still retain full transcript/tool/debug bodies when they only need derived state or bounded previews?

Investigation plan

• Reproduce a heavy multi-agent orchestration session similar to Heap: main-process OOM under heavy multi-agent orchestration — follow-up to #288 #327 with the current incident journal enabled.
• Capture or collect the artifacts linked by the current appRunId: incidents.jsonl, heartbeat.json, heap snapshot if heap.pressure fires, Crashpad minidump if native OOM wins the race, and any manual debug bundle tail.
• Run scripts/analyze-heapsnapshot.mjs --owners against the main-process snapshot and record the top retained owners.
• Capture renderer heap separately for long resumed transcripts; Main-process heap unbounded — full transcript + tool-output text retained in memory (681MB, 92% strings) #288 explicitly left renderer SessionRuntime.entries / _atp.source body retention as a separate remaining class.
• Compare main vs renderer retainers before changing code. Do not assume the Main-process heap unbounded — full transcript + tool-output text retained in memory (681MB, 92% strings) #288 owner recurred.
• Audit known high-risk owner families only after evidence points there: orchestration run output/state, built-in MCP request/result retention, proxy/debug capture buffers, SessionRuntime.entries, transcript parser sidecars, xterm/screen buffers, and per-session lifecycle maps.

Expected fix shapes

• Prefer derive-and-drop accumulators over retaining raw entries, mirroring fix(#288): bound CodexSubAgentTracker retained entries + clear on stop (Codex twin of #300) #317/fix(#288): derive-and-drop sub-agent state (stop retaining raw entries — 85% of heap) #322.
• Keep bounded previews and lazy-load full bodies from disk when UI/debug flows need them.
• Add byte caps, not only entry-count caps, for any retained tool result or transcript body.
• Avoid V8 slice()-based truncation for long strings; force real copies where truncation is meant to release the parent string (fix(#288): make sub-agent body truncation actually free memory (V8 slice trap + toolUseResult miss) #321).
• Keep diagnostic collection bounded: new incident/heap tooling must not become another retained-memory source.

Acceptance

• A fresh issue/PR comment records before/after heap evidence from at least one main-process snapshot and, if relevant, one renderer snapshot.
• The top remaining retained owner(s) are named with dominator evidence, not inferred from symptoms.
• Heavy orchestration no longer shows unbounded main-process heap growth in the reproduced scenario.
• Long resumed transcripts do not retain full bodies in renderer memory unless the visible UI actually needs them.
• Any new retention/capping rule has a focused regression test or a documented measurement fixture.
• The existing incident/Crashpad/heap snapshot paths still work after optimization.

Non-goals

• Do not duplicate Heap: main-process OOM under heavy multi-agent orchestration — follow-up to #288 #327 by treating every OOM symptom as this issue. Heap: main-process OOM under heavy multi-agent orchestration — follow-up to #288 #327 remains the broad crash tracker.
• Do not fold the whole General optimization sweep — reduce memory use and improve app-wide performance #103 performance sweep into this. CPU churn, render memoization, and generic IPC volume belong there unless they directly explain retained heap.
• Do not raise V8 heap limits as the fix. That would hide the growth instead of bounding it.

Related: #48, #76, #86, #87, #103, #288, #310, #317, #318, #319, #320, #321, #322, #327, #355, #363, #364.

[Juliusolsson05]

Progress from today's merged wave — three of this issue's children shipped and closed:

• Add byte caps to orchestration MCP outputs and closed-agent tombstones #373 → PR fix(orchestration): byte-cap MCP outputs, tombstones, and transcript tail rings #510: orchestration MCP outputs, closed-agent tombstones, and transcript tail rings are byte-capped (per-message 4K / per-agent 24K / total 120K defaults, summary-first degradation, truncated metadata). The 5×–7× payload mirroring in every orchestration response is reduced to one full capped copy + excerpts.
• Window live renderer transcript/runtime state by bytes, not only initial history count #375 → PR feat(renderer): live transcript window + memory instrumentation + raw-payload compaction #511: live renderer transcript window (2000→1500 entries AND 32→24 MiB byte budget, with pair-integrity/ownership/marker safety constraints), per-session memory gauges on the perf journal (entries/semantic/feed-debug/ghost/tool-map/seen-set bytes + global code-block registry and Monaco model counts), and raw semantic/feed-debug payload compaction behind AGENT_CODE_DEV_DEBUG.
• Add build and crash-classifier provenance to incident artifacts #374 → PR feat(incident): build + classifier provenance in run manifests, incidents, and debug bundles #509: every incident artifact now records build provenance + classifier version/flags, so the next heap investigation can trust which code produced its evidence.

The new gauges are exactly the "evidence collection" this issue's second pass wants — the next incident should come with per-session, per-category retained-bytes data instead of heap-snapshot archaeology. Remaining siblings still open: #369 (stalled Codex flows), #370 (crash breadcrumbs), #372 (Codex proxy mirroring pressure).