feat(agent): TinyAgents migration wave 2 — microcompact upstream, session shadow reads, budget dedupe, replay RPC

docs/tinyagents-full-migration-plan/99-deletion-ledger.md

@@ -18,6 +18,15 @@ they land.
 - [x] `context/pipeline.rs` (454) + `context/guard.rs` (236, keep stats structs) — 03.1
 - [x] `context/tool_result_budget.rs` (172) — 03.1
 - [x] `harness/payload_summarizer.rs` (490) — 01.4
+- [x] `tinyagents/middleware.rs::MicrocompactMiddleware` struct + impl (~46) —
+      W2-microcompact (2026-07-03): upstreamed into the vendored crate as
+      `tinyagents::harness::middleware::MicrocompactMiddleware`
+      (`tinyhumansai/tinyagents@feat/microcompact-middleware`, gitlink bumped);
+      OpenHuman now constructs the crate type with `CLEARED_PLACEHOLDER` and
+      events off, so behavior is byte-identical. The in-house struct + impl are
+      deleted; the retained OpenHuman tests assert parity against the crate type.
+      Was the C3-corrected/C5 "extract-then-delete" item (the local microcompact
+      was NOT 1.5.0-superseded; this PR did the extraction).
 
 ## Deletable after SDK-surface adoption
 
@@ -84,6 +93,19 @@ they land.
       crate-internal `agent/harness/turn_subagent_usage.rs` (176) task-local — 06
       (live until crate budget/run-tree accounting avoids duplicate
       `UsageRecorded` and covers parent-turn rollups)
+      W2-budget-dedupe (2026-07-03): dedupe guard landed — the event bridge now
+      records a model call's `UsageRecorded` exactly once, keyed on the run-scoped
+      iteration, so the observe-only crate `BudgetMiddleware`'s re-emit can't
+      double-count (`observability::OpenhumanEventBridge::record_usage`, `[budget]`).
+      Crate `BudgetMiddleware` installed OBSERVE-ONLY (empty `BudgetLimits`) at
+      `tinyagents/mod.rs`; local `CostBudgetMiddleware` demoted to a
+      divergence-logging shadow (`[budget_shadow]`, `after_agent`) but STILL
+      authoritative for enforcement. Flip criteria (must ALL hold before deleting
+      this row): (1) ≥ 500 parent+subagent turns with zero `[budget_shadow]`
+      divergence; (2) crate pricing table wired for money budgets; (3) run-tree
+      rollup via a shared `BudgetTracker` replacing the `turn_subagent_usage`
+      task-local. See the flip-criteria comment at the `tinyagents/mod.rs`
+      registration site.
 - [ ] `agent/dispatcher.rs` (609) + `harness/parse.rs` (833) legacy tool-call
       parsing — after XML/P-format transcripts read from the store and no
       live path parses provider text (04.2 + verify)
@@ -94,6 +116,15 @@ they land.
 
 ## Deletable after session-store cutover (04.2 phase 4)
 
+> **04.2 phase 2 landed (W2-shadow-reads, 2026-07-03):** a store-backed shadow
+> reader runs beside the legacy transcript reader, normalizes both sides via
+> `session_import/convert.rs`, and logs `[session_shadow_read]` divergence
+> (compact, no-PII). Legacy stays authoritative; gated by
+> `agent.session_shadow_reads` (default OFF) + `OPENHUMAN_SESSION_SHADOW_READS`
+> kill switch. The rows below stay `[ ]` until the shadow logs are
+> divergence-clean across the fixture matrix and reads are flipped to the store
+> (phase 3), which is the precondition for these deletions.
+
 - [ ] `session/transcript.rs` (1347) + tests (978)
 - [ ] `session/migration.rs` (373) + tests
 - [ ] `session/turn/session_io.rs` (391)

docs/tinyagents-full-migration-plan/HANDOFF-2026-07-03.md

@@ -102,50 +102,69 @@ All slice branches were created FROM `feat/tinyagents-c0-15-baseline`
   / `mod.rs` when merged together — merge C0 first, then slices one at a
   time, resolving against C0's tree.
 
-## Wave 2 status (2026-07-03, late)
+## Wave 1 (merged)
 
 All five wave-1 slice branches were MERGED into
-`feat/tinyagents-c0-15-baseline` (light conflicts only; fmt commit on top;
-50 targeted tests green post-merge) and pushed to PR **#4473**, which now
-carries the entire wave-1 scope.
-
-Wave 2 was launched as a 4-agent workflow but **died on the monthly spend
-limit** before any work landed. Its scoped slices (prompts preserved in the
-session workflow script `tinyagents-continuation-wave2-*.js`, resumable via
-`resumeFromRunId: wf_d6f5d26d-7e4`):
-
-1. `W2-shadow-reads` — 04.2 phase 2: store-backed shadow reader, compare with
-   legacy render, log divergence, legacy stays authoritative
-   (flag `agent.session_shadow_reads`, env kill switch).
-2. `W2-budget-dedupe` — single-owner `UsageRecorded` recording (dedupe guard)
-   → install crate `BudgetMiddleware` observe-only; local `CostBudgetMiddleware`
-   demoted to divergence-logging shadow; flip criteria documented.
-3. `W2-microcompact-upstream` — implement microcompact IN `vendor/tinyagents`
-   (branch `feat/microcompact-middleware`), push submodule upstream, then swap
-   OpenHuman to the crate version + delete local (gitlink bump ONLY if the
-   submodule push succeeded).
-4. `W2-replay-rpc` — `openhuman.agent_run_events` (paged, `next_offset`),
-   `agent_run_status`, `agent_runs_active` controllers over the C4 journal/
-   status seams, registry pattern.
+`feat/tinyagents-c0-15-baseline` and pushed to PR **#4473**, which landed the
+entire wave-1 scope into `tinyhumansai/openhuman:main`.
+
+## Wave 2 status (2026-07-04) — DONE, on `feat/tinyagents-wave2`
+
+All four wave-2 slices are implemented and integrated onto ONE branch
+`feat/tinyagents-wave2` (off `upstream/main`), one focused commit each, and
+opened as a single combined PR. Adapter-first throughout: legacy authoritative,
+crate features shadow + log divergence, deletions gated on proven parity.
+
+1. `W2-microcompact-upstream` — **done.** The generic
+   `MicrocompactMiddleware` (caller-supplied placeholder, opt-in `Compressed`
+   event, idempotent tool-body clearing) was implemented IN the vendored crate
+   and pushed to `tinyhumansai/tinyagents@feat/microcompact-middleware`
+   (commit `ac73382`) BEFORE the gitlink bump. OpenHuman swapped to the crate
+   type (constructed with `CLEARED_PLACEHOLDER`, events off → byte-identical);
+   local struct+impl deleted, OpenHuman tests retargeted as the parity contract.
+   Crate tests: 5 green. Ledger row ticked.
+2. `W2-shadow-reads` — **done.** 04.2 phase 2 store-backed shadow reader beside
+   the legacy transcript reader; `[session_shadow_read]` divergence logging;
+   flag `agent.session_shadow_reads` (default OFF) + `OPENHUMAN_SESSION_SHADOW_READS`
+   kill switch. Legacy authoritative. (Flip → reads-from-store is phase 3, the
+   ~9k-line deletion unlock.)
+3. `W2-budget-dedupe` — **done.** Event bridge records each model call's usage
+   exactly once (dedupe guard keyed on the run-scoped iteration cursor); crate
+   `BudgetMiddleware` installed observe-only (empty `BudgetLimits`); local
+   `CostBudgetMiddleware` demoted to a `[budget_shadow]` divergence logger, still
+   authoritative for enforcement. Three flip criteria documented at the
+   `tinyagents/mod.rs` registration site + in the ledger.
+4. `W2-replay-rpc` — **done.** Read-only `openhuman.agent_run_events` (paged,
+   `next_offset`, capped limit), `agent_run_status`, `agent_runs_active`
+   controllers over the C4 journal/status seams; direct `AgentObservation`/
+   `HarnessRunStatus` serde projection (no PII); registered via
+   `src/core/all.rs` per the controller-migration checklist. (Resolves the
+   "replay RPC unexposed" C4 follow-up.)
+
+**Test note:** targeted crate tests (microcompact) ran green locally; the full
+core-crate test binary would not link locally under this box's memory ceiling
+(single-rustc codegen thrash), so `cargo check --lib --tests` is the local
+compile gate and CI runs the actual targeted + full suites + coverage.
 
 ## Merge / PR state
 
-- Wave-1 execution branches are LOCAL (not pushed) except as noted below.
-- C0 PR: see PR link in the section below / `gh pr list --repo
-  tinyhumansai/openhuman --author @me`.
-- Git etiquette (user rules): push to `origin` (senamakel fork), PR against
-  `upstream` (tinyhumansai) with `--head senamakel:<branch>`; explicit
-  `git add <paths>` only; never commit on main.
-
-## Suggested next steps (wave 2)
-
-1. Merge C0 PR; rebase + push + PR the five slice branches (stack on C0).
-2. 04.2 shadow reads → read cutover (biggest deletion unlock, ~9k lines gated
-   on it, incl. dispatcher/parse/pformat).
-3. C5 upstream extractions INTO `vendor/tinyagents` (now editable in-tree):
-   microcompact (new — see corrections), multimodal resolver, dialect layer,
-   overflow-to-artifact, hooks traits.
-4. C3 remainder: UsageRecorded de-dup → crate `BudgetMiddleware` → delete
-   local `CostBudgetMiddleware`; exposure-shadow parity audit → flip owner.
-5. Flip C2 shadows to authoritative once divergence logs are clean; wire the
-   goals migration helper to boot.
+- `feat/tinyagents-wave2` pushed to `origin` (senamakel fork); ONE combined PR
+  vs `upstream` (`--head senamakel:feat/tinyagents-wave2`): **PR #4483**
+  (`tinyhumansai/openhuman#4483`).
+- Submodule branch `feat/microcompact-middleware` pushed to
+  `tinyhumansai/tinyagents`; gitlink bumped to `ac73382`. Fresh worktrees must
+  `git submodule update --init vendor/tinyagents`.
+- Git etiquette (user rules): push to `origin`, PR against `upstream` with
+  `--head senamakel:<branch>`; explicit `git add <paths>`; never commit on main.
+
+## Suggested next steps (wave 3)
+
+1. Flip `session_shadow_reads` → reads-from-store once the fixture-matrix
+   divergence logs are clean (biggest deletion unlock, ~9k incl.
+   dispatcher/parse/pformat).
+2. Flip crate `BudgetMiddleware` → enforcing owner once the 3 documented
+   criteria hold; delete local `CostBudgetMiddleware` + `turn_subagent_usage.rs`.
+3. Open a tinyagents PR for `feat/microcompact-middleware` (currently a pushed
+   branch, not merged) so the gitlink can later track a tagged release.
+4. Continue C5 upstream extractions (multimodal resolver, dialect layer,
+   overflow-to-artifact, hooks traits) + C2 shadow → authoritative flips.

src/core/all.rs

@@ -131,6 +131,10 @@ fn build_registered_controllers() -> Vec<RegisteredController> {
     controllers.extend(crate::openhuman::webview_apis::all_webview_apis_registered_controllers());
     // Agent definition and prompt inspection
     controllers.extend(crate::openhuman::agent::all_agent_registered_controllers());
+    // Read-only agent run replay + status over the durable journal/status seams
+    // (agent_run_events / agent_run_status / agent_runs_active).
+    controllers
+        .extend(crate::openhuman::tinyagents::replay::all_agent_replay_registered_controllers());
     // Persistent agent profiles (flavours): name, soul, memory sources, skills, MCP, connectors.
     controllers.extend(crate::openhuman::profiles::all_profiles_registered_controllers());
     // User-facing agent registry: defaults, enablement, custom agents, tool policy.
@@ -383,6 +387,8 @@ fn build_declared_controller_schemas() -> Vec<ControllerSchema> {
     schemas.extend(crate::openhuman::mcp_registry::all_mcp_registry_controller_schemas());
     schemas.extend(crate::openhuman::webview_apis::all_webview_apis_controller_schemas());
     schemas.extend(crate::openhuman::agent::all_agent_controller_schemas());
+    // Read-only agent run replay + status controllers (workstream 05.x).
+    schemas.extend(crate::openhuman::tinyagents::replay::all_agent_replay_controller_schemas());
     schemas.extend(crate::openhuman::profiles::all_profiles_controller_schemas());
     schemas.extend(crate::openhuman::agent_registry::all_agent_registry_controller_schemas());
     schemas.extend(crate::openhuman::agent_experience::all_agent_experience_controller_schemas());

src/openhuman/agent/harness/session/turn/session_io.rs

@@ -34,6 +34,11 @@ impl Agent {
                         }
                         let loaded_count = session.messages.len();
                         log::info!("[transcript] loaded {} messages for resume", loaded_count);
+                        // Best-effort store-backed shadow read (issue #4249,
+                        // 04.2 phase 2). Observes + logs divergence only; the
+                        // legacy transcript just loaded stays authoritative and
+                        // is what feeds the resume below. Gated OFF by default.
+                        self.maybe_shadow_read_session_store(&path, &session);
                         let bounded = self.bound_cached_transcript_messages(session.messages);
                         if bounded.len() < loaded_count {
                             log::warn!(
@@ -322,6 +327,59 @@ impl Agent {
         });
     }
 
+    /// Store-backed **shadow read** of a just-loaded session transcript.
+    ///
+    /// Beside the legacy authoritative reader (`try_load_session_transcript`),
+    /// read the same session back from the TinyAgents journal store, normalize
+    /// both sides through the importer's `session_import::convert` machinery,
+    /// compare, and log any divergence (`[session_shadow_read]`, issue #4249,
+    /// 04.2 phase 2). Additive and gated on the default-**OFF**
+    /// `AgentConfig::session_shadow_reads` flag
+    /// (`OPENHUMAN_SESSION_SHADOW_READS` is a kill switch): when disabled this
+    /// is a cheap early return.
+    ///
+    /// The legacy transcript stays authoritative — this only observes. The
+    /// comparison runs on a spawned background task so it never slows the
+    /// authoritative read, and every store-read error is treated as "no shadow
+    /// available" (logged at debug), never propagated.
+    fn maybe_shadow_read_session_store(
+        &self,
+        path: &std::path::Path,
+        session: &transcript::SessionTranscript,
+    ) {
+        use crate::openhuman::session_import::live;
+
+        // Config flag (default OFF) gates the shadow read; the env kill switch
+        // can still force it off. `self.config` is the effective per-agent config.
+        if !live::shadow_reads_enabled(self.config.session_shadow_reads) {
+            return;
+        }
+
+        // Same session key the write side / importer use: the transcript stem.
+        let Some(stem) = path
+            .file_stem()
+            .and_then(|s| s.to_str())
+            .map(str::to_string)
+        else {
+            log::debug!(
+                "[session_shadow_read] skipped: no file stem for {}",
+                path.display()
+            );
+            return;
+        };
+
+        let workspace = self.workspace_dir.clone();
+        let transcript = session.clone();
+        log::debug!(
+            "[session_shadow_read] scheduled stem={stem} workspace={} legacy_messages={}",
+            workspace.display(),
+            transcript.messages.len()
+        );
+        tokio::spawn(async move {
+            let _ = live::shadow_read_compare(&workspace, &stem, &transcript).await;
+        });
+    }
+
     // ─────────────────────────────────────────────────────────────────
     // Session-memory extraction.
     // ─────────────────────────────────────────────────────────────────

src/openhuman/config/schema/agent.rs

@@ -259,12 +259,36 @@ pub struct AgentConfig {
     /// [`crate::openhuman::session_import::live::dual_write_enabled`].
     #[serde(default = "default_session_dual_write")]
     pub session_dual_write: bool,
+
+    /// Store-backed **shadow read** of a resumed session's messages: on the
+    /// legacy transcript read path (`session/turn/session_io.rs` →
+    /// `try_load_session_transcript`), also read the same session back from the
+    /// TinyAgents journal (`{workspace}/tinyagents_store/journal`), normalize
+    /// both sides through the importer's `session_import::convert` machinery,
+    /// compare, and log any divergence (`[session_shadow_read]`, issue #4249,
+    /// sessions 04.2 phase 2).
+    ///
+    /// Defaults **OFF** (unlike `session_dual_write`, which defaults ON): this
+    /// is an observation-only parity probe with no product effect. The legacy
+    /// JSONL read stays authoritative — the shadow read only observes and logs
+    /// on a background task; a store-read failure is treated as "no shadow
+    /// available" and never breaks or slows the authoritative read. The
+    /// `OPENHUMAN_SESSION_SHADOW_READS` env var is a pure **kill switch**: a
+    /// falsy value (`0`/`false`/`no`/`off`/`disable`) forces the shadow read
+    /// OFF regardless of config; it can never force it ON. See
+    /// [`crate::openhuman::session_import::live::shadow_reads_enabled`].
+    #[serde(default = "default_session_shadow_reads")]
+    pub session_shadow_reads: bool,
 }
 
 fn default_session_dual_write() -> bool {
     true
 }
 
+fn default_session_shadow_reads() -> bool {
+    false
+}
+
 fn default_tool_result_budget_bytes() -> usize {
     crate::openhuman::context::DEFAULT_TOOL_RESULT_BUDGET_BYTES
 }
@@ -395,6 +419,7 @@ impl Default for AgentConfig {
             tool_result_budget_bytes: default_tool_result_budget_bytes(),
             agent_timeout_secs: default_agent_timeout_secs(),
             session_dual_write: default_session_dual_write(),
+            session_shadow_reads: default_session_shadow_reads(),
         }
     }
 }