Epic: evolve ir toward the ir_09 Composable Search Agent — layering decision + role map

[thorwhalen]

Decision

Layering: hybrid. A new orchestration layer on top owns the agent; ir grows a tight set of retrieval-adjacent seams that each independently improve ir's own non-agentic search. ir stays the retrieval substrate ('retrieval-first, extensible into RAG without being one by default'); the new layer depends ON ir and ir never imports it.

The decision rule for every seam: does this make ir's single-shot, agent-free search measurably better? Yes -> it lands in ir. If it is the back-edge / cross-corpus reasoning that makes a system an agent rather than a DAG -> it lives in the new layer.

Role map (ir_09 §3) vs ir 0.1.11 (verified this session)

Role	ir_09 §3	Status in ir	Where it goes
Retriever (one query, one corpus)	dense/BM25/hybrid/graph/SQL/web	present — ir.retrieve.search (dense/lexical/hybrid)	ir (leaf); graph/SQL/web retrievers -> new layer
Reranker (final ordering)	swappable	present — search(rerank=ef.Reranker), per-corpus	ir at leaf; cross-source rerank at fan-in -> new layer
Evaluator/Critic	relevant + sufficient + refinement	partial — Selection has relevant/abstain + signals	ir owns verdict SHAPE (+ derived sufficient); the loop/refinement -> new layer
Formulator	rewrite/decompose/HyDE/filter-synth	absent — query embedded as-is	ir (identity-default seam; improves ir's own recall)
Planner/Orchestrator	decompose + source-select + budget + terminate	absent	new layer
Citer/Verifier	confirm pointer supports claim	absent (needs generation ir omits)	new layer (srag)

Value types: Result already = SearchHit+Disclosure (pointer-passing, §5, implemented in ir). Query/SubTask/LowLevelQuery/Judgement are agent-layer types (ir keeps a plain str query + a derived sufficient signal).

What ir owns (this epic's ir-side issues)

1. Formulator seam — injectable formulate=, identity default; make_llm_formulator mirroring make_llm_selector. Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32
2. ir.as_retriever(corpus) — the corpus->Retriever adapter matching ir_09's leaf contract. Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33
3. registry.retrievers() view + multi-corpus discover — a read-projection Mapping[name, Retriever] + federated single-shot search. Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34
4. sufficient signal on Selection — derived, model-free; refinement stays OUT. Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35
5. Injectable resource store on disclose + SearchHit.to_dict() — formalize §5 pointer-passing over dol/URLs; serialization-clean leaf. Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36

Each is strictly additive, keyword-only, defaults to today's behavior, verified against the existing eval harness. None breaks backcompat.

What the new layer owns (stub issue)

Planner/Orchestrator, the control loop with the back-edge (evaluator->reformulate), the budget governor, the live Mapping[name, Retriever] source registry across heterogeneous backends, Citer/Verifier, and the SingleContext-vs-MultiAgent orchestrator split. New-layer stub: #37.

Two load-bearing boundaries (guard in every review)

• Formulate returns QUERIES, never SubTasks. Decomposition into goal+sources is the Planner -> agent layer.
• ir gains a sufficient SIGNAL, never a refinement directive or a loop. The back-edge (ir_09 §4) is what makes it an agent; it must not live in ir.

Build order

Matches ir_09 §8: (1) Retriever adapter + registry view [#2, #3]; (2) new-layer single-context orchestrator with trivial planner + pass-through evaluator wrapping today's search/select/disclose [stub]; (3) Formulator + an LLM Evaluator returning sufficient/refinement, turn on the back-edge [#1 in ir; loop in new layer]; (4) move rerank to fan-in + Citer in srag [new layer]; (5) budget governor + run-log [new layer]; (6) multi-agent orchestrator if breadth justifies [new layer].

Links

• ir_09 design doc: misc/docs/ir_09 -- A Composable Search Agent ...md
• ir-side issues: Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32, Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33, Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34, Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35, Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36
• New-layer stub: New: orchestration layer on top of ir — the Composable Search Agent (Planner + back-edge loop + multi-corpus + Citer) #37

[thorwhalen]

ir-side seams all shipped ✅ (epic complete on the ir side):

• Publish a stable Retriever adapter — ir.as_retriever(corpus) -> Callable matching ir_09's leaf contract #33 as_retriever + Make disclosure's resource-store contract explicit and injectable (BodyLoader -> Mapping resource store) + SearchHit.to_dict() #36 SearchHit.to_dict / disclose(store=) — PR feat: agent-ready retrieval seams — as_retriever, SearchHit.to_dict, disclose(store=) #39 (ir 0.1.12)
• Project the corpus registry into a Mapping[name, Retriever] view + multi-corpus discover (single-shot, no loop) #34 registry.retrievers() view + Extend Selection toward a Judgement-shaped result — add a derived sufficient signal (keep refinement OUT) #35 Selection.sufficient — PR feat: registry.retrievers() Mapping view + Selection.sufficient signal #40 (ir 0.1.13)
• Add an injectable Formulator seam to ir.search / ir.discover (identity default; opt-in rewrite/expand/HyDE) #32 formulate= seam + make_llm_formulator — PR feat(retrieve): injectable query-formulation seam — formulate= + make_llm_formulator (#32) #41

The agent layer foundation landed in raglab (thorwhalen/raglab#1, raglab 0.2.2): value types + role Protocols + SingleContextAgent (with the back-edge) + a no-LLM thin slice running end-to-end over ir.as_retriever. Remaining epic work is raglab-side (thorwhalen/raglab#2): wire the LLM Formulator (consume ir.make_llm_formulator) + LLM Evaluator (consume Selection.sufficient) to turn on the back-edge, then reranker-at-fan-in, budget/observability, and the MultiAgentAgent. Cross-source merge stays raglab's fan-in (not ir).