Synopsis surfaces: LLM-derived summaries as indexed surfaces (document-summary routing fuel)

[thorwhalen]

Problem

The motivating case for linked retrieval (ADR #43): run an LLM over each document to produce a synopsis, index the synopses, and route from a synopsis match down to the document's chunks. ir's data model reserved the slot — "synopsis" is a documented surface kind and the strategy docstrings name "AI synopsis / problem-class surfaces" as the extension point — but no strategy produces them. This is report 12's document-summary-index pattern (build-time LLM cost, ≈free at query time) and the natural fuel for the collapsed-tree policy (#47).

Scope

• A synopsis-producing strategy decorator/wrapper (e.g. with_synopsis(strategy, *, synthesize=None)): wraps any IndexingStrategy, adds one synopsis surface per artifact. synthesize: Callable[[Artifact], str] is injectable; built lazily on oa only when omitted (the make_llm_* idiom — import ir stays offline; tests inject doubles).
• Build-time cost is the regime (report 12: summary indexing ≈30 LLM calls/doc happens once); the ledger's incremental rebuild means only new/changed artifacts get synthesized.
• Synopsis text rides the normal embed/index path (it's just a surface); surfaces={"synopsis"} search works immediately; PARENT/CHILD edges to the artifact's chunks land in the links view (links: typed-edge view on CorpusStore + GraphStore protocol (semantic link graph) #46) at build time.
• Persistence/staleness: synopses are derived state — stamp the synthesizer identity (model/prompt hash) the way embedder_id/strategy_id are stamped, so a prompt change triggers re-synthesis, not silent staleness.

Acceptance criteria

• build(source, strategy=with_synopsis(Chunked())) with an injected fake synthesizer indexes a synopsis surface per artifact, hermetically.
• Search restricted to synopsis surfaces + traverse (traverse(query, store, policy) — pluggable graph traversal with operator-enforced safety #47) routes to the right chunks end-to-end.
• Incremental rebuild re-synthesizes only changed artifacts; synthesizer identity stamped and staleness-checked.
• Offline import preserved; no oa at import time.

Size: M. Depends on #46 (edges) and feeds #47 (routing policy). Capability 1 (ir). Refs ADR #43, report 12.

[thorwhalen]

Design decision (confirming before coding).

Module. New ir/synopsis.py (an operator module beside expand.py/formulate.py/graph.py):

• with_synopsis(strategy, *, synthesize=None, synthesizer_id=None, synopsis_kind="synopsis") -> IndexingStrategy — wraps any strategy, adds one synopsis surface per artifact.
• make_llm_synthesizer(*, summarize=None, prompt=SYNOPSIS_PROMPT, model=None, synthesizer_id=None, **prompt_function_kwargs) -> Synthesizer — the lazy-oa default (mirrors make_llm_formulator/make_llm_selector; oa imported only inside the call, so import ir stays offline).
• Synthesizer = Callable[[Artifact], str].

Surface placement. The synopsis is prepended (plan position 0) so it is the first summary surface → the collapsed-tree router (the #48 intent: route from a synopsis match). On with_synopsis(Package()) the synopsis (not the terse description) routes. An empty synopsis (synth returns "") is dropped — the artifact keeps its other surfaces.

Staleness — reuse the existing ledger mechanism, no new bookkeeping. The wrapper exposes its identity as scalar attrs (synthesizer_id, synopsis_kind) and holds the inner strategy. I make index._strategy_id recurse into nested strategy attributes (a small, generic change: any attr with a decompose method folds in its own _strategy_id). So strategy_id encodes (inner strategy + params) + synthesizer identity → a model/prompt change OR an inner-param change re-synthesizes exactly the affected artifacts through the normal incremental path. Default synthesizer_id = oa:{model}:{sha(prompt)[:12]}; an injected double should pass synthesizer_id= (else falls back to its __qualname__).

Routing is surface-level, not links-view edges. collapsed_tree (#47) descends synopsis→chunks within an artifact via records_for_artifact (same artifact_id), so synopsis routing needs no edges. I'm therefore not writing synopsis→chunk PARENT/CHILD edges into the #46 links view: that view is artifact-keyed with artifact-id targets (cross-artifact REF/PARENT), whereas synopsis→chunk is surface→surface within one artifact — a different grain. This refines the issue body's wording. (Cross-artifact synopsis edges, if ever wanted, are just another edge_extractor.)

Known interaction (documenting, not fixing here). Eager edge ingest (build(edge_extractor=...)) calls decompose for every artifact each build, so combining with_synopsis with edge_extractor= re-runs synthesis every rebuild (synthesis lives in decompose). The common case — synopsis routing with no edge_extractor — stays fully incremental (the AC). I'll document the caveat; decoupling edge extraction from full decompose is a separate build() refactor I can file if wanted.

Proceeding on this.

[thorwhalen]

Shipped in ir 0.1.22 (squash-merge 746d859, PR #55).

ir.with_synopsis(strategy, *, synthesize=None) adds one LLM-derived synopsis surface per artifact (prepended → the collapsed-tree router); make_llm_synthesizer is the lazy-oa default (import ir stays offline). Staleness reuses the ledger — index._strategy_id now recurses into nested strategies, folding inner-strategy params + synthesizer identity into strategy_id, so a model/prompt/inner change re-synthesizes only the affected artifacts.

Adversarial review (12 agents, every finding verified) caught two should-fixes — both contradicting docstrings shipped in the PR — fixed before merge:

1. text_key alignment — the default synthesizer summarized text_of(raw) with no text_key, so with_synopsis(Chunked(text_key='body')) summarized the wrong field. Now threads the inner strategy's text_key.
2. non-identifiable synthesizer — an unnamed lambda / local closure aliased to a shared <lambda>/<locals> qualname → swapping it was silently stale. Now warns and uses a sentinel id (named functions + explicit/stamped ids still track).

Plus 4 test-hardening nits (Package router precedence, mutation-resistant offline guarantee, default-id content-stability, non-str guard, file-backed round-trip). 365 tests, lint/format clean.

This completes the cap-1 graph arc (#46 links → #47 traverse → #48 synopsis). Next per ADR #43 sequencing: budget governor + run-log (raglab), then PurposeStore.