Deterministic insights and review-only suggestions

[tony]

Status. Stacked on #47 — retarget to master after it lands. The insight/suggestion schema, CLI groups, and MCP tools are written, and the sketch columns (simhash_hex, minhash_json, token_count, quality_flags) are computed and persisted. Two honest gaps to close: (1) it inherits #47's unlanded cache-consult keystone (both build on the same DbStore); and (2) InsightEngine.run_similarity today clusters only exact normalized-hash duplicates — the SimHash/MinHash → LSH-bucket → Jaccard-verify → union-find near-duplicate cascade this PR headlines is not yet consuming the persisted sketches. The design below is that remaining cascade, stdlib-first.

Motivation — why deterministic insights + review-only suggestions

Once history is a durable local read model (#47), the higher-value questions are interpretive, not retrieval: which prompts are duplicates or the same intent across projects (variant edges), which recurring instruction is absent from a given AGENTS.md / skill surface (omission findings), and which of those should become a reviewable instruction-change suggestion. The need is to surface recurring asks, forgotten-but-similar past conversations, and instruction-change candidates — without an LLM and without unbounded pairwise comparison.

LLM-free-by-default is a trust requirement: results must be reproducible and evidence-backed, and normal search must stay a read-only local evidence surface. An LLM reaches this surface only by explicitly invoking a CLI/MCP tool, never implicitly (ADR 0017/0018). Semantic/vector backends stay an optional later extra, never a required dependency.

The problem being solved

1. Non-quadratic, deterministic candidate generation. Global similarity over thousands of prompts can't be O(n²) — and agentgrep already has the anti-pattern in-tree twice: ranking.py:collapse_near_duplicates is a nested WRatio loop, and RapidFuzz's process.cdist materializes a full len(a)×len(b) score matrix (process_py.py). Both are correct for a small already-retrieved result page and wrong for global clustering. The discipline that makes it cheap is cpython's difflib.get_close_matches: a strict cheap→expensive admissible cascade (difflib.py).
2. Bounded by construction. Variant edges grow superlinearly on duplicate-heavy histories, so every listing pages with a default limit and reports totals rather than materializing full result sets — mirrored into the MCP payloads and the graph render.
3. A hard review boundary. Suggestions turn evidence into proposed instruction changes for AGENTS.md / skills. They render for human review as a diff and never auto-edit files or call an LLM (ADR 0004's query/plan/execute/sink separation, applied to a mutation-shaped surface).

Design inspiration

The deterministic similarity cascade — map difflib's discipline onto the persisted sketches. get_close_matches runs real_quick_ratio (O(1) length bound) → quick_ratio (multiset overlap) → ratio (full), admitting only survivors to each stage. Port that shape at scale: LSH-band collision (cheapest gate over the stored minhash_json) → SimHash-Hamming / MinHash-overlap estimate (medium) → exact token-set Jaccard verify (authoritative) → dict-based union-find clustering (~15 lines, path compression). Add difflib's free prunes: a length-ratio gate min(|A|,|B|)/max(|A|,|B|) ≥ threshold before the Jaccard verify (bounds), and an autojunk-style corpus-document-frequency drop of ubiquitous shingles (__chain_b). semhash validates the clustering shape — greedy keep-first-representative over the candidate set (semhash.py). All sketches are hashlib-only (never the process-randomized hash()), with the algorithm + params recorded as provenance so a run is re-derivable.

The optional semantic tier — always an extra over the same SQLite file. Gate it behind agentgrep[semantic]: a pure-numpy static embedding (model2vec: tokenize → gather → mean-pool, no torch — model.py) behind an Encoder Protocol whose default is the substring/no-op path; vicinity's default backend is exact brute-force numpy cosine (basic.py), so no ANN dependency for the common case. When installed, store vectors in the same DB via sqlite-vec's vec0 virtual table (README) and back the FTS candidate generator with external-content FTS5, exactly as sqlite-rag keeps everything in one SQLite file (database.py).

Review-only suggestions — propose-don't-apply, borrowed from git and jj. Render each suggestion as a difflib.unified_diff against the live target and persist a pre-image fingerprint on the artifact — the way a unified diff's extended header records blob identity (diff-generate-patch.adoc). A future opt-in suggestions apply should mirror git apply: a --check dry-run that reports applies-cleanly / drift without touching the file (git-apply.adoc), atomic write on a clean pre-image match, and refuse-on-drift by default (atomicity). Make it undoable like jj — append a content-hashed apply-op so suggestions apply --undo restores the pre-image and history is never destroyed (op_store.rs, operation/restore.rs). Because rendered evidence is prompt-derived, route it through a span-based redactor (presidio's analyze→anonymize split; a small built-in recognizer set for emails, home-dir → ~, tokens — replace.py).

Rendering + MCP DX, pi-minimal, one emitter → three surfaces. Keep to_mermaid(run), rank_rows(run), to_unified_diff(suggestion) as pure functions returning str/rows. Emit the insight graph as a fenced mermaid flowchart LR (clusters as subgraph, variant edges as A -->|0.82| B, one classDef per cluster), bounded like Mermaid's own maxEdges cap with a terse %% N more omitted footer (grammar). Human CLI mode stays calm — Rich Table(box=None, show_edge=False) (box.py); the TUI feeds the same rows into the existing OptionList. insights_list / suggestions_list return typed Pydantic models (auto output_schema + structured_content, readOnlyHint) with per-run artifacts as templated resources agentgrep://insights/{run_id} (ToolResult, templates); an eventual interactive suggestions apply maps naturally to Context.elicit for the human-approve step (context.py).

---

Stacked on #47 — retarget to master after it merges; until then the diff below is insights/suggestions only.

Summary

• Add deterministic feature and artifact storage: the DB index gains a record_features table (simhash/minhash/quality flags) plus insight-run, cluster, variant-edge, omission-finding, and suggestion schema. db sync --features defer|inline defers expensive feature extraction by default, with a batched missing-feature refresh API for insight runs.
• Add a deterministic insights engine (ADR 0017): agentgrep insights analyze|list|explain computes similarity variant edges (simhash/minhash candidates confirmed by token-set Jaccard) and omission findings against a --target file, persisting runs, clusters, and evidence rows with provenance — no LLM involved.
• Add live analyze progress: multi-line stderr progress with feature-refresh and artifact-write counters, Enter-to-exit, and clean JSON stdout.
• Add bounded persisted listings: insights list pages with a default limit backed by a confidence-order index, explain is count-only, and listing payloads render as human summaries by default with explicit --json/--ndjson.
• Add review-only suggestion artifacts (ADR 0018): agentgrep suggestions list|show|render turns insight evidence into persisted instruction-change suggestions for AGENTS.md/skill surfaces. Suggestions render for human review; agentgrep never edits instruction files or calls an LLM automatically.
• Add read-only MCP tools insights_list and suggestions_list with bounded listing payloads and result totals.

Changes by area

Storage

src/agentgrep/db.py: feature builders, the feature/artifact schema, --features sync modes with deferred counters, refresh_missing_features with progress reporting, and similarity-row iteration.

Engines

src/agentgrep/insights.py: InsightEngine with similarity and omission analysis, bounded listings, and count APIs. src/agentgrep/suggestions.py: SuggestionEngine producing persisted, review-only SuggestionArtifact rows tied to insight evidence.

CLI

src/agentgrep/cli/parser.py, src/agentgrep/cli/render.py: the insights and suggestions command groups, analyze progress rendering, bounded listing and suggestion formatters, parse-time --target requirement for omission runs.

MCP

src/agentgrep/mcp/tools/insight_tools.py, src/agentgrep/mcp/models.py: two read-only tools with pydantic response models and capability registration.

Docs

ADRs 0006–0007; per-command CLI pages for insights and suggestions; the /insights/ feature section; MCP tool documentation.

Design decisions

• Deterministic before semantic: similarity uses simhash/minhash candidate generation confirmed by Jaccard overlap, so insight runs are reproducible and evidence-backed; semantic backends such as LanceDB remain an optional later addition, not a required dependency (ADR 0017).
• Hard review boundary for suggestions: suggestion workflows never edit AGENTS.md/skills and never invoke an LLM on their own — an LLM may only reach this surface by explicitly calling the CLI/MCP tools (ADR 0018).
• Features are deferred secondary state: sync writes the ledger, records, and FTS index fast by default and lets insight runs batch-refresh missing features.
• Listings are bounded by construction: variant edges grow superlinearly on duplicate-heavy histories, so list surfaces page with a default limit and report totals instead of materializing full result sets.

Test plan

• tests/test_db_index.py — feature modes, batched feature refresh with progress, deferred counters
• tests/test_cache_cli.py — insights/suggestions CLI contracts: bounded listings, human summaries, analyze progress, early exit, --target requirement, JSON/NDJSON stdout
• tests/test_agentgrep_mcp.py — bounded MCP listing payloads with totals
• tests/test_widgets.py — rendered command-page docs for both groups
• Full gate per commit: ruff check / ruff format, ty check, uv run pytest (incl. doctests), just build-docs