Persistent knowledge base: dedicated Knowledge node schema + extraction pipeline + knowledge_search/add/forget tools

[murari316]

Summary

LM2's auto-injection (Phase 4: symbol_context/get_doc_context auto-inject) and confidence decay (Phase 3) are already implemented and working. What's missing is a dedicated, queryable knowledge layer on top of them: a real Knowledge node type in the graph, a pipeline to extract facts from saved sessions, and MCP tools to search/add/forget knowledge directly rather than relying purely on incidental auto-injection.

Three categories of knowledge (from design discussion): codebase facts (e.g. "this module intentionally duplicates X because Y"), interaction facts (user preferences, workflow quirks), and decision facts (architectural choices and their rationale — similar in spirit to what save_session's decisions field captures today, but queryable independently of any one session).

Design decisions (from prior design session)

• Storage: graph nodes in the existing Kuzu+Cozo DBs, not a separate store. Rationale: infigraph already has graph infrastructure, and knowledge naturally links to existing Symbol/File nodes (e.g. a knowledge fact about "why this function is structured this way" should be an edge from a Knowledge node to the relevant Symbol node).
• Confidence model: starts at 0.7 on extraction. User-corrected facts jump to 1.0. Decays 0.05/week if unused. Re-confirmed usage resets to 0.9. Anything that decays below 0.3 gets archived (not deleted).
• Retrieval: two paths — (1) auto-inject into symbol_context/get_doc_context responses (zero-effort, facts appear alongside code context automatically), and (2) a dedicated knowledge_search tool for explicit queries.
• Scope for v1: project-scoped, not team-shared. A sync/sharing layer is an explicit non-goal until v1 proves out.
• Scale assumption: designed around ~10K max facts per project. If that's insufficient in practice, needs pruning or tiered storage — not designed yet.

Open question (blocking, needs a decision before implementation)

Extraction approach is undecided: rule-based, LLM-assisted, or hybrid. Suggested path: start rule-based (extract from save_session's structured decisions/constraints/assumptions fields, which are already semi-structured), and layer in LLM-assisted extraction from free-text narrative later if rule-based proves too shallow.

Scope (proposed, not yet built)

1. Knowledge node schema in Kuzu + Cozo (with a HAS_KNOWLEDGE edge type linking to Symbol/File nodes).
2. Extraction pipeline: turn saved session data into Knowledge nodes (approach per the open question above).
3. New MCP tools: knowledge_search, knowledge_add, knowledge_forget.
4. Confidence decay job (weekly, per the model above).

Related work already shipped

LM2 Phases 3 (confidence decay) and 4 (auto-injection + selective indexing) are already implemented — this issue is specifically about the parts that were designed but not yet built: the dedicated node schema, the extraction pipeline, and the explicit query/mutation tools.

[murari316]

Additional detail recovered from session narrative (2026-06-29)

More is already implemented than the original issue stated. By 2026-06-29, LM2 had 20 mechanisms with full test coverage (all passing), including several relevant to persistent knowledge specifically:

• Symbol clustering (already built): co-occurrence tracking (record_cooccurrence, writes pairs to JSON), cluster construction via union-find (build_symbol_clusters, configurable min_cooccurrence threshold), O(1) cluster-peer lookup (load_cluster_peers), and a cluster-boost mechanism (apply_cluster_boost, +0.1 score to peer symbols) wired into memory_context's ranking.
• Auto-trigger consolidation (already built): tool_save_session checks embedding-store size after each save; once it exceeds 50 entries, tool_consolidate_memory runs automatically with a default similarity threshold of 0.7, and the consolidation result is appended to the save response.
• Auto-injection (already built, confirmed working): both the "skip connection" (auto_inject_session_context in graph.rs, wired into symbol_context and get_doc_context) and the "input gate" (score_session_value in session.rs, sets initial confidence at save time) were verified already implemented and were given dedicated test coverage (6 new tests) during this session.

This issue's original scope (dedicated Knowledge node schema distinct from sessions, extraction pipeline, knowledge_search/add/forget tools) is still the accurate remaining gap — the above is context on how much of the surrounding LM2 infrastructure is already solid, so the new work here can build directly on it rather than starting from scratch.

Previously undocumented, concrete design question with a recommendation: multi-dev knowledge merge

The original issue said knowledge base is "project-scoped, not team-shared" for v1, treating team-sharing as an explicit non-goal. The recovered narrative shows this was actually discussed further with a specific recommendation on record, worth capturing before this is built:

Question raised: if/when knowledge does need to be shared across multiple developers working on the same project, how do their individually-accumulated knowledge bases merge? Four options were discussed (only the winning one was recorded in detail):

Recommended: Option D — JSON files as the git-mergeable source of truth, with Kuzu as a rebuild-on-index query layer. This mirrors the pattern already used for named sessions (which are stored as JSON files, e.g. named_persistent-knowledge.json, and are human/git-diffable) rather than living solely inside the graph DB. The graph DB becomes a derived, rebuildable index over the JSON source of truth, not the canonical store — which is exactly what makes multi-dev merging tractable, since JSON files merge via normal git conflict resolution while a binary graph DB does not.

This was awaiting explicit user decision before implementation as of the last save on this topic — worth resolving as part of scoping this issue, since it affects the storage design (issue currently says "graph nodes in the existing Kuzu+Cozo DBs" as the sole store, which would need to become "JSON source of truth + Kuzu/Cozo as a derived query index" if Option D is adopted).