feat: reasoning-training pipeline — mine thinking → distill patterns → inject (run 007)#82
Merged
Merged
Conversation
added 13 commits
July 16, 2026 22:33
Brain-scoped reasoning_traces staging buffer mirroring the tool_events pattern, for the reasoning-training mining pipeline (run 007 Faza 1). - sqlite_schema: SCHEMA_VERSION 39->40, (39,40) migration + SCHEMA base block - surrealdb/schema: DEFINE TABLE reasoning_traces SCHEMAFULL + fields/indexes - SQLiteReasoningTracesMixin + SurrealDBReasoningTracesMixin (dedup by trace_hash: SQLite UNIQUE + INSERT OR IGNORE, SurrealDB pre-filter SELECT) - base.py graceful no-op defaults; InMemoryStorage functional impl; wired into SQLiteStorage / SurrealDBStorage - tests: real-DB SQLite + mock SurrealDB mixin (live-verified on SurrealDB v3.2.0)
Add the frozen ReasoningTrainingConfig dataclass (mining + injection settings,
opt-in/off by default) to UnifiedConfig, with config.toml + env support
(run 007 Faza 2).
- ReasoningTrainingConfig (all plan fields: mining/injection toggles, mining_models
globs, injection_map target->source, categories, trace/scan/retention/cap limits,
distiller + confidence knobs, redact_secrets)
- load: [reasoning_training] TOML section + [reasoning_training.injection_map] subtable,
env overrides SURREAL_MEMORY_REASONING_{MINING,INJECTION,MODELS,INJECTION_MAP}
- save: emits the section incl. injection_map subtable; adds _sanitize_toml_glob so
glob model names (claude-opus-*) survive serialization (the strict _sanitize_toml_str
would drop the '*')
- tests: defaults, from/to_dict roundtrip, min_confidence clamp, injection_map dict/list
forms, TOML glob+injection_map roundtrip, env overrides, empty-env no-op
Mine model thinking blocks from ~/.claude transcripts into the reasoning_traces staging table, driven by consolidation (run 007 Faza 3). - engine/reasoning_miner.py: scan_transcripts (path-guarded to ~/.claude, incremental last_line resume, trace_hash dedup, model normalize/denylist/glob, <synthetic>/opus skip, min/max char gating) + ingest_reasoning_traces - privacy: redaction-before-stage of BOTH thinking content AND user task_context via a reasoning-tuned pattern set (min_severity=2 + Bearer/prose/vendor-key patterns); opt-in (mining_enabled) only; signatures never stored - consolidation: PROCESS_REASONING_TRACES strategy (tier 1, guarded by mining_enabled + dry_run, excluded from scheduled defaults like LEARN_HABITS) + ConsolidationReport.reasoning_traces_ingested - per-line size cap before json.loads (crafted-transcript DoS guard) - tests: 21 miner/consolidation cases; live-verified against SurrealDB v3.2.0
Distill staged reasoning_traces into ReasoningBank pattern fibers (run 007 Faza 4), driven by consolidation's SUMMARIZE tier. - engine/reasoning_distiller.py: segment thinking into ~12 closed-vocabulary reasoning moves; classify category (bge-m3 seed-centroid cosine, keyword fallback, "other" excluded from coverage); cluster within (model, category) by cosine>=0.83 (embeddings) or move-set Jaccard>=0.6 (fallback) via UnionFind; build a ReasoningBank pattern (title/description=medoid/strategy=LCS moves) and materialize a fiber(_reasoning_pattern) + CONCEPT neuron + EFFECTIVE_FOR synapse, idempotent by a trace-set _reasoning_signature; reasoning_coverage(). - fully fail-soft: provider down -> keyword classification + move-set clustering. - consolidation: LEARN_REASONING strategy (SUMMARIZE tier, guarded by mining_enabled + dry_run, excluded from scheduled defaults) + handler wrapped in try/except + ConsolidationReport.reasoning_patterns_learned (+summary lines). - tests: 16 cases incl. cap-conservation, signature-collision, fail-soft embedder-raises; live-verified materialization on SurrealDB v3.2.0.
engine/reasoning_injection.py: resolve_active_model fallback chain (payload -> transcript tail -> env -> ~/.claude/settings.json -> default, alias-expanded, date-normalized) and build_injection_context (injection_map glob/default match, source-model pattern fibers ranked by confidence*frequency, <=2/category + injection_max_patterns + char budget, markdown block) plus session-scoped idempotency markers. hooks/session_start.py: append the reasoning block after the memories block, opt-in via reasoning_training.injection_enabled, exception-isolated per block. unified_config._get_surrealdb_storage: reuse the cached instance only when its _conn is live, else reinitialize (mirrors the SQLite cache) so the memories block closing shared storage no longer leaves the reasoning block a dead handle. Security: transcript_path (untrusted hook stdin) is allowlisted to ~/.claude via resolve()+is_relative_to; stat/read faults degrade to the fallback chain. Truncation of the bounded pattern-fiber fetch is warned, not silent.
hooks/user_prompt_submit.py: new UserPromptSubmit hook. From the 2nd prompt on,
the active model is resolvable from the transcript tail, so this injects the
model's learned reasoning strategies (opt-in via injection_enabled). Shares the
once-per-session marker with SessionStart, so the two never double-inject; always
exits 0 so it can never block the prompt.
engine/reasoning_injection.get_reasoning_context: shared orchestrator (config
guard -> already_injected short-circuit -> resolve_active_model -> open storage ->
build_injection_context -> always close -> mark_injected), used by both hooks.
session_start.get_reasoning_injection is now a thin delegate to it.
Registration mirrors SessionStart: pyproject entry point smem-hook-user-prompt-
submit + .claude-plugin/hooks/hooks.json UserPromptSubmit entry. (setup.py/
doctor.py wire only the original PreCompact/Stop/PostToolUse into settings.json;
SessionStart-class hooks are plugin-manifest only, so U6 matches that.)
Output contract fix (verified against the installed claude-code binary): context
reaches the model ONLY via hookSpecificOutput.additionalContext. Both hooks now
emit that envelope; the prior session_start {"type":"context","content"} form was
JSON-parsed with no recognized field, so U5's SessionStart injection (and the
memories block) were silently invisible to the model. Now fixed for both.
server/routes/reasoning_training.py: new local-only dashboard router
(/api/dashboard/reasoning) — GET /status (config + per-model trace/pattern
stats + per-model category coverage + mining-job state), PUT /config (partial
update with 422 validation: no-thinking models, glob/literal injection sources,
category charset, ranges), POST /mine (background job, 400 if mining disabled,
per-brain 409), GET /patterns[+/{id}], DELETE /patterns[/{id}|?model=],
DELETE /traces?model= (privacy wipe). Registered in routes/__init__ + app.py.
storage: new delete_reasoning_traces_by_model across base/sqlite/surrealdb/memory
(parameterized, empty-model no-op; SurrealDB live-verified on v3.2.0).
unified_config.create_isolated_storage: a fresh, non-cached SurrealDB instance so
the background mining job holds its own brain context — a concurrent request's
set_brain() on the shared singleton can no longer cross-write its graph writes
into the wrong brain. Mining job state is keyed per brain (no cross-brain block
or status leak). reasoning_distiller now honors mining_models in distillation,
not just ingestion, so POST /mine {models} restricts the whole run.
Mining runs ingest+distill directly (not ConsolidationEngine) so backfill/models
overrides are honored; gated on mining_enabled (privacy). Reviewed by
real-db-test-runner (live), security-reviewer, python-reviewer (all APPROVE).
dashboard/src/features/reasoning/: ReasoningPage (KPI cards, per-model coverage
bars + per-category badges, empty/loading/error states) composing MiningConfigCard
(mining toggle, per-model source checkboxes disabled for no-thinking models, Run
mining with backfill + 409/400 handling, per-model trace wipe), InjectionMappingCard
(injection toggle, source→target rows with stable-id keys, duplicate-target guard),
and PatternsTable (model/category filters, pagination with shrink-clamp, per-pattern
and per-model delete via ConfirmDialog).
api/hooks/useReasoning.ts: status (polls 3s while mining runs), optimistic config
update (no toggle flicker), trigger mining, patterns list, and pattern/traces deletes
with correct cache invalidation. api/client.ts: apiErrorMessage() parses FastAPI
{detail} so toasts show the message, not raw JSON. api/types.ts: interfaces mirroring
the U7 Pydantic models. Wired route (App.tsx), nav (Sidebar.tsx, GraduationCap), and
i18n (en.json). Rebuilt static/dist.
Patterns table refreshes when a mining run finishes. e2e (dashboard/e2e/reasoning.spec.ts,
2 tests) passes; tsc + eslint clean. react-reviewer APPROVE (2 HIGH + 5 MED fixed).
cli/commands/reasoning.py: new Typer sub-app `smem reasoning` with status (trace/pattern stats + per-model coverage), mine (--backfill/--dry-run/--models, gated on mining_enabled unless --force), patterns (--model/--category), and clear (--model privacy wipe). Registered in cli/main.py. Mirrors commands/habits.py. mcp/reasoning_handler.py: new ReasoningHandler mixin exposing smem_reasoning (actions status/mine/patterns/config), registered in mcp/server.py + schema in tool_schemas.py (58 tools now). mine runs on create_isolated_storage so a concurrent MCP call (HTTP transport is not serialized) can't redirect its graph writes into the wrong brain; gated on mining_enabled. Both surfaces are thin wrappers over the U3/U4/U7 engine + storage layer (get_reasoning_stats, reasoning_coverage, ingest+distill honoring mining_models, delete_reasoning_traces_by_model). Tests: test_cli_reasoning (8) + test_mcp_reasoning (7) + test_mcp count bump. python-reviewer APPROVE (1 HIGH + 2 MED fixed).
- bump cross-cutting assertions the feature legitimately shifted: SCHEMA_VERSION 39->40 (U1 reasoning_traces) in 5 tests; MCP full-tier tool count 57->58 (U9 smem_reasoning) in test_tool_tiers - conftest: import the real surrealdb SDK early so a unit-test module's `sys.modules["surrealdb"]=MagicMock()` stub can't leak into the live SurrealDB tests in the same serial session (no-op when SDK absent) - integration/test_surrealdb_synapse_migration: update stale v8 stamp assertions to v9 (SCHEMA_VERSION/TARGET_VERSION became 9 in v2.9.0, after this test file's last edit) and tolerate a retryable SurrealDB write-conflict in the two-parallel-migration race (exactly-once is still asserted independently via the count checks)
…ining # Conflicts: # tests/unit/test_surrealdb_store.py
…oning / smem reasoning)
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Privacy-preserving reasoning-training pipeline: mines model
thinkingblocks from~/.claudetranscripts, distills them into ReasoningBank pattern fibers, and injects the learned strategies into
other models' sessions. Fully opt-in (
reasoning_training.mining_enabled/injection_enabled,both default off). Surfaces across dashboard, CLI (
smem reasoning), and MCP (smem_reasoning).Units
21faa2freasoning_tracesstaging storage (4 backends) + schema v39→40 migrationa330262ReasoningTrainingConfig(frozen) + TOML/env wiring497a855reasoning_miner(scan/redact/ingest) +PROCESS_REASONING_TRACESb261ceareasoning_distiller(segment→classify→cluster→pattern fibers) +LEARN_REASONING4822968reasoning_injection+ SessionStart wiring + SurrealDB conn-liveness fix028ad9fget_reasoning_context+ hook output-contract fixe442e94/api/dashboard/reasoningrouter +delete_reasoning_traces_by_model+create_isolated_storage398a74dfeatures/reasoning/(page + 3 cards + hooks + route/nav/i18n + e2e)ca8295dsmem reasoningCLI +smem_reasoningMCP toolfb08965Test plan
uv run make verifyGREEN (CI-equivalent, noSURREALDB_URL): 6532 passed, 84 skipped, 1 xfailed,coverage 69.72% (≥67% gate), ruff + format + mypy(src) + security all clean.
Release notes / decisions needed before tag
main(2.10.5); version NOT bumped —v2.11.0is already taken by the parallelrelease-2.11.0branch, so a target version (e.g. 2.12.0) needs to be chosen before tagging.release-2.11.0); reconcile on merge ordering.Follow-ups (non-blocking, REVISIT)
_source_modelfilter infind_fibers(U5) to remove the post-LIMIT filter class.require_local_request(U7/M2) — spawned as a separate task.