Conversation
…nfig (#118) Inert pre-v1 scaffolding for semantic search: migration 016 trace_embeddings (local-only index), HTTPLLMClient.Embed (OpenAI /embeddings, batched), embedding BLOB codec, and the cortex.md search: config block with ValidateSearch. No query path or MCP changes yet (phases 2-4); off by default. Extensively tested: cross-platform build, -race x2 + stress x5, fuzzers, adversarial review, and a testbot MCP playbook (4/4).
…CLI (#119) Embedding-lifecycle core: Cortex.EmbedBackfill (idempotent/resumable/force/limit/model-change), EmbeddingStatus, the Embedder interface, and noema embeddings status|backfill. Off by default; no query path yet. Write-time async hooks deferred to phase 2b.
…120) Adds cortex.SemanticSearch/SemanticSimilar (cosine over stored vectors, finiteness + dim-mismatch skip), actor bump wrappers, and noema search/similar --semantic. MCP mode, vector cache, and hybrid RRF deferred to later phases.
search_traces and find_similar_traces gain a mode arg (lexical|semantic|hybrid) with graceful fallback to lexical. Agents can now drive semantic search over MCP. Hybrid returns semantic results until phase 4.
hybrid mode now does real Reciprocal Rank Fusion of FTS5 (BM25) + embedding (cosine) rankings (weight from search.hybrid_weight). Adds lexicalRanked, rrfFuse, HybridSearch/HybridSimilar, CLI --hybrid, MCP mode=hybrid. Completes the lexical|semantic|hybrid triad.
…123) Under serve, a periodic idempotent backfill maintainer keeps the embedding index fresh automatically (no per-mutation hook; never blocks writes). internal/embed.Maintainer + SearchConfig.EmbedIntervalSeconds, gated on the background-work lock.
…124) README semantic-search section + CLI/MCP tables; CLAUDE.md semantic section (replaces stale no-semantic-search line); AGENTS.md generator + get_instructions note the mode arg.
v0.14.0 security-review hardening: enforce MaxSearchQueryLen on the hybrid lexical + embed-query paths (DoS guard regression); make the MCP degradation note generic so the embedding endpoint URL/host isn't leaked to clients (logged server-side instead).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.14.0 — semantic & hybrid search
nextis a clean 7-ahead / 0-behind ofmain; squash-merge as one release commit, then tagv0.14.0.Adds an opt-in semantic search layer alongside lexical FTS5 — pure-Go (BLOB-stored embeddings + in-Go cosine; no CGo, no vector extension, no external DB), local-only (never federated), and degrade-safe (falls back to lexical when unconfigured/unreachable).
Highlights
search:config block incortex.md(off by default; reuses the consolidation LLM endpoint config).noema embeddings status|backfill;noema search/similar --semantic|--hybrid.search_traces/find_similar_tracesgain amodearg (lexical | semantic | hybrid) with graceful lexical fallback.hybrid_weight).016_trace_embeddings(new table; non-destructive, not federated).Validation
Extensively tested incl.
-raceand fuzz; validated end-to-end on a live local embedding model (semantic returned results where lexical returned none). Docs updated across README, CLAUDE.md, AGENTS.md, andget_instructions.Deferred (non-blocking): in-memory vector cache (speed at scale),
max_charsgraceful degradation polish.🤖 Generated with Claude Code