feat(search): BM25 lexical fallback via LadybugDB FTS

README.md

@@ -39,7 +39,7 @@ The rest of this README is the install, the tool/command orientation, and the re
 pip install java-codebase-rag
 ```
 
-Python **3.11+** required, on **Linux, macOS, and Windows**. On Linux, Windows, and **Apple Silicon** Macs every native dependency (LanceDB, LadybugDB/kuzu, CocoIndex) ships a wheel and you get the full semantic + graph search. **Intel Macs (x86_64) install graph-only**: PyTorch ≥2.3 and LanceDB ≥0.26 dropped macOS Intel wheels, so the vector stack is auto-excluded via PEP 508 markers — `pip install java-codebase-rag` works out of the box, the graph layer (`find` / `describe` / `neighbors` / `resolve`) is fully usable, and the `search` tool falls back to **lexical (keyword) search** over the symbol graph (same tool contract, keyword-ranked instead of semantic; an advisory notes the mode). Semantic/vector search needs Apple Silicon, Linux, or Windows. After install, `java-codebase-rag --help` should print the CLI groups.
+Python **3.11+** required, on **Linux, macOS, and Windows**. On Linux, Windows, and **Apple Silicon** Macs every native dependency (LanceDB, LadybugDB, CocoIndex) ships a wheel and you get the full semantic + graph search. **Intel Macs (x86_64) install graph-only**: PyTorch ≥2.3 and LanceDB ≥0.26 dropped macOS Intel wheels, so the vector stack is auto-excluded via PEP 508 markers — `pip install java-codebase-rag` works out of the box, the graph layer (`find` / `describe` / `neighbors` / `resolve`) is fully usable, and the `search` tool falls back to **lexical search** over the symbol graph — BM25-ranked over a LadybugDB full-text index (same tool contract, keyword-ranked instead of semantic; an advisory notes the mode). Semantic/vector search needs Apple Silicon, Linux, or Windows. After install, `java-codebase-rag --help` should print the CLI groups.
 The package includes the CocoIndex lifecycle dependency used by `init`, `increment`, `reprocess`, and `erase` on platforms that have it (it is absent on Intel Mac).
 
 ### Interactive setup (recommended)

docs/AGENT-GUIDE.md

@@ -207,7 +207,7 @@ Prefer **`resolve` → `describe(id=…)`** over **`describe(fqn=…)`** when an
 
 Ranked chunk retrieval. Args: `query`, `table` (`java`|`sql`|`yaml`|`all`, default `java`), `hybrid` (bool), `limit` (default 5), `offset`, `path_contains`, optional `filter` (symbol-applicable `NodeFilter` only), optional `chunks` (bool, default `false`). Returns one row per `primary_type_fqn` (symbol/type) by default; set `chunks=true` to restore chunk-level output. When deduped, each hit includes a `chunks` field (≥1) indicating how many chunks were collapsed into that hit.
 
-> **Intel Mac (graph-only) installs:** `search` runs the **lexical backend** — keyword relevance over the symbol graph instead of embeddings, behind this same contract. Same `query`/`table`/`filter`/`limit`/`chunks` behavior; results are keyword-ranked (not semantic), `hybrid` is ignored, `sql`/`yaml` tables aren't indexed (only Java symbols), and an `advisories` entry + `lexical_mode=true` flag note the mode. Structural discovery (`find`/`describe`/`neighbors`/`resolve`) is unaffected.
+> **Intel Mac (graph-only) installs:** `search` runs the **lexical backend** — BM25 keyword ranking over the symbol graph's LadybugDB full-text index instead of embeddings, behind this same contract. Same `query`/`table`/`filter`/`limit`/`chunks` behavior; results are keyword-ranked (not semantic), `hybrid` is ignored, `sql`/`yaml` tables aren't indexed (only Java symbols), and an `advisories` entry + `lexical_mode=true` flag note the mode. Structural discovery (`find`/`describe`/`neighbors`/`resolve`) is unaffected.
 
 #### `find`
 

docs/ARCHITECTURE.md

@@ -68,7 +68,7 @@ java_codebase_rag/pipeline.py
 ```
 MCP tool call (server.py)  ──asyncio.to_thread──▶  mcp_v2.*
   ├─ search ─▶ search_lancedb.run_search    (vector / hybrid; optional graph-expand + RRF rank fusion)
-  │            └─ lancedb import absent (Intel Mac) → search_lexical (keyword over graph)
+  │            └─ lancedb import absent (Intel Mac) → search_lexical (BM25 over Symbol FTS index; heuristic scan fallback)
   ├─ find / describe / neighbors ─▶ ladybug_queries.LadybugGraph   (Cypher)
   └─ resolve ─▶ resolve_service.resolve_v2   (cascade → status one | many | none)
        on empty ─▶ absence_diagnosis.diagnose   → verdict + (optional) proof
@@ -77,13 +77,13 @@ MCP tool call (server.py)  ──asyncio.to_thread──▶  mcp_v2.*
 
 | Tool | Backing | Notes |
 | --- | --- | --- |
-| `search` | Lance vector/hybrid, or lexical fallback | dedup by FQN; role weights via `search_scoring` |
+| `search` | Lance vector/hybrid, or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring` |
 | `find` | Ladybug Cypher | required `NodeFilter`; strict per-kind frame |
 | `describe` | Ladybug Cypher | node record + `edge_summary` (composed/override rollups) |
 | `neighbors` | Ladybug Cypher | one hop; `direction` + `edge_types` required; dot-key composed edges |
 | `resolve` | Ladybug Cypher | per-kind generators exact→fuzzy; cap 10 candidates |
 
-**Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs.
+**Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_scoring`. The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs.
 
 ## Stores
 
@@ -108,7 +108,7 @@ Dev workflow (editable install, test-reset ritual, full-suite discipline) — se
 
 | Constant | Value / location |
 | --- | --- |
-| `ONTOLOGY_VERSION` | `18` — `ast_java.py:87` |
+| `ONTOLOGY_VERSION` | `19` — `ast_java.py:87` |
 | `LANCE_TABLE_NAMES` | 3 tables — `java_codebase_rag/lance_optimize.py:35` |
 | Graph passes | 6 (labels `build_ast_graph.py:83`) |
 | Incremental cap | `expansion_cap=50` — `build_ast_graph.py:3800` |
@@ -117,4 +117,4 @@ Dev workflow (editable install, test-reset ritual, full-suite discipline) — se
 
 ## TL;DR
 
-Two stores built in lockstep — LanceDB vectors via CocoIndex, LadybugDB graph via a 6-pass tree-sitter build — queried by 5 MCP tools that split cleanly: `search` → vector/lexical, `find`/`describe`/`neighbors`/`resolve` → Cypher. Hints and absence wrap every response; `ONTOLOGY_VERSION=18` is the rebuild/staleness contract. Contributors extend via `EDGE_SCHEMA` + builder passes, and bump the version on any semantic change.
+Two stores built in lockstep — LanceDB vectors via CocoIndex, LadybugDB graph via a 6-pass tree-sitter build — queried by 5 MCP tools that split cleanly: `search` → vector/lexical, `find`/`describe`/`neighbors`/`resolve` → Cypher. Hints and absence wrap every response; `ONTOLOGY_VERSION=19` is the rebuild/staleness contract. Contributors extend via `EDGE_SCHEMA` + builder passes, and bump the version on any semantic change.

docs/CONFIGURATION.md

@@ -437,15 +437,17 @@ Combined, these pull `processClientMessage` / `pickEligibleOperator` / `onOperat
 
 #### Graph-only (macOS Intel) lexical ranking
 
-On Intel Mac installs the vector stack is absent (see `README.md`), so `search` runs the **lexical backend** — keyword relevance over the symbol graph instead of embeddings, behind the same tool contract. Ranking components, normalized into a `[0,1]` score:
+On Intel Mac installs the vector stack is absent (see `README.md`), so `search` runs the **lexical backend** — keyword ranking over the symbol graph instead of embeddings, behind the same tool contract. It is **BM25-first**: at index time every `Symbol` gets a `search_text` column (camelCase-split tokens of name + fqn + signature + annotations + capabilities, tokenized with the same splitter the query path uses) and a LadybugDB FTS index (`sym_fts`, Okapi BM25, porter stemmer) over it. At query time `QUERY_FTS_INDEX` fetches the top-K candidates DB-side, which are then re-ranked in Python by the heuristic below and deduped by FQN.
+
+The heuristic re-rank decides final order (what `--explain` reports as `relevance=` / `name=` / `type=` / `fqn=`, plus a `bm25=` component for the index score):
 
 - **Name token overlap** — strongest signal (weight `0.45`); full query-token coverage of the declaration name scores `1.0`.
 - **Type-name overlap** — `+0.05`/hit, capped `+0.10` (same convention as the vector symbol bonus above).
 - **FQN / package overlap** — weight `0.20`.
 - **Signature / annotation / capability text overlap** — weight `0.15`.
 - **Role weights** — the same table above applies as a tie-breaker/booster.
 
-Rows with **no keyword overlap** are dropped — role alone never qualifies a hit (it only reorders matches). Locking `role=` / `exclude_roles` still skips the role weight. `--explain` surfaces `name=` / `type=` / `fqn=` / `relevance=` components. `sql` / `yaml` tables aren't indexed in graph-only mode (only Java symbols are), and `hybrid` is ignored (lexical-only).
+BM25 only selects the candidate pool (so large repos no longer hit the old bounded Python scan); the heuristic decides order. If the FTS index or extension is unavailable (older graph, or an offline first run where `INSTALL FTS` can't fetch it), the backend falls back to that bounded scan over `Symbol` rows using the same heuristic. Locking `role=` / `exclude_roles` skips the role weight. `sql` / `yaml` tables aren't indexed in graph-only mode (only Java symbols are), and `hybrid` is ignored (lexical-only).
 
 ### Debugging empty `context_before` / `context_after`
 

src/java_codebase_rag/ast/ast_java.py

@@ -84,7 +84,8 @@
 # Phase 11: `EDGE_SCHEMA` in `java_ontology.py` (canonical edge navigation schema; v14 re-index).
 # Phase 12: CALLS `callee_declaring_role`, supertype-walk dedup, pass3 unresolved counters (v15 re-index).
 # Bumps whenever extraction / enrichment semantics change.
-ONTOLOGY_VERSION = 18
+# Phase 13: Symbol.search_text + LadybugDB FTS (Okapi BM25) index for lexical search (v19 re-index).
+ONTOLOGY_VERSION = 19
 
 ROLE_ANNOTATIONS: dict[str, str] = {
     # Spring Web

src/java_codebase_rag/graph/build_ast_graph.py

@@ -68,6 +68,7 @@
     symbol_id,
 )
 from java_codebase_rag.graph.path_filtering import LayeredIgnore, iter_java_source_files
+from java_codebase_rag.search.search_scoring import SYMBOL_FTS_INDEX as _SYMBOL_FTS_INDEX, _split_identifier
 from java_codebase_rag.graph.java_ontology import (
     CLIENT_KIND_FEIGN_METHOD,
     CLIENT_KIND_REST_TEMPLATE,
@@ -801,6 +802,12 @@ def _delete_file_scope(
     Producer nodes use DETACH DELETE as a safety net for any edges missed in
     Phase 1.
     """
+    # Symbol DELETEs below maintain the FTS index, which needs the extension loaded on
+    # this connection. Idempotent + best-effort (no-op when no index exists / FTS absent).
+    try:
+        conn.execute("LOAD EXTENSION FTS")
+    except Exception:
+        pass
     scope_files = changed_files | dependent_files
     scope_list = list(scope_files)
     changed_list = list(changed_files)
@@ -2927,7 +2934,8 @@ def _micro_factor(member: MemberEntry | None) -> float:
     "start_byte INT64, end_byte INT64, "
     "modifiers STRING[], annotations STRING[], capabilities STRING[], "
     "role STRING, signature STRING, parent_id STRING, resolved BOOLEAN, "
-    "generated BOOLEAN, generated_by STRING"
+    "generated BOOLEAN, generated_by STRING, "
+    "search_text STRING"
     ")"
 )
 
@@ -3050,7 +3058,28 @@ def _micro_factor(member: MemberEntry | None) -> float:
 )
 
 
+def _drop_symbol_fts_index_if_present(conn: ladybug.Connection) -> None:
+    """Drop the Symbol FTS index so the table drop below succeeds.
+
+    LadybugDB refuses ``DROP TABLE Symbol`` while an FTS index references it ("Cannot
+    delete node table ... referenced by index"), so the index must go first. No-op when
+    FTS is unavailable (offline first-run) or the index was never created — in those
+    cases nothing blocks the table drop. Best-effort: any failure is swallowed.
+    """
+    try:
+        conn.execute("LOAD EXTENSION FTS")
+        existing = conn.execute("CALL SHOW_INDEXES() RETURN index_name")
+        names: set[str] = set()
+        while existing.has_next():
+            names.add(existing.get_next()[0])
+        if _SYMBOL_FTS_INDEX in names:
+            conn.execute(f"CALL DROP_FTS_INDEX('Symbol', '{_SYMBOL_FTS_INDEX}')")
+    except Exception:
+        pass
+
+
 def _drop_all(conn: ladybug.Connection) -> None:
+    _drop_symbol_fts_index_if_present(conn)
     for stmt in (
         "DROP TABLE IF EXISTS DECLARES_CLIENT",
         "DROP TABLE IF EXISTS DECLARES_PRODUCER",
@@ -3101,6 +3130,80 @@ def _create_schema(conn: ladybug.Connection) -> None:
         conn.execute(stmt)
 
 
+_IDENT_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*")
+
+
+def _compute_symbol_search_text(
+    name: str,
+    fqn: str,
+    signature: str,
+    annotations: list[str],
+    capabilities: list[str],
+    package: str = "",
+) -> str:
+    """Build the BM25-indexable token soup for a Symbol (fork A).
+
+    camelCase / snake_case identifiers are split into lowercase tokens via the SAME
+    ``_split_identifier`` the query path uses (index- and query-time tokenization
+    must agree), then space-joined into one STRING that LadybugDB FTS porter-stems.
+    Name + fqn carry the strongest discovery signal; signature / annotations /
+    capabilities / package segments are weaker corroborators. Members reach this via
+    the same ``_node_row`` constructor as types.
+    """
+    toks: list[str] = []
+    for field in (name, fqn, signature, package):
+        for m in _IDENT_RE.findall(str(field or "")):
+            toks.extend(_split_identifier(m))
+    for lst in (annotations, capabilities):
+        for item in (lst or []):
+            for m in _IDENT_RE.findall(str(item)):
+                toks.extend(_split_identifier(m))
+    seen: set[str] = set()
+    out: list[str] = []
+    for t in toks:
+        if len(t) >= 2 and t not in seen:
+            seen.add(t)
+            out.append(t)
+    return " ".join(out)
+
+
+def _ensure_symbol_fts_index(conn: ladybug.Connection, *, verbose: bool) -> None:
+    """Best-effort create the Symbol FTS (Okapi BM25) index if absent (fork A).
+
+    Idempotent: skips when the index already exists. The FTS extension is fetched
+    from extension.ladybugdb.com on first ``INSTALL FTS`` (cached locally after); an
+    offline first run fails softly — ``run_lexical_search`` then falls back to the
+    heuristic over the bare Symbol scan. The index auto-maintains on later
+    COPY / MERGE / DELETE (verified), so this only runs on full builds and as a
+    cheap self-heal at the end of an incremental rebuild.
+    """
+    try:
+        try:
+            conn.execute("INSTALL FTS")  # already-installed raises; swallow
+        except Exception:
+            pass
+        conn.execute("LOAD EXTENSION FTS")
+        existing = conn.execute("CALL SHOW_INDEXES() RETURN index_name")
+        names: set[str] = set()
+        while existing.has_next():
+            names.add(existing.get_next()[0])
+        if _SYMBOL_FTS_INDEX not in names:
+            conn.execute(
+                f"CALL CREATE_FTS_INDEX('Symbol', '{_SYMBOL_FTS_INDEX}', "
+                "['search_text'], stemmer := 'porter')"
+            )
+            if verbose:
+                _verbose_stderr_line(
+                    f"[graph] fts · created Symbol {_SYMBOL_FTS_INDEX} (BM25 over search_text)"
+                )
+    except Exception as e:
+        if verbose:
+            _verbose_stderr_line(
+                f"[graph] fts · unavailable ({type(e).__name__}: {e}); "
+                "lexical search will use the heuristic scan"
+            )
+
+
 def _node_row(**kwargs) -> dict:
     base = {
         "kind": "", "name": "", "fqn": "", "package": "",
@@ -3109,9 +3212,16 @@ def _node_row(**kwargs) -> dict:
         "start_byte": 0, "end_byte": 0,
         "modifiers": [], "annotations": [], "capabilities": [],
         "role": "OTHER", "signature": "", "parent_id": "", "resolved": True,
-        "generated": False, "generated_by": None,
+        "generated": False, "generated_by": None, "search_text": "",
     }
     base.update(kwargs)
+    # Derive the BM25 token soup unless the caller set it explicitly.
+    if not base.get("search_text"):
+        base["search_text"] = _compute_symbol_search_text(
+            base.get("name", ""), base.get("fqn", ""), base.get("signature", ""),
+            base.get("annotations", []), base.get("capabilities", []),
+            base.get("package", ""),
+        )
     return base
 
 
@@ -3161,7 +3271,7 @@ def _existing_node_ids(conn: ladybug.Connection) -> set[str]:
     "id", "kind", "name", "fqn", "package", "module", "microservice",
     "filename", "start_line", "end_line", "start_byte", "end_byte",
     "modifiers", "annotations", "capabilities", "role", "signature", "parent_id", "resolved",
-    "generated", "generated_by"
+    "generated", "generated_by", "search_text"
 ]
 
 # Type declaration kinds. Tuple (not set) so the rendered SQL `IN` clause is
@@ -3185,7 +3295,7 @@ def _existing_node_ids(conn: ladybug.Connection) -> set[str]:
     "n.modifiers = $modifiers, n.annotations = $annotations, "
     "n.capabilities = $capabilities, n.role = $role, "
     "n.signature = $signature, n.parent_id = $parent_id, n.resolved = $resolved, "
-    "n.generated = $generated, n.generated_by = $generated_by"
+    "n.generated = $generated, n.generated_by = $generated_by, n.search_text = $search_text"
 )
 
 # Refresh every mutable Route field on an existing Route node by id. Mirrors the
@@ -3835,6 +3945,13 @@ def incremental_rebuild(
 
     db = ladybug.Database(str(ladybug_path))
     conn = ladybug.Connection(db)
+    # If a Symbol FTS index exists, the scoped DELETE/MERGE/COPY below must maintain it,
+    # which needs the FTS extension loaded on THIS connection. Best-effort: when FTS is
+    # unavailable no index exists, so there's nothing to maintain and DML proceeds fine.
+    try:
+        conn.execute("LOAD EXTENSION FTS")
+    except Exception:
+        pass
 
     # Check ontology version
     try:
@@ -4007,6 +4124,7 @@ def incremental_rebuild(
 
         # Update GraphMeta
         _write_meta(conn, tables_for_global, source_root)
+        _ensure_symbol_fts_index(conn, verbose=verbose)
 
         # Remove crash marker
         crash_marker_path.unlink(missing_ok=True)
@@ -4236,6 +4354,7 @@ def write_ladybug(
         if verbose:
             _verbose_stderr_line(f"[graph] writing · routes/exposes written in {time.time() - t2:.2f}s")
         _write_meta(conn, tables, source_root)
+        _ensure_symbol_fts_index(conn, verbose=verbose)
         conn.close()
         db.close()