feat(search): lexical fallback for graph-only / macOS Intel (PR-LEX)

.github/workflows/test.yml

@@ -19,13 +19,19 @@ jobs:
         # before this matrix. Promote an OS to a hard gate by dropping it from
         # the continue-on-error expression once it is reliably green.
         #
-        # macos-13 (Intel x86_64) is the only runner that proves the graph-only
-        # install/boot path (PEP 508 markers gate the vector trio off there).
-        # TEMPORARILY DISABLED: the leg was pending indefinitely. Re-add
-        # `macos-13` to the list below to restore Intel graph-only coverage;
-        # the graph-only boot is still covered cross-platform by
-        # tests/test_graph_only_boot.py (which blocks the vector modules in-process).
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        # macos-15-intel (x86_64) is the only runner that proves the graph-only
+        # install/boot path AND the lexical-search fallback (search_lexical): the
+        # PEP 508 markers in pyproject.toml gate the vector trio
+        # (cocoindex[lancedb]/lancedb/sentence-transformers) off on Intel, so this
+        # leg installs graph-only and every vector test file skips via
+        # pytest.importorskip. This is the only CI coverage for the graph-only /
+        # lexical path (the rest of the matrix is arm64/x64-with-vectors). NOTE:
+        # macos-13 (the previous Intel label) was retired by GitHub on 2025-12-04,
+        # which is why this leg previously hung "pending indefinitely";
+        # macos-15-intel is its supported replacement, available until 2027-08.
+        # Cross-platform graph-only boot is also covered by
+        # tests/test_graph_only_boot.py (blocks the vector modules in-process).
+        os: [ubuntu-latest, macos-latest, macos-15-intel, windows-latest]
     continue-on-error: ${{ matrix.os != 'ubuntu-latest' }}
     runs-on: ${{ matrix.os }}
     steps:
@@ -71,7 +77,7 @@ jobs:
         if: steps.changes.outputs.code == 'true'
         run: python scripts/generate_edge_navigation.py --check
       - name: Cache HuggingFace models
-        if: steps.changes.outputs.code == 'true' && matrix.os != 'macos-13'
+        if: steps.changes.outputs.code == 'true' && matrix.os != 'macos-15-intel'
         uses: actions/cache@v4
         with:
           path: ~/.cache/huggingface

README.md

@@ -41,7 +41,7 @@ The rest of this README is the install, the tool/command orientation, and the re
 pip install java-codebase-rag
 ```
 
-Python **3.11+** required, on **Linux, macOS, and Windows**. On Linux, Windows, and **Apple Silicon** Macs every native dependency (LanceDB, LadybugDB/kuzu, CocoIndex) ships a wheel and you get the full semantic + graph search. **Intel Macs (x86_64) install graph-only**: PyTorch ≥2.3 and LanceDB ≥0.26 dropped macOS Intel wheels, so the vector stack is auto-excluded via PEP 508 markers — `pip install java-codebase-rag` works out of the box, the graph layer (`find` / `describe` / `neighbors` / `resolve`) is fully usable, and the `search` (semantic) tool reports it is unavailable rather than failing. After install, `java-codebase-rag --help` should print the CLI groups.
+Python **3.11+** required, on **Linux, macOS, and Windows**. On Linux, Windows, and **Apple Silicon** Macs every native dependency (LanceDB, LadybugDB/kuzu, CocoIndex) ships a wheel and you get the full semantic + graph search. **Intel Macs (x86_64) install graph-only**: PyTorch ≥2.3 and LanceDB ≥0.26 dropped macOS Intel wheels, so the vector stack is auto-excluded via PEP 508 markers — `pip install java-codebase-rag` works out of the box, the graph layer (`find` / `describe` / `neighbors` / `resolve`) is fully usable, and the `search` tool falls back to **lexical (keyword) search** over the symbol graph (same tool contract, keyword-ranked instead of semantic; an advisory notes the mode). Semantic/vector search needs Apple Silicon, Linux, or Windows. After install, `java-codebase-rag --help` should print the CLI groups.
 The package includes the CocoIndex lifecycle dependency used by `init`, `increment`, `reprocess`, and `erase` on platforms that have it (it is absent on Intel Mac).
 
 ### Interactive setup (recommended)

docs/AGENT-GUIDE.md

@@ -192,6 +192,8 @@ Prefer **`resolve` → `describe(id=…)`** over **`describe(fqn=…)`** when an
 
 Ranked chunk retrieval. Args: `query`, `table` (`java`|`sql`|`yaml`|`all`, default `java`), `hybrid` (bool), `limit` (default 5), `offset`, `path_contains`, optional `filter` (symbol-applicable `NodeFilter` only), optional `chunks` (bool, default `false`). Returns one row per `primary_type_fqn` (symbol/type) by default; set `chunks=true` to restore chunk-level output. When deduped, each hit includes a `chunks` field (≥1) indicating how many chunks were collapsed into that hit.
 
+> **Intel Mac (graph-only) installs:** `search` runs the **lexical backend** — keyword relevance over the symbol graph instead of embeddings, behind this same contract. Same `query`/`table`/`filter`/`limit`/`chunks` behavior; results are keyword-ranked (not semantic), `hybrid` is ignored, `sql`/`yaml` tables aren't indexed (only Java symbols), and an `advisories` entry + `lexical_mode=true` flag note the mode. Structural discovery (`find`/`describe`/`neighbors`/`resolve`) is unaffected.
+
 #### `find`
 
 Exact listing for one kind. Args: `kind` (`symbol`|`route`|`client`|`producer`), **`filter`** (required object), `limit` (default 25), `offset`. Returns `NodeRef` rows (`id`, `kind`, `fqn`, `microservice`, `module`, `role` on symbols, `symbol_kind` on symbols).

docs/CONFIGURATION.md

@@ -361,6 +361,18 @@ On top of role weights, Java chunks receive a **symbol-match bonus** (exposed as
 
 Combined, these pull `processClientMessage` / `pickEligibleOperator` / `onOperatorAssigned` chunks — and the classes that own them — above ones that only enqueue or configure. Like role weights, the bonus is **skipped when the caller locks `role=`**.
 
+#### Graph-only (macOS Intel) lexical ranking
+
+On Intel Mac installs the vector stack is absent (see `README.md`), so `search` runs the **lexical backend** — keyword relevance over the symbol graph instead of embeddings, behind the same tool contract. Ranking components, normalized into a `[0,1]` score:
+
+- **Name token overlap** — strongest signal (weight `0.45`); full query-token coverage of the declaration name scores `1.0`.
+- **Type-name overlap** — `+0.05`/hit, capped `+0.10` (same convention as the vector symbol bonus above).
+- **FQN / package overlap** — weight `0.20`.
+- **Signature / annotation / capability text overlap** — weight `0.15`.
+- **Role weights** — the same table above applies as a tie-breaker/booster.
+
+Rows with **no keyword overlap** are dropped — role alone never qualifies a hit (it only reorders matches). Locking `role=` / `exclude_roles` still skips the role weight. `--explain` surfaces `name=` / `type=` / `fqn=` / `relevance=` components. `sql` / `yaml` tables aren't indexed in graph-only mode (only Java symbols are), and `hybrid` is ignored (lexical-only).
+
 ### Debugging empty `context_before` / `context_after`
 
 If `context_neighbors=1` returns empty context strings, set `JAVA_CODEBASE_RAG_DEBUG_CONTEXT=1` in the MCP server env before launching. The server logs (to stderr) why expansion bailed: missing schema columns, empty bucket scan, chunk not found in bucket, or underlying scan error. Typical causes are (a) a stale server that hasn't reloaded after a reindex, or (b) an index missing `range_start` / `range_end` columns — the code falls back to exact-text matching, so re-running fixes it.

java_codebase_rag/jrag.py

@@ -4321,13 +4321,20 @@ def _cmd_search(args: argparse.Namespace) -> int:
             d["kind"] = "search_hit"
         # Add explain token when --explain is set
         if args.explain:
-            from search_lancedb import explain_score_components
+            # search_lancedb is unimportable on graph-only (macOS Intel) installs —
+            # lancedb/sentence-transformers are excluded by the PEP 508 markers in
+            # pyproject.toml, and this module imports them at module top. Importing the
+            # explain renderer from there would crash `jrag search ... --explain` on the
+            # exact Intel install that runs the lexical path. search_scoring is
+            # dependency-free and always installed, so the explain import works everywhere.
+            from search_scoring import explain_score_components
             comps = d.get("score_components")
             d["explain"] = explain_score_components(
                 comps,
                 role=d.get("role"),
                 hybrid=bool(args.hybrid),
                 graph_expanded=False,
+                lexical=bool(getattr(out, "lexical_mode", False)),
             )
         hit_dicts.append(d)
 

mcp_v2.py

@@ -520,6 +520,10 @@ class SearchOutput(BaseModel):
     )
     advisories: list[str] = Field(default_factory=list, description="Pure informational text with no tool call suggestion")
     hints_structured: list[StructuredHint] = Field(default_factory=list, description=MCP_HINTS_STRUCTURED_FIELD_DESCRIPTION)
+    lexical_mode: bool = Field(
+        default=False,
+        description="True when results come from the graph-only lexical (keyword) backend instead of semantic/vector search.",
+    )
 
 
 class FindOutput(BaseModel):
@@ -850,97 +854,131 @@ def search_v2(
         if nf and (err := _nodefilter_applicability_error("symbol", nf)):
             _log_fail_loud("applicability")
             return SearchOutput(success=False, message=err, advisories=[], limit=None, offset=None)
-        if run_search is None:
-            # Graph-only install (no torch/lancedb): the vector stack is absent. Return a
-            # clean failure rather than crashing so the server keeps serving graph tools.
-            return SearchOutput(
-                success=False,
-                message="Vector search unavailable: graph-only mode (vector stack not installed).",
-                advisories=[],
-                limit=None,
-                offset=None,
-            )
-        # hybrid + table='all' is unsupported (hybrid fuses vector+FTS on ONE
-        # table); fail fast with a clean envelope BEFORE loading the embedding
-        # model. run_search also guards this — this is the user-facing fast path.
-        if hybrid and table == "all":
-            return SearchOutput(
-                success=False,
-                message="hybrid search requires a single table; use java, sql, or yaml (not all)",
-                advisories=[],
-                limit=None,
-                offset=None,
-            )
-        model_name = resolved_sbert_model_for_process_env(SBERT_MODEL)
-        device = os.environ.get("SBERT_DEVICE") or None
-        model = _get_sentence_transformer(model_name, device)
-        uri = os.environ.get("JAVA_CODEBASE_RAG_INDEX_DIR", "").strip() or str(
-            (Path.cwd() / ".java-codebase-rag").resolve()
-        )
-        uri_path = Path(uri)
-        if not uri.startswith(("s3://", "gs://", "az://")) and uri_path.exists():
-            uri = str(uri_path.resolve())
-        table_keys = list(TABLES) if table == "all" else [table]
-
-        # Graceful fallback: if hybrid=True and FTS index is missing (old index),
-        # retry with hybrid=False and return vector-only results with an advisory.
         advisories: list[str] = []
-        try:
-            rows = run_search(
+        lexical_mode = run_search is None
+        if lexical_mode:
+            # Graph-only install (macOS Intel: no torch/lancedb). Fall back to lexical
+            # (keyword) search over the symbol graph that graph-only mode already builds.
+            # run_lexical_search returns rows in the same shape as run_search, so the
+            # shared row->hit loop below works unchanged. It raises (message contains
+            # "lexical search unavailable") when no graph exists — caught by the outer
+            # try -> success=False. It returns [] for sql/yaml (advisory below) and for
+            # empty-but-valid results.
+            try:
+                from search_lexical import run_lexical_search
+            except ImportError:  # pragma: no cover - search_lexical has no heavy deps
+                run_lexical_search = None  # type: ignore[assignment]
+            if run_lexical_search is None:
+                return SearchOutput(
+                    success=False,
+                    message="search unavailable: graph-only mode and lexical backend not importable.",
+                    advisories=[],
+                    limit=None,
+                    offset=None,
+                )
+            advisories.append(
+                "lexical (graph-only) mode — keyword ranking only; "
+                "semantic/vector search requires Apple Silicon, Linux, or Windows"
+            )
+            if table in ("sql", "yaml", "all"):
+                advisories.append(
+                    "sql/yaml tables are not indexed in graph-only mode; only Java symbols were searched"
+                )
+            if hybrid:
+                advisories.append("hybrid is ignored in graph-only lexical mode")
+            rows = run_lexical_search(
                 query,
-                uri=uri,
-                table_keys=table_keys,
-                hybrid=hybrid,
+                table=table,
                 limit=limit,
                 offset=offset,
-                path_substring=path_contains,
-                model_name=model_name,
-                device=device,
-                model=model,
-                # Push the NodeFilter structural predicates into the LanceDB query so
-                # they apply BEFORE pagination (issue #353) — previously they were only
-                # a post-filter on the already-paginated page, which could shrink or
-                # empty filtered pages even when many matches existed deeper in the
-                # ranking. _node_matches_filter below still re-checks every row (it
-                # covers the non-pushdownable fields and is the contract guarantee).
-                role=nf.role if nf else None,
-                module=nf.module if nf else None,
-                microservice=nf.microservice if nf else None,
-                capability=nf.capability if nf else None,
-                exclude_roles=nf.exclude_roles if nf else None,
-                dedup_by_fqn=dedup,
+                path_contains=path_contains,
+                filter=nf,
+                explain=explain,
+                dedup=dedup,
+                advisories=advisories,
+                graph=graph,
+            )
+        else:
+            # hybrid + table='all' is unsupported (hybrid fuses vector+FTS on ONE
+            # table); fail fast with a clean envelope BEFORE loading the embedding
+            # model. run_search also guards this — this is the user-facing fast path.
+            if hybrid and table == "all":
+                return SearchOutput(
+                    success=False,
+                    message="hybrid search requires a single table; use java, sql, or yaml (not all)",
+                    advisories=[],
+                    limit=None,
+                    offset=None,
+                )
+            model_name = resolved_sbert_model_for_process_env(SBERT_MODEL)
+            device = os.environ.get("SBERT_DEVICE") or None
+            model = _get_sentence_transformer(model_name, device)
+            uri = os.environ.get("JAVA_CODEBASE_RAG_INDEX_DIR", "").strip() or str(
+                (Path.cwd() / ".java-codebase-rag").resolve()
             )
-        except Exception as exc:
-            # Check if this is a missing-FTS error (old index built before PR-SEARCH-3)
-            exc_text = str(exc).lower()
-            is_fts_missing = "full text search" in exc_text or "inverted index" in exc_text
-            if hybrid and is_fts_missing:
-                # Retry with vector-only search
+            uri_path = Path(uri)
+            if not uri.startswith(("s3://", "gs://", "az://")) and uri_path.exists():
+                uri = str(uri_path.resolve())
+            table_keys = list(TABLES) if table == "all" else [table]
+
+            # Graceful fallback: if hybrid=True and FTS index is missing (old index),
+            # retry with hybrid=False and return vector-only results with an advisory.
+            try:
                 rows = run_search(
                     query,
                     uri=uri,
                     table_keys=table_keys,
-                    hybrid=False,  # Fallback to vector-only
+                    hybrid=hybrid,
                     limit=limit,
                     offset=offset,
                     path_substring=path_contains,
                     model_name=model_name,
                     device=device,
                     model=model,
+                    # Push the NodeFilter structural predicates into the LanceDB query so
+                    # they apply BEFORE pagination (issue #353) — previously they were only
+                    # a post-filter on the already-paginated page, which could shrink or
+                    # empty filtered pages even when many matches existed deeper in the
+                    # ranking. _node_matches_filter below still re-checks every row (it
+                    # covers the non-pushdownable fields and is the contract guarantee).
                     role=nf.role if nf else None,
                     module=nf.module if nf else None,
                     microservice=nf.microservice if nf else None,
                     capability=nf.capability if nf else None,
                     exclude_roles=nf.exclude_roles if nf else None,
                     dedup_by_fqn=dedup,
                 )
-                advisories.append(
-                    f"hybrid unavailable on table '{table}' (FTS index missing on this index built before "
-                    f"PR-SEARCH-3); fell back to vector-only — reindex to enable hybrid"
-                )
-            else:
-                # Non-FTS error: surface as structured failure
-                raise
+            except Exception as exc:
+                # Check if this is a missing-FTS error (old index built before PR-SEARCH-3)
+                exc_text = str(exc).lower()
+                is_fts_missing = "full text search" in exc_text or "inverted index" in exc_text
+                if hybrid and is_fts_missing:
+                    # Retry with vector-only search
+                    rows = run_search(
+                        query,
+                        uri=uri,
+                        table_keys=table_keys,
+                        hybrid=False,  # Fallback to vector-only
+                        limit=limit,
+                        offset=offset,
+                        path_substring=path_contains,
+                        model_name=model_name,
+                        device=device,
+                        model=model,
+                        role=nf.role if nf else None,
+                        module=nf.module if nf else None,
+                        microservice=nf.microservice if nf else None,
+                        capability=nf.capability if nf else None,
+                        exclude_roles=nf.exclude_roles if nf else None,
+                        dedup_by_fqn=dedup,
+                    )
+                    advisories.append(
+                        f"hybrid unavailable on table '{table}' (FTS index missing on this index built before "
+                        f"PR-SEARCH-3); fell back to vector-only — reindex to enable hybrid"
+                    )
+                else:
+                    # Non-FTS error: surface as structured failure
+                    raise
         hits: list[SearchHit] = []
         for row in rows:
             if path_contains and path_contains not in str(row.get("filename") or ""):
@@ -964,6 +1002,7 @@ def search_v2(
             offset=offset,
             advisories=advisories + raw_advisories,  # Merge fallback + hints advisories
             hints_structured=_to_structured_hints(raw_struct),
+            lexical_mode=lexical_mode,
         )
     except Exception as exc:
         return SearchOutput(success=False, message=str(exc), advisories=[], limit=None, offset=None)