feat(search): lexical fallback for graph-only / macOS Intel (PR-LEX)

[HumanBean17]

Problem

On macOS Intel (x86_64) the vector stack is uninstallable — torch>=2.3 and lancedb>=0.26 both dropped macOS x86_64 wheels — so installs run graph-only and search returns an empty success=False envelope. Vectors can't come back cleanly: the last Intel torch (2.2.2) caps at Python 3.11, while requires-python = ">=3.11" actively supports 3.12/3.13.

Change

Make search return ranked keyword results on Intel via a new lexical backend over the symbol graph that graph-only mode already builds. Same tool contract — same SearchHit/SearchOutput, same NodeFilter, same CLI — just keyword-ranked instead of semantic, with an advisory noting the mode. No reindex, no new heavy deps; Apple Silicon/Linux/Windows are unchanged.

• search_scoring.py (new) — dependency-free scoring/dedup primitives extracted from search_lancedb.py (which imports lancedb at module top and is unimportable on Intel), so both backends share them. Re-exported from search_lancedb for backward compat. explain_score_components gains a lexical= flag.
• search_lexical.py (new) — run_lexical_search(): LadybugDB Symbol query with role/path pushdown, name/type/fqn/signature relevance scoring (normalized [0,1]), real source snippets read from disk (graph source_root, signature fallback), _dedup_by_fqn, and the same row-dict shape as run_search. Raises a clean "lexical search unavailable" error when no graph exists.
• mcp_v2.py — the run_search is None branch now dispatches to lexical and converges on the existing row→hit loop; SearchOutput.lexical_mode; advisories for lexical mode / sql-yaml-not-indexed / hybrid-ignored.
• pyproject.toml — register search_lexical + search_scoring as py-modules so the installed jrag CLI can import them from outside the repo. (Without this the real Intel path ships broken — caught by the full suite; an in-process sim missed it because the repo root was on sys.path.)
• Tests (tests/test_search_lexical.py, 10 cases; updated tests/test_graph_only_boot.py) + docs (README, CONFIGURATION, AGENT-GUIDE).

Verification

• Affected suites green: test_graph_only_boot, test_search_lexical, test_search_lancedb, test_search_lancedb_capability, test_lance_optimize, test_mcp_v2, test_mcp_v2_compose.
• Full suite: 1168 passed, 14 skipped.
• End-to-end via the installed jrag CLI from /tmp with the vector stack blocked (simulating macOS Intel):

  $ jrag search "distribution chunk service" --limit 3
  DistributionChunkService  @svc  role=SERVICE  file=svc/DistributionChunkService.java:2  score=0.880  chunks=4
  next: jrag inspect svc.DistributionChunkService
  

Notes / out of scope

• Ranking is heuristic keyword overlap (name > fqn > signature/annotation), not BM25. Upgradable to SQLite FTS5 later without changing the tool contract.
• Only Java Symbol nodes are searched; sql/yaml LanceDB tables don't exist in graph-only mode (advisory). Route/Client/Producer remain reachable via find/resolve/neighbors.
• Did not pursue pinning the old vector stack (stale, unsupported, Python-3.11-only).

🤖 Generated with Claude Code

[HumanBean17]

Addressed all three findings from the code review (commit `be6976e`). Each is a one/few-line fix with a regression test verified to catch the bug — both were invisible in dev/CI because the vector stack is installed there and the fixture corpus is small.

C1 (Critical) — jrag search --explain crashed on macOS Intel. jrag.py imported explain_score_components from search_lancedb, which is unimportable on graph-only installs (lancedb/sentence-transformers excluded by the PEP 508 markers). The re-export inside search_lancedb can't help — you can't import a name from a module that itself fails to load. Fixed: import from the dependency-free search_scoring. Test asserts the CLI source imports from search_scoring and that blocking the vector stack leaves search_scoring importable while search_lancedb is not.

C2 (Critical) — keyword ranking missed most matches on real repos. The lexical backend reused the vector-path overfetch limit (4× page) on an unordered Cypher scan with no ORDER BY. That's correct on the vector path (LanceDB returns rows pre-ranked by similarity) but on the lexical scan it returned only the first ~20 symbols in arbitrary storage order, so a plain search "distribution" on a >20-symbol repo silently missed its best match. Fixed: fetch the full candidate pool up to the safety cap (_CANDIDATE_LIMIT_CAP) and rank in Python. Test builds >20 distinct matching symbols and asserts they all return (verified: returns exactly 20 under the old formula, all 30 under the fix).

I1 (Important) — member-level fqn_contains matches were dropped. The shared _node_matches_filter re-checks fqn_contains against row['fqn'] (falling back to primary_type_fqn). The lexical row set only primary_type_fqn, so a member node Type#method(...) accepted by the Cypher pushdown was dropped by the post-filter. Fixed: added the raw fqn key. Test drives the bug end-to-end — runs the search, then re-checks each row through the same _node_matches_filter the real search_v2 loop uses (verified: fails with fqns=[None] without the key).

Verification: test_search_lexical.py (13) + test_graph_only_boot.py (4) green; test_mcp_v2.py + test_mcp_v2_compose.py (163) green; ruff clean on changed files.

The two Minor/Recommendation items from the review (test for table="all"; doc note on the 5000-symbol ceiling) are noted but deferred — happy to fold them in if wanted.

🤖 Generated with Claude Code

[HumanBean17]

Update: second review pass (3 parallel reviewers) + Intel CI, all pushed.

CI — Intel macOS leg re-enabled (`a30757c`). The previous `macos-13` leg was disabled because GitHub retired the macos-13 image on 2025-12-04 (the job hung pending a runner that no longer existed). Re-enabled with `macos-15-intel` — GitHub's designated x86_64 migration image (available until 2027-08). This is now the only CI matrix leg that exercises the graph-only install path and the lexical fallback (vector tests skip via `pytest.importorskip`); it's non-blocking (continue-on-error) initially. This is exactly the coverage gap that let the C1 Intel-only crash slip through.

C2 test hardened (`8a8a6eb`). Two reviewers independently found the C2 regression test was a false sentinel — with `limit=50` the buggy page-derived formula computes `200 ≥ 30`-symbol pool, so it passed green under the bug. A deterministic behavioral test is impossible here (the buggy `(limit+offset)*4` always overfetches enough to fill the windowed page; only the nondeterministic symbol identity differs). Replaced with a white-box test asserting the fetch LIMIT equals `_CANDIDATE_LIMIT_CAP` independent of page size — verified to capture `[20,80]` and fail under the bug.

Cap-truncation advisory (`8a8a6eb'). On a large repo the 5000-row scan cap returns an arbitrary, storage-order-dependent subset with no signal. `run_lexical_search` now appends an advisory when the fetch hits the cap (mode advisories reordered to lead the message). Covered by fire/silent tests.

End-to-end dispatch test added (`8a8a6eb`). Drives the real `search_v2` down the lexical branch against a fixture graph — covers the shared row→hit loop, `lexical_mode`, the graph-only advisory, `SearchHit.fqn`, and `--explain` (gaps the unit tests over `run_lexical_search` don't reach).

Reviewer verdicts: import/packaging — merge, no fixes; correctness — no production bugs (scoring math verified, no double-truncation/dedup, all fixes clean); test-quality — with fixes (the C2 sentinel + coverage gaps, now addressed). Remaining deferred minors: offset/empty-query/`table="all"` unit tests, negative-raw cosmetic, `role_w` reuse — noted, low priority.

🤖 Generated with Claude Code

[HumanBean17]

macOS-15-Intel CI: 17 failures → fixed (dea9ea2)

All 17 failures on the new macos-15-intel leg were vector-stack-dependent tests that never self-skipped on a graph-only install — none related to the lexical-search feature. Two fix strategies:

Skip (vector-specific contracts; the lexical path has its own coverage in test_search_lexical.py):

• test_mcp_v2.py — @needs_vectors on 9 search tests (no-lance-index failure envelope, hybrid+all fail-fast, hybrid FTS fallback, explain/dedup/chunks). These monkeypatch run_search → vector path → embed the query → need sentence_transformers.
• test_mcp_v2_compose.py — added the needs_vectors marker (mirroring test_mcp_v2.py) + applied to 2 search tests.
• test_brownfield_overrides.py — importorskip("cocoindex") on the 2 Lance chunk-schema pre-flights, importorskip("lancedb") on the round-trip.
• test_lance_optimize.py — importorskip("lancedb") on the TABLES-constant parity test.

Path-aware (graph-only is now a supported mode → keep coverage there rather than skip):

• test_java_codebase_rag_cli.py — _vector_stack_available(); test_increment_first_run_falls_back_to_full and test_cli_tables_lists_known_table now assert the graph-only-correct output (increment notes "vectors skipped"; tables is {} but graph is reported). Vector-path behavior unchanged.

Verified locally by simulating graph-only (sys.modules[m]=None flips both find_spec and import, so needs_vectors and importorskip both trigger): the 15 vector-only tests SKIP; running them normally they PASS; the 2 CLI tests pass on the vector path. Collection clean (264 tests). Ruff clean on changed lines (2 pre-existing import types F401 in test_lance_optimize.py left untouched — out of scope, and the workflow does not gate on ruff).

Note: this leg is continue-on-error (non-blocking) per the existing policy until it is reliably green across a few merges.