feat(search): redesign jrag search — honest scores, dedup, hybrid, guidance (Phase 0)

docs/AGENT-GUIDE.md

@@ -190,7 +190,7 @@ Prefer **`resolve` → `describe(id=…)`** over **`describe(fqn=…)`** when an
 
 #### `search`
 
-Ranked chunk retrieval. Args: `query`, `table` (`java`|`sql`|`yaml`|`all`, default `java`), `hybrid` (bool), `limit` (default 5), `offset`, `path_contains`, optional `filter` (symbol-applicable `NodeFilter` only).
+Ranked chunk retrieval. Args: `query`, `table` (`java`|`sql`|`yaml`|`all`, default `java`), `hybrid` (bool), `limit` (default 5), `offset`, `path_contains`, optional `filter` (symbol-applicable `NodeFilter` only), optional `chunks` (bool, default `false`). Returns one row per `primary_type_fqn` (symbol/type) by default; set `chunks=true` to restore chunk-level output. When deduped, each hit includes a `chunks` field (≥1) indicating how many chunks were collapsed into that hit.
 
 #### `find`
 

docs/CONFIGURATION.md

@@ -238,6 +238,7 @@ async_producer_overrides:
 - **The config file may live anywhere under your project, including a subdirectory of the Java tree.** Both the CLI (`init` / `increment` / `reprocess`) and the MCP server walk up from cwd to find `.java-codebase-rag.yml`, then resolve `source_root` and `index_dir` relative to the config file's directory. So a config living in `my-context/` next to `source_root: ../` and `index_dir: ../.java-codebase-rag` resolves identically for the CLI and the MCP server. If the config lives in a *sibling* dir of the Java tree (not an ancestor of where the agent works), the index remembers its location via `.java-codebase-rag/config_source` — see [Config file discovery (walk-up)](#config-file-discovery-walk-up). Keep the file under your project (not `$HOME`); set `JAVA_CODEBASE_RAG_SOURCE_ROOT` (MCP) or `--source-root` (CLI) only to override the discovered location.
 - **Don't commit secrets** into this YAML — it sits next to your source tree and is read by every operator who clones it.
 - **Rebuild after editing brownfield overrides.** Run a full `java-codebase-rag reprocess` (no flags) so Lance and LadybugDB stay coherent, or use `--graph-only` / `--vectors-only` when you know only one store needs invalidation. Editing `embedding.model` requires a vector rebuild (`reprocess` or `--vectors-only`).
+- **Reprocess after upgrading for per-symbol dedup and FTS.** Existing indexes should be reprocessed (`java-codebase-rag reprocess`) to get (a) per-symbol dedup re-tagging (collapsing multiple chunks of the same type into one hit) and (b) the index-time full-text search index that makes `--hybrid` work on yaml/sql tables.
 - **Diagnose what's loaded.** `java-codebase-rag meta` prints the resolved config and each value's `*_source` (`cli` / `env` / `yaml` / `default`) — see `embedding_model_source`, `embedding_device_source`, `index_dir_source`.
 - **`embedding.model` and `$` in directory names.** `expandvars` treats `$VAR` / `${VAR}` like the shell. HuggingFace hub ids never contain `$`. If a local filesystem path contains a literal `$` in a directory name, use an absolute path that avoids `$`-expansion patterns, or expect `expandvars` to interpret `$` sequences.
 

docs/JAVA-CODEBASE-RAG-CLI.md

@@ -345,3 +345,42 @@ Prefer **`java-codebase-rag reprocess --graph-only`** when you only need Ladybug
 - [README.md](../README.md) — env vars, MCP tool table, ignore layout.
 - [CODEBASE_REQUIREMENTS.md](./CODEBASE_REQUIREMENTS.md) — repo layout, brownfield, when to rebuild.
 - [MANUAL-VERIFICATION-CHECKLIST.md](./MANUAL-VERIFICATION-CHECKLIST.md) — phased checks that mix CLI + MCP.
+
+## `jrag` command — search CLI
+
+The **`jrag`** command provides semantic search over the LanceDB index. This is the CLI surface for search (see MCP `search` tool in AGENT-GUIDE.md for the programmatic interface).
+
+### `jrag search`
+
+Semantic search via natural language queries. Returns one row per symbol/type by default; use `--chunks` to restore chunk-level output.
+
+```bash
+# Basic search (deduped by default)
+jrag search "authentication service"
+
+# Show all chunks (no dedup)
+jrag search "authentication service" --chunks
+
+# Hybrid search (vector + keyword)
+jrag search "login" --hybrid
+
+# With score breakdown
+jrag search "controller" --explain
+
+# With pagination
+jrag search "service" --limit 20 --offset 20
+```
+
+**Key flags:**
+- `--table {java,sql,yaml,all}` — Which content table to search (default: `java`).
+- `--hybrid` — Enable vector + keyword hybrid search (single table only).
+- `--explain` — Include score breakdown (distance, role weight, symbol bonus).
+- `--chunks` — Show every chunk (default collapses to one row per symbol/type).
+- `--limit N` — Max hits to return (default 10).
+- `--offset N` — Skip N hits (pagination).
+- `--min-score N` — Drop hits below this score floor (default 0.0).
+- `--path-contains SUBSTR` — Narrow to chunks whose filename contains this substring.
+- `--role ROLE` — Filter by role (e.g., `CONTROLLER`, `SERVICE`).
+- `--framework FRAMEWORK` — Filter by framework (e.g., `spring_mvc`, `webflux`).
+
+**Breaking change (PR-SEARCH-2):** By default, `jrag search` now returns one row per `primary_type_fqn` (symbol/type) to prevent a single type from flooding the page. The `--chunks` flag restores the previous chunk-level output. When deduped, each hit shows a `chunks=N` field indicating how many chunks were collapsed into that hit.

java_codebase_rag/jrag.py

@@ -1058,6 +1058,9 @@ def _core_parser() -> argparse.ArgumentParser:
     search.add_argument(
         "--hybrid", action="store_true", help="Enable vector+keyword hybrid search."
     )
+    search.add_argument(
+        "--explain", action="store_true", help="Show score breakdown per hit."
+    )
     search.add_argument(
         "--path-contains", type=str, default=None, dest="path_contains",
         help="Narrow to chunks whose filename contains this substring.",
@@ -1088,6 +1091,11 @@ def _core_parser() -> argparse.ArgumentParser:
         default=0,
         help="Page offset (passed to search_v2; paginated via +1-fetch).",
     )
+    search.add_argument(
+        "--chunks",
+        action="store_true",
+        help="Show every chunk (default collapses to one row per symbol/type).",
+    )
     search.set_defaults(handler=_cmd_search, auto_scope=True)
 
     return parser
@@ -4025,6 +4033,65 @@ def _cmd_overview(args: argparse.Namespace) -> int:
 # ============================================================================
 
 
+def _zero_result_guidance(args: argparse.Namespace, graph) -> str | None:
+    """Hint where matches live when a filtered search returns 0 results.
+
+    Runs ONE cheap unfiltered probe (limit 10) and tallies the filtered
+    dimension across the probe hits, so an agent who filtered to e.g.
+    ``--role SERVICE`` and got nothing learns the matches are under
+    COMPONENT/OTHER instead of guessing. Returns None when no guidance
+    applies: no recognizable single-dimension filter set, the probe is
+    empty (truly no matches for this query), or the probe itself errored
+    (non-fatal — the empty result still renders).
+    """
+    import mcp_v2
+    from collections import Counter
+
+    from java_codebase_rag.jrag_envelope import normalize_enum
+
+    # Only the common single-dimension filters get guidance; first set wins.
+    dims: list[tuple[str, str, str, str]] = []
+    if args.role:
+        dims.append(("role", "role", "roles", normalize_enum(args.role, kind="role")))
+    if args.service:
+        dims.append(("microservice", "service", "services", args.service))
+    if args.module:
+        dims.append(("module", "module", "modules", args.module))
+    if not dims:
+        return None
+    attr, flag, plural, value = dims[0]
+
+    try:
+        probe = mcp_v2.search_v2(
+            args.query,
+            table=args.table,
+            hybrid=args.hybrid,
+            limit=10,
+            offset=0,
+            path_contains=args.path_contains,
+            filter=None,
+            explain=False,
+            graph=graph,
+        )
+    except Exception:
+        return None
+    if not probe.success or not probe.results:
+        return None
+
+    counts: Counter = Counter(getattr(h, attr, None) for h in probe.results)
+    counts.pop(None, None)
+    if not counts:
+        return None
+    total = sum(counts.values())
+    top = counts.most_common(3)
+    alts = ", ".join(f"{v} ({c})" for v, c in top)
+    suggestion = top[0][0]
+    return (
+        f"0 results with --{flag} {value}; {total} matches exist under other {plural}: "
+        f"{alts} — try --{flag} {suggestion}"
+    )
+
+
 def _cmd_search(args: argparse.Namespace) -> int:
     """search <query> — semantic search via search_v2 over Lance tables.
 
@@ -4054,6 +4121,19 @@ def _cmd_search(args: argparse.Namespace) -> int:
 
     limit = min(args.limit if args.limit is not None else 20, 499)
 
+    # --limit 0: short-circuit to a clean empty page. mark_truncated(rows, 0)
+    # would otherwise report truncated=True (a unit test pins the helper's
+    # current behavior, so we fix this in the handler, not the helper), and
+    # there is nothing to search — skip the embedding-model load entirely.
+    if limit == 0:
+        env = Envelope(
+            status="ok", nodes={}, truncated=False,
+            warnings=_auto_scope_notice(args),
+        )
+        next_actions_hook(env)
+        print(render(env, fmt=args.format, detail=args.detail, noun="search"))
+        return 0
+
     # Build NodeFilter from flags (same set as `find` filter mode).
     filter_dict: dict = {}
     if args.service:
@@ -4107,7 +4187,9 @@ def _cmd_search(args: argparse.Namespace) -> int:
         offset=args.offset,
         path_contains=args.path_contains,
         filter=node_filter,
+        explain=args.explain,
         graph=graph,
+        dedup=not getattr(args, "chunks", False),
     )
 
     if not out.success:
@@ -4134,6 +4216,16 @@ def _cmd_search(args: argparse.Namespace) -> int:
             d["id"] = d.get("chunk_id") or d.get("symbol_id") or d.get("fqn") or ""
         if "kind" not in d:
             d["kind"] = "search_hit"
+        # Add explain token when --explain is set
+        if args.explain:
+            from search_lancedb import explain_score_components
+            comps = d.get("score_components")
+            d["explain"] = explain_score_components(
+                comps,
+                role=d.get("role"),
+                hybrid=bool(args.hybrid),
+                graph_expanded=False,
+            )
         hit_dicts.append(d)
 
     # --framework POST-filter: the graph stores `framework` only on Route nodes,
@@ -4162,6 +4254,13 @@ def _cmd_search(args: argparse.Namespace) -> int:
             f"--framework {framework_want!r} filtered out all {framework_dropped} hit(s); "
             f"no symbol's declaring type matched the framework's characteristic annotations"
         )
+    # Zero-result guidance: when a structural filter emptied the page, run one
+    # cheap unfiltered probe and point at where matches actually live (e.g.
+    # "--role SERVICE" returned 0 but matches are under COMPONENT/OTHER).
+    if not hit_dicts and filter_dict:
+        guidance = _zero_result_guidance(args, graph)
+        if guidance:
+            warnings.append(guidance)
     env = Envelope(
         status="ok", nodes=nodes, truncated=truncated,
         warnings=warnings + _auto_scope_notice(args),

java_codebase_rag/jrag_envelope.py

@@ -778,7 +778,7 @@ def next_actions_hook(
 # brief + location / classification / ranking. ``file`` is the composed
 # ``filename:start_line`` display field (see :func:`_compose_file`).
 _NORMAL_NODE_KEYS: frozenset[str] = _BRIEF_NODE_KEYS | frozenset(
-    {"module", "role", "symbol_kind", "framework", "file", "score"}
+    {"module", "role", "symbol_kind", "framework", "file", "score", "explain", "chunks"}
 )
 
 # Edge attrs the text renderers read at the default level (target id variants

java_codebase_rag/jrag_render.py

@@ -70,6 +70,8 @@
     "framework",
     "file",
     "score",
+    "explain",
+    "chunks",
 )
 
 # Identity-adjacent extras shown inline at ``--detail brief``. ``score`` is the
@@ -87,6 +89,13 @@
 )
 
 
+def _format_inline_value(value: Any) -> str:
+    """Format a value for inline rendering: round floats to 3 decimals, others verbatim."""
+    if isinstance(value, float):
+        return f"{value:.3f}"
+    return str(value)
+
+
 def _next_action_lines(envelope: Envelope) -> list[str]:
     """Build up to 2 ``next: <hint>`` lines from ``agent_next_actions``.
 
@@ -266,17 +275,17 @@ def _render_listing(envelope: Envelope, *, noun: str, detail: str = "normal") ->
         # of every non-identity key (signature/annotations/snippet/...).
         if detail == "brief":
             extras = [
-                f"{key}={node[key]}"
+                f"{key}={_format_inline_value(node[key])}"
                 for key in _BRIEF_INLINE_EXTRAS
                 if key in node and node[key] not in ("", None)
             ]
             if extras:
                 line += "  " + "  ".join(extras)
         elif detail == "normal":
             extras = [
-                f"{key}={node[key]}"
+                f"{key}={_format_inline_value(node[key])}"
                 for key in _NORMAL_INLINE_EXTRAS
-                if key in node and node[key] not in ("", None)
+                if key in node and node[key] not in ("", None) and not (key == "chunks" and node[key] == 1)
             ]
             if extras:
                 line += "  " + "  ".join(extras)

java_codebase_rag/lance_optimize.py

@@ -184,6 +184,24 @@ async def optimize_lance_tables(
 
                 if last_exc is None:
                     results[name] = "ok"
+                    # Best-effort FTS index at index time (PR-SEARCH-3) so hybrid
+                    # search works on all tables (java/sql/yaml) without a
+                    # first-query race. Failure is non-fatal — the lazy
+                    # ensure_text_fts_index in search_lancedb.py is the runtime
+                    # fallback — so it never alters the "ok" optimize status; we
+                    # only log the skip when verbose.
+                    try:
+                        from lancedb.index import FTS
+                        await table.create_index("text", config=FTS(), replace=True)
+                    except Exception as exc:
+                        low = str(exc).lower()
+                        if not any(
+                            w in low for w in ("exist", "duplicate", "already", "same name")
+                        ) and not quiet:
+                            print(
+                                f"java-codebase-rag: optimize: {name} fts skipped: {exc}",
+                                file=sys.stderr,
+                            )
                     if not quiet:
                         print(
                             f"java-codebase-rag: optimize: {name} ok",