Recall quality on a real 18k store: honest measurements, early experiments, looking for the lever that moves it drastically

[Marsu6996]

We are measuring and trying to drastically improve recall quality on a real production store (18k records, French, custom Solon-large HTTP embedding provider), against a 63-query human-labelled gold set evaluated through the daemon path. Opening this issue to share findings as we go and to discuss directions — this is very much work in progress.

What we measured

• Baseline (v2.5.1, exact index): top-1 12.7%, top-5 25.4%. The move from ANN to the exact index changed almost nothing on this benchmark — retrieval approximation was not the bottleneck.
• ~90% of top-5 hits were raw episodic turns, even when curated doc:* chunks (ingested via upload/teach) containing the answer sat just below the cutoff: ranking does not distinguish knowledge from conversation.
• The vocabulary gap dominates failures: a query phrased with words absent from the stored text ("versionnage" vs a record talking about semver/release/bump) leaves the target stranded at deep ranks.

What we tried so far (branches on our fork)

• Source-aware weighting (m1-source-weighting): bounded soft boost of doc:*/semantic-tier records at final ranking, never a filter. Factor swept on the gold set; 1.05 optimal.
• IDF-gated hybrid lexical⊕semantic (m2-hybrid-lexical): extends the in-tree BM25 LexicalIndex with FR tokenisation + deterministic synonyms, fires only on rare tokens (IDF ≥ 4), RRF fusion; RRF-k swept, 240 optimal.

Honest outcome: these two combined only moved top-1 from 12.7% to 14.3% (top-5 did better: 25.4% → 34.9%, and six previously unfindable queries now surface). That is far from what we consider acceptable — our target is near-assured top-5 recall.

Where we are heading

We are actively working on this and looking for the lever that moves the score drastically, not incrementally. Candidates on our list: a much larger gold set to understand the remaining failures, chunking/section-aware ingestion, and a measured reranker. We are open to any and all ideas — engine-side, ingestion-side, evaluation-side, or approaches we have not considered at all. If you have thoughts on where the engine itself could help most, or results from your own stores, we are very interested.

🤖 Designed by Marsu — Refined by Claude.

[CodeAbra]

Thanks for measuring this properly. A 63-query gold set against a real 18k store is more than we had, and both of your diagnoses were correct. Both are now fixed in the engine.

Knowledge-aware ranking. Curated doc:* chunks get a soft boost at final rank, default 1.05 (IAI_MCP_TIER_BOOST, 1.0 disables). Your factor survived essentially intact. One difference from your m1 branch: semantic summaries are treated separately from teach chunks. They boost only when the literal_preservation profile knob is relaxed past its strong default, because that knob is precisely the raw-versus-summary preference and a default install has to keep raw originals on top. Both classes stand down for verbatim cues. Your main symptom, doc chunks stranded below raw conversation turns, is fixed on default config.

Lexical fusion. The BM25 index now feeds recall. Your IDF gate was the right instinct and it shipped: fusion fires only on genuinely rare tokens (IAI_MCP_LEX_MIN_IDF, default 4.0), warm hits join the seed frontier and earn a decaying rank bonus, and IAI_MCP_LEX_FUSION_OFF=true kills the lane. One constraint your m2 branch did not have to honor: the recall path never pays an O(corpus) decrypt rebuild. The index is built by scoped search or the nightly warm-up and kept current by a cheap per-insert feed, so a cold index contributes nothing rather than stalling the query.

Your report also surfaced a defect we would not have found on our own. The exact-cosine authority merge was replacing a hit's pipeline score with raw cosine, and the re-sort after it flattened every authority-head candidate back into pure cosine order. Community and other rank signals were being silently discarded for the whole top-10-by-cosine set. The surviving hit now keeps the higher score, so authority still guarantees inclusion without wiping the ordering.

On what we can and cannot claim: our tests prove these mechanisms fire and stay bounded. They do not reproduce your numbers. We have no French corpus and no gold set of that shape, which is exactly what your fork has and we do not. If you re-run your 63 queries on 2.6.0 we would like to see the delta, including if it turns out small. Worth flagging that your Solon-large setup sits off our default path (the engine is tuned for bge-small-en), so embedder and language fit remains a variable on your side that these two levers do not address.

Both shipped in 2.6.0.

[Marsu6996]

Thank you — genuinely. Shipping both levers, crediting the prototype, and turning our report into a bug fix you'd have had no reason to find otherwise: that's a lovely way to run a project, and it's appreciated.

I re-ran the 63-query gold set on 2.6.0 through the daemon path, on the same 18k French/Solon store. Honest delta, exactly as you asked — including that it's small:

	top-1	top-5
v2.5.1 stock	12.7%	25.4%
v2.6.0 (default)	11.1%	23.8%
our m1+m2 fork	14.3%	34.9%

So on our corpus 2.6.0 lands right around stock, a touch under. I want to stress up front: this is almost certainly a "us" problem, not a "you" problem — you've said French is off your default path, and you're right that nobody should reshape an engine tuned for bge-small-en around one French user. I went digging to understand why, purely so the finding is useful to you, and it turned out to be more interesting than "just tokenisation":

I grafted our FR tokenisation + synonym expansion onto 2.6.0's LexicalIndex and forced the index to stay warm — and recall didn't move at all. Two design choices explain it, and both are perfectly reasonable for your target use case:

1. The lexical query() is an AND-of-tokens intersection. On a vocabulary-gap French cue it returns 1–3 very specific docs and drops the target that only carries the synonym of the query word. Our m2 lane instead scores BM25-sum (OR) and returns ~200 candidates.
2. Fusion re-ranks the existing pool rather than widening it — a lexical hit is only used if rid in id_to_idx, i.e. if the semantic lane already surfaced it. So a target that only the lexical lane can find never enters the candidate set. Our m2 unions the BM25 candidates directly into the pool before ranking.

In other words, for our case the load-bearing move wasn't the FR synonyms — it was letting the lexical lane widen the pool, not just re-order it. I completely understand that pool-widening runs against the "recall never pays an O(corpus) rebuild" guarantee you (rightly) protect, so this may simply be a fork-shaped difference rather than something upstream should adopt. But if you ever wanted an optional "lexical seeds may enter the frontier" path gated behind a flag, I'd be very happy to prototype it against our gold set and share the numbers either way.

Either way — thank you again for the care you put into this. It's a pleasure to build on.

🤖 Designed by Marsu — Refined by Claude.

[danielhertz1999-bit]

Picking up on the "widen vs. re-rank" distinction from the thread above — traced it down to the exact gate and have a draft fix + tests, sharing in case it's useful.

Root cause, pinned to code

The gate is in _recall_core (src/iai_mcp/pipeline.py), in the multi-seed widening block:

_lex_seed_ids = sorted(lex_rank, key=lex_rank.get)
if fts_hits or _lex_seed_ids:
    _fts_seed_ids = [
        rid
        for rid in list(fts_hits) + _lex_seed_ids
        if rid in id_to_idx and rid not in seed_ids   # <-- here
    ]

id_to_idx is built from pool_ids, and a BM25 hit only becomes a spread-seed (or later earns its += LEX_FUSION_W / (1 + rank) bonus in the scoring loop) if it's already in pool_ids. If it isn't, it's filtered out right there — the lexical lane can re-rank, never widen.

One layer deeper than that, though: on the AWAKE path, graph inside _recall_core isn't the full corpus graph. core/__init__.py builds it fresh, per call, from a K_CANDIDATES-bounded ANN query plus 1-/2-hop neighbours and a rich-club cap (graph = MemoryGraph(); for _rec in _candidate_recs.values(): graph.add_node(...)). So pool_ids absence doesn't mean "the graph structurally excludes this" — it means "the ANN/hop pass didn't happen to surface it this call." That distinction matters for the fix: checking graph-node membership before widening (which I tried first) just re-imposes the same ANN cutoff the fix is supposed to lift.

Proposed fix

Mirrors the existing confidence-escalation widen (escalate_recall_candidates, unioned into pool_ids/pool_embs/cosine_top_indices earlier in the same function) but keyed by id instead of by vector: for BM25 hits in lex_rank not already in pool_ids, fetch by id (store.get), compute cosine against the cue, and union into the pool. Bounded cost — lex_rank is already capped at LEX_FUSION_K (32) and gated to idf >= min_idf, so at most 32 extra by-id reads, never an O(corpus) scan. Gated behind a new opt-in flag, IAI_MCP_LEX_WIDEN_POOL (default off), so shipped behavior is unchanged unless someone turns it on.

Diff (also attached as a file):

--- a/src/iai_mcp/pipeline.py
+++ b/src/iai_mcp/pipeline.py
@@ -995,6 +995,68 @@ def _recall_core(
             _arousal_max_hops_used = spread_hops
             _arousal_mode_bias_adjust = 0.0
 
+    # Bounded lexical-only widen (opt-in experiment for #85): `graph` here is
+    # the PER-CALL candidate graph the caller built from a K_CANDIDATES-
+    # bounded ANN query plus its 1-/2-hop neighbours and a rich-club cap (see
+    # core/__init__.py) -- it is NOT the full corpus. So a BM25 hit absent
+    # from `pool_ids` only ever means "the ANN/hop pass didn't happen to
+    # surface it this call", never a deliberate structural exclusion; fetching
+    # it by id and folding it in is the fix, not a violation of some
+    # graph-authority invariant. Bounded cost: lex_rank is already capped at
+    # LEX_FUSION_K (32) entries and gated to idf >= min_idf, so this is at
+    # most 32 extra by-id store reads, never an O(corpus) scan.
+    #
+    # Known open question for review: unlike the ANN widen above, this does
+    # not cross-check the contradiction/anti-hits exclusion set (built later
+    # in this function, from contradicts_outgoing). The lexical index only
+    # ever indexes tombstoned_at IS NULL rows, so a hard-deleted record can
+    # never reach lex_rank -- but a *contradicted-but-live* record (meant to
+    # surface only via the anti-hits path) could in principle widen in here.
+    # Left for a maintainer call on whether threading contradicts_dst_set
+    # earlier in the function is worth it for this path.
+    if lex_rank and os.environ.get("IAI_MCP_LEX_WIDEN_POOL") == "true":
+        _existing_pool_ids = set(pool_ids)
+        _lex_widen_ids = [
+            rid for rid in sorted(lex_rank, key=lex_rank.get)
+            if rid not in _existing_pool_ids
+        ]
+        if _lex_widen_ids:
+            _lex_widened_records: dict = {}
+            for rid in _lex_widen_ids:
+                try:
+                    rec = store.get(rid)
+                except Exception as exc:  # noqa: BLE001 -- widen is a bonus, never a dependency
+                    logger.debug("lex_widen_fetch_failed: %s", exc)
+                    continue
+                if rec is not None and getattr(rec, "embedding", None):
+                    _lex_widened_records[rid] = rec
+                    if records_cache is not None and rid not in records_cache:
+                        records_cache[rid] = rec
+            _new_lex_ids = list(_lex_widened_records.keys())
+            if _new_lex_ids:
+                _new_lex_embs = np.asarray(
+                    [list(_lex_widened_records[rid].embedding) for rid in _new_lex_ids],
+                    dtype=np.float32,
+                )
+                _new_lex_norms = np.linalg.norm(_new_lex_embs, axis=1, keepdims=True)
+                _new_lex_norms[_new_lex_norms == 0.0] = 1.0
+                _new_lex_embs = _new_lex_embs / _new_lex_norms
+                _new_lex_cos = np.matmul(_new_lex_embs, cue_vec).astype(np.float32)
+
+                _widen_start = len(pool_ids)
+                pool_ids = list(pool_ids) + _new_lex_ids
+                pool_embs = (
+                    np.concatenate([pool_embs, _new_lex_embs], axis=0)
+                    if pool_embs.size else _new_lex_embs
+                )
+                shared_cos = np.concatenate([shared_cos, _new_lex_cos])
+                cosine_top_indices = np.union1d(
+                    cosine_top_indices,
+                    np.arange(_widen_start, len(pool_ids), dtype=np.int64),
+                ).astype(np.int64)
+        if trace_mark is not None:
+            trace_mark("lex_widen")
+
     id_to_idx = {rid: i for i, rid in enumerate(pool_ids)}
 
     gate_member_embeddings: dict[UUID, np.ndarray] = {

Tests

Added two cases to tests/test_recall_rank_fusion.py. The existing _seed_vocab_gap_world fixture (5 fillers + 1 target) never actually exercises the ANN cutoff, since 6 records all fit under the real K_CANDIDATES=200 default — so I added a variant that monkeypatches K_CANDIDATES down to n_fillers to force a deterministic cutoff, then asserts the target is invisible by default and surfaces with the flag on:

--- a/tests/test_recall_rank_fusion.py
+++ b/tests/test_recall_rank_fusion.py
@@ -266,3 +266,67 @@ def test_common_word_cue_gated_out_of_fusion(tmp_path, monkeypatch):
     assert store.lexical_query_warm("zephyrblatt", min_idf=0.5), (
         "a rare-token cue must pass the gate"
     )
+
+
+def _seed_cutoff_world(store: MemoryStore, monkeypatch, *, n_fillers: int):
+    """`n_fillers` cosine-dominant fillers plus one vocabulary-gap target.
+    Every filler outranks the target on raw cosine, so patching
+    ``pipeline.K_CANDIDATES`` down to ``n_fillers`` deterministically pushes
+    the target out of the cosine top-K -- unlike ``_seed_vocab_gap_world``'s
+    5-filler world, which is smaller than the real K_CANDIDATES=200 default
+    and so never actually exercises the cutoff."""
+    filler_texts = [
+        f"weakly related note number {i} about general status" for i in range(n_fillers)
+    ]
+    target_text = "the zephyrblatt rotor calibration constant lives in the flux config"
+    mapping = {t: _mix(0, 2 + i, 0.25) for i, t in enumerate(filler_texts)}
+    mapping[target_text] = _unit(9)
+    _install_embedder(monkeypatch, mapping)
+    fillers = [_rec(t, mapping[t]) for t in filler_texts]
+    target = _rec(target_text, _unit(9))
+    for r in fillers + [target]:
+        store.insert(r)
+    flush_record_buffer(store)
+    return fillers, target
+
+
+def test_lex_widen_pool_off_by_default_misses_cutoff_target(tmp_path, monkeypatch):
+    """Baseline for the #85 gap: a BM25 hit that ranks outside the raw
+    cosine top-K is invisible to recall by default, no matter how strongly
+    lexical fusion would rank it -- fusion re-ranks the existing pool, it
+    does not widen it. See pipeline.py's lex_rank / id_to_idx discussion."""
+    from iai_mcp import pipeline
+
+    monkeypatch.setenv("IAI_MCP_LEX_MIN_IDF", "0.5")
+    monkeypatch.delenv("IAI_MCP_LEX_WIDEN_POOL", raising=False)
+    monkeypatch.setattr(pipeline, "K_CANDIDATES", 5)
+    store = MemoryStore(path=tmp_path)
+    _fillers, target = _seed_cutoff_world(store, monkeypatch, n_fillers=5)
+
+    store.lexical_search("warm the lane", k=1)
+    resp = _dispatch(store, "zephyrblatt config", _unit(0))
+    hit_ids = {h["record_id"] for h in resp["hits"]}
+    assert str(target.id) not in hit_ids, (
+        "a target beyond the cosine cutoff must NOT surface without the widen: "
+        f"{sorted(hit_ids)}"
+    )
+
+
+def test_lex_widen_pool_flag_surfaces_cutoff_target(tmp_path, monkeypatch):
+    """With IAI_MCP_LEX_WIDEN_POOL=true, the same cutoff-excluded target
+    from the test above must surface -- the lexical lane now widens the
+    pool instead of only re-ranking what cosine already let through."""
+    from iai_mcp import pipeline
+
+    monkeypatch.setenv("IAI_MCP_LEX_MIN_IDF", "0.5")
+    monkeypatch.setenv("IAI_MCP_LEX_WIDEN_POOL", "true")
+    monkeypatch.setattr(pipeline, "K_CANDIDATES", 5)
+    store = MemoryStore(path=tmp_path)
+    _fillers, target = _seed_cutoff_world(store, monkeypatch, n_fillers=5)
+
+    store.lexical_search("warm the lane", k=1)
+    resp = _dispatch(store, "zephyrblatt config", _unit(0))
+    assert resp["hits"][0]["record_id"] == str(target.id), (
+        "the widen must pull the cutoff-excluded target back in and BM25 "
+        f"fusion must still rank it first: {[h['record_id'] for h in resp['hits']]}"
+    )

Verification

Ran tests/test_recall_rank_fusion.py against main (7e5b2a9) with the patch applied, using the compiled extension from the iai-pme 2.7.1 PyPI wheel. Both new tests pass consistently across repeated runs. Full file: 11/11 passed on a clean run; on two other runs, 1-2 pre-existing, unrelated tests (test_semantic_boost_stands_down_under_strong_literal_preservation, test_tier_boost_env_disables) failed intermittently — confirmed this reproduces on clean, unpatched main too (ran it twice: 2 failures, then 0), so it's an existing flake in the suite, not something this change causes or fixes. Flagging in case it's not already on your radar.

Happy to open this as a PR instead of a comment if that's more useful — let me know. And the open question above (contradiction/anti-hits interaction) is a real one I don't have a confident answer to; deferring to whoever knows that part of the design best.

---

Drafted with Claude Code, on behalf of @danielhertz1999-bit.

[CodeAbra]

3.0.0 is out, and it's the first release where re-measuring your gold set is worth your time rather than a way to reproduce 12.7% with more steps. Four things landed against the three findings in your opening post.

The vocabulary gap. Capture now extracts entity anchors from every stored turn — backticked identifiers, at-prefixed handles, dotted and dashed names, CamelCase, mid-sentence proper nouns — and writes them as entity: tags that the ranker matches on. For a corpus captured before this shipped, iai-mcp entity-backfill --apply anchors it retroactively, and --refresh recomputes the whole thing. This doesn't close a semantic gap like "versionnage" vs "semver", but it does close the one where a query names a thing and the record names the same thing.

Ranking not distinguishing knowledge from conversation. Every served hit now carries the arithmetic that decided its score, reconciled against the printed reason by a property test. So instead of inferring why a doc:* chunk sat below an episodic turn, you can read which term won. That's not a fix for the weighting itself — it's the instrument you'd need to argue for one with numbers.

Conversational noise drowning curated chunks. iai-mcp blob-quarantine tombstones machine-notification blobs — task notifications, system notices, command envelopes — that share vocabulary with real records and outrank them. Dry-run by default. If your 18k store has any agent-loop residue in it, this is the cheapest thing to try first.

A bug that may have been part of your ceiling. memory_recall was ignoring a caller-supplied cue_embedding on the warm path, as the tool schema had always promised it would honour. With a custom HTTP provider that is exactly the path that mattered: the cue was being re-embedded server-side instead of using the vector you supplied. Fixed, with seed-stable regression tests.

And the thing that actually changes your situation. Your store is French, and until today we were English-only — which is why you were running Solon through the HTTP provider in the first place. iai lang add fr now opts a store into a built-in multilingual embedder (multilingual-e5-small, pinned, used raw). So there is a second honest comparison available on your own corpus and your own 63 queries: your Solon provider against our native multilingual path, same gold set, same daemon.

One caveat, stated up front: non-Latin scripts get no lexical contribution yet, but French is Latin so that doesn't touch you.

If you have the appetite, the measurement I'd most like to see is blob-quarantine --apply plus entity-backfill --apply, then the same 63 queries. And I want to say plainly: the numbers may not move. If they don't, that's still the most useful result anyone can give us, because 63 human-labelled queries on a live 18k non-English store exist in exactly one place and it isn't here.

[Marsu6996]

Thanks, Areg — we ran the requested measurements on isolated clones of the same production-store lineage, now at 21,905 records, using the same 63-query gold set. All runs completed 63/63 queries with no errors.

One correctness issue surfaced first: in v3.0.0, blob-quarantine --apply sets tombstoned_at but does not clear the denormalized live flag. On our clone, all 242 quarantined blobs remained live=1.

We applied the minimal fix — updating tombstoned_at and live=0 atomically — and added regression coverage for both the SQL invariant and recall exclusion. Afterward, 242/242 blobs had live=0, and a token unique to a quarantined blob appeared in neither hits nor anti-hits.

entity-backfill --apply then processed 14,990 records and added 89,136 anchors. Subsequent dry-runs found nothing left to process.

Controlled Solon A/B:

Variant	Top-1	Top-5	Top-10	MRR
v3 before maintenance	12.70%	28.57%	28.57%	0.1853
corrected quarantine + entity backfill	14.29%	28.57%	28.57%	0.1953

So the combined maintenance moved one query into rank 1, but did not change top-5 or top-10.

We also repeated the provider comparison through the same isolated v3 daemon build/path:

Provider	Top-1	Top-5	Top-10	MRR	Median
Solon HTTP 1024D	12.70%	26.98%	28.57%	0.1866	0.781s
Native multilingual FR 384D	3.17%	12.70%	15.87%	0.0696	0.210s

The native multilingual path is roughly 3.7× faster here, but the quality loss is too large, so we are keeping Solon for this corpus.

I have the minimal quarantine fix and its regression tests ready. Happy to send the patch or open a focused PR if useful.

Marsu