Tracking: 231-package relevance run — turn the FP/FN findings into ranking + eval improvements across ir / ef / raglab

[thorwhalen]

Context

On 2026-06-18 we classified all 231 local Python packages for relevance to two themes — (a) embeddings / semantic-search / NLP / vectorization and (b) graphs / networks — by using ir hybrid+dense retrieval to gather candidates, then independently LLM-classifying every package (from pyproject description + README + dependency list) into graded levels {core, strong, uses-tools, tangential, none}, then adversarially verifying. That gives us a ground-truth labeling to score ir's ranking against.

Final verified group sizes (out of 231): embeddings {core 11, strong 22, uses-tools 6, tangential 51}; graphs {core 5, strong 21, uses-tools 3, tangential 32}.

How ir indexes a package today (the Package strategy)

One description surface = "{name}: {pyproject [project].description}", plus readme_chunk surfaces (README split into overlapping chunks). Dependencies, owner, name, has_readme are stored as FILTER fields (metadata, NOT embedded). Default embedder = all-MiniLM-L6-v2 (384-dim). No docstrings, no source, no Trove classifiers. ir.with_synopsis (an LLM topical-summary surface) is OPT-IN and was NOT used in this run.

What the run surfaced

FALSE NEGATIVES (genuinely relevant, ranked low / near-zero) — nearly all have an empty/thin pyproject description, so ir had only a weak README signal:

• embeddings: chromadol (0.02, a vector-DB data-object-layer), http_cosmo_prep (0.01, ships an embeddings service).
• graphs: kroki (0.29, renders graphviz/mermaid digraphs), xcosmo (0.29, cosmograph viz), cosmo_data_prep (no score, node-link graph data), allude (no score, depends on the meshed graph lib), unbox (no score, builds an import-dependency graph), lexis (0.23, WordNet semantic network).
• thin-description CORE packages surviving on README alone (fragile): imbed, meshed, linked (all empty descriptions).

FALSE POSITIVES (ranked high, off-theme):

• embeddings: ir over-weights the AUDIO/SIGNAL modality and the literal token "search" — su (0.51, speech I/O), csm (0.46), voxy (0.44), theremin/slink (0.45), audio/music DSP (audiolazy/loopyng/hear/taped/hum/atonal/tonal/accompy, 0.38–0.42), haggle (0.43, Kaggle access), farepy (0.39, flight search).
• graphs: ir conflates graph/network with generic pipeline/flow/stream/DAG-mention prose. WORST CASE: au ranked Foundation: config/XDG dirs + core types #2 overall (0.51) but is an async-task framework ("DAG" appears in one feature bullet). Also strand (0.445), reci (0.442), creek (0.427), know (0.411), mv (0.400), odus (0.430), and ef/imbed (0.46/0.34 — an embedding flow, a pipeline not a graph lib).

Critical insight

The single most discriminative signal our human-style classifier used was the dependency list, which ir does NOT use for ranking (only as a hard filter field). Depending on ef/imbed/vd/grub/sentence-transformers/transformers/torch/openai/oa/aix/sklearn ⇒ embeddings-relevant; depending on meshed/linked/networkx/graphviz/igraph/dagapp/cosmograph ⇒ graph-relevant. Deps cleanly separate the "uses-tools" packages from both the cores and the modality/keyword distractors.

Dataset

The labeling is the durable artifact of the run. Schema + hosting are specified in the eval-harness child issue (graded PackageRelevanceCase, RELEVANCE_LEVELS/LEVEL_GAINS, to_graded_qrels). The data lives privacy-aware at the private benchmark repo thorwhalen/ir-eval-data (access-controlled) — package_relevance_labels.jsonl (full 231-package graded gold labeling), named_sets.json (per-theme distractors + hard_positives), and benchmark_analysis.json (frozen all-MiniLM-L6-v2 baseline: precision@K = recall@K ≈ 0.42 for both themes). Clone with repo access; package names are not mirrored into this public repo. (some package names are private; the repos are public, so only a small public-name fixture is committed). the private benchmark repo thorwhalen/ir-eval-data (access-controlled) — package_relevance_labels.jsonl (full 231-package graded gold labeling), named_sets.json (per-theme distractors + hard_positives), and benchmark_analysis.json (frozen all-MiniLM-L6-v2 baseline: precision@K = recall@K ≈ 0.42 for both themes). Clone with repo access; package names are not mirrored into this public repo. is the single indirection reused by every child issue below.

Boundary (why each fix lands where it does)

• ef = the embedding spine — any "try a different/better model" or instruction-prefixed embedding belongs here. ir needs zero API change for those (it already passes input_type).
• ir = the retrieval substrate — indexing surfaces, dense/lexical/hybrid + fusion, the link graph + traversal, single-shot rerank hook, the offline eval harness. Decision rule (epic Epic: evolve ir toward the ir_09 Composable Search Agent — layering decision + role map #38): a change lands in ir IFF it makes ir's single-shot, agent-free search measurably better. No back-edge / control loop in ir.
• raglab = the agent on top of ir — Planner, the evaluator→reformulate back-edge, graded multi-level judging of the whole corpus.

Child issues (index)

ef (embedder spine)

• ef: InstructionEmbedder wrapper + sentence-transformers instruction-prefix support (asymmetric query/doc), so the already-plumbed input_type actually re-weights toward topical intent.
• ef: graded multi-model embedding benchmark sweep + FP-on-distractors / hard-positive-recall metrics in ef.evaluation.

ir (retrieval substrate)

• ir: embed the dependency list as a first-class deps surface in ir.strategy.Package (deps-as-text), keeping the filter field.
• ir: conditional synopsis — auto-apply with_synopsis only to thin-description packages.
• ir: dependency-as-ranking-signal — reverse-dependency WalkPolicy in ir.traverse + a post-fusion dep-evidence boost in ir.retrieve (seed/lib-set supplied by the caller; ir never decides domain-relevance).
• ir: wire ef's cross-encoder reranker as a calibrated precision stage through ir's single-shot rerank= hook.
• ir: graded eval harness — extend ir.eval with graded PackageRelevanceCase + named-distractor FP-rate / hard-positive-recall metrics + an A/B compare_indexings regression gate. This issue HOLDS the private benchmark repo thorwhalen/ir-eval-data (access-controlled) — package_relevance_labels.jsonl (full 231-package graded gold labeling), named_sets.json (per-theme distractors + hard_positives), and benchmark_analysis.json (frozen all-MiniLM-L6-v2 baseline: precision@K = recall@K ≈ 0.42 for both themes). Clone with repo access; package names are not mirrored into this public repo..

raglab (agent)

• raglab: CorpusLabeler recipe + GradedRubricEvaluator (per-candidate graded level with a Self-RAG is_supported axis + forced evidence quote), the reusable form of this very run.

Threads

#1 (roadmap), #38 (epic / decision rule / role map), #28/#29 (DEFERRED storage-unification & Inspect-AI scorer), ir ADR#43 / report 12-14 (synopsis & PurposeStore), thorwhalen/raglab#2 (build-order roadmap), thorwhalen/raglab#5 (rank-fair budget), thorwhalen/raglab#6 (PurposeStore).

https://claude.ai/code/session_01D229oNHVN1drd1mdbQL5MV

---

Child issues

Group 0 — Unblocker (materializes the dataset + baseline scoreboard; everything scores against it)

• Graded eval harness: PackageRelevanceCase + named-distractor FP-rate / hard-positive-recall metrics + compare_indexings A/B regression gate (holds the dataset pointer) #66 (merged via Add graded package-relevance eval harness to ir.eval (#66) #67) — Graded eval harness: PackageRelevanceCase + named-distractor FP-rate / hard-positive-recall metrics + compare_indexings A/B regression gate (holds the dataset pointer)

Group 1 — ir indexing & ranking experiments (parallel; scored via Group 0)

• Embed the dependency list as a first-class 'deps' surface in ir.strategy.Package (deps-as-text), keeping the filter field #62 (merged via Add opt-in embedded deps surface to ir.strategy.Package (#62) #69) — Embed the dependency list as a first-class 'deps' surface in ir.strategy.Package (deps-as-text), keeping the filter field
• Conditional synopsis: auto-apply ir.with_synopsis only to thin-description packages to rescue empty-pyproject false negatives #63 — Conditional synopsis: auto-apply ir.with_synopsis only to thin-description packages to rescue empty-pyproject false negatives
• Dependency-as-ranking-signal: reverse-dependency WalkPolicy (ir.traverse) + post-fusion dep-evidence boost (ir.retrieve), seeded by the caller #64 — Dependency-as-ranking-signal: reverse-dependency WalkPolicy (ir.traverse) + post-fusion dep-evidence boost (ir.retrieve), seeded by the caller
• Wire ef's cross-encoder reranker as a calibrated precision stage through ir's single-shot rerank hook (audio/DAG distractor suppression) #65 — Wire ef's cross-encoder reranker as a calibrated precision stage through ir's single-shot rerank hook (audio/DAG distractor suppression)

Group 2 — ef embedder work (the model lever)

• Add an InstructionEmbedder wrapper + instruction-prefix support to the sentence-transformers adapter (task-conditioned, asymmetric query/doc) thorwhalen/ef#34 — Add an InstructionEmbedder wrapper + instruction-prefix support to the sentence-transformers adapter (task-conditioned, asymmetric query/doc)
• Add a graded multi-model embedding benchmark sweep + FP-on-distractors / hard-positive-recall metrics to ef.evaluation thorwhalen/ef#35 — Add a graded multi-model embedding benchmark sweep + FP-on-distractors / hard-positive-recall metrics to ef.evaluation

Group 3 — raglab agent (codify the workflow we ran)

• CorpusLabeler recipe + GradedRubricEvaluator: graded multi-level judging of every candidate with a Self-RAG is_supported axis, forced evidence quote, gap-driven reformulation thorwhalen/raglab#8 — CorpusLabeler recipe + GradedRubricEvaluator: graded multi-level judging of every candidate with a Self-RAG is_supported axis, forced evidence quote, gap-driven reformulation

Dataset: private repo thorwhalen/ir-eval-data (the benchmark dataset referenced throughout the child issues).

[thorwhalen]

✅ Unblocker landed — #66 merged via #67 (squash). The graded harness (PackageRelevanceCase, to_graded_qrels, named-set FP-rate / hard-positive-recall, compare_indexings + .regressions() gate) is in ir.eval; 16 new tests, full suite green (415).

Frozen all-MiniLM-L6-v2 / hybrid baseline (in the private ir-eval-data repo, baseline_report.json):

theme	nDCG@20	distractor fp_rate@20	hard-positive recall@20
embeddings	0.80	0.14	0.40
graphs	0.74	0.22	0.00

The harness reproduces the failure numerically (graphs recovers none of its hard-positives at 20; aggregate nDCG stays high — exactly why the named-set metrics, not nDCG, are the diagnostic). Group-1/2/3 candidate issues (#62–#65, #34/#35, #8) now have a scoreboard to score against.