fix: scale per-term exact/prefix score tiers by squared term coverage (#1602)#1724
Open
fkhawajagh wants to merge 3 commits into
Open
fix: scale per-term exact/prefix score tiers by squared term coverage (#1602)#1724fkhawajagh wants to merge 3 commits into
fkhawajagh wants to merge 3 commits into
Conversation
…+ footer Lead with what keeps a first-time visitor moving toward install/community instead of a terminal exit. Top strip trimmed 9 -> 5: PyPI, Downloads, CI, Discord, YC S26 (product -> adoption -> health, then the one converting click, with YC last: its credibility value is in being seen, not intercepting the front-door slot). Moved Sponsor, X and the book to a new "Community and links" footer (higher-intent readers who reach the bottom), added a contextual book link under "Learn more", and dropped the LinkedIn badge (near-zero value to a repo visitor; the org profile/site already carry it). Trendshift unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Maintainer preference: drop the CI-passing badge from the hero strip and keep LinkedIn visible up top instead. Top strip stays 5: PyPI, Downloads, Discord, LinkedIn, YC S26 — trust signals lead, Discord is the converting click, and the two company/social links group at the tail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…verage (fixes Graphify-Labs#1602) In a multi-term query, a single generic term that exactly equals a short leaf label (query term "home" vs a home() controller action, "list" vs a list() function) receives _EXACT_MATCH_BONUS x IDF and outscores every node that matches several of the query's terms by roughly 10x. _pick_seeds' 20% gap cutoff then discards the genuinely relevant candidates, so BFS explores the neighborhood of one unrelated leaf. Graphify-Labs#1596 mitigated the downstream symptom by guaranteeing one seed per distinct term; this fixes the root ranking: nothing in _score_nodes rewarded matching more of the query. _score_nodes now scales the per-term exact/prefix tier contributions by squared term coverage: tiered * (matched_terms / total_terms) ** 2. - Squared, not linear, because the exact tier is 10x the prefix tier; at linear coverage a 1-of-10-terms exact hit still edges out a 3-of-10-terms prefix+substring match. The regression test pins this ratio coupling: retuning either tier constant re-fails it. - Coverage counts label hits (exact/prefix/substring) only. Source-path hits still score but do not count: a colliding leaf that lives in the target's own directory (the common case) must not win back its exact tier via path fragments. - Substring and source bonuses stay unscaled, as does the full-query tier (a full-label match already implies full coverage). - Query tokens are now deduped, order-preserving, as _pick_seeds already does: a repeated query word must not double-count every tier, and with coverage scaling it would also inflate the matched ratio. - Single-term and full-coverage queries are unchanged (coverage == 1), so identifier lookups keep exact-match dominance (test_score_nodes_coverage_full_coverage_query_is_unchanged pins the exact pre-change score). On a real 5,700-node graph (the Graphify-Labs#1602 repros): the "home" collision drops from 6087.9 to 63.2 and the relevant cluster takes rank 1-2, so seeds recover it inside the gap window instead of relying solely on the per-term guarantee; the "list" collision drops from 4871.7 to 194.9 with the intended identifier back inside the gap window. Identifier and well-formed natural-language queries produce identical seeds before and after. Known residual, documented in _pick_seeds' docstring: a relevant node matched only via interior substrings still scores below a dampened collision (flat tier ~1 x IDF vs exact/n^2), which is exactly why the Graphify-Labs#1596 per-term seed guarantee remains load-bearing alongside this. Adds a regression test reproducing the Graphify-Labs#1602 second repro (the query contains the target's literal identifier yet three same-directory list() leaves outrank it) and a no-change test pinning full-coverage scores. tests/test_serve.py (78 tests) and ruff both pass; the full suite's only failures are environment-specific and reproduce identically on the unpatched base.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1602. Follows up on the discussion there; builds on the per-term seed guarantee from #1596 (
d56ee83) and fixes the scoring half that was left open.Root cause
In a multi-term query, a single generic term that exactly equals a short leaf label (
homevs ahome()controller action,listvs alist()function) receives_EXACT_MATCH_BONUS x IDFand outscores every node matching several of the query's terms by roughly 10x._pick_seeds' 20% gap cutoff then discards the genuinely relevant candidates. #1596 guarantees each term still contributes a seed, which stops the starvation; but nothing in_score_nodesrewarded matching more of the query, so the collision still owned the top of the ranking (andpath-style consumers ofscored[0]have no per-term backstop).The change
_score_nodesnow scales the per-term exact/prefix tier contributions by squared term coverage:Design decisions, each of which came out of testing against a real graph (below):
_pick_seedsalready does. A repeated query word previously double-added every tier; under coverage scaling it would also inflate the matched ratio, quadratically restoring the collision (a query withlisttwice re-ranked thelist()leaf above the intended target, 154.0 vs 84).Results on a real graph
Production graph: 5,700 nodes / 7,899 edges (Phoenix/Elixir codebase plus design docs; the graph both #1602 repros came from). Methodology: run
_query_terms -> _score_nodes -> _pick_seeds(the_query_graph_textflow) on basev8(8a608d1, which already includes #1596) and on this branch, and diff the top of the ranking and the seed set.Identifier queries: byte-identical
ClientLive.IndexandDashboardLiveproduce identical rankings and identical seeds before and after. Example (ClientLive.Index):(Partial-coverage neighbors like
ClientLive.Formscale down as intended; rank order and seeds are unchanged.)Natural-language query: seeds byte-identical
dashboard KPI alertsseeds['Dashboard COI Alerts Integration', 'dashboard_live.ex', 'Dashboard (/) Rebuild Implementation Plan', 'Placement-location seam for overlap alerts', 'kpi.ex']both before and after.Repro 1:
dashboard home page LiveView content KPI cards alerts today gridBefore (v8 with #1596):
After:
The relevant cluster is now inside the gap window on its own merit, not only via the per-term guarantee, and it also becomes the top-ranked answer for
scored[0]consumers.Repro 2:
ClientLive.Index clients list columnsBefore: three
list()nodes at 4871.72 vsClientLive.Indexat 600.72 (8.1x, outside the 20% gap window; seeded only via the per-term guarantee).After:
list()drops to 194.87 (25x reduction) andClientLive.Indexat 101.52 is back inside the gap window.list()still ranks first (it is a legitimate 1-of-5 exact match), but its dominance collapses from 8.1x to 1.9x and the intended identifier is seeded through the gap window itself.Known residual (why #1596 stays load-bearing)
A relevant node matched only via interior substrings (camelCase mid-label terms) still scores below a dampened collision: flat substring scores are ~1 x IDF per term vs
exact/n^2for the collision. The per-term seed guarantee from #1596 covers exactly this class, which is now documented in_pick_seeds' docstring alongside the coverage note. A symmetric coverage boost for the flat tier would be a larger behavioral change than this PR wants to take on; happy to explore it separately if you think it is worth it.One more trade-off worth naming: on very long queries (10+ content terms after stopword filtering), a node matching exactly one term via prefix (
100/n^2) can score below a node matching one term via substring (flat 1.0). Both are weak candidates for such queries, and the per-term guarantee still seeds each term's best node (single-term coverage == 1), but it is a place where the eval could be extended if you want stricter tier monotonicity.Tests
list()leaves outrank it on basev8. Fails onv8, passes with this change, with a 30% score margin so IDF drift will not flake it.query "FooBarService"class), so identifier dominance is provably untouched.tests/test_serve.py: 78/78 pass.ruff check: clean. The full suite's only failures on this machine are environment-specific (local provider env vars, XAML grammar) and reproduce identically on the unpatched base.