Skip to content

fix: scale per-term exact/prefix score tiers by squared term coverage (#1602)#1724

Open
fkhawajagh wants to merge 3 commits into
Graphify-Labs:v8from
fkhawajagh:fix/1602-term-coverage-scaling
Open

fix: scale per-term exact/prefix score tiers by squared term coverage (#1602)#1724
fkhawajagh wants to merge 3 commits into
Graphify-Labs:v8from
fkhawajagh:fix/1602-term-coverage-scaling

Conversation

@fkhawajagh

Copy link
Copy Markdown

Fixes #1602. Follows up on the discussion there; builds on the per-term seed guarantee from #1596 (d56ee83) and fixes the scoring half that was left open.

Root cause

In a multi-term query, a single generic term that exactly equals a short leaf label (home vs a home() controller action, list vs a list() function) receives _EXACT_MATCH_BONUS x IDF and outscores every node matching several of the query's terms by roughly 10x. _pick_seeds' 20% gap cutoff then discards the genuinely relevant candidates. #1596 guarantees each term still contributes a seed, which stops the starvation; but nothing in _score_nodes rewarded matching more of the query, so the collision still owned the top of the ranking (and path-style consumers of scored[0] have no per-term backstop).

The change

_score_nodes now scales the per-term exact/prefix tier contributions by squared term coverage:

score += tiered * (matched_terms / total_terms) ** 2

Design decisions, each of which came out of testing against a real graph (below):

  1. Squared, not linear. The exact tier is 10x the prefix tier; at linear coverage a 1-of-10-terms exact hit still edges out a 3-of-10-terms prefix+substring match. The regression test pins this coupling: retuning either tier constant re-fails it.
  2. Coverage counts label hits only (exact/prefix/substring). Source-path hits still score, but do not count toward coverage. Without this, a colliding leaf that lives in the target's own directory (the common case) wins back most of its exact tier via path fragments: on the regression scenario the leaf scored 122.5 vs the target's 66.3 when path hits counted, vs 30.9 / 40.1 when they do not.
  3. Substring and source bonuses stay unscaled, as does the full-query tier (a full-label match already implies full coverage).
  4. Query tokens are deduped, order-preserving, as _pick_seeds already does. A repeated query word previously double-added every tier; under coverage scaling it would also inflate the matched ratio, quadratically restoring the collision (a query with list twice re-ranked the list() leaf above the intended target, 154.0 vs 84).
  5. Single-term and full-coverage queries are unchanged (coverage == 1), so identifier lookups keep exact-match dominance. A test pins the exact pre-change score.

Results on a real graph

Production graph: 5,700 nodes / 7,899 edges (Phoenix/Elixir codebase plus design docs; the graph both #1602 repros came from). Methodology: run _query_terms -> _score_nodes -> _pick_seeds (the _query_graph_text flow) on base v8 (8a608d1, which already includes #1596) and on this branch, and diff the top of the ranking and the seed set.

Identifier queries: byte-identical

ClientLive.Index and DashboardLive produce identical rankings and identical seeds before and after. Example (ClientLive.Index):

rank before after
1 60028.71 ClientLive.Index 60028.71 ClientLive.Index
2 60026.57 ClientLive.Index (doc) 60026.57 ClientLive.Index (doc)
3 594.28 ClientLive.Form 148.57 ClientLive.Form

(Partial-coverage neighbors like ClientLive.Form scale down as intended; rank order and seeds are unchanged.)

Natural-language query: seeds byte-identical

dashboard KPI alerts seeds ['Dashboard COI Alerts Integration', 'dashboard_live.ex', 'Dashboard (/) Rebuild Implementation Plan', 'Placement-location seam for overlap alerts', 'kpi.ex'] both before and after.

Repro 1: dashboard home page LiveView content KPI cards alerts today grid

Before (v8 with #1596):

6087.92  home()                                   <- 1-term exact collision, rank 1
 703.97  grid_view()
 703.97  gridline_size_pct()
 626.73  Home / Today Dashboard (Iris)            <- intended cluster, below gap window
 621.12  Home Today Mock
seeds: home() first; cluster recovered only via the per-term guarantee

After:

  72.94  Home / Today Dashboard (Iris)            <- intended cluster, rank 1
  71.92  Dashboard (/) Rebuild to Home-Today Mock (Design)
  63.23  home()                                   <- collision dampened 6087.9 -> 63.2 (96x)
  36.91  Home Today Mock
seeds: ['Home / Today Dashboard (Iris)', 'Dashboard (/) Rebuild to Home-Today Mock (Design)', 'home()', ...]

The relevant cluster is now inside the gap window on its own merit, not only via the per-term guarantee, and it also becomes the top-ranked answer for scored[0] consumers.

Repro 2: ClientLive.Index clients list columns

Before: three list() nodes at 4871.72 vs ClientLive.Index at 600.72 (8.1x, outside the 20% gap window; seeded only via the per-term guarantee).

After: list() drops to 194.87 (25x reduction) and ClientLive.Index at 101.52 is back inside the gap window. list() still ranks first (it is a legitimate 1-of-5 exact match), but its dominance collapses from 8.1x to 1.9x and the intended identifier is seeded through the gap window itself.

Known residual (why #1596 stays load-bearing)

A relevant node matched only via interior substrings (camelCase mid-label terms) still scores below a dampened collision: flat substring scores are ~1 x IDF per term vs exact/n^2 for the collision. The per-term seed guarantee from #1596 covers exactly this class, which is now documented in _pick_seeds' docstring alongside the coverage note. A symmetric coverage boost for the flat tier would be a larger behavioral change than this PR wants to take on; happy to explore it separately if you think it is worth it.

One more trade-off worth naming: on very long queries (10+ content terms after stopword filtering), a node matching exactly one term via prefix (100/n^2) can score below a node matching one term via substring (flat 1.0). Both are weak candidates for such queries, and the per-term guarantee still seeds each term's best node (single-term coverage == 1), but it is a place where the eval could be extended if you want stricter tier monotonicity.

Tests

  • New regression test reproducing the second query: single generic term exact-matching a short leaf label hijacks seed selection in multi-term queries #1602 repro: the query contains the target's literal identifier, yet three same-directory list() leaves outrank it on base v8. Fails on v8, passes with this change, with a 30% score margin so IDF drift will not flake it.
  • New no-change test pinning the exact full-coverage score (query "FooBarService" class), so identifier dominance is provably untouched.
  • tests/test_serve.py: 78/78 pass. ruff check: clean. The full suite's only failures on this machine are environment-specific (local provider env vars, XAML grammar) and reproduce identically on the unpatched base.

safishamsi and others added 3 commits July 6, 2026 17:05
…+ footer

Lead with what keeps a first-time visitor moving toward install/community
instead of a terminal exit. Top strip trimmed 9 -> 5: PyPI, Downloads, CI,
Discord, YC S26 (product -> adoption -> health, then the one converting click,
with YC last: its credibility value is in being seen, not intercepting the
front-door slot). Moved Sponsor, X and the book to a new "Community and links"
footer (higher-intent readers who reach the bottom), added a contextual book
link under "Learn more", and dropped the LinkedIn badge (near-zero value to a
repo visitor; the org profile/site already carry it). Trendshift unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Maintainer preference: drop the CI-passing badge from the hero strip and keep
LinkedIn visible up top instead. Top strip stays 5: PyPI, Downloads, Discord,
LinkedIn, YC S26 — trust signals lead, Discord is the converting click, and the
two company/social links group at the tail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…verage (fixes Graphify-Labs#1602)

In a multi-term query, a single generic term that exactly equals a
short leaf label (query term "home" vs a home() controller action,
"list" vs a list() function) receives _EXACT_MATCH_BONUS x IDF and
outscores every node that matches several of the query's terms by
roughly 10x. _pick_seeds' 20% gap cutoff then discards the genuinely
relevant candidates, so BFS explores the neighborhood of one unrelated
leaf. Graphify-Labs#1596 mitigated the downstream symptom by guaranteeing one seed
per distinct term; this fixes the root ranking: nothing in _score_nodes
rewarded matching more of the query.

_score_nodes now scales the per-term exact/prefix tier contributions by
squared term coverage: tiered * (matched_terms / total_terms) ** 2.

- Squared, not linear, because the exact tier is 10x the prefix tier;
  at linear coverage a 1-of-10-terms exact hit still edges out a
  3-of-10-terms prefix+substring match. The regression test pins this
  ratio coupling: retuning either tier constant re-fails it.
- Coverage counts label hits (exact/prefix/substring) only. Source-path
  hits still score but do not count: a colliding leaf that lives in the
  target's own directory (the common case) must not win back its exact
  tier via path fragments.
- Substring and source bonuses stay unscaled, as does the full-query
  tier (a full-label match already implies full coverage).
- Query tokens are now deduped, order-preserving, as _pick_seeds
  already does: a repeated query word must not double-count every tier,
  and with coverage scaling it would also inflate the matched ratio.
- Single-term and full-coverage queries are unchanged (coverage == 1),
  so identifier lookups keep exact-match dominance
  (test_score_nodes_coverage_full_coverage_query_is_unchanged pins the
  exact pre-change score).

On a real 5,700-node graph (the Graphify-Labs#1602 repros): the "home" collision
drops from 6087.9 to 63.2 and the relevant cluster takes rank 1-2, so
seeds recover it inside the gap window instead of relying solely on the
per-term guarantee; the "list" collision drops from 4871.7 to 194.9
with the intended identifier back inside the gap window. Identifier
and well-formed natural-language queries produce identical seeds
before and after.

Known residual, documented in _pick_seeds' docstring: a relevant node
matched only via interior substrings still scores below a dampened
collision (flat tier ~1 x IDF vs exact/n^2), which is exactly why the
Graphify-Labs#1596 per-term seed guarantee remains load-bearing alongside this.

Adds a regression test reproducing the Graphify-Labs#1602 second repro (the query
contains the target's literal identifier yet three same-directory
list() leaves outrank it) and a no-change test pinning full-coverage
scores.

tests/test_serve.py (78 tests) and ruff both pass; the full suite's
only failures are environment-specific and reproduce identically on
the unpatched base.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query: single generic term exact-matching a short leaf label hijacks seed selection in multi-term queries

2 participants