Skip to content

feat(gitree): add vector search for concepts#1463

Merged
Evanfeenstra merged 4 commits into
mainfrom
feat/gitree-concept-vector-search
Jul 10, 2026
Merged

feat(gitree): add vector search for concepts#1463
Evanfeenstra merged 4 commits into
mainfrom
feat/gitree-concept-vector-search

Conversation

@Evanfeenstra

Copy link
Copy Markdown
Collaborator

Summary

Adds semantic (vector) search for Concepts. Previously only Clues had embeddings; concepts were only retrievable by ID or plain text matching. This mirrors the existing clue search infrastructure.

What & why

  • What to embed: name + description (concise, high signal) rather than documentation, which can be arbitrarily long and would be silently truncated at the model's 512-token limit / diluted by mean-pooling.
  • Concept nodes are already labeled :Data_Bank:Concept, so no new schema/index work is needed to store embeddings.

Changes

  • types.ts — add embedding?: number[] to Concept.
  • store/utils.tsconceptEmbeddingText() / computeConceptEmbedding() helpers (embed name + "\n\n" + description via vectorizeQuery).
  • saveConcept (GraphStorage + FileSystemStore) — computes/stores the embedding on every write, so both create and description-update paths stay current. Falls back to any existing embedding on failure.
  • searchConcepts() — added to the Storage abstract and both impls:
    • GraphStorage: brute-force gds.similarity.cosine over :Concept nodes (concept counts are small, so a full scan is fast and gives exact scores — same pattern as searchClues).
    • FileSystemStore: in-memory cosine.
  • POST /gitree/search-concepts route + search-concepts CLI command. Results are flat ConceptSummary objects with a merged score.
  • Startup migration: backfillConceptEmbeddings() runs at the end of initialize() (alongside the existing backfillPRFileEdges), embedding any pre-existing concepts missing an embedding. Becomes a cheap no-op scan once all concepts are embedded — no manual script required.

API

curl -X POST http://localhost:3355/gitree/search-concepts \
  -H "Content-Type: application/json" \
  -d '{"query": "authentication"}'

Returns:

{
  "query": "authentication",
  "repo": "all",
  "count": 1,
  "results": [
    { "id": "owner/repo/auth-system", "name": "Authentication System", "description": "...", "prCount": 2, "commitCount": 0, "lastUpdated": "...", "hasDocumentation": true, "score": 0.82 }
  ]
}

Testing

  • tsc --noEmit passes.

Notes

  • Backfill runs sequentially and loads the BGE model in-process; fine for hundreds/low-thousands of concepts. Can switch to vectorizeBatch if a much larger legacy corpus is expected.

Embed Concept name + description (BGE-small, 384-dim) so concepts are
discoverable via semantic search, mirroring the existing clue search.

- saveConcept computes/stores embeddings on every write (create + update)
- searchConcepts() on Storage: brute-force cosine over :Concept nodes
  (GraphStorage) and in-memory cosine (FileSystemStore)
- POST /gitree/search-concepts route + search-concepts CLI command,
  returning flat ConceptSummary results with a merged score
- backfillConceptEmbeddings() runs on initialize() to embed pre-existing
  concepts (cheap no-op scan once all are embedded)
…ponse

Use { concepts, total } instead of { results, count }.
Return ranked results without a hard cutoff by default. Guard the
FileSystemStore path to exclude concepts without embeddings so 0.0 is safe.
@Evanfeenstra Evanfeenstra merged commit c3a0be6 into main Jul 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant