feat: classify, tag, and filter generated Java sources#404
Merged
Conversation
HumanBean17
force-pushed
the
feat/generated-source-indexing
branch
from
8e5d69b to
569dc8b
Compare
…ion plan Proposal: classify generated Java sources (content-based, file-level), tag both the Lance chunk index and graph Symbol nodes, surface the tag on results, and add exclude_generated/generated_only filters — equal-treatment default, no ranking penalty (role model already down-ranks DTOs/mappers). Plan: 6 TDD tasks (detect, tag Lance, tag graph, surface, filter, docs). Co-Authored-By: Claude <noreply@anthropic.com>
ONTOLOGY_VERSION 17->18 (T2) + new generated/generated_by columns (T2/T3) left drift in version-anchored tests and the committed graph baseline: - test_ontology_version_bumped_to_17 -> _to_18 (assert 18) - test_jrag_status: assert ontology_version == 18 - graph_baseline_bank_chat.json: graph_meta.ontology_version 17->18 (node/edge/counts_json unchanged - T3 added columns, not nodes/edges) - test_brownfield_overrides: JavaLanceChunk(...) passes generated/generated_by Co-Authored-By: Claude <noreply@anthropic.com>
- Add exclude_generated and generated_only boolean flags to both engines - Lance SQL: column-guarded predicates (generated = true / IS NULL OR false) - MCP/Kùzu: Cypher predicates + Python post-filter via _node_matches_filter - CLI: --exclude-generated and --generated-only flags - NodeFilter: added fields with applicability for symbol kind - Fix _populated_nodefilter_fields to skip False boolean fields Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ivity
NodeFilter has generated_only/exclude_generated (extra="forbid"), not a
`generated` field. Fix AGENT-GUIDE filter examples and the NodeFilter keys
table to use the real field names (a filter={"generated":...} call would
error), and drop the false "mutually exclusive" claim on --generated-only
(the flags are independent store_true).
Co-Authored-By: Claude <noreply@anthropic.com>
…ed 17 The incremental floor was hardcoded to version < 17, but ontology is now 18: an existing ontology-17 graph passed the gate, then errored in _load_existing_types (RETURN ... s.generated, s.generated_by against a schema lacking those columns) and recovered via exception fallback — correct outcome, but noisy stderr and inconsistent with Lance's clean column guard. Gate on version < ONTOLOGY_VERSION so stale graphs rebuild cleanly. (Final whole-branch review finding.) Co-Authored-By: Claude <noreply@anthropic.com>
…t coverage - Fix detection precision: check @generated value/comments before FQN early-return (graph_enrich.py) - Preserve generated_by as None not "" end-to-end (build_ast_graph.py) - Add false-positive + precision tests (test_generated_detection.py) - Create dedicated generated-sample graph fixture for MCP filter tests (conftest.py, test_generated_filter.py) - Cleanup: remove tautological assertion, move import re to module level (test_graph_generated_node.py, graph_enrich.py) - Add lombok to generated_by families docs (AGENT-GUIDE.md) - DB investigation confirms LadybugDB/Kùzu preserves None as NULL (no coercion) All 210 tests passing. Co-Authored-By: Claude <noreply@anthropic.com>
_lexical_where (graph-only / macOS-Intel lexical search path) now pushes generated_only / exclude_generated into Cypher pre-pagination, matching _symbol_where_from_filter and the Lance SQL engine. Previously these were applied only by the _node_matches_filter post-filter on the lexical path (correct, but not pre-paginated). Parity test extended with generated cases. Co-Authored-By: Claude <noreply@anthropic.com>
…, cleanup, docs note - Precompile regex patterns at module load (graph_enrich.py) - DRY load_generated_detection with spec table loop (graph_enrich.py) - Simplify defensive code in graph stub seeding (build_ast_graph.py) - Comment NodeFilter bool-skip rationale (mcp_v2.py) - Document @generated simple-name tradeoff (docs) Co-Authored-By: Claude <noreply@anthropic.com>
HumanBean17
force-pushed
the
feat/generated-source-indexing
branch
from
93f0d53 to
b2852af
Compare
test_generated_surface.py referenced Edge in the neighbors assertion without importing it (NameError on ubuntu, where neighbors returns edges; vacuously passed locally where results were empty). The two search tests also require lancedb but didn't skip on graph-only installs (macOS Intel), failing with ModuleNotFoundError. Import Edge and add pytest.importorskip('lancedb') to the search tests, matching the codebase's existing graph-only skip pattern.
Co-Authored-By: Claude <noreply@anthropic.com>
Module-level 'import lancedb' in test_generated_filter.py and test_lancedb_generated_column.py crashed at collection on graph-only installs (no vector stack) before pytest.importorskip could skip — macos-15-intel failed with ModuleNotFoundError. test_generated_filter.py: drop the unused bare import; the module-level pytest.importorskip('lancedb') now gates the module. test_lancedb_generated_column.py: replace 'import lancedb' with 'lancedb = pytest.importorskip("lancedb")' (binds it where present, skips on graph-only; lancedb.connect is used in-body).
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generated Java sources (OpenAPI, jsonschema2pojo, protobuf, MapStruct, wsimport, QueryDSL, jOOQ, Immutables, AutoValue) are now a first-class tagged, filterable dimension — fully retrievable (closes the agent dead-end when resolving a generated type) with no ranking penalty by default.
classify_java_fileingraph_enrich.py—@Generatedannotations + generator header banners. Extensible via agenerated_detectionsection in.java-codebase-rag.yml. Content-based, not path-based (a real package namedgeneratedis never misclassified).generated/generated_byon Lance chunks and graphSymbolnodes (ONTOLOGY_VERSION 17→18).search/find/describe/neighborsresult + the search CLI hint.exclude_generated/generated_only(MCPNodeFilter+ CLI flags), mirroringrole/exclude_rolesthrough both the Lance SQL and Kùzu Cypher + post-filter engines.Equal-treatment default: generated sources are indexed and returned exactly like hand-written code — not ranked down, not excluded from graph traversal. The existing role-aware ranking already down-ranks non-actionable roles (DTOs/mappers), which covers most generated code; the tag + opt-in filter let the agent decide.
Migration
Schema change (ontology 17→18). Existing indexes must be reprocessed (
java-codebase-rag reprocess) to populategenerated/generated_by; old indexes keep working (columns absent → not selected) until then.Design & plan
propose/GENERATED-SOURCE-INDEXING-PROPOSE.mdplans/active/PLAN-GENERATED-SOURCE-INDEXING.mdTests
Full suite green: 1187 passed, 14 skipped. New tests cover detection (8 scenarios + config overrides), Lance tagging, graph tagging + incremental stub preservation, surfacing (search/find/describe/neighbors + CLI hint + missing-field tolerance), and filters (both engines, against a real Lance index).
Commits
6 TDD tasks (detect → tag Lance → tag graph → surface → filter → docs) + fixes: ontology-18 version-anchor/baseline green-up, Lance test coverage, docs field-name correction, and a final-review fix (incremental ontology floor gates on
ONTOLOGY_VERSION).🤖 Generated with Claude Code