Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions internal/tools/lens.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ func RegisterQueryLFXLens(server *mcp.Server) {
Description: `Ask natural language questions about a project's data using ad-hoc SQL generation.

Always use this tool for:
- All membership questions (e.g. "current members", "membership revenue by tier", "churn rate")
- Membership questions (e.g. "current members", "membership revenue by tier", "churn rate"), EXCEPT country/region
breakdowns, which use query_lfx_semantic_layer
- Maintainer names or maintainer+activities data joins, where activities data is the code activities model
with code contributions, PRs, commits etc (e.g. "top maintainers by contributions", "who maintains Kubernetes?").
IMPORTANT: activities data (contributors, PRs, code contributions etc) not involving maintainers should use query_lfx_semantic_layer.
Expand Down Expand Up @@ -71,7 +72,7 @@ Tips:
// QueryLFXLensArgs defines the input for query_lfx_lens.
type QueryLFXLensArgs struct {
ProjectSlug string `json:"project_slug" jsonschema:"Project slug from search_projects (e.g. 'cncf') (required)"`
Input string `json:"input" jsonschema:"Natural language question. Always use for memberships, maintainer names/trends, open-ended analysis, subproject questions, cross-domain joins, and exploratory questions. Takes 15-30s. (required)"`
Input string `json:"input" jsonschema:"Natural language question. Always use for memberships (except country/region breakdowns), maintainer names/trends, open-ended analysis, subproject questions, cross-domain joins, and exploratory questions. Takes 15-30s. (required)"`
}

type lensWorkflowAdditional struct {
Expand Down Expand Up @@ -161,7 +162,7 @@ const (
Best for direct, well-scoped questions: totals, counts, averages, breakdowns by a single dimension, and time series (e.g. "total activities for CNCF", "active maintainers by organization", "health score trend by month", "total enrollments by course"). This is also the right tool for contributor/activity questions — it has full contributor data including names, organizations, and activity breakdowns.

Use query_lfx_lens INSTEAD for:
- All membership questions (memberships model works better with ad-hoc SQL)
- Membership questions, EXCEPT country/region breakdowns (see the country/region tip)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 88c461e. Confirmed against a running server that the action property's description does ship in tools/list next to the tool description, so clients were seeing both the new regional carve-out and the old unconditional "use query_lfx_lens for memberships" instruction.

The schema text now reads "For memberships (except country/region breakdowns), ..." to match. I also added a schemaPropertyDescription helper and assert the registered schema's action description directly, rather than only the tool description, so the two can't drift apart again.

- Maintainer names, maintainer+contribution (activities data) joins, or maintainer trends
- Open-ended or exploratory analysis (e.g. "which projects need attention?")
- Questions involving subprojects (e.g. "health scores by project")
Expand Down Expand Up @@ -197,6 +198,7 @@ Tips:
- Contributors and code-related data (commits, PRs, insertions, deletions) are in the activities model — search for "activities" in list_metrics.
IMPORTANT: Questions about contributors and code-related topics that do not involve maintainers should prefer this tool.
- Events metrics use project_name rather than project_slug for filtering.
- Country/region breakdowns belong here for contributors, organizations, memberships, event registrations and enrollments — even when the topic would otherwise route to query_lfx_lens. A person's country uses country__* (e.g. country__lf_region); an organization's HQ uses organization_lf_region. Membership metrics don't inline dimensions, so call get_dimensions with search "country" or "region".
- `

// semanticLayerSlotSearchProjects: search_projects guidance.
Expand Down Expand Up @@ -232,7 +234,7 @@ func RegisterSemanticLayer(server *mcp.Server) {
// SemanticLayerLFXLensArgs defines the input for the unified semantic layer tool.
type SemanticLayerLFXLensArgs struct {
ProjectSlug string `json:"project_slug,omitempty" jsonschema:"Optional project slug from search_projects (e.g. 'cncf'). When provided, where-clause project filters are validated against that foundation's subtree. May be omitted for global or cross-foundation queries."`
Action string `json:"action" jsonschema:"Required. Start with list_metrics — often enough to go straight to query. Best for activities, maintainer counts, health scores, projects, events, education. For memberships, maintainer names/trends, open-ended, subproject, or exploratory questions use query_lfx_lens instead. Values: list_metrics, get_dimensions, query, describe"`
Action string `json:"action" jsonschema:"Required. Start with list_metrics — often enough to go straight to query. Best for activities, maintainer counts, health scores, projects, events, education. For memberships (except country/region breakdowns), maintainer names/trends, open-ended, subproject, or exploratory questions use query_lfx_lens instead. Values: list_metrics, get_dimensions, query, describe"`
Target string `json:"target,omitempty" jsonschema:"For action=describe only: which action to get help for (e.g. 'query')"`
Metrics string `json:"metrics,omitempty" jsonschema:"Comma-separated metric names from list_metrics (for get_dimensions and query)"`
Search string `json:"search,omitempty" jsonschema:"Search term to filter results (for list_metrics and get_dimensions)"`
Expand Down
69 changes: 66 additions & 3 deletions internal/tools/lens_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,14 @@ func TestSemanticLayerDescription(t *testing.T) {
"may be omitted for global or cross-foundation questions",
"For membership metrics, a Linux Foundation ('tlf') filter only captures direct LF memberships",
"Activity metrics are fanned out to foundations",
// Regional guidance: country/region questions must route here for every
// topic, including memberships, whose dimensions are not inlined.
"Country/region breakdowns belong here for contributors, organizations, memberships, event registrations and enrollments",
"even when the topic would otherwise route to query_lfx_lens",
"country__lf_region",
"organization_lf_region",
"call get_dimensions with search \"country\" or \"region\"",
"Membership questions, EXCEPT country/region breakdowns",
} {
if !strings.Contains(semanticLayerDescription, want) {
t.Errorf("description missing %q", want)
Expand All @@ -229,12 +237,17 @@ func TestSemanticLayerDescription(t *testing.T) {

func listSemanticLayerTool(t *testing.T) *mcp.Tool {
t.Helper()
return listRegisteredTool(t, "query_lfx_semantic_layer", RegisterSemanticLayer)
}

func listRegisteredTool(t *testing.T, name string, register func(*mcp.Server)) *mcp.Tool {
t.Helper()

server := mcp.NewServer(&mcp.Implementation{
Name: "test-server",
Version: "0.0.1",
}, nil)
RegisterSemanticLayer(server)
register(server)

ctx := context.Background()
clientTransport, serverTransport := mcp.NewInMemoryTransports()
Expand All @@ -256,11 +269,11 @@ func listSemanticLayerTool(t *testing.T) *mcp.Tool {
t.Fatalf("ListTools failed: %v", err)
}
for _, tool := range res.Tools {
if tool.Name == "query_lfx_semantic_layer" {
if tool.Name == name {
return tool
}
}
t.Fatal("query_lfx_semantic_layer not found in tool list")
t.Fatalf("%s not found in tool list", name)
return nil
}

Expand All @@ -279,6 +292,30 @@ func schemaRequired(t *testing.T, tool *mcp.Tool) []string {
return schema.Required
}

// schemaPropertyDescription returns the description a client sees for one
// input-schema property. These travel with tools/list alongside the tool
// description, so guidance in them must not contradict it.
func schemaPropertyDescription(t *testing.T, tool *mcp.Tool, property string) string {
t.Helper()
raw, err := json.Marshal(tool.InputSchema)
if err != nil {
t.Fatalf("failed to marshal input schema: %v", err)
}
var schema struct {
Properties map[string]struct {
Description string `json:"description"`
} `json:"properties"`
}
if err := json.Unmarshal(raw, &schema); err != nil {
t.Fatalf("failed to parse input schema: %v", err)
}
prop, ok := schema.Properties[property]
if !ok {
t.Fatalf("input schema has no %q property", property)
}
return prop.Description
}

func TestRegisterSemanticLayer_Schema(t *testing.T) {
tool := listSemanticLayerTool(t)

Expand All @@ -295,6 +332,32 @@ func TestRegisterSemanticLayer_Schema(t *testing.T) {
if !strings.Contains(tool.Description, "project_slug is optional") {
t.Error("description missing optional project_slug wording")
}

// The action property's own guidance ships with tools/list, so its
// membership routing must carry the same regional exception as the tool
// description — otherwise clients get contradictory instructions.
action := schemaPropertyDescription(t, tool, "action")
if !strings.Contains(action, "For memberships (except country/region breakdowns)") {
t.Errorf("action schema description missing the regional exception: %q", action)
}
}

// TestQueryLFXLensDescription_RegionalException guards the other half of the
// routing contract: query_lfx_lens claims memberships, and both its
// description and its input schema ship with tools/list. If they keep saying
// "always use for memberships" unconditionally, clients get instructions that
// contradict the semantic layer's regional carve-out.
func TestQueryLFXLensDescription_RegionalException(t *testing.T) {
tool := listRegisteredTool(t, "query_lfx_lens", RegisterQueryLFXLens)

if !strings.Contains(tool.Description, "EXCEPT country/region") {
t.Errorf("query_lfx_lens description missing the regional exception: %q", tool.Description)
}

input := schemaPropertyDescription(t, tool, "input")
if !strings.Contains(input, "memberships (except country/region breakdowns)") {
t.Errorf("query_lfx_lens input schema missing the regional exception: %q", input)
}
}

func contains(list []string, want string) bool {
Expand Down
Loading