-
Notifications
You must be signed in to change notification settings - Fork 29
feat: add Compass catalog manifests for all agentic packs #133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
ebce94d
c008708
f8e52a9
656cfe1
df03ed4
3deefb1
3bf6e63
d0532ae
d884c97
5125295
a69a5fd
7e301fc
b913638
c6b8d80
b476511
8f60a02
8c2289e
77c4cb3
386f8a0
ac06903
4ca7656
44e705a
25c03b5
6aa42bb
c3ace00
a40e79b
d312972
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,14 +12,16 @@ Contributors work here to create, improve, and validate skills. An internal proc | |
|
|
||
| ``` | ||
| agentic-plugins/ | ||
| ├── catalog-info.yaml # Root Location — entry point for Compass ingestion | ||
| ├── system.yaml # System entity (agentic-plugins) — models the whole repo | ||
| ├── rh-sre/ # Site Reliability Engineering pack (reference implementation) | ||
| ├── rh-developer/ # Developer tools pack | ||
| ├── ocp-admin/ # OpenShift administration pack | ||
| ├── rh-virt/ # Virtualization management pack | ||
| ├── rh-basic/ # Getting started pack | ||
| ├── rh-ai-engineer/ # AI Engineering pack | ||
| ├── rh-automation/ # Automation pack | ||
| ├── rh-support-engineer/ # Support engineering pack | ||
| ├── mcps/ # MCP server Compass manifests (System + 9 MCPServer entities) | ||
| ├── eval/ # Skill evaluation reports (report.json + report.md per skill) | ||
| ├── scripts/ # Validation and CI helper scripts | ||
| ├── catalog/ # JSON Schema for .catalog/collection.yaml validation | ||
|
|
@@ -37,6 +39,8 @@ Each pack is persona-specific and follows this structure: | |
|
|
||
| ``` | ||
| <pack-name>/ | ||
| ├── catalog-info.yaml # Location entity — indexes this pack's Compass manifests | ||
| ├── <pack-name>-plugin.yaml # AiResource (type: plugin) — defines the pack itself | ||
| ├── AGENTS.md # AI Context Module instruction routing (persona, skills, rules) | ||
| ├── README.md # Pack description, persona, target marketplaces | ||
| ├── mcps.json # MCP server configurations (uses env vars for credentials) | ||
|
|
@@ -45,14 +49,120 @@ Each pack is persona-specific and follows this structure: | |
| │ └── collection.json # Deterministic JSON mirror of collection.yaml | ||
| ├── skills/ # Specialized task executors (including orchestration skills) | ||
| │ └── <skill>/ | ||
| │ └── SKILL.md # Skill definition with YAML frontmatter | ||
| │ ├── SKILL.md # Skill definition with YAML frontmatter | ||
| │ └── catalog-info.yaml # AiResource (type: skill) — Compass manifest | ||
| └── docs/ # AI-optimized knowledge base (optional, rh-sre reference) | ||
| ``` | ||
|
|
||
| ### Relationship with the Catalog | ||
|
|
||
| Each pack's `.catalog/` directory contains metadata that describes the pack for the marketplace. This metadata stays here, alongside the skills it describes. The catalog build process reads it from this repo to assemble the unified marketplace. The golden sources are always `SKILL.md`, `AGENTS.md`, `README.md`, and `mcps.json` — `.catalog/` is derived from them, never the other way around. | ||
|
|
||
| ### Compass / Backstage Manifests | ||
|
|
||
| The repository is registered in [Red Hat Compass](https://compass.redhat.com) (internal Backstage instance) through a hierarchy of Backstage entity manifests. All manifests use `apiVersion: backstage.io/v1alpha1` except MCPServer entities which use `apiVersion: mcp/v1beta1`. | ||
|
|
||
| #### Entity Kinds | ||
|
|
||
| | Kind | Purpose | spec.type | Count | | ||
| |------|---------|-----------|-------| | ||
| | **Location** | Index that references other manifest files | — | 9 (1 root + 7 packs + 1 mcps) | | ||
| | **System** | Top-level grouping for the repository and MCP servers | — | 2 (`agentic-plugins`, `rh-agentic-plugins-mcps`) | | ||
| | **AiResource** | Skills and pack definitions | `plugin` (packs) / `skill` (skills) / `rule` | 7 packs + 75 skills | | ||
| | **MCPServer** | MCP server configurations | `local` | 9 | | ||
|
|
||
| #### Location Hierarchy | ||
|
|
||
| Compass ingests a single root Location. Everything else is discovered through delegation: | ||
|
|
||
| ``` | ||
| catalog-info.yaml (root Location) | ||
| ├── system.yaml → System: agentic-plugins | ||
| ├── ocp-admin/catalog-info.yaml → Location → ocp-admin-plugin.yaml + 7 skills | ||
| ├── rh-sre/catalog-info.yaml → Location → rh-sre-plugin.yaml + 13 skills | ||
| ├── rh-virt/catalog-info.yaml → Location → rh-virt-plugin.yaml + 10 skills | ||
| ├── rh-developer/catalog-info.yaml → Location → rh-developer-plugin.yaml + 17 skills | ||
| ├── rh-basic/catalog-info.yaml → Location → rh-basic-plugin.yaml + 6 skills | ||
| ├── rh-ai-engineer/catalog-info.yaml → Location → rh-ai-engineer-plugin.yaml + 11 skills | ||
| ├── rh-automation/catalog-info.yaml → Location → rh-automation-plugin.yaml + 11 skills | ||
| └── mcps/catalog-info.yaml → Location → system.yaml + 9 MCPServers | ||
| ``` | ||
|
|
||
| #### Entity Relationships | ||
|
|
||
| All relationships use `dependsOn` / `dependencyOf`: | ||
|
|
||
| > **Why not `partOf`/`hasPart`?** Compass only processes `partOf`/`hasPart` for standard Backstage kinds (Component, API, Resource). Custom kinds like `AiResource` and `MCPServer` can store these fields in `spec`, but they are not processed into the relation graph. Only `dependsOn`/`dependencyOf` and `ownedBy`/`ownerOf` generate actual relations for custom kinds. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems a bug and we should instead try to fix it rather than finding a workaround. WDYT? |
||
|
|
||
| Concrete relationships in this repo: | ||
|
|
||
| - **Skill → Plugin**: `dependsOn` / `dependencyOf` (a skill belongs to its plugin) | ||
| - **Plugin → System**: `dependsOn` / `dependencyOf` (a plugin belongs to the system) | ||
| - **MCPServer → System**: `dependsOn` / `dependencyOf` (an MCP server belongs to the system) | ||
| - **Skill → MCPServer**: `dependsOn` / `dependencyOf` (a skill uses an MCP server) | ||
| - **Plugin → MCPServer**: `dependsOn` / `dependencyOf` (a plugin uses an MCP server) | ||
| - **Skill → Skill**: `dependsOn` / `dependencyOf` (orchestration skills invoke other skills) | ||
| - **All entities → Group**: `spec.owner: group:redhat/ai5-marketplace` | ||
|
|
||
| > **Bidirectional declaration policy:** Compass does not auto-generate inverse relations for custom entity kinds such as `AiResource` and `MCPServer` (tracked as COMPASS-1288). Until this is resolved upstream, we explicitly declare **both directions** of every relationship in our manifests. For example, if a skill declares `dependsOn: [airesource:rh-sre/rh-sre]`, the plugin must also declare `dependencyOf: [airesource:rh-sre/<skill>]`. When adding or modifying a relationship, always update both the source and target manifests. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some of them are not needed and may require hard maintenance, please consider this to define your design. |
||
|
|
||
| #### Namespaces | ||
|
|
||
| Each pack uses its own namespace matching the pack name (e.g., `rh-sre`, `ocp-admin`). All MCP servers share the namespace `agentic-plugins`. The root System `agentic-plugins` uses the `default` namespace. | ||
|
|
||
| #### Entity Reference Formats | ||
|
|
||
| - Skills: `airesource:<pack-namespace>/<skill-name>` | ||
| - Pack plugins: `airesource:<pack-namespace>/<pack-name>` | ||
| - MCP servers: `mcpserver:agentic-plugins/<server-name>` | ||
|
|
||
| #### Adding Compass Manifests for a New Skill | ||
|
r2dedios marked this conversation as resolved.
|
||
|
|
||
| When adding a skill, create `skills/<skill-name>/catalog-info.yaml`: | ||
| ```yaml | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: AiResource | ||
| metadata: | ||
| name: <skill-name> | ||
| namespace: <pack-name> | ||
| title: <Skill Title> | ||
| description: > | ||
| <skill description> | ||
| labels: | ||
| distribution: external | ||
| annotations: | ||
| backstage.io/source-location: >- | ||
| url:https://github.com/RHEcosystemAppEng/agentic-plugins/blob/main/<pack>/skills/<skill>/SKILL.md | ||
| tags: | ||
| - ai-skill | ||
| links: | ||
| - url: https://github.com/RHEcosystemAppEng/agentic-plugins | ||
| title: Source Repository | ||
| icon: github | ||
| spec: | ||
| type: skill | ||
| lifecycle: beta | ||
| owner: group:redhat/ai5-marketplace | ||
| disciplines: | ||
| - <discipline> | ||
| categories: | ||
| - <category> | ||
| agents: | ||
| - claude-code | ||
| - opencode | ||
| - cursor | ||
| dependsOn: | ||
| - airesource:<pack-name>/<pack-name> | ||
| # Add mcpserver and airesource (skill) dependencies as needed | ||
| - mcpserver:agentic-plugins/<server-name> | ||
| ``` | ||
|
|
||
| Then update **both sides** of every relationship: | ||
| 1. Add the file as a target in the pack's `catalog-info.yaml` Location | ||
| 2. Add the skill to the plugin's `dependencyOf` list in `<pack>-plugin.yaml` | ||
| 3. Add the skill to each referenced MCP server's `dependencyOf` list in `mcps/<server>.yaml` | ||
| 4. If the skill depends on other skills, add `dependencyOf` entries in those skills' manifests | ||
|
|
||
| ## Contributing | ||
|
|
||
| Skills are added directly to this repository, inside an existing pack. The contributor opens a PR, skills are reviewed and merged, and maintainers own them from that point. Use `/agentic-contribution-skill` in Claude Code or follow [CONTRIBUTING.md](CONTRIBUTING.md). | ||
|
|
@@ -165,6 +275,7 @@ last_updated: YYYY-MM-DD | |
|
|
||
| ### Files | ||
| - Skills: `skills/<skill-name>/SKILL.md` (uppercase SKILL.md) | ||
| - Compass manifests: `catalog-info.yaml` (Locations), `<pack-name>-plugin.yaml` (pack AiResource), `system.yaml` (System entities) | ||
| - Docs: Lowercase with dashes, categorized by directory | ||
|
|
||
| ## Development Workflow | ||
|
|
@@ -176,7 +287,11 @@ last_updated: YYYY-MM-DD | |
| 3. Add `AGENTS.md` with persona, skill-first rule, intent routing table, MCP servers, and global rules (see [rh-ai-engineer/AGENTS.md](rh-ai-engineer/AGENTS.md) for reference) | ||
| 4. Create `skills/` directory | ||
| 5. Add `mcps.json` when the pack integrates MCP servers (use `${VAR}` for secrets) | ||
| 6. Update main `README.md` table with link | ||
| 6. Create Compass manifests: | ||
| - `<pack-name>-plugin.yaml` — AiResource with `type: plugin`, `system: agentic-plugins` | ||
| - `catalog-info.yaml` — Location targeting the plugin file and all skill catalog-info.yaml files | ||
| 7. Add the pack's `catalog-info.yaml` as a target in the root `catalog-info.yaml` | ||
| 8. Update main `README.md` table with link | ||
|
|
||
| ### Adding a Skill | ||
|
|
||
|
|
@@ -192,8 +307,10 @@ last_updated: YYYY-MM-DD | |
| - Dependencies declaration | ||
| 4. Include concrete examples and complete error handling | ||
| 5. Update the pack's `AGENTS.md` intent routing table to include the new skill | ||
| 6. Test with `Skill` tool invocation | ||
| 7. Validate with `uv run python scripts/validate_skills_tier1.py <pack>/skills/<skill-name>/SKILL.md` | ||
| 6. Create `skills/<skill-name>/catalog-info.yaml` Compass manifest (see "Adding Compass Manifests for a New Skill") | ||
| 7. Add the skill's `catalog-info.yaml` as a target in the pack's `catalog-info.yaml` Location | ||
| 8. Test with `Skill` tool invocation | ||
| 9. Validate with `uv run python scripts/validate_skills_tier1.py <pack>/skills/<skill-name>/SKILL.md` | ||
|
|
||
| **Collection-Specific Standards:** | ||
| - **rh-virt**: Follow `rh-virt/SKILL_TEMPLATE.md` for enhanced quality standards including mandatory Common Issues and Example Usage sections | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| apiVersion: backstage.io/v1alpha1 | ||
| kind: Location | ||
| metadata: | ||
| name: agentic-plugins | ||
| title: Agentic Plugins | ||
| description: > | ||
| Root catalog entry point for the agentic-plugins repository. | ||
| Points to all agentic pack locations and MCP server manifests | ||
| for Compass ingestion. | ||
| annotations: | ||
| agentic-plugins.redhat.com/version: "1.0" | ||
| agentic-plugins.redhat.com/repo: "agentic-plugins" | ||
| spec: | ||
| targets: | ||
| # Repository-level system | ||
| - ./system.yaml | ||
| # Agentic Pack Locations | ||
| - ./ocp-admin/catalog-info.yaml | ||
| - ./rh-sre/catalog-info.yaml | ||
| - ./rh-virt/catalog-info.yaml | ||
| - ./rh-developer/catalog-info.yaml | ||
| - ./rh-basic/catalog-info.yaml | ||
| - ./rh-ai-engineer/catalog-info.yaml | ||
| - ./rh-automation/catalog-info.yaml | ||
| # MCP Servers Location | ||
| - ./mcps/catalog-info.yaml |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| apiVersion: mcp/v1beta1 | ||
| kind: MCPServer | ||
| metadata: | ||
| namespace: agentic-plugins | ||
| name: ai-observability | ||
| title: AI Observability MCP Server | ||
| description: > | ||
| AI observability MCP server for monitoring and debugging AI/ML model | ||
| inference, performance metrics, and serving runtime health on | ||
| Red Hat OpenShift AI. | ||
| labels: | ||
| distribution: external | ||
| annotations: | ||
| backstage.io/source-location: >- | ||
| url:https://github.com/RHEcosystemAppEng/agentic-plugins/blob/main/rh-ai-engineer/mcps.json | ||
| tags: | ||
| - ai | ||
| - observability | ||
| - machine-learning | ||
| - monitoring | ||
| - mcp-server | ||
| links: | ||
| - url: https://github.com/rh-ai-quickstart/ai-observability-summarizer | ||
| title: Upstream Repository | ||
| icon: github | ||
| spec: | ||
| type: remote | ||
| lifecycle: beta | ||
| owner: group:redhat/ai5-marketplace | ||
| system: default/agentic-plugins | ||
| primitives: | ||
| - type: tool | ||
| name: list_models | ||
| description: List all available AI models for analysis. | ||
| - type: tool | ||
| name: list_vllm_namespaces | ||
| description: Get list of monitored vLLM Kubernetes namespaces with observability data. | ||
| - type: tool | ||
| name: get_model_config | ||
| description: Get available LLM models configured for summarization and analysis. | ||
| - type: tool | ||
| name: get_vllm_metrics_tool | ||
| description: Get available vLLM metrics with friendly names and PromQL queries. | ||
| - type: tool | ||
| name: fetch_vllm_metrics_data | ||
| description: Fetch vLLM metrics data with time-series for dashboard display. | ||
| - type: tool | ||
| name: analyze_vllm | ||
| description: Analyze vLLM metrics and generate an AI summary using an LLM. | ||
| - type: tool | ||
| name: calculate_metrics | ||
| description: Calculate statistics (avg, min, max, latest) for provided metrics data. | ||
| - type: tool | ||
| name: list_summarization_models | ||
| description: List all configured summarization models with metadata from runtime configuration. | ||
| - type: tool | ||
| name: get_gpu_info | ||
| description: Get cluster GPU information from Prometheus DCGM metrics. | ||
| - type: tool | ||
| name: get_deployment_info | ||
| description: Get model deployment details for a model in a namespace. | ||
| - type: tool | ||
| name: chat_vllm | ||
| description: Chat about vLLM metrics by asking follow-up questions on analyzed data. | ||
| - type: tool | ||
| name: analyze_openshift | ||
| description: Analyze OpenShift metrics for a category and scope with LLM-generated summary. | ||
| - type: tool | ||
| name: fetch_openshift_metrics_data | ||
| description: Fetch OpenShift metrics data for dashboard visualization without LLM analysis. | ||
| - type: tool | ||
| name: list_openshift_namespaces | ||
| description: Get list of all OpenShift namespaces observed in Prometheus. | ||
| - type: tool | ||
| name: list_openshift_metric_groups | ||
| description: Return available OpenShift metric group categories for cluster-wide analysis. | ||
| - type: tool | ||
| name: list_openshift_namespace_metric_groups | ||
| description: Return OpenShift metric groups that support namespace-scoped analysis. | ||
| - type: tool | ||
| name: chat_openshift | ||
| description: Chat about OpenShift metrics for a specific category and scope. | ||
| - type: tool | ||
| name: search_metrics | ||
| description: Search for Prometheus metrics by pattern using semantic understanding. | ||
| - type: tool | ||
| name: get_metric_metadata | ||
| description: Get detailed metadata for a specific Prometheus metric. | ||
| - type: tool | ||
| name: get_label_values | ||
| description: Get all possible values for a specific label of a Prometheus metric. | ||
| - type: tool | ||
| name: execute_promql | ||
| description: Execute a PromQL query and return structured results. | ||
| - type: tool | ||
| name: explain_results | ||
| description: Explain PromQL query results in natural language. | ||
| - type: tool | ||
| name: suggest_queries | ||
| description: Suggest related PromQL queries based on user intent. | ||
| - type: tool | ||
| name: select_best_metric | ||
| description: Select the best metric for a user question using semantic analysis. | ||
| - type: tool | ||
| name: find_best_metric_with_metadata_v2 | ||
| description: Enhanced metric selection with improved semantic analysis and query suggestions. | ||
| - type: tool | ||
| name: find_best_metric_with_metadata | ||
| description: Find the best metric for a user question using comprehensive metadata analysis. | ||
| - type: tool | ||
| name: get_metrics_categories | ||
| description: Get all metric categories with summary counts, priorities, and example queries. | ||
| - type: tool | ||
| name: search_metrics_by_category | ||
| description: Search metrics filtered by category IDs and priority levels. | ||
| - type: tool | ||
| name: get_category_metrics_detail | ||
| description: Get category summaries or detailed metrics catalog for a single category. | ||
| - type: tool | ||
| name: convert_time_to_promql_duration | ||
| description: Convert decimal hours to Prometheus duration format for use in PromQL queries. | ||
| - type: tool | ||
| name: query_tempo_tool | ||
| description: Search Tempo traces by TraceQL query, service, operation, and time range. | ||
| - type: tool | ||
| name: get_trace_details_tool | ||
| description: Get detailed trace information including spans by trace ID. | ||
| - type: tool | ||
| name: chat_tempo_tool | ||
| description: Conversational interface for Tempo trace analysis using natural language questions. | ||
| - type: tool | ||
| name: korrel8r_query_objects | ||
| description: Execute a Korrel8r domain query and return correlated objects. | ||
| - type: tool | ||
| name: korrel8r_get_correlated | ||
| description: Return correlated objects (alerts, logs, traces, metrics) for a Korrel8r query. | ||
| - type: tool | ||
| name: get_correlated_logs | ||
| description: Fetch application and infrastructure logs for a namespace or pod via Korrel8r. | ||
| - type: tool | ||
| name: chat | ||
| description: Chat with AI models using observability tools with progress tracking. | ||
| - type: tool | ||
| name: validate_api_key | ||
| description: Validate an API key for a given provider by making a minimal server-side request. | ||
| - type: tool | ||
| name: save_api_key | ||
| description: Save or update a provider API key in a Kubernetes Secret. | ||
| - type: tool | ||
| name: check_provider_secret | ||
| description: Check if a provider API key secret exists in the cluster. | ||
| - type: tool | ||
| name: delete_provider_secret | ||
| description: Delete a provider API key secret from the cluster. | ||
| - type: tool | ||
| name: list_provider_models | ||
| description: Query a provider API to list available models for configuration. | ||
| - type: tool | ||
| name: add_model_to_config | ||
| description: Add a new model to the model configuration by updating the Kubernetes ConfigMap. | ||
| - type: tool | ||
| name: update_maas_model_api_key | ||
| description: Update API key and optionally endpoint for an existing MaaS model. | ||
| packages: [] | ||
| remotes: | ||
| - url: https://ai-observability.example.com/mcp | ||
| dependsOn: | ||
| - system:default/agentic-plugins | ||
| dependencyOf: | ||
|
r2dedios marked this conversation as resolved.
|
||
| - airesource:rh-ai-engineer/ai-observability | ||
| - airesource:rh-ai-engineer/debug-inference | ||
| - airesource:rh-ai-engineer/ds-project-setup | ||
| - airesource:rh-ai-engineer/guardrails-config | ||
| - airesource:rh-ai-engineer/model-deploy | ||
| - airesource:rh-ai-engineer/model-monitor | ||
| - airesource:rh-ai-engineer/model-registry | ||
| - airesource:rh-ai-engineer/nim-setup | ||
| - airesource:rh-ai-engineer/pipeline-manage | ||
| - airesource:rh-ai-engineer/rh-ai-engineer | ||
| - airesource:rh-ai-engineer/serving-runtime-config | ||
| - airesource:rh-ai-engineer/workbench-manage | ||
Uh oh!
There was an error while loading. Please reload this page.