Add x_RecommendationCategory and x_RecommendationSubcategory enrichments to v1_4 Recommendations#2194
Conversation
Adds two hub-toolkit enrichment columns to the Recommendations dataset in the FOCUS 1.4 hub schema, normalized to a fixed hub taxonomy so reports can group recommendations consistently across providers. - IngestionSetup_v1_4.kql: normalize x_RecommendationCategory into the 5 top-level values (Cost, Operational Excellence, Performance, Reliability, Security); map Azure Advisor's HighAvailability -> Reliability; derive x_RecommendationSubcategory from the documented taxonomy with 'Other' as the fallback. Add both columns to Recommendations_final_v1_4. - HubSetup_v1_4.kql: include both columns in the Recommendations_v1_4() project list. - data-model.md: document the allowed values for both columns and the normalization behavior. - HubsFocusSchemas.Tests.ps1: assert columns exist in the final table, the transform projects them, and the case() expression covers all 5 top-level categories. - changelog.md: note the new enrichment columns under FinOps hubs v15.0.0. Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Taxonomy review — "Pick one" resolutions (Low Utilization, Idle Resources, Unused Capacity)The PR resolved both of the user's open "pick one" questions and the standalone-Unused-Capacity question. The picks are reasonable but the boundary between
Folding
Item 3 is fundamentally different from 1 and 2 — the user has already paid for it, the resource isn't actually idle from the provider's POV, and the remediation lives in commitment portfolio management, not resource cleanup. Lumping unused commitment hours under Recommendation: Either (a) keep 🤖 [AI][Claude Code] Taxonomy review |
Taxonomy review — Cross-category overlap and routing ambiguityA single subcategory list per parent category is clean, but several subcategories sit on category boundaries where ingestion will have to make a judgment call. These need either (a) tie-breaker rules documented next to the taxonomy, or (b) renames that make the boundary obvious.
Without disambiguation rules, two Hub deployments ingesting the same Advisor stream could classify identical recommendations differently depending on which 🤖 [AI][Claude Code] Taxonomy review |
Taxonomy review — Missing subcategoriesThe Cost category is the most consequential one for a FinOps Hub, and several very common cost-recommendation types have no home in the current taxonomy and would all fall into Critical gaps (would already be hit by Azure Advisor, Cost Management, and partner tools today):
Important gaps:
Cross-cloud / multi-cloud allocation — the user's question implied this is in scope. If the hub will normalize AWS Cost Explorer / GCP Recommender recs (per the FOCUS framing), categories like "Reserved Instance modification" (AWS) and "Committed Use Discount" (GCP) already map cleanly to existing Cost subcategories — that's fine. But "Savings Plans coverage gap", "Compute Optimizer rightsizing", and "GCP Recommender idle VM" need test coverage to confirm the mapping holds. Worth adding test cases. The taxonomy currently captures the FinOps Framework's "Workload Optimization" and "Rate Optimization" capabilities reasonably well but misses most of "Architecting for Cloud" (license, tier, transfer) and parts of "Allocation". 🤖 [AI][Claude Code] Taxonomy review |
Taxonomy review — Naming consistency and label hygieneThe taxonomy is mostly noun-phrased and Title-Cased, but a few inconsistencies will create friction for downstream consumers (filters, dropdowns, KQL Hyphen vs space inconsistency:
Either keep the hyphen for
This is the most jarring inconsistency. Suggest renaming to Verb vs noun phrasing: mostly noun-phrased, which is good. The two outliers worth checking:
Capitalization — "Other" placement —
🤖 [AI][Claude Code] Taxonomy review |
Taxonomy review — KQL mapping logic and default behaviorThe Asymmetric defaults — category vs subcategory:
This asymmetry is documented in the PR description, but it has a usability consequence: a row can legitimately have Suggest one of:
Reservation default to Hard-coded reservation subcategory — the fallback
If the hub ever ingests reservation utilization recs under the same
No mapping for Microsoft Defender for Cloud / Microsoft Sentinel recs — these flow through a different API and category schema ( Subcategory 🤖 [AI][Claude Code] Taxonomy review |
Taxonomy review — Documentation completeness and canonical-vs-open statusThe taxonomy is documented in Is this a closed enum or an open vocabulary? The PR description says "fixed hub taxonomy" but the docs don't explicitly say "this list is canonical and will not change without a schema version bump". Without that statement:
Suggest adding to
No queryable form — the taxonomy lives in (a) the KQL
Today, a Power BI author building a Recommendations report has to either (a) re-type the list from the docs, or (b) Mapping table to source values is missing — the docs say "Azure Advisor's
This becomes essential when you add a second source (Defender for Cloud, AWS Trusted Advisor, GCP Recommender). Changelog entry — the entry calls out the columns but not the taxonomy decisions ( 🤖 [AI][Claude Code] Taxonomy review |
Resolves conflicts from the upstream IngestionSetup_v1_4.kql split (D9): - IngestionSetup_v1_4.kql: take the phase7 split (Recommendations and other supplemental datasets moved to IngestionSetup_v1_4_Datasets.kql) - IngestionSetup_v1_4_Datasets.kql: re-apply this PR's Recommendations category/subcategory normalization where the transform now lives - HubsFocusSchemas.Tests.ps1: keep the phase7 rewrite; taxonomy assertions auto-merged as an additional context - changelog.md: keep both sides' entries; ms.date set to today Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and enforce category coherence Addresses the W9 review feedback on the recommendation taxonomy: - Declare x_RecommendationSubcategory in all 22 single-purpose hub-native Resource Graph queries (Idle Resources, SKU Modernization, Spot Eligibility, License Optimization, Storage Tiering) so they no longer fall back to 'Other'. AdvisorCost is per-row heterogeneous and keeps the transform fallback. - Replace the flat subcategory allow-list in Recommendations_transform_v1_4 with a SubcategoryTaxonomy datatable of (subcategory, parent) pairs, and enforce coherence: unknown categories are backfilled from a valid subcategory's parent, and a subcategory only passes through when its parent matches the resolved category. - Make defaults symmetric and deliberate: both columns always have a value and unmapped values become 'Other' for both. - Taxonomy hygiene: add License Optimization and Storage Tiering (both used by hub queries today); rename Identity & Access -> Identity and Access and Negotiated Discount Enablement -> Negotiated Discounts; keep unused commitment capacity under Commitment Discount Utilization instead of overloading Idle Resources. - Document the canonical taxonomy contract, source-value mapping table, normalization rules, and cross-category tie-breakers in data-model.md; document subcategory guidance for custom queries in configure-recommendations.md. - Rework the taxonomy test context: canonical taxonomy as test source of truth validated against both the KQL datatable and the docs, coherence rule assertions, and per-query subcategory validation for all 22 hub query files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
W9 fix pass — responses to the 6 taxonomy review commentsAll 6 review comments are addressed below. Merged 1. Low Utilization / Idle Resources / Unused Capacity overlap — addressed (option a + b)Took the recommendation: 2. Cross-category routing ambiguity — addressed via documented tie-breakers; renames deliberately deferredTie-breaker rules are documented inline in the taxonomy list in 3. Missing subcategories — partially addressed, grounded in what the repo produces todayAdded 4. Naming hygiene — addressed
5. KQL defaults and coherence — addressedThe asymmetry is gone. Decision (reflected in 6. Canonical-enum documentation — addressed; queryable form deferred
🤖 [AI][Claude Code] W9 fix pass |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Synced with phase 7, completing the dev sync through the full FOCUS 1.4 stack. Clean merge, no conflicts (changelog entries auto-merged). This pulls in from upstream: the Verified: |
|
Hubs + ADX (managed) deployed to |
🛠️ Description
Adds two hub-toolkit enrichment columns to the
Recommendationsdataset in the FOCUS 1.4 hub schema (v1_4):x_RecommendationCategoryandx_RecommendationSubcategory, normalized to a fixed, documented hub taxonomy so reports can group recommendations consistently across providers.HighAvailability→Reliability.x_RecommendationDetails(see mapping table below) instead of falling back toOther.Otherwhen unmapped (symmetric defaults).data-model.md(with a stability statement, source-value mapping table, and cross-category tie-breakers) and enforced by tests that validate the KQL datatable, the docs, and every query file against a single canonical list.Query → subcategory mapping
All hub-native queries hardcode
x_RecommendationCategory = 'Cost'. Subcategories:Idle ResourcesSKU ModernizationLicense OptimizationSpot EligibilityStorage TieringOtheruntil a per-type mapping (for example, keyed onx_RecommendationTypeId) is addedReservation recommendations from Cost Management exports default to
Commitment Discount Coverage.Decisions
Low Utilization= running but underused (resize/consolidate);Idle Resources= doing no useful work, including stopped, unattached, and orphaned resources (stop/deallocate/delete). Unused commitment capacity is explicitly routed toCommitment Discount Utilizationrather than overloadingIdle Resources.License OptimizationandStorage Tieringunder Cost — both are used by hub-native queries today. Other suggested subcategories (data transfer, cost allocation, marketplace) are noted in the docs as future candidates; nothing in the repo produces them yet.Identity & Access→Identity and Access(the only ampersand in the list) andNegotiated Discount Enablement→Negotiated Discounts.Otheris now an allowed category fallback too, so both columns always have a value.High Availaibility→High Availabilitytypo from the design notes.Files changed
.../Analytics/scripts/IngestionSetup_v1_4_Datasets.kql— taxonomy normalization inRecommendations_transform_v1_4()(aSubcategoryTaxonomydatatable of (subcategory, parent) pairs + coherence rules) and both columns inRecommendations_final_v1_4(the Recommendations transform moved here with the D9 file split).../Analytics/scripts/HubSetup_v1_4.kql— both columns in theRecommendations_v1_4()projection.../Recommendations/queries/*.json— 22 query files declarex_RecommendationSubcategoryinx_RecommendationDetailsdocs-mslearn/toolkit/hubs/data-model.md— canonical taxonomy, stability statement, source mapping table, normalization rules, tie-breakersdocs-mslearn/toolkit/hubs/configure-recommendations.md— subcategory guidance for custom query authorssrc/powershell/Tests/Unit/HubsFocusSchemas.Tests.ps1—Recommendations taxonomy (W9)context (42 tests) validating KQL, docs, and query files against one canonical taxonomydocs-mslearn/toolkit/changelog.md— entry under FinOps hubs v15.0.0Verification
Invoke-Pester ./src/powershell/Tests/Unit/*— 2097 total, 2093 passed, 0 failed, 4 skipped.Build-Toolkit finops-hubcompletes with exit code 0; both v1_4 ingestion setup scripts are under the Bicep 131KloadTextContent()limit (91,445 and 40,494 chars including these changes).Recommendations()returning reservation rows with categoryCost/ subcategoryCommitment Discount Coverage, AdvisorHighAvailabilityrows asReliability, and hub-native query rows with their mapped subcategories — not yet verified against a live deployment.Stacked on #2136 (flanakin/focus14-phase7-tests-changelog); merges up the stack toward
flanakin/focus14.📋 Checklist
🔬 How did you test this change?
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
🤖 Generated with Claude Code