Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql)#2126
Conversation
Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/ Recommendations/Transactions transforms and final tables renamed for FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns: AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator split cost allocation), ContractApplied (per-row contract commitment application), ServiceProviderName + HostProviderName (replacing the deprecated ProviderName/PublisherName, with empty-fallback for back compat). Costs_raw now carries the new columns so downstream v1_2 transforms keep working and v1_3 transforms can read them directly. The v1_3 file is wired into the Bicep deployment alongside v1_0 and v1_2. The unversioned Costs() function still aliases to v1_2 — phase 2 adds Costs_v1_3() and retargets Latest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates FOCUS 1.3 and 1.4 column additions into a single v1_4 ingestion setup file. Removes the intermediate v1_3 file, adds ContractCommitment_raw table definition, and updates app.bicep and .build.config to reference v1_4. Co-Authored-By: Claude <noreply@anthropic.com>
flanakin
left a comment
There was a problem hiding this comment.
Changes here need to also be applied to the other 3 FOCUS 1.4 PRs
| @@ -359,6 +455,11 @@ | |||
|
|
|||
| // Costs_raw table -- Redefine all columns | |||
| .alter table Costs_raw ( | |||
There was a problem hiding this comment.
FOCUS 1.4 has a requirement to include ALL native columns. This will significantly bloat the table size for cross-cloud data. Given we don't have a full set of columns for this yet, we need to have a way to ingest any additional columns either by auto-accepting new columns on ingestion or dumping them into a single x_ column. We need to research options and weigh pros/cons. This can happen in a separate PR, but don't resolve this without a link to that PR.
There was a problem hiding this comment.
🤖 [AI][Claude Code] 🤔 Needs discussion
Filed a tracking issue to scope the research: #2159 — Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat. The issue captures the three candidate approaches (auto-accept new columns, single x_NativeColumns JSON, hybrid) and explicitly scopes implementation to a separate PR.
This PR ships the FOCUS 1.4 schema with the columns we know today; native-columns handling will follow #2159's resolution. Leaving this thread unresolved per your request until the follow-up PR is linked.
- Pluralize ContractCommitments_raw / _transform_v1_4 / _final_v1_4 / mapping / update policy - Add BillingPeriods_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add InvoiceDetails_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails + InvoiceDetailId to Costs_raw, Costs_final_v1_4, and Costs_transform_v1_4 - Fix x_Source* annotations on ContractCommitments_raw to Hubs v1_4+ - Update Costs_raw ProviderName/PublisherName annotations to FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) - Update Costs_raw Region annotation: deprecated -> removed - File #2159 for the FOCUS 1.4 native-columns architecture research 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
|
🤖 [AI][Claude Code] PR Update Summary Addressed: 7 thread(s)
Key changes:
Cascading next: Same pluralization + new BillingPeriods/InvoiceDetails datasets will be added to PRs #2128 (hub functions), #2135 (plugin docs), and #2136 (tests + changelog). |
- Rename ContractCommitment_v1_4() -> ContractCommitments_v1_4() (and update folder label) - Add BillingPeriods_v1_4() and InvoiceDetails_v1_4() hub functions - Add BillingPeriods(), ContractCommitments() (renamed), and InvoiceDetails() unversioned aliases in HubSetup_Latest.kql - Add CommitmentProgramEligibilityDetails + 12 ContractCommitment* + InvoiceDetailId column defaults to Costs_final_v1_0 and Costs_final_v1_2 union arms (back-compat shims) and add them to the project list 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
- Rename ContractCommitment() -> ContractCommitments() in skill references and agent docs - Add BillingPeriods() and InvoiceDetails() to the function list and dedicated agent doc sections - Mention the new FOCUS 1.4 per-row columns (CommitmentProgramEligibilityDetails, 12 ContractCommitment* columns, InvoiceDetailId) - Update output-style guidance to reference all three supplemental datasets 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
Tests: - Pluralize ContractCommitments_raw / _final_v1_4 / _v1_4 assertions - Assert new BillingPeriods_raw + InvoiceDetails_raw datasets exist with their FOCUS 1.4 columns - Assert new BillingPeriods_v1_4(), InvoiceDetails_v1_4(), ContractCommitments_v1_4() hub functions - Assert HubSetup_Latest aliases for BillingPeriods(), ContractCommitments(), InvoiceDetails() - Assert Costs_raw and Costs_final_v1_4 include the 14 new FOCUS 1.4 columns - Assert NO singular ContractCommitment_raw / ContractCommitment_v1_4 / ContractCommitment() exists - Assert Costs_raw keeps removed ProviderName/PublisherName/Region for back compat Changelog: - Expand v15.0.0 entry with the 14 new FOCUS 1.4 columns and the three new supplemental datasets (ContractCommitments, BillingPeriods, InvoiceDetails) - Update plugin entry to list ContractCommitments() / BillingPeriods() / InvoiceDetails() 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
| ProviderName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) | ||
| PublisherName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) | ||
| Region: string, // FOCUS 0.5-1.0-preview (removed; kept for back compat) |
There was a problem hiding this comment.
| ProviderName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) | |
| PublisherName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) | |
| Region: string, // FOCUS 0.5-1.0-preview (removed; kept for back compat) | |
| ProviderName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat) | |
| PublisherName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat) | |
| Region: string, // FOCUS 0.5-1.0-preview (removed; keep for back-compat) |
| // FOCUS 1.3 | ||
| isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity, | ||
| // FOCUS 1.0-preview, 1.0 | ||
| isempty(CommitmentDiscountStatus), real(null), | ||
| CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)), | ||
| CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio, | ||
| real(null) | ||
| ) | ||
| | extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case( | ||
| // FOCUS 1.3 |
There was a problem hiding this comment.
| // FOCUS 1.3 | |
| isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity, | |
| // FOCUS 1.0-preview, 1.0 | |
| isempty(CommitmentDiscountStatus), real(null), | |
| CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)), | |
| CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio, | |
| real(null) | |
| ) | |
| | extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case( | |
| // FOCUS 1.3 | |
| // FOCUS 1.2+ | |
| isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity, | |
| // FOCUS 1.0-preview, 1.0 | |
| isempty(CommitmentDiscountStatus), real(null), | |
| CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)), | |
| CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio, | |
| real(null) | |
| ) | |
| | extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case( | |
| // FOCUS 1.2+ |
| // | ||
| // Pricing | ||
| | extend old_x_AmortizationClass = x_AmortizationClass, x_AmortizationClass = case( | ||
| // FOCUS 1.3 |
There was a problem hiding this comment.
| // FOCUS 1.3 | |
| // FOCUS 1.2+ |
| ) | ||
| | extend x_ConsumedCoreHours = iff(ConsumedUnit == 'Hours' and isnotempty(x_SkuCoreCount), x_SkuCoreCount * ConsumedQuantity, real(null)) | ||
| | extend SkuPriceDetails = case( | ||
| // FOCUS 1.3 |
There was a problem hiding this comment.
| // FOCUS 1.3 | |
| // FOCUS 1.2+ |
| ``` | ||
|
|
||
|
|
||
| //===| BillingPeriods |================================================================================================= |
There was a problem hiding this comment.
BillingPeriods should be first in this file (alphabetized list)
| ContractApplied = parse_json(ContractApplied), // FOCUS 1.3+ | ||
| ContractCommitmentBenefitCategory = column_ifexists('ContractCommitmentBenefitCategory', ''), // FOCUS 1.4+ | ||
| ContractCommitmentCreated = column_ifexists('ContractCommitmentCreated', datetime(null)), // FOCUS 1.4+ | ||
| ContractCommitmentDiscountPercentage = column_ifexists('ContractCommitmentDiscountPercentage', real(null)), // FOCUS 1.4+ | ||
| ContractCommitmentDurationType = column_ifexists('ContractCommitmentDurationType', ''), // FOCUS 1.4+ | ||
| ContractCommitmentFulfillmentInterval = column_ifexists('ContractCommitmentFulfillmentInterval', ''), // FOCUS 1.4+ | ||
| ContractCommitmentLastUpdated = column_ifexists('ContractCommitmentLastUpdated', datetime(null)), // FOCUS 1.4+ | ||
| ContractCommitmentLifecycleStatus = column_ifexists('ContractCommitmentLifecycleStatus', ''), // FOCUS 1.4+ | ||
| ContractCommitmentModel = column_ifexists('ContractCommitmentModel', ''), // FOCUS 1.4+ | ||
| ContractCommitmentOfferCategory = column_ifexists('ContractCommitmentOfferCategory', ''), // FOCUS 1.4+ | ||
| ContractCommitmentPaymentInterval = column_ifexists('ContractCommitmentPaymentInterval', ''), // FOCUS 1.4+ | ||
| ContractCommitmentPaymentModel = column_ifexists('ContractCommitmentPaymentModel', ''), // FOCUS 1.4+ | ||
| ContractCommitmentPaymentUpfrontPercentage = column_ifexists('ContractCommitmentPaymentUpfrontPercentage', real(null)), // FOCUS 1.4+ |
There was a problem hiding this comment.
Port from older FOCUS exports
| ContractedUnitPrice = coalesce(ContractedUnitPrice, x_OnDemandUnitPrice), | ||
| EffectiveCost, | ||
| HostProviderName = iff(isempty(HostProviderName), PublisherName, HostProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat | ||
| InvoiceDetailId = column_ifexists('InvoiceDetailId', ''), // FOCUS 1.4+ |
There was a problem hiding this comment.
Port from older FOCUS exports
| PricingCurrency = coalesce(PricingCurrency, x_PricingCurrency), | ||
| PricingQuantity, | ||
| PricingUnit, | ||
| // ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName. |
There was a problem hiding this comment.
| // ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName. |
| ResourceType, | ||
| ServiceCategory, | ||
| ServiceName, | ||
| ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat |
There was a problem hiding this comment.
| ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat | |
| ServiceProviderName = iff(isempty(ServiceProviderName), PublisherName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat |
| PricingCurrency: string, | ||
| PricingQuantity: real, | ||
| PricingUnit: string, | ||
| // ProviderName and PublisherName removed in FOCUS 1.4 |
There was a problem hiding this comment.
| // ProviderName and PublisherName removed in FOCUS 1.4 |
- D4 revised: new FOCUS datasets in scope per owner review direction on #2126 - D3 refined: provider-name mapping non-null guarantee; swap defect noted - D9 added: split IngestionSetup_v1_4.kql for the Bicep 131K loadTextContent limit - §6: dispositions + blocking fix lists for all 6 reviewed PRs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fix swapped HostProviderName/ServiceProviderName fallbacks and guarantee non-null ServiceProviderName (C1) - Project all 22 new Costs_raw columns from the ActualCosts/AmortizedCosts C360 transforms in v1_0, v1_2, and v1_4 (C2) - Disable v1_2 update policies and add deprecation docstrings pointing to the _v1_4() transforms (C3) - Detect native FOCUS 1.3/1.4 exports in x_SourceVersion and treat 1.4 as the current version for x_SourceChanges (H1) - Always preserve removed ProviderName/PublisherName values in x_SourceValues (M1) - Apply review suggestions: raw table annotation wording, FOCUS 1.2+ comment fixes, remove stale placement comments - Split IngestionSetup_v1_4.kql into IngestionSetup_v1_4.kql (Prices + Costs) and IngestionSetup_v1_4_Datasets.kql (remaining datasets, alphabetized) to stay under the Bicep loadTextContent() 131,072-character limit; register the new file in app.bicep and .build.config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed 30f926d addressing the round-2 review feedback and the remaining W2 fix list. Mapping of changes to review comments: Applied
Also in this push (W2 fix list)
Deliberately not done (tracked follow-ups — native-column research, #2159)
Verification: |
Resolved IngestionSetup_v1_2.kql docstring conflict by combining dev's AmortizedCost copy-paste fix with the DEPRECATED annotation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renames 12 unprefixed columns (BenefitCategory, Created, Model, ...) to their spec IDs (ContractCommitmentBenefitCategory, ...) and adds the two missing columns (ContractCommitmentDescription, ServiceProviderName), bringing the dataset to the full 30 FOCUS columns across the raw table, parquet mapping, transform, and final table. Hub x_ add-on columns are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Synced with dev and applied the Q4 schema fix:
Verified: |
🛠️ Description
Adds the FOCUS 1.4 ingestion schema (
v1_4) for FinOps hubs, converting all currently supported export versions up to FOCUS 1.4 and pre-wiring detection for native 1.3/1.4 exports.IngestionSetup_v1_4.kql(Prices, Costs, ActualCosts/AmortizedCosts C360 transforms) andIngestionSetup_v1_4_Datasets.kql(BillingPeriods, CommitmentDiscountUsage, ContractCommitments, InvoiceDetails, Recommendations, Transactions) — split into two self-contained files to stay under the BiceploadTextContent()131,072-character limit; each dataset's transform, final table, and update policy live in the same file.Costs_rawandContractCommitments_rawinIngestionSetup_RawTables.kql.ProviderName/PublisherNametoHostProviderName/ServiceProviderName(falls back to the deprecated columns; guarantees non-nullServiceProviderNameper spec); original values are always preserved inx_SourceValues.x_SourceVersion(byServiceProviderName/HostProviderName/InvoiceDetailIdpresence) and treats1.4as the current version when labelingx_SourceChanges.Costs_rawcolumns from the C360 (ActualCosts/AmortizedCosts) transforms in v1_0, v1_2, and v1_4 so the transactional update policies validate against the raw table schema._v1_4()transforms (same pattern as the v1_0 → v1_2 upgrade).Analytics/app.bicepand the.build.configFabric KQL bundle.New FOCUS 1.4 datasets (BillingPeriods, ContractCommitments, InvoiceDetails) define schema in advance; they remain empty until Cost Management ships an export for them.
Verification
Build-Toolkit finops-hubsucceeds (bicep build +loadTextContent; both v1_4 files under the 131,072-character limit: 91,437 and 37,175 bytes).Costs_rawcolumn set (166 columns; verified programmatically).IngestionSetup_v1_2.kqlhas zero enabled update policies; v1_4 files have 4 + 6 enabled policies.finops-hub-fabric-setup-Ingestion.kql) includes both v1_4 files.HubsIngestionQueries.Tests.ps1: 192 passed, 0 failed.📋 Checklist
🔬 How did you test this change?
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
🤖 Generated with Claude Code