From dc16998de3b02eb73bf648b8c6ca3db839c92032 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 14:58:33 -0700 Subject: [PATCH 1/3] docs(hubs): add FOCUS 1.4 schema documentation - upgrade.md: add v1_4 to the dashboard compatibility matrix, document the 22 new and 2 removed Costs columns (FOCUS 1.3 vs 1.4), list the new BillingPeriods (6), ContractCommitments (28), and InvoiceDetails (22) dataset columns, and note the v1_2 transform deprecation - compatibility.md: add the FinOps toolkit 15 / hubs v1_4 / FOCUS 1.4 compatibility row and clarify export-to-schema conversion - data-processing.md: update the transform narrative to v1_4, add v15 Costs transform notes and new dataset transform sections, and list the new raw datasets in the ingestion container docs - data-dictionary.md: add entries for the new FOCUS 1.3/1.4 Costs columns and removal/successor notes on ProviderName and PublisherName Co-Authored-By: Claude Fable 5 --- docs-mslearn/toolkit/help/data-dictionary.md | 30 +++++- docs-mslearn/toolkit/hubs/compatibility.md | 29 ++--- docs-mslearn/toolkit/hubs/data-processing.md | 63 +++++++++-- docs-mslearn/toolkit/hubs/upgrade.md | 105 +++++++++++++------ 4 files changed, 173 insertions(+), 54 deletions(-) diff --git a/docs-mslearn/toolkit/help/data-dictionary.md b/docs-mslearn/toolkit/help/data-dictionary.md index 58a8f31c3..d651e5356 100644 --- a/docs-mslearn/toolkit/help/data-dictionary.md +++ b/docs-mslearn/toolkit/help/data-dictionary.md @@ -3,7 +3,7 @@ title: FinOps toolkit data dictionary description: This article describes the column names found in FinOps toolkit solutions, including columns from Cost Management and FOCUS. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/15/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -26,7 +26,8 @@ Quick links:   [B](#b)   [C](#c)   [D](#d) -  [E](#e) +  [E](#e) +  [H](#h)   [I](#i)   [L](#l)   [M](#m) @@ -42,6 +43,11 @@ Quick links: | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AccountName | Name of the identity responsible for billing for this subscription. It's your EA enrollment account owner or Microsoft Online Subscription Agreement (MOSA) account admin. Not applicable to a Microsoft Customer Agreement (MCA). | | AccountOwnerId | Email address of the identity responsible for billing for this subscription. It's your EA enrollment account owner or MOSA account admin. Not applicable to MCA. | +| AllocatedMethodDetails | Details about the method or rule used to allocate the charge identified by `AllocatedMethodId`. Formatted as a JSON object. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedMethodId | Unique identifier of the method or rule used to allocate the charge. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedResourceId | Unique identifier of the resource the charge was allocated to. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedResourceName | Name of the resource the charge was allocated to. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedTags | Tags associated with the target the charge was allocated to. Formatted as a JSON object. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | | AmortizationClass | Indicates how a row relates to amortization to identify and distinguish between the charges before and after amortization is applied. Derived from `ChargeCategory` and `CommitmentDiscountStatus` when not provided by Cost Management. | | AvailabilityZone | Area within a resource location used for high availability. Not available for all services. Not included in Microsoft Cloud cost data. | | BilledCost | Amount owed for the charge after any applied discounts. If using FinOps hubs, you need to include the Cost Management connector to see all billed costs. Maps to CostInBillingCurrency for actual cost in Cost Management. | @@ -91,6 +97,7 @@ Quick links: | CommitmentDiscountUtilizationPotential | Derived. Amount that could have been applied to a commitment, but may not have been. This is generally the same as `CommitmentDiscountUtilizationAmount`, except for the unused charges. Uses cost for savings plans and quantity for reservations. | | CommitmentCostSavings | See [`CommitmentDiscountSavings`](#c). | | CommitmentCostSavingsRunningTotal | See [`CommitmentDiscountSavingsRunningTotal`](#c). | +| CommitmentProgramEligibilityDetails | Details about the commitment discount programs the charge is eligible for. Formatted as a JSON object. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | CommitmentUnitPriceSavings | Derived. Amount the unit price was reduced for commitment discounts. Does not include negotiated discounts. Formula: `ContractedUnitPrice - x_EffectiveUnitPrice`. | | CommitmentUtilization | See `CommitmentDiscountUtilization`. | | CommitmentUtilizationAmount | See `CommitmentDiscountUtilizationAmount`. | @@ -99,6 +106,19 @@ Quick links: | ConsumedQuantity | Volume of a given SKU associated with a resource or service used, based on the Consumed Unit. | | ConsumedUnit | Provider-specified measurement unit indicating how a provider measures usage of a given SKU associated with a resource or service. Maps to **UnitOfMeasure** in Cost Management. | | ConsumedService | Azure Resource Manager resource provider namespace. | +| ContractApplied | Indicates the contract terms applied to the charge. Formatted as a JSON object. Introduced in FOCUS 1.3 with a revised format in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentBenefitCategory | Primary type of benefit received for the contract commitment, like a discount, entitlement, or availability guarantee. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentCreated | Date and time when the contract commitment was created. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentDiscountPercentage | Percentage discount provided by the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentDurationType | Categorical length of the contract commitment (for example, "1 Year" or "3 Years") based on the commercial agreement or pricing model. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentFulfillmentInterval | Period used to measure and reset fulfillment of the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentLastUpdated | Date and time when the contract commitment was last updated. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentLifecycleStatus | Current lifecycle status of the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentModel | Operational behavior and consumption flexibility of the contract commitment, distinguishing "use it or lose it" obligations from flexible, aggregate-based agreements. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentOfferCategory | Indicates whether the pricing and terms of the contract commitment are based on a standard, publicly accessible offering or were privately negotiated. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentInterval | Interval at which payments are made for the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentModel | Financial settlement structure of the contract commitment, like a single upfront payment, recurring charges, or a combination of both. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentUpfrontPercentage | Percentage of the contract commitment cost paid upfront. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | ContractedCost | Cost calculated by multiplying contracted unit price and the corresponding Pricing Quantity. | | ContractedUnitPrice | The agreed-upon unit price for a single Pricing Unit of the associated SKU, inclusive of negotiated discounts, if present, while excluding negotiated commitment discounts or any other discounts. | | CostAllocationRuleName | Name of the Microsoft Cost Management cost allocation rule that generated this charge. Cost allocation is used to move or split shared charges. | @@ -113,7 +133,9 @@ Quick links: | EffectiveCostRunningTotal | Derived. Used in Power BI to calculate the total `EffectiveCost` for the reporting period up to and including the specified day. For instance, the cost for the 15th will include the cost from the 1st through the 15th. | | EffectivePricingCost | `EffectiveCost` in the pricing currency. | | EffectiveUnitPrice | Amortized price per unit after commitment discounts. | +| HostProviderName | Name of the provider that operates the infrastructure where the resource runs or the service is provided. Always "Microsoft" for Cost Management data. Introduced in FOCUS 1.3 as a replacement for `ProviderName`, which was removed in FOCUS 1.4. | | IncrementalRefreshDate | Derived. Numeric version of the `ChargePeriodStart` column to simplify setup for incremental refresh. | +| InvoiceDetailId | Unique identifier of the invoice line item the charge relates to. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | InvoiceId | Unique identifier for the invoice the charge is included in. Only available for closed months after the invoice is published. Renamed from **x_InvoiceId** in FOCUS 1.2. | | InvoiceIssuerId | Unique identifier of the organization that generated the invoice. | | InvoiceIssuerName | Name of the organization that generated the invoice. Only supported for CSP accounts. Not supported for EA or MCA accounts that a partner manages due to data not being provided by Cost Management. | @@ -142,9 +164,10 @@ Quick links: | PricingSubcategory | Describes the kind of pricing model used for a charge within a specific `PricingCategory`. | | PricingUnit | Derived. Indicates what measurement type is used by the `PricingQuantity`. Extracted from **UnitOfMeasure** in Cost Management. | | PricingUnitDescription | Describes the measurement type is used by the `PricingQuantity`. Maps to **UnitOfMeasure** in Cost Management. | +| ProviderName | Name of the organization that made the resources or services available for purchase. Removed from the Costs dataset in FOCUS 1.4 (FinOps hubs `v1_4` schema) in favor of `HostProviderName` and `ServiceProviderName`. The `Costs_v1_0()` and `Costs_v1_2()` functions still return this column. Still included in the CommitmentDiscountUsage, Recommendations, and Transactions datasets. | | PublisherCategory | Indicates whether a charge is from a cloud provider or third-party Marketplace vendor. Allowed values: `Cloud Provider` and `Vendor`. Maps to PublisherType in Cost Management. | | PublisherId | Unique identifier for the organization that created the product that got used or purchased. | -| PublisherName | Name of the organization that created the product that got used or purchased. | +| ~PublisherName~ (removed) | Name of the organization that created the product that got used or purchased. Removed from the Costs dataset in FOCUS 1.4 (FinOps hubs `v1_4` schema). Use `ServiceProviderName` instead. The `v1_0` and `v1_2` schema functions still return this column. | | ~Region~ (removed) | Isolated geographic area where a resource is provisioned in and/or a service is provided from. Replaced with `RegionId` and `RegionName` in v0.4 to align with FOCUS 1.0. | | RegionId | Provider-assigned identifier for an isolated geographic area where a resource is provisioned or a service is provided. | | RegionName | Name of an isolated geographic area where a resource is provisioned or a service is provided. | @@ -166,6 +189,7 @@ Quick links: | ServiceCategory | Top-level category for the `ServiceName`. This column aligns with the FOCUS requirements. | | ServiceModel | Indicates what aspects of the service are managed by the provider – infrastructure only, the platform custom code runs on, or the entire service. Derived via the FinOps toolkit Services open data file based on the `x_ResourceType` column when not specified by Cost Management. | | ServiceName | Name of the service the resource type is a part of. This column aligns with the FOCUS requirements. | +| ServiceProviderName | Name of the organization that makes the service available to consumers. The Marketplace publisher for Marketplace charges; otherwise, "Microsoft". Introduced in FOCUS 1.3 as a replacement for `PublisherName`, which was removed in FOCUS 1.4. | | ServiceSubcategory | Secondary classification of the ServiceCategory for a service based on its core function. Derived via the FinOps toolkit Services open data file based on the `x_ResourceType` column when not specified by Cost Management. | | SkuCoreCount | Derived. Indicates the number of virtual CPUs used by this resource. Extracted from `x_SkuDetails`. Used for Azure Hybrid Benefit reports. | | SkuCPUs | See `SkuCoreCount` | diff --git a/docs-mslearn/toolkit/hubs/compatibility.md b/docs-mslearn/toolkit/hubs/compatibility.md index dadbf0e04..37f612f49 100644 --- a/docs-mslearn/toolkit/hubs/compatibility.md +++ b/docs-mslearn/toolkit/hubs/compatibility.md @@ -3,7 +3,7 @@ title: Compatibility guide description: Learn which Power BI report versions are compatible with each FinOps hubs version to ensure seamless upgrades and data integrity. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/15/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -31,19 +31,20 @@ The storage path is important for any client that's utilizing the same datasets. The Power BI version refers to the Power BI reports made available within that specific version of the FinOps toolkit. If you customized an older version of the reports, make note of the version you started with, as that is what is useful here. -| Cost Management exports | FinOps hubs | Storage path | Power BI | Notes | -| ----------------------------------------- | ----------- | ---------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -| FOCUS costs, prices, reservation datasets | 12+ | {dataset}/{yyyy}/{MM}/{scope} | 12+² | KQL reports use the FinOps hubs v1_2 schema for FOCUS 1.2 | -| FOCUS costs, prices, reservation datasets | 0.8+ | {dataset}/{yyyy}/{MM}/{scope} | 0.7-0.11² | | -| FOCUS costs, prices, reservation datasets | 0.7 | {dataset}/{yyyy}/{MM}/{scope}¹ | 0.7+² | Storage path updated to new dataset names that support joining multiple related datasets together | -| FOCUS costs, prices, reservation datasets | 0.6 | {export-type}/{yyyy}/{MM}/{scope} | 0.6+ | Reservation recommendations pulled from hub storage | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.6 | {export-type}/{yyyy}/{MM}/{scope}¹ | 0.6+² | Storage path updated; Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.5 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.4 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Supports a mix of FOCUS 1.0 and 1.0-preview(v1) data; Reservation recommendations pulled from the Cost Management connector for Power BI | -| FOCUS 1.0-preview(v1) only | 0.4 | {scope}/{yyyyMM}/{export-type}¹ | 0.2+ | Storage path updated | -| FOCUS 1.0-preview(v1) only | 0.2 - 0.3 | {path}/{yyyyMM}/{export-type} | 0.2+² | Switched to FOCUS data only | -| Actual or Amortized costs (not both) | 0.1 - 0.1.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA and MCA | -| Actual or Amortized costs (not both) | 0.0.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA only | +| Cost Management exports | FinOps hubs | Storage path | Power BI | Notes | +| ----------------------------------------- | ----------- | ---------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| FOCUS costs, prices, reservation datasets | 15+ | {dataset}/{yyyy}/{MM}/{scope} | 15+² | KQL reports use the FinOps hubs v1_4 schema for FOCUS 1.4; Cost Management exports up to FOCUS 1.2, which hubs converts to FOCUS 1.4 during ingestion | +| FOCUS costs, prices, reservation datasets | 12-14 | {dataset}/{yyyy}/{MM}/{scope} | 12+² | KQL reports use the FinOps hubs v1_2 schema for FOCUS 1.2 | +| FOCUS costs, prices, reservation datasets | 0.8+ | {dataset}/{yyyy}/{MM}/{scope} | 0.7-0.11² | | +| FOCUS costs, prices, reservation datasets | 0.7 | {dataset}/{yyyy}/{MM}/{scope}¹ | 0.7+² | Storage path updated to new dataset names that support joining multiple related datasets together | +| FOCUS costs, prices, reservation datasets | 0.6 | {export-type}/{yyyy}/{MM}/{scope} | 0.6+ | Reservation recommendations pulled from hub storage | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.6 | {export-type}/{yyyy}/{MM}/{scope}¹ | 0.6+² | Storage path updated; Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.5 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.4 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Supports a mix of FOCUS 1.0 and 1.0-preview(v1) data; Reservation recommendations pulled from the Cost Management connector for Power BI | +| FOCUS 1.0-preview(v1) only | 0.4 | {scope}/{yyyyMM}/{export-type}¹ | 0.2+ | Storage path updated | +| FOCUS 1.0-preview(v1) only | 0.2 - 0.3 | {path}/{yyyyMM}/{export-type} | 0.2+² | Switched to FOCUS data only | +| Actual or Amortized costs (not both) | 0.1 - 0.1.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA and MCA | +| Actual or Amortized costs (not both) | 0.0.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA only | ¹ When storage paths update, there's a risk that re-exported data lands in a new place and might cause duplicate data. To resolve, remove the old folders in the **ingestion** container.
diff --git a/docs-mslearn/toolkit/hubs/data-processing.md b/docs-mslearn/toolkit/hubs/data-processing.md index 9c3e1cabb..4a1eb95a8 100644 --- a/docs-mslearn/toolkit/hubs/data-processing.md +++ b/docs-mslearn/toolkit/hubs/data-processing.md @@ -3,7 +3,7 @@ title: FinOps hubs data processing description: Learn how FinOps hubs process data, including scope setup, data normalization, and optimization, to enhance cost management and analysis. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/15/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -50,16 +50,16 @@ The following diagram depicts the end-to-end data ingestion process within FinOp 2. The **ingestion_ETL_dataExplorer** pipeline ingests data into the `{dataset}_raw` table in the Data Explorer. - The dataset name is the first folder in the **ingestion** container. - All raw tables are in the **Ingestion** database in Data Explorer. - 3. When data is ingested into raw tables in Data Explorer, an update policy copies the data into the corresponding `{dataset}_final_v1_0` table using the `{dataset}_transform_v1_0()` function to normalize all data to align to FOCUS 1.0. + 3. When data is ingested into raw tables in Data Explorer, an update policy copies the data into the corresponding `{dataset}_final_v1_4` table using the `{dataset}_transform_v1_4()` function to normalize all data to align to FOCUS 1.4. 4. After ingestion, the **ingestion_ETL_dataExplorer** pipeline performs some cleanup, including purging data in the final table that is past the data retention period. - As of 0.7, Data Explorer applies data retention in raw tables while data retention in final tables is applied by the ingestion pipeline. If data ingestion stops, historical data isn't purged. - Data retention can be configured during the template deployment or manually in the **config/settings.json** file in storage. 6. (Optional) The **queries_DailySchedule** trigger runs the **queries_ExecuteETL** pipeline once per day to query Azure Resource Graph for additional recommendations and saves results to the **ingestion/Recommendations** folder. [Learn more](#azure-resource-graph-recommendations). 7. Reports and other tools like Power BI read data from Data Explorer or the **ingestion** container. - Data in Data Explorer can be read from the **Hub** database. - - Use the `{dataset}()` function to use the latest schema. + - Use the `{dataset}()` function to use the latest schema (`v1_4` as of FinOps hubs v15). - This function is useful for quick exploration, but may introduce breaking changes as the FinOps hub instance is updated. - - Use the `{dataset}_v1_0()` function to use the FOCUS 1.0 schema. + - Use the versioned functions, like `{dataset}_v1_0()`, `{dataset}_v1_2()`, or `{dataset}_v1_4()`, to use a specific FOCUS schema version. - Versioned function schemas shouldn't change over time but values may change if the data source changes those values. - Avoid using the **Ingestion** database for queries. While not explicitly prohibited, the **Ingestion** database should be considered an internal area for staging and data preparation. - Data in storage can be read from `ingestion////`. @@ -78,7 +78,7 @@ The following diagram depicts the HubsRecommendations data flow: 1. The **queries_DailySchedule** trigger runs once per day. 2. The **queries_ExecuteETL** pipeline iterates through all query files in the **config/queries** storage folder. 3. The **queries_ETL_ingestion** pipeline executes each query against Azure Resource Graph, deduplicates results, and saves data as parquet in the **ingestion/Recommendations** folder. -4. (If using Azure Data Explorer) Data is ingested into the `Recommendations_raw` table and transformed using the `Recommendations_transform_v1_2()` function. +4. (If using Azure Data Explorer) Data is ingested into the `Recommendations_raw` table and transformed using the `Recommendations_transform_v1_4()` function. HubsRecommendations includes the following query types: @@ -100,7 +100,7 @@ HubsRecommendations includes the following query types: ## About Data Explorer ingestion -When data is ingested into Data Explorer, the `{dataset}_transform_v1_0()` functions apply transform rules in the **Ingestion** database. Each dataset has a different set of transform rules covered in the following sections. +When data is ingested into Data Explorer, the `{dataset}_transform_v1_4()` functions apply transform rules in the **Ingestion** database. Each dataset has a different set of transform rules covered in the following sections. For a list of requested changes, ideas under consideration, and open questions about the underlying Cost Management datasets, see [issue #1111](https://github.com/microsoft/finops-toolkit/issues/1111). Leave comments on that issue if you find opportunities to address any concerns or to voice your support for any of the specific issues. @@ -183,6 +183,54 @@ Transforms: - `x_CommitmentDiscountPercent` - `x_TotalDiscountPercent` - Added `x_SourceValues` to track data changes during ingestion. +- v15+: + - Convert FOCUS 1.0-preview, 1.0, and 1.2 data to FOCUS 1.4. + - Replaced `ProviderName` and `PublisherName` with the FOCUS 1.3 columns `HostProviderName` and `ServiceProviderName`. + - `HostProviderName` falls back to `ProviderName` when not provided. + - `ServiceProviderName` falls back to `PublisherName` (the Marketplace publisher), then `ProviderName`, then "Microsoft" to meet the FOCUS non-null requirement. + - The `{dataset}_v1_0()` and `{dataset}_v1_2()` functions still return `ProviderName` and `PublisherName` by down-converting from the new columns. + - Added the FOCUS 1.3 cost allocation columns: `AllocatedMethodDetails`, `AllocatedMethodId`, `AllocatedResourceId`, `AllocatedResourceName`, `AllocatedTags`, and `ContractApplied`. + - Added the FOCUS 1.4 columns `CommitmentProgramEligibilityDetails`, `InvoiceDetailId`, and 12 `ContractCommitment*` columns. + - These columns are passed through from the export when present and remain empty until Cost Management supports exporting FOCUS 1.4 data. + +### Billing period data transforms + +Supported datasets: + +- FOCUS 1.4 BillingPeriod (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. + +### Contract commitment data transforms + +Supported datasets: + +- FOCUS 1.4 ContractCommitment (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. + +### Invoice detail data transforms + +Supported datasets: + +- FOCUS 1.4 InvoiceDetail (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. ### Price data transforms @@ -279,8 +327,11 @@ ingestion/{dataset}/{date-folder-path}/{scope-id-path}/{ingestion-id}__{original - `ingestion` is the container where the data pipeline saves data. - `{dataset}` is the exported dataset type. If ingesting into Azure Data Explorer, the **Ingestion** database must have a matching, case-sensitive "\_raw" table (for example, "Costs_raw"). FinOps hubs support the following datasets in this release: + - **BillingPeriods** - FOCUS 1.4 billing period data. No Cost Management export available yet. - **CommitmentDiscountUsage** - Cost Management reservation details export. + - **ContractCommitments** - FOCUS 1.4 contract commitment data. No Cost Management export available yet. - **Costs** - FOCUS cost and usage data. + - **InvoiceDetails** - FOCUS 1.4 invoice detail data. No Cost Management export available yet. - **Prices** - Cost Management price sheet export. - **Recommendations** - Cost Management reservation recommendations export. - **Transactions** - Cost Management reservation transactions export. diff --git a/docs-mslearn/toolkit/hubs/upgrade.md b/docs-mslearn/toolkit/hubs/upgrade.md index eb7b112d6..47c745623 100644 --- a/docs-mslearn/toolkit/hubs/upgrade.md +++ b/docs-mslearn/toolkit/hubs/upgrade.md @@ -3,7 +3,7 @@ title: Upgrade your FinOps hubs description: Learn how to upgrade your existing FinOps hub instance to the latest version, including necessary steps and considerations. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/15/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit @@ -130,11 +130,12 @@ For more information, see [Set up Power BI reports](../power-bi/setup.md). The Data Explorer dashboard was introduced with Data Explorer support in 0.7 and also works with Microsoft Fabric since 0.10. Generally, the dashboard does not need to be updated once deployed unless you want to take advantage of new features. To upgrade the dashboard, replace the existing dashboard with the latest dashboard template. -Each version of the dashboard is configured to work with a specific FinOps hub schema version (v1_0 or v1_2). Schema versions ensure backwards compatibility across FOCUS dataset versions from different providers. Older dashboard versions will continue to work after upgrading to the latest version of FinOps hubs, but newer dashboard versions may not work with older FinOps hub versions. The following table outlines the supported combinations. +Each version of the dashboard is configured to work with a specific FinOps hub schema version (v1_0, v1_2, or v1_4). Schema versions ensure backwards compatibility across FOCUS dataset versions from different providers. Older dashboard versions will continue to work after upgrading to the latest version of FinOps hubs, but newer dashboard versions may not work with older FinOps hub versions. The following table outlines the supported combinations. | Dashboard version | FinOps hubs schema | FinOps hubs version | | ----------------- | ------------------ | ------------------- | -| 12+ | v1_2 | 12+ | +| 15+ | v1_4 | 15+ | +| 12-14 | v1_2 | 12+ | | 0.7-0.11 | v1_0 | 0.7+ | @@ -152,6 +153,8 @@ Replace the use of deprecated columns and functions: | Introduced | Retired | Deprecated | Replacement | | ---------- | ------- | ------------------------------------------- | --------------------------------------------------- | +| 0.7 | 15 | `Costs().ProviderName` | `Costs().HostProviderName` | +| 0.7 | 15 | `Costs().PublisherName` | `Costs().ServiceProviderName` | | 0.7 | 12 | `Costs().x_InvoiceId` | `Costs().InvoiceId` | | 0.7 | 12 | `Costs().x_PricingCurrency` | `Costs().PricingCurrency` | | 0.7 | 12 | `Costs().x_SkuMeterName` | `Costs().SkuMeter` | @@ -166,34 +169,74 @@ If using unversioned functions or updating from the `v1_0` schema version, revie If updating queries to use a newer schema version, use the following table to understand the changes introduced in each schema version for each managed dataset. -| Dataset | Schema | Column | Notes | -| ----------------------- | ------ | ------------------------------------- | ----------------------------------------------- | -| (All) | v1_2 | All `decimal` columns | Changed to `real` | -| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountQuantity` | New custom column | -| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountUnit` | New custom column | -| CommitmentDiscountUsage | v1_2 | `ServiceSubcategory` | New custom column | -| Costs | v1_2 | `CapacityReservationId` | New with FOCUS 1.2 | -| Costs | v1_2 | `CapacityReservationStatus` | New with FOCUS 1.2 | -| Costs | v1_2 | `CommitmentDiscountQuantity` | New with FOCUS 1.2 | -| Costs | v1_2 | `CommitmentDiscountUnit` | New with FOCUS 1.2 | -| Cost | v1_2 | `ServiceSubcategory` | New with FOCUS 1.2 | -| Cost | v1_2 | `SkuPriceDetails` | New with FOCUS 1.2; derived from `x_SkuDetails` | -| Costs | v1_2 | `x_AmortizationClass` | New with Cost Management FOCUS 1.2-preview | -| Costs | v1_2 | `x_CommitmentDiscountNormalizedRatio` | New with Cost Management FOCUS 1.2-preview | -| Costs | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | -| Costs | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | -| Costs | v1_2 | `x_ServiceModel` | New custom column | -| Costs | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | -| Prices | v1_2 | `CommitmentDiscountUnit` | New custom column | -| Prices | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | -| Prices | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | -| Recommendations | v1_2 | `ResourceId` | New custom column | -| Recommendations | v1_2 | `ResourceName` | New custom column | -| Recommendations | v1_2 | `ResourceType` | New custom column | -| Recommendations | v1_2 | `SubAccountName` | New custom column | -| Recommendations | v1_2 | `x_RecommendationDetails` | New custom column | -| Recommendations | v1_2 | `x_ResourceGroupName` | New custom column | -| Transactions | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | +| Dataset | Schema | Column | Notes | +| ----------------------- | ------ | -------------------------------------------- | -------------------------------------------------- | +| (All) | v1_2 | All `decimal` columns | Changed to `real` | +| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountQuantity` | New custom column | +| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountUnit` | New custom column | +| CommitmentDiscountUsage | v1_2 | `ServiceSubcategory` | New custom column | +| Costs | v1_2 | `CapacityReservationId` | New with FOCUS 1.2 | +| Costs | v1_2 | `CapacityReservationStatus` | New with FOCUS 1.2 | +| Costs | v1_2 | `CommitmentDiscountQuantity` | New with FOCUS 1.2 | +| Costs | v1_2 | `CommitmentDiscountUnit` | New with FOCUS 1.2 | +| Costs | v1_2 | `ServiceSubcategory` | New with FOCUS 1.2 | +| Costs | v1_2 | `SkuPriceDetails` | New with FOCUS 1.2; derived from `x_SkuDetails` | +| Costs | v1_2 | `x_AmortizationClass` | New with Cost Management FOCUS 1.2-preview | +| Costs | v1_2 | `x_CommitmentDiscountNormalizedRatio` | New with Cost Management FOCUS 1.2-preview | +| Costs | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | +| Costs | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | +| Costs | v1_2 | `x_ServiceModel` | New custom column | +| Costs | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | +| Costs | v1_4 | `AllocatedMethodDetails` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedMethodId` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedResourceId` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedResourceName` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedTags` | New with FOCUS 1.3 | +| Costs | v1_4 | `CommitmentProgramEligibilityDetails` | New with FOCUS 1.4 | +| Costs | v1_4 | `ContractApplied` | New with FOCUS 1.3 | +| Costs | v1_4 | `ContractCommitmentBenefitCategory` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentCreated` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentDiscountPercentage` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentDurationType` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentFulfillmentInterval` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentLastUpdated` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentLifecycleStatus` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentModel` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentOfferCategory` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentInterval` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentModel` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentUpfrontPercentage` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `HostProviderName` | New with FOCUS 1.3; replaces `ProviderName` | +| Costs | v1_4 | `InvoiceDetailId` | New with FOCUS 1.4 | +| Costs | v1_4 | `ProviderName` | Removed with FOCUS 1.4; use `HostProviderName`² | +| Costs | v1_4 | `PublisherName` | Removed with FOCUS 1.4; use `ServiceProviderName`² | +| Costs | v1_4 | `ServiceProviderName` | New with FOCUS 1.3; replaces `PublisherName` | +| Prices | v1_2 | `CommitmentDiscountUnit` | New custom column | +| Prices | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | +| Prices | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | +| Recommendations | v1_2 | `ResourceId` | New custom column | +| Recommendations | v1_2 | `ResourceName` | New custom column | +| Recommendations | v1_2 | `ResourceType` | New custom column | +| Recommendations | v1_2 | `SubAccountName` | New custom column | +| Recommendations | v1_2 | `x_RecommendationDetails` | New custom column | +| Recommendations | v1_2 | `x_ResourceGroupName` | New custom column | +| Transactions | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | + +¹ The `ContractCommitment*` columns are defined in the FOCUS 1.4 ContractCommitment dataset and included in the Costs dataset to provide per-charge contract commitment context. They remain empty until Cost Management supports exporting FOCUS 1.4 data.
+ +² The `{dataset}_v1_0()` and `{dataset}_v1_2()` functions still return `ProviderName` and `PublisherName` for FOCUS 1.4 data by down-converting from `HostProviderName` and `ServiceProviderName`.
+ +The Prices, Recommendations, Transactions, and CommitmentDiscountUsage datasets didn't change between the `v1_2` and `v1_4` schema versions. + +The `v1_4` schema also introduced three new managed datasets defined by FOCUS 1.4. These datasets remain empty until Cost Management supports exporting FOCUS 1.4 data: + +- **BillingPeriods** includes 6 FOCUS columns: `BillingPeriodCreated`, `BillingPeriodEnd`, `BillingPeriodLastUpdated`, `BillingPeriodStart`, `BillingPeriodStatus`, and `InvoiceIssuerName`. +- **ContractCommitments** includes 28 FOCUS columns: `BenefitCategory`, `BillingCurrency`, `ContractCommitmentApplicability`, `ContractCommitmentCategory`, `ContractCommitmentCost`, `ContractCommitmentId`, `ContractCommitmentPeriodEnd`, `ContractCommitmentPeriodStart`, `ContractCommitmentQuantity`, `ContractCommitmentType`, `ContractCommitmentUnit`, `ContractId`, `ContractPeriodEnd`, `ContractPeriodStart`, `Created`, `DiscountPercentage`, `DurationType`, `FulfillmentInterval`, `InvoiceIssuerName`, `LastUpdated`, `LifecycleStatus`, `Model`, `OfferCategory`, `PaymentInterval`, `PaymentModel`, `PaymentUpfrontPercentage`, `PricingCurrency`, and `PricingCurrencyContractCommitmentCost`. +- **InvoiceDetails** includes 22 FOCUS columns: `BilledCost`, `BillingAccountId`, `BillingCurrency`, `BillingPeriodEnd`, `BillingPeriodStart`, `ChargeCategory`, `InvoiceDetailCreated`, `InvoiceDetailDescription`, `InvoiceDetailGrain`, `InvoiceDetailId`, `InvoiceDetailLastUpdated`, `InvoiceId`, `InvoiceIssueDate`, `InvoiceIssueStatus`, `InvoiceIssuerName`, `PaymentCurrency`, `PaymentCurrencyBilledCost`, `PaymentCurrencyInvoiceDetailId`, `PaymentDueDate`, `PaymentTerms`, `PurchaseOrderNumber`, and `ReferenceInvoiceId`. + +Each new dataset also includes 5 hub columns: `x_IngestionTime`, `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion`. + +As of FinOps hubs v15 (schema version `v1_4`), the FOCUS 1.2 transform functions are deprecated and the `v1_2` update policies are disabled. New data is transformed to FOCUS 1.4 and lands in the `{dataset}_final_v1_4` tables. Data ingested under older releases remains in the `{dataset}_final_v1_0` and `{dataset}_final_v1_2` tables and is still available through both versioned functions, like `Costs_v1_2()`, and unversioned functions, which up-convert older data to the latest schema.
From c54ea88de1d6c090679b3a781d497659cd60c2f2 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 17:26:02 -0700 Subject: [PATCH 2/3] docs(hubs): update ContractCommitments column count and IDs for FOCUS 1.4 rename The Q4 schema change renamed ContractCommitments dataset columns to spec-prefixed FOCUS 1.4 IDs and added ContractCommitmentDescription and ServiceProviderName, bringing the dataset from 28 to 30 FOCUS columns. Sync upgrade.md and changelog.md (post-merge) to the new column list, verified against ContractCommitments_final_v1_4 in IngestionSetup_v1_4_Datasets.kql. Also fix changelog.md's stale ms.date. Co-Authored-By: Claude Fable 5 --- docs-mslearn/toolkit/changelog.md | 2 +- docs-mslearn/toolkit/hubs/upgrade.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index e168935bc..6484e4a02 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 07/16/2026 +ms.date: 07/15/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/upgrade.md b/docs-mslearn/toolkit/hubs/upgrade.md index 47c745623..d61fe6ae5 100644 --- a/docs-mslearn/toolkit/hubs/upgrade.md +++ b/docs-mslearn/toolkit/hubs/upgrade.md @@ -231,7 +231,7 @@ The Prices, Recommendations, Transactions, and CommitmentDiscountUsage datasets The `v1_4` schema also introduced three new managed datasets defined by FOCUS 1.4. These datasets remain empty until Cost Management supports exporting FOCUS 1.4 data: - **BillingPeriods** includes 6 FOCUS columns: `BillingPeriodCreated`, `BillingPeriodEnd`, `BillingPeriodLastUpdated`, `BillingPeriodStart`, `BillingPeriodStatus`, and `InvoiceIssuerName`. -- **ContractCommitments** includes 28 FOCUS columns: `BenefitCategory`, `BillingCurrency`, `ContractCommitmentApplicability`, `ContractCommitmentCategory`, `ContractCommitmentCost`, `ContractCommitmentId`, `ContractCommitmentPeriodEnd`, `ContractCommitmentPeriodStart`, `ContractCommitmentQuantity`, `ContractCommitmentType`, `ContractCommitmentUnit`, `ContractId`, `ContractPeriodEnd`, `ContractPeriodStart`, `Created`, `DiscountPercentage`, `DurationType`, `FulfillmentInterval`, `InvoiceIssuerName`, `LastUpdated`, `LifecycleStatus`, `Model`, `OfferCategory`, `PaymentInterval`, `PaymentModel`, `PaymentUpfrontPercentage`, `PricingCurrency`, and `PricingCurrencyContractCommitmentCost`. +- **ContractCommitments** includes 30 FOCUS columns: `BillingCurrency`, `ContractCommitmentApplicability`, `ContractCommitmentBenefitCategory`, `ContractCommitmentCategory`, `ContractCommitmentCost`, `ContractCommitmentCreated`, `ContractCommitmentDescription`, `ContractCommitmentDiscountPercentage`, `ContractCommitmentDurationType`, `ContractCommitmentFulfillmentInterval`, `ContractCommitmentId`, `ContractCommitmentLastUpdated`, `ContractCommitmentLifecycleStatus`, `ContractCommitmentModel`, `ContractCommitmentOfferCategory`, `ContractCommitmentPaymentInterval`, `ContractCommitmentPaymentModel`, `ContractCommitmentPaymentUpfrontPercentage`, `ContractCommitmentPeriodEnd`, `ContractCommitmentPeriodStart`, `ContractCommitmentQuantity`, `ContractCommitmentType`, `ContractCommitmentUnit`, `ContractId`, `ContractPeriodEnd`, `ContractPeriodStart`, `InvoiceIssuerName`, `PricingCurrency`, `PricingCurrencyContractCommitmentCost`, and `ServiceProviderName`. - **InvoiceDetails** includes 22 FOCUS columns: `BilledCost`, `BillingAccountId`, `BillingCurrency`, `BillingPeriodEnd`, `BillingPeriodStart`, `ChargeCategory`, `InvoiceDetailCreated`, `InvoiceDetailDescription`, `InvoiceDetailGrain`, `InvoiceDetailId`, `InvoiceDetailLastUpdated`, `InvoiceId`, `InvoiceIssueDate`, `InvoiceIssueStatus`, `InvoiceIssuerName`, `PaymentCurrency`, `PaymentCurrencyBilledCost`, `PaymentCurrencyInvoiceDetailId`, `PaymentDueDate`, `PaymentTerms`, `PurchaseOrderNumber`, and `ReferenceInvoiceId`. Each new dataset also includes 5 hub columns: `x_IngestionTime`, `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion`. From 5bb6f89959e18c9a39d49ea2744abab789dcada7 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 16 Jul 2026 00:27:16 +0000 Subject: [PATCH 3/3] chore: Update ms.date in docs-mslearn files --- docs-mslearn/toolkit/help/data-dictionary.md | 2 +- docs-mslearn/toolkit/hubs/compatibility.md | 2 +- docs-mslearn/toolkit/hubs/data-processing.md | 2 +- docs-mslearn/toolkit/hubs/upgrade.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs-mslearn/toolkit/help/data-dictionary.md b/docs-mslearn/toolkit/help/data-dictionary.md index 6011e7e68..64d11902d 100644 --- a/docs-mslearn/toolkit/help/data-dictionary.md +++ b/docs-mslearn/toolkit/help/data-dictionary.md @@ -3,7 +3,7 @@ title: FinOps toolkit data dictionary description: This article describes the column names found in FinOps toolkit solutions, including columns from Cost Management and FOCUS. author: flanakin ms.author: micflan -ms.date: 07/15/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/compatibility.md b/docs-mslearn/toolkit/hubs/compatibility.md index 37f612f49..17fa1e2a5 100644 --- a/docs-mslearn/toolkit/hubs/compatibility.md +++ b/docs-mslearn/toolkit/hubs/compatibility.md @@ -3,7 +3,7 @@ title: Compatibility guide description: Learn which Power BI report versions are compatible with each FinOps hubs version to ensure seamless upgrades and data integrity. author: flanakin ms.author: micflan -ms.date: 07/15/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/data-processing.md b/docs-mslearn/toolkit/hubs/data-processing.md index 4a1eb95a8..ceedf55ce 100644 --- a/docs-mslearn/toolkit/hubs/data-processing.md +++ b/docs-mslearn/toolkit/hubs/data-processing.md @@ -3,7 +3,7 @@ title: FinOps hubs data processing description: Learn how FinOps hubs process data, including scope setup, data normalization, and optimization, to enhance cost management and analysis. author: flanakin ms.author: micflan -ms.date: 07/15/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/upgrade.md b/docs-mslearn/toolkit/hubs/upgrade.md index d61fe6ae5..c1ace5606 100644 --- a/docs-mslearn/toolkit/hubs/upgrade.md +++ b/docs-mslearn/toolkit/hubs/upgrade.md @@ -3,7 +3,7 @@ title: Upgrade your FinOps hubs description: Learn how to upgrade your existing FinOps hub instance to the latest version, including necessary steps and considerations. author: flanakin ms.author: micflan -ms.date: 07/15/2026 +ms.date: 07/16/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit