diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index e168935bc..6484e4a02 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 07/16/2026 +ms.date: 07/15/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/help/data-dictionary.md b/docs-mslearn/toolkit/help/data-dictionary.md index 5c0fe49b5..64d11902d 100644 --- a/docs-mslearn/toolkit/help/data-dictionary.md +++ b/docs-mslearn/toolkit/help/data-dictionary.md @@ -3,7 +3,7 @@ title: FinOps toolkit data dictionary description: This article describes the column names found in FinOps toolkit solutions, including columns from Cost Management and FOCUS. author: flanakin ms.author: micflan -ms.date: 04/29/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -26,7 +26,8 @@ Quick links:   [B](#b)   [C](#c)   [D](#d) -  [E](#e) +  [E](#e) +  [H](#h)   [I](#i)   [L](#l)   [M](#m) @@ -42,6 +43,11 @@ Quick links: | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AccountName | Name of the identity responsible for billing for this subscription. It's your EA enrollment account owner or Microsoft Online Subscription Agreement (MOSA) account admin. Not applicable to a Microsoft Customer Agreement (MCA). | | AccountOwnerId | Email address of the identity responsible for billing for this subscription. It's your EA enrollment account owner or MOSA account admin. Not applicable to MCA. | +| AllocatedMethodDetails | Details about the method or rule used to allocate the charge identified by `AllocatedMethodId`. Formatted as a JSON object. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedMethodId | Unique identifier of the method or rule used to allocate the charge. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedResourceId | Unique identifier of the resource the charge was allocated to. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedResourceName | Name of the resource the charge was allocated to. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | +| AllocatedTags | Tags associated with the target the charge was allocated to. Formatted as a JSON object. Introduced in FOCUS 1.3. Not currently populated by Cost Management. | | AmortizationClass | Indicates how a row relates to amortization to identify and distinguish between the charges before and after amortization is applied. Derived from `ChargeCategory` and `CommitmentDiscountStatus` when not provided by Cost Management. | | AvailabilityZone | Area within a resource location used for high availability. Not available for all services. Not included in Microsoft Cloud cost data. | | BilledCost | A charge serving as the basis for invoicing, inclusive of all reduced rates and discounts while excluding the amortization of upfront charges (one-time or recurring). See [`EffectiveCost`](#e) for the amortized equivalent. | @@ -91,6 +97,7 @@ Quick links: | CommitmentDiscountUtilizationPotential | Derived. Amount that could have been applied to a commitment, but may not have been. This is generally the same as `CommitmentDiscountUtilizationAmount`, except for the unused charges. Uses cost for savings plans and quantity for reservations. | | CommitmentCostSavings | See [`CommitmentDiscountSavings`](#c). | | CommitmentCostSavingsRunningTotal | See [`CommitmentDiscountSavingsRunningTotal`](#c). | +| CommitmentProgramEligibilityDetails | Details about the commitment discount programs the charge is eligible for. Formatted as a JSON object. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | CommitmentUnitPriceSavings | Derived. Amount the unit price was reduced for commitment discounts. Does not include negotiated discounts. Formula: `ContractedUnitPrice - x_EffectiveUnitPrice`. | | CommitmentUtilization | See `CommitmentDiscountUtilization`. | | CommitmentUtilizationAmount | See `CommitmentDiscountUtilizationAmount`. | @@ -99,6 +106,19 @@ Quick links: | ConsumedQuantity | Volume of a given SKU associated with a resource or service used, based on the Consumed Unit. | | ConsumedUnit | Provider-specified measurement unit indicating how a provider measures usage of a given SKU associated with a resource or service. Maps to **UnitOfMeasure** in Cost Management. | | ConsumedService | Azure Resource Manager resource provider namespace. | +| ContractApplied | Indicates the contract terms applied to the charge. Formatted as a JSON object. Introduced in FOCUS 1.3 with a revised format in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentBenefitCategory | Primary type of benefit received for the contract commitment, like a discount, entitlement, or availability guarantee. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentCreated | Date and time when the contract commitment was created. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentDiscountPercentage | Percentage discount provided by the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentDurationType | Categorical length of the contract commitment (for example, "1 Year" or "3 Years") based on the commercial agreement or pricing model. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentFulfillmentInterval | Period used to measure and reset fulfillment of the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentLastUpdated | Date and time when the contract commitment was last updated. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentLifecycleStatus | Current lifecycle status of the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentModel | Operational behavior and consumption flexibility of the contract commitment, distinguishing "use it or lose it" obligations from flexible, aggregate-based agreements. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentOfferCategory | Indicates whether the pricing and terms of the contract commitment are based on a standard, publicly accessible offering or were privately negotiated. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentInterval | Interval at which payments are made for the contract commitment. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentModel | Financial settlement structure of the contract commitment, like a single upfront payment, recurring charges, or a combination of both. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | +| ContractCommitmentPaymentUpfrontPercentage | Percentage of the contract commitment cost paid upfront. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | ContractedCost | Cost calculated by multiplying contracted unit price and the corresponding Pricing Quantity. | | ContractedUnitPrice | The agreed-upon unit price for a single Pricing Unit of the associated SKU, inclusive of negotiated discounts, if present, while excluding negotiated commitment discounts or any other discounts. | | CostAllocationRuleName | Name of the Microsoft Cost Management cost allocation rule that generated this charge. Cost allocation is used to move or split shared charges. | @@ -113,7 +133,9 @@ Quick links: | EffectiveCostRunningTotal | Derived. Used in Power BI to calculate the total `EffectiveCost` for the reporting period up to and including the specified day. For instance, the cost for the 15th will include the cost from the 1st through the 15th. | | EffectivePricingCost | `EffectiveCost` in the pricing currency. | | EffectiveUnitPrice | Amortized price per unit after commitment discounts. | +| HostProviderName | Name of the provider that operates the infrastructure where the resource runs or the service is provided. Always "Microsoft" for Cost Management data. Introduced in FOCUS 1.3 as a replacement for `ProviderName`, which was removed in FOCUS 1.4. | | IncrementalRefreshDate | Derived. Numeric version of the `ChargePeriodStart` column to simplify setup for incremental refresh. | +| InvoiceDetailId | Unique identifier of the invoice line item the charge relates to. Introduced in FOCUS 1.4. Not currently populated by Cost Management. | | InvoiceId | Unique identifier for the invoice the charge is included in. Only available for closed months after the invoice is published. Renamed from **x_InvoiceId** in FOCUS 1.2. | | InvoiceIssuerId | Unique identifier of the organization that generated the invoice. | | InvoiceIssuerName | Name of the organization that generated the invoice. Only supported for CSP accounts. Not supported for EA or MCA accounts that a partner manages due to data not being provided by Cost Management. | @@ -142,9 +164,10 @@ Quick links: | PricingSubcategory | Describes the kind of pricing model used for a charge within a specific `PricingCategory`. | | PricingUnit | Derived. Indicates what measurement type is used by the `PricingQuantity`. Extracted from **UnitOfMeasure** in Cost Management. | | PricingUnitDescription | Describes the measurement type used by the `PricingQuantity`, including pricing block size. This is what is used in the price list and on the invoice. Maps to **UnitOfMeasure** in Cost Management. | +| ProviderName | Name of the organization that made the resources or services available for purchase. Removed from the Costs dataset in FOCUS 1.4 (FinOps hubs `v1_4` schema) in favor of `HostProviderName` and `ServiceProviderName`. The `Costs_v1_0()` and `Costs_v1_2()` functions still return this column. Still included in the CommitmentDiscountUsage, Recommendations, and Transactions datasets. | | PublisherCategory | Indicates whether a charge is from a cloud provider or third-party Marketplace vendor. Allowed values: `Cloud Provider` and `Vendor`. Maps to PublisherType in Cost Management. | | PublisherId | Unique identifier for the organization that created the product that got used or purchased. | -| PublisherName | Name of the organization that created the product that got used or purchased. | +| ~PublisherName~ (removed) | Name of the organization that created the product that got used or purchased. Removed from the Costs dataset in FOCUS 1.4 (FinOps hubs `v1_4` schema). Use `ServiceProviderName` instead. The `v1_0` and `v1_2` schema functions still return this column. | | ~Region~ (removed) | Isolated geographic area where a resource is provisioned in and/or a service is provided from. Replaced with `RegionId` and `RegionName` in v0.4 to align with FOCUS 1.0. | | RegionId | Provider-assigned identifier for an isolated geographic area where a resource is provisioned or a service is provided. | | RegionName | Name of an isolated geographic area where a resource is provisioned or a service is provided. | @@ -166,6 +189,7 @@ Quick links: | ServiceCategory | Top-level category for the `ServiceName`. This column aligns with the FOCUS requirements. | | ServiceModel | Indicates what aspects of the service are managed by the provider – infrastructure only, the platform custom code runs on, or the entire service. Derived via the FinOps toolkit Services open data file based on the `x_ResourceType` column when not specified by Cost Management. | | ServiceName | Name of the service the resource type is a part of. This column aligns with the FOCUS requirements. | +| ServiceProviderName | Name of the organization that makes the service available to consumers. The Marketplace publisher for Marketplace charges; otherwise, "Microsoft". Introduced in FOCUS 1.3 as a replacement for `PublisherName`, which was removed in FOCUS 1.4. | | ServiceSubcategory | Secondary classification of the ServiceCategory for a service based on its core function. Derived via the FinOps toolkit Services open data file based on the `x_ResourceType` column when not specified by Cost Management. | | SkuCoreCount | Derived. Indicates the number of virtual CPUs used by this resource. Extracted from `x_SkuDetails`. Used for Azure Hybrid Benefit reports. | | SkuCPUs | See `SkuCoreCount` | diff --git a/docs-mslearn/toolkit/hubs/compatibility.md b/docs-mslearn/toolkit/hubs/compatibility.md index dadbf0e04..17fa1e2a5 100644 --- a/docs-mslearn/toolkit/hubs/compatibility.md +++ b/docs-mslearn/toolkit/hubs/compatibility.md @@ -3,7 +3,7 @@ title: Compatibility guide description: Learn which Power BI report versions are compatible with each FinOps hubs version to ensure seamless upgrades and data integrity. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -31,19 +31,20 @@ The storage path is important for any client that's utilizing the same datasets. The Power BI version refers to the Power BI reports made available within that specific version of the FinOps toolkit. If you customized an older version of the reports, make note of the version you started with, as that is what is useful here. -| Cost Management exports | FinOps hubs | Storage path | Power BI | Notes | -| ----------------------------------------- | ----------- | ---------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -| FOCUS costs, prices, reservation datasets | 12+ | {dataset}/{yyyy}/{MM}/{scope} | 12+² | KQL reports use the FinOps hubs v1_2 schema for FOCUS 1.2 | -| FOCUS costs, prices, reservation datasets | 0.8+ | {dataset}/{yyyy}/{MM}/{scope} | 0.7-0.11² | | -| FOCUS costs, prices, reservation datasets | 0.7 | {dataset}/{yyyy}/{MM}/{scope}¹ | 0.7+² | Storage path updated to new dataset names that support joining multiple related datasets together | -| FOCUS costs, prices, reservation datasets | 0.6 | {export-type}/{yyyy}/{MM}/{scope} | 0.6+ | Reservation recommendations pulled from hub storage | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.6 | {export-type}/{yyyy}/{MM}/{scope}¹ | 0.6+² | Storage path updated; Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.5 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | -| FOCUS 1.0-preview(v1) or 1.0 costs | 0.4 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Supports a mix of FOCUS 1.0 and 1.0-preview(v1) data; Reservation recommendations pulled from the Cost Management connector for Power BI | -| FOCUS 1.0-preview(v1) only | 0.4 | {scope}/{yyyyMM}/{export-type}¹ | 0.2+ | Storage path updated | -| FOCUS 1.0-preview(v1) only | 0.2 - 0.3 | {path}/{yyyyMM}/{export-type} | 0.2+² | Switched to FOCUS data only | -| Actual or Amortized costs (not both) | 0.1 - 0.1.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA and MCA | -| Actual or Amortized costs (not both) | 0.0.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA only | +| Cost Management exports | FinOps hubs | Storage path | Power BI | Notes | +| ----------------------------------------- | ----------- | ---------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| FOCUS costs, prices, reservation datasets | 15+ | {dataset}/{yyyy}/{MM}/{scope} | 15+² | KQL reports use the FinOps hubs v1_4 schema for FOCUS 1.4; Cost Management exports up to FOCUS 1.2, which hubs converts to FOCUS 1.4 during ingestion | +| FOCUS costs, prices, reservation datasets | 12-14 | {dataset}/{yyyy}/{MM}/{scope} | 12+² | KQL reports use the FinOps hubs v1_2 schema for FOCUS 1.2 | +| FOCUS costs, prices, reservation datasets | 0.8+ | {dataset}/{yyyy}/{MM}/{scope} | 0.7-0.11² | | +| FOCUS costs, prices, reservation datasets | 0.7 | {dataset}/{yyyy}/{MM}/{scope}¹ | 0.7+² | Storage path updated to new dataset names that support joining multiple related datasets together | +| FOCUS costs, prices, reservation datasets | 0.6 | {export-type}/{yyyy}/{MM}/{scope} | 0.6+ | Reservation recommendations pulled from hub storage | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.6 | {export-type}/{yyyy}/{MM}/{scope}¹ | 0.6+² | Storage path updated; Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.5 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Reservation recommendations pulled from a separate, non-hub storage URL (or excluded from report) | +| FOCUS 1.0-preview(v1) or 1.0 costs | 0.4 | {scope}/{yyyyMM}/{export-type} | 0.4+² | Supports a mix of FOCUS 1.0 and 1.0-preview(v1) data; Reservation recommendations pulled from the Cost Management connector for Power BI | +| FOCUS 1.0-preview(v1) only | 0.4 | {scope}/{yyyyMM}/{export-type}¹ | 0.2+ | Storage path updated | +| FOCUS 1.0-preview(v1) only | 0.2 - 0.3 | {path}/{yyyyMM}/{export-type} | 0.2+² | Switched to FOCUS data only | +| Actual or Amortized costs (not both) | 0.1 - 0.1.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA and MCA | +| Actual or Amortized costs (not both) | 0.0.1 | {path}/{yyyyMM}/{export-type} | 0.0.1 - 0.1.1 | EA only | ¹ When storage paths update, there's a risk that re-exported data lands in a new place and might cause duplicate data. To resolve, remove the old folders in the **ingestion** container.
diff --git a/docs-mslearn/toolkit/hubs/data-processing.md b/docs-mslearn/toolkit/hubs/data-processing.md index 9c3e1cabb..ceedf55ce 100644 --- a/docs-mslearn/toolkit/hubs/data-processing.md +++ b/docs-mslearn/toolkit/hubs/data-processing.md @@ -3,7 +3,7 @@ title: FinOps hubs data processing description: Learn how FinOps hubs process data, including scope setup, data normalization, and optimization, to enhance cost management and analysis. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/16/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -50,16 +50,16 @@ The following diagram depicts the end-to-end data ingestion process within FinOp 2. The **ingestion_ETL_dataExplorer** pipeline ingests data into the `{dataset}_raw` table in the Data Explorer. - The dataset name is the first folder in the **ingestion** container. - All raw tables are in the **Ingestion** database in Data Explorer. - 3. When data is ingested into raw tables in Data Explorer, an update policy copies the data into the corresponding `{dataset}_final_v1_0` table using the `{dataset}_transform_v1_0()` function to normalize all data to align to FOCUS 1.0. + 3. When data is ingested into raw tables in Data Explorer, an update policy copies the data into the corresponding `{dataset}_final_v1_4` table using the `{dataset}_transform_v1_4()` function to normalize all data to align to FOCUS 1.4. 4. After ingestion, the **ingestion_ETL_dataExplorer** pipeline performs some cleanup, including purging data in the final table that is past the data retention period. - As of 0.7, Data Explorer applies data retention in raw tables while data retention in final tables is applied by the ingestion pipeline. If data ingestion stops, historical data isn't purged. - Data retention can be configured during the template deployment or manually in the **config/settings.json** file in storage. 6. (Optional) The **queries_DailySchedule** trigger runs the **queries_ExecuteETL** pipeline once per day to query Azure Resource Graph for additional recommendations and saves results to the **ingestion/Recommendations** folder. [Learn more](#azure-resource-graph-recommendations). 7. Reports and other tools like Power BI read data from Data Explorer or the **ingestion** container. - Data in Data Explorer can be read from the **Hub** database. - - Use the `{dataset}()` function to use the latest schema. + - Use the `{dataset}()` function to use the latest schema (`v1_4` as of FinOps hubs v15). - This function is useful for quick exploration, but may introduce breaking changes as the FinOps hub instance is updated. - - Use the `{dataset}_v1_0()` function to use the FOCUS 1.0 schema. + - Use the versioned functions, like `{dataset}_v1_0()`, `{dataset}_v1_2()`, or `{dataset}_v1_4()`, to use a specific FOCUS schema version. - Versioned function schemas shouldn't change over time but values may change if the data source changes those values. - Avoid using the **Ingestion** database for queries. While not explicitly prohibited, the **Ingestion** database should be considered an internal area for staging and data preparation. - Data in storage can be read from `ingestion////`. @@ -78,7 +78,7 @@ The following diagram depicts the HubsRecommendations data flow: 1. The **queries_DailySchedule** trigger runs once per day. 2. The **queries_ExecuteETL** pipeline iterates through all query files in the **config/queries** storage folder. 3. The **queries_ETL_ingestion** pipeline executes each query against Azure Resource Graph, deduplicates results, and saves data as parquet in the **ingestion/Recommendations** folder. -4. (If using Azure Data Explorer) Data is ingested into the `Recommendations_raw` table and transformed using the `Recommendations_transform_v1_2()` function. +4. (If using Azure Data Explorer) Data is ingested into the `Recommendations_raw` table and transformed using the `Recommendations_transform_v1_4()` function. HubsRecommendations includes the following query types: @@ -100,7 +100,7 @@ HubsRecommendations includes the following query types: ## About Data Explorer ingestion -When data is ingested into Data Explorer, the `{dataset}_transform_v1_0()` functions apply transform rules in the **Ingestion** database. Each dataset has a different set of transform rules covered in the following sections. +When data is ingested into Data Explorer, the `{dataset}_transform_v1_4()` functions apply transform rules in the **Ingestion** database. Each dataset has a different set of transform rules covered in the following sections. For a list of requested changes, ideas under consideration, and open questions about the underlying Cost Management datasets, see [issue #1111](https://github.com/microsoft/finops-toolkit/issues/1111). Leave comments on that issue if you find opportunities to address any concerns or to voice your support for any of the specific issues. @@ -183,6 +183,54 @@ Transforms: - `x_CommitmentDiscountPercent` - `x_TotalDiscountPercent` - Added `x_SourceValues` to track data changes during ingestion. +- v15+: + - Convert FOCUS 1.0-preview, 1.0, and 1.2 data to FOCUS 1.4. + - Replaced `ProviderName` and `PublisherName` with the FOCUS 1.3 columns `HostProviderName` and `ServiceProviderName`. + - `HostProviderName` falls back to `ProviderName` when not provided. + - `ServiceProviderName` falls back to `PublisherName` (the Marketplace publisher), then `ProviderName`, then "Microsoft" to meet the FOCUS non-null requirement. + - The `{dataset}_v1_0()` and `{dataset}_v1_2()` functions still return `ProviderName` and `PublisherName` by down-converting from the new columns. + - Added the FOCUS 1.3 cost allocation columns: `AllocatedMethodDetails`, `AllocatedMethodId`, `AllocatedResourceId`, `AllocatedResourceName`, `AllocatedTags`, and `ContractApplied`. + - Added the FOCUS 1.4 columns `CommitmentProgramEligibilityDetails`, `InvoiceDetailId`, and 12 `ContractCommitment*` columns. + - These columns are passed through from the export when present and remain empty until Cost Management supports exporting FOCUS 1.4 data. + +### Billing period data transforms + +Supported datasets: + +- FOCUS 1.4 BillingPeriod (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. + +### Contract commitment data transforms + +Supported datasets: + +- FOCUS 1.4 ContractCommitment (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. + +### Invoice detail data transforms + +Supported datasets: + +- FOCUS 1.4 InvoiceDetail (no Cost Management export available yet) + +Transforms: + +- v15+: + - Align columns to FOCUS 1.4. + - Add `x_IngestionTime` to indicate when the row was last updated. + - Add `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion` to identify the original ingested dataset. ### Price data transforms @@ -279,8 +327,11 @@ ingestion/{dataset}/{date-folder-path}/{scope-id-path}/{ingestion-id}__{original - `ingestion` is the container where the data pipeline saves data. - `{dataset}` is the exported dataset type. If ingesting into Azure Data Explorer, the **Ingestion** database must have a matching, case-sensitive "\_raw" table (for example, "Costs_raw"). FinOps hubs support the following datasets in this release: + - **BillingPeriods** - FOCUS 1.4 billing period data. No Cost Management export available yet. - **CommitmentDiscountUsage** - Cost Management reservation details export. + - **ContractCommitments** - FOCUS 1.4 contract commitment data. No Cost Management export available yet. - **Costs** - FOCUS cost and usage data. + - **InvoiceDetails** - FOCUS 1.4 invoice detail data. No Cost Management export available yet. - **Prices** - Cost Management price sheet export. - **Recommendations** - Cost Management reservation recommendations export. - **Transactions** - Cost Management reservation transactions export. diff --git a/docs-mslearn/toolkit/hubs/upgrade.md b/docs-mslearn/toolkit/hubs/upgrade.md index eb7b112d6..c1ace5606 100644 --- a/docs-mslearn/toolkit/hubs/upgrade.md +++ b/docs-mslearn/toolkit/hubs/upgrade.md @@ -3,7 +3,7 @@ title: Upgrade your FinOps hubs description: Learn how to upgrade your existing FinOps hub instance to the latest version, including necessary steps and considerations. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/16/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit @@ -130,11 +130,12 @@ For more information, see [Set up Power BI reports](../power-bi/setup.md). The Data Explorer dashboard was introduced with Data Explorer support in 0.7 and also works with Microsoft Fabric since 0.10. Generally, the dashboard does not need to be updated once deployed unless you want to take advantage of new features. To upgrade the dashboard, replace the existing dashboard with the latest dashboard template. -Each version of the dashboard is configured to work with a specific FinOps hub schema version (v1_0 or v1_2). Schema versions ensure backwards compatibility across FOCUS dataset versions from different providers. Older dashboard versions will continue to work after upgrading to the latest version of FinOps hubs, but newer dashboard versions may not work with older FinOps hub versions. The following table outlines the supported combinations. +Each version of the dashboard is configured to work with a specific FinOps hub schema version (v1_0, v1_2, or v1_4). Schema versions ensure backwards compatibility across FOCUS dataset versions from different providers. Older dashboard versions will continue to work after upgrading to the latest version of FinOps hubs, but newer dashboard versions may not work with older FinOps hub versions. The following table outlines the supported combinations. | Dashboard version | FinOps hubs schema | FinOps hubs version | | ----------------- | ------------------ | ------------------- | -| 12+ | v1_2 | 12+ | +| 15+ | v1_4 | 15+ | +| 12-14 | v1_2 | 12+ | | 0.7-0.11 | v1_0 | 0.7+ | @@ -152,6 +153,8 @@ Replace the use of deprecated columns and functions: | Introduced | Retired | Deprecated | Replacement | | ---------- | ------- | ------------------------------------------- | --------------------------------------------------- | +| 0.7 | 15 | `Costs().ProviderName` | `Costs().HostProviderName` | +| 0.7 | 15 | `Costs().PublisherName` | `Costs().ServiceProviderName` | | 0.7 | 12 | `Costs().x_InvoiceId` | `Costs().InvoiceId` | | 0.7 | 12 | `Costs().x_PricingCurrency` | `Costs().PricingCurrency` | | 0.7 | 12 | `Costs().x_SkuMeterName` | `Costs().SkuMeter` | @@ -166,34 +169,74 @@ If using unversioned functions or updating from the `v1_0` schema version, revie If updating queries to use a newer schema version, use the following table to understand the changes introduced in each schema version for each managed dataset. -| Dataset | Schema | Column | Notes | -| ----------------------- | ------ | ------------------------------------- | ----------------------------------------------- | -| (All) | v1_2 | All `decimal` columns | Changed to `real` | -| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountQuantity` | New custom column | -| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountUnit` | New custom column | -| CommitmentDiscountUsage | v1_2 | `ServiceSubcategory` | New custom column | -| Costs | v1_2 | `CapacityReservationId` | New with FOCUS 1.2 | -| Costs | v1_2 | `CapacityReservationStatus` | New with FOCUS 1.2 | -| Costs | v1_2 | `CommitmentDiscountQuantity` | New with FOCUS 1.2 | -| Costs | v1_2 | `CommitmentDiscountUnit` | New with FOCUS 1.2 | -| Cost | v1_2 | `ServiceSubcategory` | New with FOCUS 1.2 | -| Cost | v1_2 | `SkuPriceDetails` | New with FOCUS 1.2; derived from `x_SkuDetails` | -| Costs | v1_2 | `x_AmortizationClass` | New with Cost Management FOCUS 1.2-preview | -| Costs | v1_2 | `x_CommitmentDiscountNormalizedRatio` | New with Cost Management FOCUS 1.2-preview | -| Costs | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | -| Costs | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | -| Costs | v1_2 | `x_ServiceModel` | New custom column | -| Costs | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | -| Prices | v1_2 | `CommitmentDiscountUnit` | New custom column | -| Prices | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | -| Prices | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | -| Recommendations | v1_2 | `ResourceId` | New custom column | -| Recommendations | v1_2 | `ResourceName` | New custom column | -| Recommendations | v1_2 | `ResourceType` | New custom column | -| Recommendations | v1_2 | `SubAccountName` | New custom column | -| Recommendations | v1_2 | `x_RecommendationDetails` | New custom column | -| Recommendations | v1_2 | `x_ResourceGroupName` | New custom column | -| Transactions | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | +| Dataset | Schema | Column | Notes | +| ----------------------- | ------ | -------------------------------------------- | -------------------------------------------------- | +| (All) | v1_2 | All `decimal` columns | Changed to `real` | +| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountQuantity` | New custom column | +| CommitmentDiscountUsage | v1_2 | `CommitmentDiscountUnit` | New custom column | +| CommitmentDiscountUsage | v1_2 | `ServiceSubcategory` | New custom column | +| Costs | v1_2 | `CapacityReservationId` | New with FOCUS 1.2 | +| Costs | v1_2 | `CapacityReservationStatus` | New with FOCUS 1.2 | +| Costs | v1_2 | `CommitmentDiscountQuantity` | New with FOCUS 1.2 | +| Costs | v1_2 | `CommitmentDiscountUnit` | New with FOCUS 1.2 | +| Costs | v1_2 | `ServiceSubcategory` | New with FOCUS 1.2 | +| Costs | v1_2 | `SkuPriceDetails` | New with FOCUS 1.2; derived from `x_SkuDetails` | +| Costs | v1_2 | `x_AmortizationClass` | New with Cost Management FOCUS 1.2-preview | +| Costs | v1_2 | `x_CommitmentDiscountNormalizedRatio` | New with Cost Management FOCUS 1.2-preview | +| Costs | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | +| Costs | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | +| Costs | v1_2 | `x_ServiceModel` | New custom column | +| Costs | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | +| Costs | v1_4 | `AllocatedMethodDetails` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedMethodId` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedResourceId` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedResourceName` | New with FOCUS 1.3 | +| Costs | v1_4 | `AllocatedTags` | New with FOCUS 1.3 | +| Costs | v1_4 | `CommitmentProgramEligibilityDetails` | New with FOCUS 1.4 | +| Costs | v1_4 | `ContractApplied` | New with FOCUS 1.3 | +| Costs | v1_4 | `ContractCommitmentBenefitCategory` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentCreated` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentDiscountPercentage` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentDurationType` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentFulfillmentInterval` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentLastUpdated` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentLifecycleStatus` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentModel` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentOfferCategory` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentInterval` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentModel` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `ContractCommitmentPaymentUpfrontPercentage` | New with FOCUS 1.4¹ | +| Costs | v1_4 | `HostProviderName` | New with FOCUS 1.3; replaces `ProviderName` | +| Costs | v1_4 | `InvoiceDetailId` | New with FOCUS 1.4 | +| Costs | v1_4 | `ProviderName` | Removed with FOCUS 1.4; use `HostProviderName`² | +| Costs | v1_4 | `PublisherName` | Removed with FOCUS 1.4; use `ServiceProviderName`² | +| Costs | v1_4 | `ServiceProviderName` | New with FOCUS 1.3; replaces `PublisherName` | +| Prices | v1_2 | `CommitmentDiscountUnit` | New custom column | +| Prices | v1_2 | `x_PricingCurrency` | Renamed to `PricingCurrency` | +| Prices | v1_2 | `x_SkuMeterName` | Renamed to `SkuMeter` | +| Recommendations | v1_2 | `ResourceId` | New custom column | +| Recommendations | v1_2 | `ResourceName` | New custom column | +| Recommendations | v1_2 | `ResourceType` | New custom column | +| Recommendations | v1_2 | `SubAccountName` | New custom column | +| Recommendations | v1_2 | `x_RecommendationDetails` | New custom column | +| Recommendations | v1_2 | `x_ResourceGroupName` | New custom column | +| Transactions | v1_2 | `x_InvoiceId` | Renamed to `InvoiceId` | + +¹ The `ContractCommitment*` columns are defined in the FOCUS 1.4 ContractCommitment dataset and included in the Costs dataset to provide per-charge contract commitment context. They remain empty until Cost Management supports exporting FOCUS 1.4 data.
+ +² The `{dataset}_v1_0()` and `{dataset}_v1_2()` functions still return `ProviderName` and `PublisherName` for FOCUS 1.4 data by down-converting from `HostProviderName` and `ServiceProviderName`.
+ +The Prices, Recommendations, Transactions, and CommitmentDiscountUsage datasets didn't change between the `v1_2` and `v1_4` schema versions. + +The `v1_4` schema also introduced three new managed datasets defined by FOCUS 1.4. These datasets remain empty until Cost Management supports exporting FOCUS 1.4 data: + +- **BillingPeriods** includes 6 FOCUS columns: `BillingPeriodCreated`, `BillingPeriodEnd`, `BillingPeriodLastUpdated`, `BillingPeriodStart`, `BillingPeriodStatus`, and `InvoiceIssuerName`. +- **ContractCommitments** includes 30 FOCUS columns: `BillingCurrency`, `ContractCommitmentApplicability`, `ContractCommitmentBenefitCategory`, `ContractCommitmentCategory`, `ContractCommitmentCost`, `ContractCommitmentCreated`, `ContractCommitmentDescription`, `ContractCommitmentDiscountPercentage`, `ContractCommitmentDurationType`, `ContractCommitmentFulfillmentInterval`, `ContractCommitmentId`, `ContractCommitmentLastUpdated`, `ContractCommitmentLifecycleStatus`, `ContractCommitmentModel`, `ContractCommitmentOfferCategory`, `ContractCommitmentPaymentInterval`, `ContractCommitmentPaymentModel`, `ContractCommitmentPaymentUpfrontPercentage`, `ContractCommitmentPeriodEnd`, `ContractCommitmentPeriodStart`, `ContractCommitmentQuantity`, `ContractCommitmentType`, `ContractCommitmentUnit`, `ContractId`, `ContractPeriodEnd`, `ContractPeriodStart`, `InvoiceIssuerName`, `PricingCurrency`, `PricingCurrencyContractCommitmentCost`, and `ServiceProviderName`. +- **InvoiceDetails** includes 22 FOCUS columns: `BilledCost`, `BillingAccountId`, `BillingCurrency`, `BillingPeriodEnd`, `BillingPeriodStart`, `ChargeCategory`, `InvoiceDetailCreated`, `InvoiceDetailDescription`, `InvoiceDetailGrain`, `InvoiceDetailId`, `InvoiceDetailLastUpdated`, `InvoiceId`, `InvoiceIssueDate`, `InvoiceIssueStatus`, `InvoiceIssuerName`, `PaymentCurrency`, `PaymentCurrencyBilledCost`, `PaymentCurrencyInvoiceDetailId`, `PaymentDueDate`, `PaymentTerms`, `PurchaseOrderNumber`, and `ReferenceInvoiceId`. + +Each new dataset also includes 5 hub columns: `x_IngestionTime`, `x_SourceName`, `x_SourceProvider`, `x_SourceType`, and `x_SourceVersion`. + +As of FinOps hubs v15 (schema version `v1_4`), the FOCUS 1.2 transform functions are deprecated and the `v1_2` update policies are disabled. New data is transformed to FOCUS 1.4 and lands in the `{dataset}_final_v1_4` tables. Data ingested under older releases remains in the `{dataset}_final_v1_0` and `{dataset}_final_v1_2` tables and is still available through both versioned functions, like `Costs_v1_2()`, and unversioned functions, which up-convert older data to the latest schema.