Skip to content

Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) #2126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
flanakin wants to merge 7 commits into
base: flanakin/focus14
Choose a base branch
from
Draft

Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) #2126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
77a75b1
Add FOCUS 1.3 ingestion (#2124)
flanakin May 6, 2026
c56b616
Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql)
flanakin May 17, 2026
f8495a5
Address PR #2126 feedback
flanakin May 26, 2026
30f926d
fix: address round-2 review feedback for FOCUS 1.4 ingestion
flanakin Jul 15, 2026
14bf92c
merge: sync flanakin/focus14 (origin/dev) into phase1
flanakin Jul 15, 2026
f24f0d8
fix: port dev v1_2 KQL fixes into FOCUS 1.4 ingestion transforms
flanakin Jul 15, 2026
e7f61f2
fix: align ContractCommitments schema with FOCUS 1.4 spec column IDs
flanakin Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/templates/finops-hub/.build.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_HubInfra.kql",
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_RawTables.kql",
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql",
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql"
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql",
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_4.kql",
"modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_4_Datasets.kql"
]
},
{
Expand Down
2 changes: 2 additions & 0 deletions src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,8 @@ module ingestion_VersionedScripts '../../fx/hub-database.bicep' = if (useAzure)
scripts: {
v1_0: loadTextContent('scripts/IngestionSetup_v1_0.kql')
v1_2: loadTextContent('scripts/IngestionSetup_v1_2.kql')
v1_4: loadTextContent('scripts/IngestionSetup_v1_4.kql')
v1_4_datasets: loadTextContent('scripts/IngestionSetup_v1_4_Datasets.kql')
}
continueOnErrors: continueOnErrors
forceUpdateTag: forceUpdateTag
Expand Down
270 changes: 264 additions & 6 deletions ...es/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_RawTables.kql
View file
Open in desktop
Comment thread
flanakin marked this conversation as resolved.

Large diffs are not rendered by default.

44 changes: 44 additions & 0 deletions ...mplates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -848,6 +848,11 @@ ActualCosts_transform_v1_0()
| join kind=leftouter (Services | where isnotempty(x_ConsumedService) | summarize take_any(ServiceName), take_any(ServiceCategory) by ConsumedService = x_ConsumedService) on ConsumedService
| extend CommitmentDiscountCategory = iff(isnotempty(ReservationId), 'Usage', '') // TODO: CommitmentDiscountCategory needs to handle savings plans
| project
AllocatedMethodDetails = '',
AllocatedMethodId = '',
AllocatedResourceId = '',
AllocatedResourceName = '',
AllocatedTags = '',
AvailabilityZone = AvailabilityZone,
BilledCost = iff(x_AmortizationClass == 'Amortized Charge', real(0), Cost),
BillingAccountId = strcat('/providers/microsoft.billing/billingaccounts/', BillingAccountId, iff(BillingAccountId == BillingProfileId or isempty(BillingProfileId), '', strcat('/billingprofiles/', BillingProfileId))),
Expand Down Expand Up @@ -887,11 +892,27 @@ ActualCosts_transform_v1_0()
),
CommitmentDiscountType = iff(isnotempty(ReservationId), 'Reservation', ''),
CommitmentDiscountUnit = '',
CommitmentProgramEligibilityDetails = '',
ConsumedQuantity = PricingQuantity * x_PricingBlockSize,
ConsumedUnit = PricingUnit,
ContractApplied = '',
ContractCommitmentBenefitCategory = '',
ContractCommitmentCreated = datetime(null),
ContractCommitmentDiscountPercentage = real(null),
ContractCommitmentDurationType = '',
ContractCommitmentFulfillmentInterval = '',
ContractCommitmentLastUpdated = datetime(null),
ContractCommitmentLifecycleStatus = '',
ContractCommitmentModel = '',
ContractCommitmentOfferCategory = '',
ContractCommitmentPaymentInterval = '',
ContractCommitmentPaymentModel = '',
ContractCommitmentPaymentUpfrontPercentage = real(null),
ContractedCost = UnitPrice * PricingQuantity,
ContractedUnitPrice = UnitPrice,
EffectiveCost = iff(x_AmortizationClass == 'Principal', real(0), Cost),
HostProviderName = '',
InvoiceDetailId = '',
InvoiceId = '',
InvoiceIssuerName = 'Microsoft',
ListCost = real(null),
Expand All @@ -910,6 +931,7 @@ ActualCosts_transform_v1_0()
ResourceType,
ServiceCategory,
ServiceName,
ServiceProviderName = '',
ServiceSubcategory = '',
SkuId = '',
SkuMeter = '',
Expand Down Expand Up @@ -1070,6 +1092,11 @@ AmortizedCosts_transform_v1_0()
| join kind=leftouter (Services | where isnotempty(x_ConsumedService) | summarize take_any(ServiceName), take_any(ServiceCategory) by ConsumedService = x_ConsumedService) on ConsumedService
| extend CommitmentDiscountCategory = iff(isnotempty(ReservationId), 'Usage', '') // TODO: CommitmentDiscountCategory needs to handle savings plans
| project
AllocatedMethodDetails = '',
AllocatedMethodId = '',
AllocatedResourceId = '',
AllocatedResourceName = '',
AllocatedTags = '',
AvailabilityZone = AvailabilityZone,
BilledCost = iff(x_AmortizationClass == 'Amortized Charge', real(0), Cost),
BillingAccountId = strcat('/providers/microsoft.billing/billingaccounts/', BillingAccountId, iff(BillingAccountId == BillingProfileId or isempty(BillingProfileId), '', strcat('/billingprofiles/', BillingProfileId))),
Expand Down Expand Up @@ -1109,11 +1136,27 @@ AmortizedCosts_transform_v1_0()
),
CommitmentDiscountType = iff(isnotempty(ReservationId), 'Reservation', ''),
CommitmentDiscountUnit = '',
CommitmentProgramEligibilityDetails = '',
ConsumedQuantity = PricingQuantity * x_PricingBlockSize,
ConsumedUnit = PricingUnit,
ContractApplied = '',
ContractCommitmentBenefitCategory = '',
ContractCommitmentCreated = datetime(null),
ContractCommitmentDiscountPercentage = real(null),
ContractCommitmentDurationType = '',
ContractCommitmentFulfillmentInterval = '',
ContractCommitmentLastUpdated = datetime(null),
ContractCommitmentLifecycleStatus = '',
ContractCommitmentModel = '',
ContractCommitmentOfferCategory = '',
ContractCommitmentPaymentInterval = '',
ContractCommitmentPaymentModel = '',
ContractCommitmentPaymentUpfrontPercentage = real(null),
ContractedCost = UnitPrice * PricingQuantity,
ContractedUnitPrice = UnitPrice,
EffectiveCost = iff(x_AmortizationClass == 'Principal', real(0), Cost),
HostProviderName = '',
InvoiceDetailId = '',
InvoiceId = '',
InvoiceIssuerName = 'Microsoft',
ListCost = real(null),
Expand All @@ -1132,6 +1175,7 @@ AmortizedCosts_transform_v1_0()
ResourceType,
ServiceCategory,
ServiceName,
ServiceProviderName = '',
ServiceSubcategory = '',
SkuId = '',
SkuMeter = '',
Expand Down
72 changes: 58 additions & 14 deletions ...mplates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

// Prices_transform_v1_2 function
.create-or-alter function
with (docstring='Transforms Prices_raw into FOCUS 1.2.', folder='Prices')
with (docstring='DEPRECATED: Transforms Prices_raw into FOCUS 1.2. Use Prices_transform_v1_4() instead.', folder='Prices')
Prices_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -259,7 +259,7 @@ Prices_transform_v1_2()
.alter table Prices_final_v1_2 policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "Prices_raw",
"Query": "Prices_transform_v1_2()",
"IsTransactional": true,
Expand Down Expand Up @@ -288,7 +288,7 @@ Prices_transform_v1_2()

// Costs_transform_v1_2 function
.create-or-alter function
with (docstring='All costs transformed to FOCUS 1.2.', folder='Costs')
with (docstring='DEPRECATED: All costs transformed to FOCUS 1.2. Use Costs_transform_v1_4() instead.', folder='Costs')
Costs_transform_v1_2()
{
let checkString = (column: string, oldValue: string, newValue: string) {
Expand Down Expand Up @@ -1077,7 +1077,7 @@ Costs_transform_v1_2()
.alter table Costs_final_v1_2 policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "Costs_raw",
"Query": "Costs_transform_v1_2()",
"IsTransactional": true,
Expand All @@ -1093,7 +1093,7 @@ Costs_transform_v1_2()

// ActualCosts_transform_v1_2 function
.create-or-alter function
with (docstring='ActualCost exports transformed to FOCUS 1.2.', folder='Costs')
with (docstring='DEPRECATED: ActualCost exports transformed to FOCUS 1.2. Use ActualCosts_transform_v1_4() instead.', folder='Costs')
ActualCosts_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -1128,6 +1128,11 @@ ActualCosts_transform_v1_2()
| join kind=leftouter (Services | where isnotempty(x_ConsumedService) | summarize take_any(ServiceName), take_any(ServiceCategory), take_any(ServiceSubcategory), take_any(x_ServiceModel) by ConsumedService = x_ConsumedService) on ConsumedService
| extend CommitmentDiscountCategory = iff(isnotempty(ReservationId), 'Usage', '') // TODO: CommitmentDiscountCategory needs to handle savings plans
| project
AllocatedMethodDetails = '',
AllocatedMethodId = '',
AllocatedResourceId = '',
AllocatedResourceName = '',
AllocatedTags = '',
AvailabilityZone = AvailabilityZone,
BilledCost = iff(x_AmortizationClass == 'Amortized Charge', real(0), Cost),
BillingAccountId = strcat('/providers/microsoft.billing/billingaccounts/', BillingAccountId, iff(BillingAccountId == BillingProfileId or isempty(BillingProfileId), '', strcat('/billingprofiles/', BillingProfileId))),
Expand Down Expand Up @@ -1167,11 +1172,27 @@ ActualCosts_transform_v1_2()
),
CommitmentDiscountType = iff(isnotempty(ReservationId), 'Reservation', ''),
CommitmentDiscountUnit = '',
CommitmentProgramEligibilityDetails = '',
ConsumedQuantity = PricingQuantity * x_PricingBlockSize,
ConsumedUnit = PricingUnit,
ContractApplied = '',
ContractCommitmentBenefitCategory = '',
ContractCommitmentCreated = datetime(null),
ContractCommitmentDiscountPercentage = real(null),
ContractCommitmentDurationType = '',
ContractCommitmentFulfillmentInterval = '',
ContractCommitmentLastUpdated = datetime(null),
ContractCommitmentLifecycleStatus = '',
ContractCommitmentModel = '',
ContractCommitmentOfferCategory = '',
ContractCommitmentPaymentInterval = '',
ContractCommitmentPaymentModel = '',
ContractCommitmentPaymentUpfrontPercentage = real(null),
ContractedCost = UnitPrice * PricingQuantity,
ContractedUnitPrice = UnitPrice,
EffectiveCost = iff(x_AmortizationClass == 'Principal', real(0), Cost),
HostProviderName = '',
InvoiceDetailId = '',
InvoiceId = '',
InvoiceIssuerName = 'Microsoft',
ListCost = real(null),
Expand All @@ -1190,6 +1211,7 @@ ActualCosts_transform_v1_2()
ResourceType,
ServiceCategory,
ServiceName,
ServiceProviderName = '',
ServiceSubcategory,
SkuId = '',
SkuMeter = MeterName,
Expand Down Expand Up @@ -1299,7 +1321,7 @@ ActualCosts_transform_v1_2()
.alter table Costs_raw policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "ActualCosts_raw",
"Query": "ActualCosts_transform_v1_2()",
"IsTransactional": true,
Expand All @@ -1315,7 +1337,7 @@ ActualCosts_transform_v1_2()

// AmortizedCosts_transform_v1_2 function
.create-or-alter function
with (docstring='AmortizedCost exports transformed to FOCUS 1.2.', folder='Costs')
with (docstring='DEPRECATED: AmortizedCost exports transformed to FOCUS 1.2. Use AmortizedCosts_transform_v1_4() instead.', folder='Costs')
AmortizedCosts_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -1349,6 +1371,11 @@ AmortizedCosts_transform_v1_2()
| join kind=leftouter (Services | where isnotempty(x_ConsumedService) | summarize take_any(ServiceName), take_any(ServiceCategory), take_any(ServiceSubcategory), take_any(x_ServiceModel) by ConsumedService = x_ConsumedService) on ConsumedService
| extend CommitmentDiscountCategory = iff(isnotempty(ReservationId), 'Usage', '') // TODO: CommitmentDiscountCategory needs to handle savings plans
| project
AllocatedMethodDetails = '',
AllocatedMethodId = '',
AllocatedResourceId = '',
AllocatedResourceName = '',
AllocatedTags = '',
AvailabilityZone = AvailabilityZone,
BilledCost = iff(x_AmortizationClass == 'Amortized Charge', real(0), Cost),
BillingAccountId = strcat('/providers/microsoft.billing/billingaccounts/', BillingAccountId, iff(BillingAccountId == BillingProfileId or isempty(BillingProfileId), '', strcat('/billingprofiles/', BillingProfileId))),
Expand Down Expand Up @@ -1388,11 +1415,27 @@ AmortizedCosts_transform_v1_2()
),
CommitmentDiscountType = iff(isnotempty(ReservationId), 'Reservation', ''),
CommitmentDiscountUnit = '',
CommitmentProgramEligibilityDetails = '',
ConsumedQuantity = PricingQuantity * x_PricingBlockSize,
ConsumedUnit = PricingUnit,
ContractApplied = '',
ContractCommitmentBenefitCategory = '',
ContractCommitmentCreated = datetime(null),
ContractCommitmentDiscountPercentage = real(null),
ContractCommitmentDurationType = '',
ContractCommitmentFulfillmentInterval = '',
ContractCommitmentLastUpdated = datetime(null),
ContractCommitmentLifecycleStatus = '',
ContractCommitmentModel = '',
ContractCommitmentOfferCategory = '',
ContractCommitmentPaymentInterval = '',
ContractCommitmentPaymentModel = '',
ContractCommitmentPaymentUpfrontPercentage = real(null),
ContractedCost = UnitPrice * PricingQuantity,
ContractedUnitPrice = UnitPrice,
EffectiveCost = iff(x_AmortizationClass == 'Principal', real(0), Cost),
HostProviderName = '',
InvoiceDetailId = '',
InvoiceId = '',
InvoiceIssuerName = 'Microsoft',
ListCost = real(null),
Expand All @@ -1411,6 +1454,7 @@ AmortizedCosts_transform_v1_2()
ResourceType,
ServiceCategory,
ServiceName,
ServiceProviderName = '',
ServiceSubcategory,
SkuId = '',
SkuMeter = MeterName,
Expand Down Expand Up @@ -1520,7 +1564,7 @@ AmortizedCosts_transform_v1_2()
.alter table Costs_raw policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "AmortizedCosts_raw",
"Query": "AmortizedCosts_transform_v1_2()",
"IsTransactional": true,
Expand All @@ -1537,7 +1581,7 @@ AmortizedCosts_transform_v1_2()

// CommitmentDiscountUsage_transform_v1_2 function
.create-or-alter function
with (docstring='All commitment discount usage transformed to FOCUS 1.2. This includes reservationdeatils_raw.', folder='Commitment discounts')
with (docstring='DEPRECATED: All commitment discount usage transformed to FOCUS 1.2. This includes reservationdeatils_raw. Use CommitmentDiscountUsage_transform_v1_4() instead.', folder='Commitment discounts')
CommitmentDiscountUsage_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -1639,7 +1683,7 @@ CommitmentDiscountUsage_transform_v1_2()
.alter table CommitmentDiscountUsage_final_v1_2 policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "CommitmentDiscountUsage_raw",
"Query": "CommitmentDiscountUsage_transform_v1_2()",
"IsTransactional": true,
Expand All @@ -1656,7 +1700,7 @@ CommitmentDiscountUsage_transform_v1_2()

// Recommendations_transform_v1_2 function
.create-or-alter function
with (docstring='All recommendations transformed to FOCUS 1.2.', folder='Recommendations')
with (docstring='DEPRECATED: All recommendations transformed to FOCUS 1.2. Use Recommendations_transform_v1_4() instead.', folder='Recommendations')
Recommendations_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -1790,7 +1834,7 @@ Recommendations_transform_v1_2()
.alter table Recommendations_final_v1_2 policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "Recommendations_raw",
"Query": "Recommendations_transform_v1_2()",
"IsTransactional": true,
Expand All @@ -1807,7 +1851,7 @@ Recommendations_transform_v1_2()

// Transactions_transform_v1_2 function
.create-or-alter function
with (docstring='All transactions transformed to FOCUS 1.2.', folder='Transactions')
with (docstring='DEPRECATED: All transactions transformed to FOCUS 1.2. Use Transactions_transform_v1_4() instead.', folder='Transactions')
Transactions_transform_v1_2()
{
// NOTE: All open issues and questions are tracked @ https://github.com/microsoft/finops-toolkit/issues/1111
Expand Down Expand Up @@ -1942,7 +1986,7 @@ Transactions_transform_v1_2()
.alter table Transactions_final_v1_2 policy update
```
[{
"IsEnabled": true,
"IsEnabled": false,
"Source": "Transactions_raw",
"Query": "Transactions_transform_v1_2()",
"IsTransactional": true,
Expand Down
Loading