diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index d66a5e7cf..e168935bc 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 07/07/2026 +ms.date: 07/16/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -49,6 +49,28 @@ _Released June 2026_ - Added Claude Code plugin with skills for FinOps hubs and Azure Cost Management ([#2043](https://github.com/microsoft/finops-toolkit/pull/2043)). - Added 4 agents (CFO, FinOps practitioner, database query, hubs agent), 5 commands (`/ftk-hubs-connect`, `/ftk-hubs-healthCheck`, `/ftk-mom-report`, `/ftk-ytd-report`, `/ftk-cost-optimization`), and an output style. - Linked to the existing KQL query catalog in `src/queries/` from the plugin. +- **Changed** + - Updated plugin skill files to reflect FOCUS 1.4 hub schema, the new `ContractCommitments()`, `BillingPeriods()`, and `InvoiceDetails()` functions, and the removal of `ProviderName` / `PublisherName` ([#2120](https://github.com/microsoft/finops-toolkit/issues/2120)). + +### FinOps hubs v15.0.0 + +- **Added** + - Added full support for FOCUS 1.4 (`v1_4` schema) in Azure Data Explorer and Microsoft Fabric across all managed datasets ([#2120](https://github.com/microsoft/finops-toolkit/issues/2120)). + - When FOCUS 1.0 or 1.2 data is ingested, it's converted to FOCUS 1.4. Historical data remains in the `*_final_v1_0` and `*_final_v1_2` tables. + - New versioned Hub database functions, like `Costs_v1_4()`, were added based on the FOCUS 1.4 schema changes. + - Make sure you use versioned functions, like `Costs_v1_2()`, to avoid breaking changes between FOCUS versions. + - Added new columns to the Costs managed dataset: `AllocatedMethodId`, `AllocatedMethodDetails`, `AllocatedResourceId`, `AllocatedResourceName`, `AllocatedTags`, `ContractApplied`, `ServiceProviderName`, `HostProviderName`, `CommitmentProgramEligibilityDetails`, `InvoiceDetailId`, and 12 `ContractCommitment*` per-row columns (`ContractCommitmentBenefitCategory`, `ContractCommitmentCreated`, `ContractCommitmentDiscountPercentage`, `ContractCommitmentDurationType`, `ContractCommitmentFulfillmentInterval`, `ContractCommitmentLastUpdated`, `ContractCommitmentLifecycleStatus`, `ContractCommitmentModel`, `ContractCommitmentOfferCategory`, `ContractCommitmentPaymentInterval`, `ContractCommitmentPaymentModel`, `ContractCommitmentPaymentUpfrontPercentage`). + - Added three new FOCUS 1.4 datasets: `ContractCommitments` (30 columns, using FOCUS 1.4 specification column IDs like `ContractCommitmentBenefitCategory` and `ContractCommitmentLifecycleStatus`), `BillingPeriods` (6 columns), and `InvoiceDetails` (22 columns), each with tables and versioned and unversioned functions (for example, `ContractCommitments()` and `ContractCommitments_v1_4()`). + - These datasets remain empty until Cost Management supports exporting FOCUS 1.4 data. + - Added a static test harness for the hub database setup scripts that verifies build registration, schema version consistency, and update policy state to catch regressions before release. +- **Changed** + - Retargeted unversioned `Costs()`, `Prices()`, `CommitmentDiscountUsage()`, `Recommendations()`, and `Transactions()` aliases to their `_v1_4` counterparts. + - This is **not** a breaking change if you are following the prescribed guidance of using versioned functions. + - Updated the Data Explorer dashboard to use the `v1_4` schema. +- **Deprecated** + - Deprecated the FOCUS 1.2 transform functions and disabled the `v1_2` update policies. New data is transformed to FOCUS 1.4; data ingested under older versions remains available through versioned functions like `Costs_v1_2()`. +- **Removed** + - Removed deprecated `ProviderName` and `PublisherName` columns from the `v1_4` schema in favor of the FOCUS 1.3 replacements, `HostProviderName` and `ServiceProviderName`. The `v1_0` and `v1_2` functions still return both columns by down-converting from the new columns, and raw tables keep them for back compat. ### [FinOps hubs](hubs/finops-hubs-overview.md) v15 diff --git a/src/powershell/Tests/Unit/HubsFocusSchemas.Tests.ps1 b/src/powershell/Tests/Unit/HubsFocusSchemas.Tests.ps1 new file mode 100644 index 000000000..aed18feac --- /dev/null +++ b/src/powershell/Tests/Unit/HubsFocusSchemas.Tests.ps1 @@ -0,0 +1,527 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + Static regression harness for the FinOps hub FOCUS schema setup scripts (Analytics/scripts/*.kql). + + Covers the four D6 check categories from the FOCUS 1.4 plan, each mapped to a v1_2-era regression: + 1. Registration completeness - every versioned setup script is registered in .build.config (Fabric + bundle) and loaded in Analytics/app.bicep (prevents #1777: v1_2 script missing from Fabric build). + 2. Version-string consistency - versioned files only reference other schema versions through + allowlisted patterns, and no conflict markers or phantom versions exist (prevents stale + cross-version copy/paste references and committed merge conflict markers). + 3. Exactly one enabled update policy version - all enabled update policies live in the current + (highest) schema version; older versions are fully disabled (prevents double ingestion). + 4. Column and conversion contracts - v1_4 tables carry the FOCUS 1.4 columns, hub functions union + every schema version, and the provider column rename round-trips without inversion. + + All checks are static text analysis; no Azure Data Explorer connection is needed. +#> + +Describe 'HubsFocusSchemas' { + + BeforeDiscovery { + $repoRoot = (Resolve-Path "$PSScriptRoot/../../../..").Path + $scriptsPath = Join-Path $repoRoot 'src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts' + + # Discover versioned setup scripts dynamically so new files (e.g., a future v1_5 or another + # size-limit split file) are covered without updating this test. + $ingestionSetupFiles = @(Get-ChildItem -Path $scriptsPath -Filter 'IngestionSetup_v1_*.kql' | ForEach-Object { + @{ Name = $_.Name; FullName = $_.FullName; Version = $(if ($_.Name -match '_v1_(\d+)') { $Matches[1] }) } + }) + $hubSetupFiles = @(Get-ChildItem -Path $scriptsPath -Filter 'HubSetup_v1_*.kql' | ForEach-Object { + @{ Name = $_.Name; FullName = $_.FullName; Version = $(if ($_.Name -match '_v1_(\d+)') { $Matches[1] }) } + }) + $versionedSetupFiles = $ingestionSetupFiles + $hubSetupFiles + $registeredSetupFiles = $versionedSetupFiles + @(Get-ChildItem -Path $scriptsPath -Filter 'HubSetup_Latest.kql' | ForEach-Object { + @{ Name = $_.Name; FullName = $_.FullName; Version = $null } + }) + + # Files that define the FOCUS 1.4 side of the provider column rename (up-conversion). + $v14SetupFiles = @($versionedSetupFiles | Where-Object { $_.Version -eq '4' }) + + # Managed datasets that existed before FOCUS 1.4 (unioned across v1_0, v1_2, and v1_4). + $managedDatasets = @('CommitmentDiscountUsage', 'Costs', 'Prices', 'Recommendations', 'Transactions') + + # Supplemental datasets introduced with FOCUS 1.4 (v1_4 final tables only). + $v14OnlyDatasets = @('BillingPeriods', 'ContractCommitments', 'InvoiceDetails') + + $allDatasets = @($managedDatasets + $v14OnlyDatasets | Sort-Object) + } + + BeforeAll { + $repoRoot = (Resolve-Path "$PSScriptRoot/../../../..").Path + $hubRoot = Join-Path $repoRoot 'src/templates/finops-hub' + $scriptsPath = Join-Path $hubRoot 'modules/Microsoft.FinOpsHubs/Analytics/scripts' + $rawTablesContent = Get-Content -Path (Join-Path $scriptsPath 'IngestionSetup_RawTables.kql') -Raw + + $ingestionFiles = @{ + v1_0 = Get-Content -Path (Join-Path $scriptsPath 'IngestionSetup_v1_0.kql') -Raw + v1_2 = Get-Content -Path (Join-Path $scriptsPath 'IngestionSetup_v1_2.kql') -Raw + # FOCUS 1.4 ingestion setup is split across two files to stay under the Bicep + # loadTextContent() 131 KB limit (D9); combine them for schema-level assertions. + v1_4 = @(Get-ChildItem -Path $scriptsPath -Filter 'IngestionSetup_v1_4*.kql' | ForEach-Object { Get-Content -Path $_.FullName -Raw }) -join "`n" + } + + $hubFiles = @{ + v1_0 = Get-Content -Path (Join-Path $scriptsPath 'HubSetup_v1_0.kql') -Raw + v1_2 = Get-Content -Path (Join-Path $scriptsPath 'HubSetup_v1_2.kql') -Raw + v1_4 = Get-Content -Path (Join-Path $scriptsPath 'HubSetup_v1_4.kql') -Raw + Latest = Get-Content -Path (Join-Path $scriptsPath 'HubSetup_Latest.kql') -Raw + } + + $appBicep = Get-Content -Path (Join-Path $hubRoot 'modules/Microsoft.FinOpsHubs/Analytics/app.bicep') -Raw + $buildConfig = Get-Content -Path (Join-Path $hubRoot '.build.config') -Raw | ConvertFrom-Json + $combineKqlFiles = @($buildConfig.combineKql | ForEach-Object { $_.files }) + + # Allowlist of legitimate cross-version reference patterns. Any other reference to a schema + # version different from the file's own version fails the version-string consistency check. + # Add new entries deliberately - each one is an explicit exception, not a convenience. + $crossVersionAllowlist = @( + @{ + Pattern = "database\('Ingestion'\)\.\w+_final_v1_\d" + Reason = 'Hub setup functions union final tables from every schema version by design.' + } + @{ + Pattern = "docstring\s*=?\s*'DEPRECATED:.*Use \w+_v1_\d\(\) instead" + Reason = 'Deprecation docstrings point to the successor version function.' + } + @{ + Pattern = 'TODO: Remove x_SourceChanges in v1_3 \(or later\)' + Reason = 'Known benign TODO in IngestionSetup_v1_2.kql; v1_3 was skipped as a hub schema version.' + } + ) + + # Returns cross-version reference violations for one file: every identifier ending in _v1_ + # where N is not the file's own version and the line matches no allowlist pattern. + function Get-CrossVersionViolation([string]$FilePath, [string]$OwnVersion) + { + $violations = @() + $lineNumber = 0 + foreach ($line in [System.IO.File]::ReadAllLines($FilePath)) + { + $lineNumber++ + foreach ($token in [regex]::Matches($line, '[A-Za-z0-9]\w*_v1_(\d+)')) + { + if ($token.Groups[1].Value -eq $OwnVersion) { continue } + $isAllowed = $false + foreach ($rule in $crossVersionAllowlist) + { + if ($line -match $rule.Pattern) { $isAllowed = $true; break } + } + if (-not $isAllowed) + { + $violations += "$(Split-Path -Leaf $FilePath):${lineNumber}: '$($token.Value)' -- $($line.Trim())" + } + } + } + return $violations + } + + # Parse every update policy from the ingestion setup scripts: + # ".alter table policy update" followed by a fenced JSON array. + $updatePolicies = @() + foreach ($file in (Get-ChildItem -Path $scriptsPath -Filter 'IngestionSetup_v1_*.kql')) + { + $fileVersion = if ($file.Name -match '_v1_(\d+)') { [int]$Matches[1] } else { $null } + $content = Get-Content -Path $file.FullName -Raw + foreach ($match in [regex]::Matches($content, '(?ms)^\.alter table (?\S+) policy update[ \t]*\r?\n```[ \t]*\r?\n(?.*?)```')) + { + foreach ($policy in ($match.Groups['json'].Value | ConvertFrom-Json)) + { + $updatePolicies += @{ + File = $file.Name + Version = $fileVersion + Target = $match.Groups['target'].Value + Source = $policy.Source + IsEnabled = $policy.IsEnabled + } + } + } + } + + # Derive the current schema version instead of hardcoding it so this test survives v1_5. + $currentSchemaVersion = ($updatePolicies | ForEach-Object { $_.Version } | Measure-Object -Maximum).Maximum + } + + Context 'Registration completeness (D6 check 1)' { + + It 'Discovers versioned setup scripts via glob' { + # Guard against a broken glob silently green-washing the -ForEach tests below. + @(Get-ChildItem -Path $scriptsPath -Filter 'IngestionSetup_v1_*.kql').Count | Should -BeGreaterOrEqual 3 + @(Get-ChildItem -Path $scriptsPath -Filter 'HubSetup_v1_*.kql').Count | Should -BeGreaterOrEqual 3 + } + + It ' is registered in .build.config combineKql' -ForEach $registeredSetupFiles { + $combineKqlFiles | Should -Contain "modules/Microsoft.FinOpsHubs/Analytics/scripts/$Name" -Because "every setup script must ship in the Fabric KQL bundle (regression #1777: IngestionSetup_v1_2.kql was missing from .build.config)" + } + + It ' is loaded via loadTextContent in app.bicep' -ForEach $registeredSetupFiles { + $appBicep | Should -Match ([regex]::Escape("loadTextContent('scripts/$Name')")) -Because "every setup script must be embedded in the Data Explorer deployment" + } + + It 'Lists versioned HubSetup files before HubSetup_Latest.kql in .build.config' { + $hubBundle = @($buildConfig.combineKql | Where-Object { $_.files -match 'HubSetup_Latest\.kql' }) + $hubBundle.Count | Should -Be 1 + $files = @($hubBundle[0].files) + $latestIndex = $files.IndexOf('modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_Latest.kql') + $latestIndex | Should -BeGreaterOrEqual 0 + foreach ($file in ($files -match 'HubSetup_v1_\d+\.kql$')) + { + $files.IndexOf($file) | Should -BeLessThan $latestIndex -Because "unversioned aliases in HubSetup_Latest.kql depend on the versioned functions being created first" + } + } + } + + Context 'Version-string consistency (D6 check 2)' { + + It ' only references other schema versions via allowlisted patterns' -ForEach $versionedSetupFiles { + $violations = Get-CrossVersionViolation -FilePath $FullName -OwnVersion $Version + ($violations -join "`n") | Should -BeNullOrEmpty -Because 'cross-version identifiers outside the allowlist are usually copy/paste defects from cloning the previous version file' + } + + It 'Does not reference v1_3 anywhere in Analytics scripts' { + # v1_3 was never a hub schema version; any reference is a typo or bad copy/paste. + $violations = @() + foreach ($file in (Get-ChildItem -Path $scriptsPath -Filter '*.kql')) + { + $lineNumber = 0 + foreach ($line in [System.IO.File]::ReadAllLines($file.FullName)) + { + $lineNumber++ + if ($line -notmatch 'v1_3') { continue } + $isAllowed = $false + foreach ($rule in $crossVersionAllowlist) + { + if ($line -match $rule.Pattern) { $isAllowed = $true; break } + } + if (-not $isAllowed) + { + $violations += "$($file.Name):${lineNumber}: $($line.Trim())" + } + } + } + ($violations -join "`n") | Should -BeNullOrEmpty + } + + It 'Has no git conflict markers in finops-hub KQL or Bicep files' { + $violations = @() + foreach ($file in (Get-ChildItem -Path $hubRoot -Recurse -Include '*.kql', '*.bicep')) + { + $lineNumber = 0 + foreach ($line in [System.IO.File]::ReadAllLines($file.FullName)) + { + $lineNumber++ + if ($line -match '^(<{7} |={7}$|>{7} )') + { + $violations += "$($file.Name):${lineNumber}: $($line.Trim())" + } + } + } + ($violations -join "`n") | Should -BeNullOrEmpty -Because 'conflict markers have been committed to these files before; they break the ADX database script' + } + } + + Context 'Update policies: exactly one enabled schema version (D6 check 3)' { + + It 'Parses update policies from ' -ForEach $ingestionSetupFiles { + @($updatePolicies | Where-Object { $_.File -eq $Name }).Count | Should -BeGreaterThan 0 -Because 'every ingestion setup script defines update policies; zero parsed policies means the parser regex no longer matches the file format' + } + + It 'Enables update policies in exactly one schema version (the current one)' { + $enabledVersions = @($updatePolicies | Where-Object { $_.IsEnabled } | ForEach-Object { $_.Version } | Sort-Object -Unique) + $enabledVersions | Should -Be @($currentSchemaVersion) -Because 'enabled update policies in more than one schema version cause double ingestion; older versions must be fully disabled when a new version ships' + } + + It ' has zero enabled update policies (superseded version)' -ForEach ($ingestionSetupFiles | Where-Object { [int]$_.Version -lt (@($ingestionSetupFiles | ForEach-Object { [int]$_.Version }) | Measure-Object -Maximum).Maximum }) { + $enabled = @($updatePolicies | Where-Object { $_.File -eq $Name -and $_.IsEnabled }) + ($enabled | ForEach-Object { "$($_.Source) -> $($_.Target)" }) -join "`n" | Should -BeNullOrEmpty + } + + It 'Every raw source table has an enabled update policy in the current schema version' { + # Union of Source tables across all versions = every raw table that ever fed a transform. + # Each must still be wired to the current version (counts are flexible: Costs_raw is fed by + # ActualCosts_raw and AmortizedCosts_raw in addition to its own transform, per C360 support). + $allSources = @($updatePolicies | ForEach-Object { $_.Source } | Sort-Object -Unique) + $enabledSources = @($updatePolicies | Where-Object { $_.IsEnabled -and $_.Version -eq $currentSchemaVersion } | ForEach-Object { $_.Source } | Sort-Object -Unique) + foreach ($source in $allSources) + { + $enabledSources | Should -Contain $source -Because "data ingested into $source would silently stop flowing without an enabled update policy in the current schema version" + } + } + } + + Context 'FOCUS 1.4 columns in Costs_raw' { + + BeforeAll { + # Extract just the Costs_raw alter block (not Costs_final or any other table). + $script:costsRawBlock = if ($rawTablesContent -match '(?ms)\.alter table Costs_raw \(\r?\n(.*?)\r?\n\)') { $Matches[1] } else { '' } + } + + It 'Costs_raw block was extracted' { + $costsRawBlock | Should -Not -BeNullOrEmpty + } + + It 'Adds <_> to Costs_raw' -ForEach @( + 'AllocatedMethodId', 'AllocatedMethodDetails', 'AllocatedResourceId', + 'AllocatedResourceName', 'AllocatedTags', 'ContractApplied', + 'ServiceProviderName', 'HostProviderName', + 'CommitmentProgramEligibilityDetails', 'InvoiceDetailId', + 'ContractCommitmentBenefitCategory', 'ContractCommitmentCreated', + 'ContractCommitmentDiscountPercentage', 'ContractCommitmentDurationType', + 'ContractCommitmentFulfillmentInterval', 'ContractCommitmentLastUpdated', + 'ContractCommitmentLifecycleStatus', 'ContractCommitmentModel', + 'ContractCommitmentOfferCategory', 'ContractCommitmentPaymentInterval', + 'ContractCommitmentPaymentModel', 'ContractCommitmentPaymentUpfrontPercentage' + ) { + $costsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + + It 'Keeps deprecated <_> for back compat' -ForEach @( + 'ProviderName', 'PublisherName', 'Region' + ) { + $costsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + } + + Context 'ContractCommitments_raw exists with all FOCUS 1.4 columns' { + + BeforeAll { + # The Redefine-all-columns alter-table block is the second occurrence; match all and pick it. + $allBlocks = [regex]::Matches($rawTablesContent, '(?ms)\.alter table ContractCommitments_raw \(\r?\n(.*?)\r?\n\)') + $script:contractCommitmentsRawBlock = if ($allBlocks.Count -ge 1) { $allBlocks[$allBlocks.Count - 1].Groups[1].Value } else { '' } + } + + It 'Defines ContractCommitments_raw (plural)' { + $rawTablesContent | Should -Match '\.alter table ContractCommitments_raw \(' + } + + It 'Does NOT define singular ContractCommitment_raw' { + $rawTablesContent | Should -Not -Match '\.alter table ContractCommitment_raw \(' + } + + It 'ContractCommitments_raw column block was extracted' { + $contractCommitmentsRawBlock | Should -Not -BeNullOrEmpty + } + + It 'Includes base column <_>' -ForEach @( + 'BillingCurrency', 'ContractCommitmentCategory', 'ContractCommitmentCost', + 'ContractCommitmentDescription', 'ContractCommitmentId', 'ContractCommitmentPeriodEnd', + 'ContractCommitmentPeriodStart', 'ContractCommitmentQuantity', 'ContractCommitmentType', + 'ContractCommitmentUnit', 'ContractId', 'ContractPeriodEnd', 'ContractPeriodStart', + 'InvoiceIssuerName', 'PricingCurrency' + ) { + $contractCommitmentsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + + It 'Includes FOCUS 1.4 column <_>' -ForEach @( + 'ContractCommitmentApplicability', 'ContractCommitmentBenefitCategory', + 'ContractCommitmentCreated', 'ContractCommitmentDiscountPercentage', + 'ContractCommitmentDurationType', 'ContractCommitmentFulfillmentInterval', + 'ContractCommitmentLastUpdated', 'ContractCommitmentLifecycleStatus', + 'ContractCommitmentModel', 'ContractCommitmentOfferCategory', + 'ContractCommitmentPaymentInterval', 'ContractCommitmentPaymentModel', + 'ContractCommitmentPaymentUpfrontPercentage', 'PricingCurrencyContractCommitmentCost', + 'ServiceProviderName' + ) { + $contractCommitmentsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + + It 'Does NOT include unprefixed column <_>' -ForEach @( + 'BenefitCategory', 'Created', 'DiscountPercentage', 'DurationType', + 'FulfillmentInterval', 'LastUpdated', 'LifecycleStatus', 'Model', 'OfferCategory', + 'PaymentInterval', 'PaymentModel', 'PaymentUpfrontPercentage' + ) { + $contractCommitmentsRawBlock | Should -Not -Match "(?m)^\s+$_\s*:" + } + + It 'Defines exactly 30 FOCUS columns' { + ([regex]::Matches($contractCommitmentsRawBlock, '(?m)^\s+(?!x_)\w+\s*:')).Count | Should -Be 30 + } + } + + Context 'BillingPeriods_raw exists with FOCUS 1.4 columns' { + + BeforeAll { + $allBlocks = [regex]::Matches($rawTablesContent, '(?ms)\.alter table BillingPeriods_raw \(\r?\n(.*?)\r?\n\)') + $script:billingPeriodsRawBlock = if ($allBlocks.Count -ge 1) { $allBlocks[$allBlocks.Count - 1].Groups[1].Value } else { '' } + } + + It 'Defines BillingPeriods_raw' { + $rawTablesContent | Should -Match '\.alter table BillingPeriods_raw \(' + } + + It 'Includes column <_>' -ForEach @( + 'BillingPeriodCreated', 'BillingPeriodEnd', 'BillingPeriodLastUpdated', + 'BillingPeriodStart', 'BillingPeriodStatus', 'InvoiceIssuerName' + ) { + $billingPeriodsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + } + + Context 'InvoiceDetails_raw exists with FOCUS 1.4 columns' { + + BeforeAll { + $allBlocks = [regex]::Matches($rawTablesContent, '(?ms)\.alter table InvoiceDetails_raw \(\r?\n(.*?)\r?\n\)') + $script:invoiceDetailsRawBlock = if ($allBlocks.Count -ge 1) { $allBlocks[$allBlocks.Count - 1].Groups[1].Value } else { '' } + } + + It 'Defines InvoiceDetails_raw' { + $rawTablesContent | Should -Match '\.alter table InvoiceDetails_raw \(' + } + + It 'Includes column <_>' -ForEach @( + 'BilledCost', 'BillingAccountId', 'BillingCurrency', 'BillingPeriodEnd', + 'BillingPeriodStart', 'ChargeCategory', 'InvoiceDetailCreated', 'InvoiceDetailDescription', + 'InvoiceDetailGrain', 'InvoiceDetailId', 'InvoiceDetailLastUpdated', 'InvoiceId', + 'InvoiceIssueDate', 'InvoiceIssueStatus', 'InvoiceIssuerName', 'PaymentCurrency', + 'PaymentCurrencyBilledCost', 'PaymentCurrencyInvoiceDetailId', 'PaymentDueDate', + 'PaymentTerms', 'PurchaseOrderNumber', 'ReferenceInvoiceId' + ) { + $invoiceDetailsRawBlock | Should -Match "(?m)^\s+$_\s*:" + } + } + + Context 'IngestionSetup v1_4 transforms and final tables' { + + BeforeAll { + $script:costsFinalV14Block = if ($ingestionFiles.v1_4 -match '(?ms)\.create-merge table Costs_final_v1_4 \(\r?\n(.*?)\r?\n\)') { $Matches[1] } else { '' } + $script:contractCommitmentsFinalV14Block = if ($ingestionFiles.v1_4 -match '(?ms)\.create-merge table ContractCommitments_final_v1_4 \(\r?\n(.*?)\r?\n\)') { $Matches[1] } else { '' } + } + + It 'Defines <_>_transform_v1_4()' -ForEach @( + 'BillingPeriods', 'ContractCommitments', 'Costs', 'InvoiceDetails', 'Prices' + ) { + $ingestionFiles.v1_4 | Should -Match "$($_)_transform_v1_4\(\)" + } + + It 'Defines <_>_final_v1_4 table' -ForEach @( + 'BillingPeriods', 'ContractCommitments', 'Costs', 'InvoiceDetails', 'Prices' + ) { + $ingestionFiles.v1_4 | Should -Match "\.create-merge table $($_)_final_v1_4" + } + + It 'Does NOT define singular ContractCommitment_final_v1_4' { + $ingestionFiles.v1_4 | Should -Not -Match '\.create-merge table ContractCommitment_final_v1_4' + } + + It 'Costs_final_v1_4 block was extracted' { + $costsFinalV14Block | Should -Not -BeNullOrEmpty + } + + It 'Costs_final_v1_4 does NOT include removed <_>' -ForEach @( + 'ProviderName', 'PublisherName' + ) { + $costsFinalV14Block | Should -Not -Match "(?m)^\s+$_\s*:" + } + + It 'Costs_final_v1_4 includes new FOCUS 1.4 column <_>' -ForEach @( + 'CommitmentProgramEligibilityDetails', 'InvoiceDetailId', + 'ContractCommitmentBenefitCategory', 'ContractCommitmentCreated', + 'ContractCommitmentDiscountPercentage', 'ContractCommitmentDurationType', + 'ContractCommitmentFulfillmentInterval', 'ContractCommitmentLastUpdated', + 'ContractCommitmentLifecycleStatus', 'ContractCommitmentModel', + 'ContractCommitmentOfferCategory', 'ContractCommitmentPaymentInterval', + 'ContractCommitmentPaymentModel', 'ContractCommitmentPaymentUpfrontPercentage' + ) { + $costsFinalV14Block | Should -Match "(?m)^\s+$_\s*:" + } + + It 'ContractCommitments_final_v1_4 includes FOCUS 1.4 column <_>' -ForEach @( + 'ContractCommitmentApplicability', 'ContractCommitmentBenefitCategory', + 'ContractCommitmentCreated', 'ContractCommitmentDescription', + 'ContractCommitmentDiscountPercentage', 'ContractCommitmentDurationType', + 'ContractCommitmentFulfillmentInterval', 'ContractCommitmentLastUpdated', + 'ContractCommitmentLifecycleStatus', 'ContractCommitmentModel', + 'ContractCommitmentOfferCategory', 'ContractCommitmentPaymentInterval', + 'ContractCommitmentPaymentModel', 'ContractCommitmentPaymentUpfrontPercentage', + 'PricingCurrencyContractCommitmentCost', 'ServiceProviderName' + ) { + $contractCommitmentsFinalV14Block | Should -Match "(?m)^\s+$_\s*:" + } + + It 'ContractCommitments_final_v1_4 defines exactly 30 FOCUS columns' { + ([regex]::Matches($contractCommitmentsFinalV14Block, '(?m)^\s+(?!x_)\w+\s*:')).Count | Should -Be 30 + } + } + + Context 'HubSetup_v1_4.kql' { + + It 'Defines <_>_v1_4()' -ForEach $allDatasets { + $hubFiles.v1_4 | Should -Match "$($_)_v1_4\(\)" + } + + It 'Does NOT define singular ContractCommitment_v1_4' { + $hubFiles.v1_4 | Should -Not -Match 'ContractCommitment_v1_4\(\)' + } + + It 'Unions <_> final tables from all three schema versions exactly once each' -ForEach $managedDatasets { + foreach ($version in @('v1_0', 'v1_2', 'v1_4')) + { + [regex]::Matches($hubFiles.v1_4, "database\('Ingestion'\)\.$($_)_final_$version\b").Count | Should -Be 1 -Because "$_ data ingested under the $version schema must surface exactly once in the v1_4 hub function" + } + } + + It 'Reads <_> (new in FOCUS 1.4) from the v1_4 final table only' -ForEach $v14OnlyDatasets { + [regex]::Matches($hubFiles.v1_4, "database\('Ingestion'\)\.$($_)_final_v1_4\b").Count | Should -Be 1 + $hubFiles.v1_4 | Should -Not -Match "$($_)_final_v1_0" + $hubFiles.v1_4 | Should -Not -Match "$($_)_final_v1_2" + } + } + + Context 'HubSetup_v1_0.kql / HubSetup_v1_2.kql back-compat arms' { + + It 'HubSetup_v1_0.kql unions <_>_final_v1_4 (down-converted for old reports)' -ForEach $managedDatasets { + [regex]::Matches($hubFiles.v1_0, "database\('Ingestion'\)\.$($_)_final_v1_4\b").Count | Should -Be 1 -Because "data ingested under the v1_4 schema must remain visible to consumers still on the v1_0 functions" + } + + It 'HubSetup_v1_2.kql unions <_>_final_v1_4 (down-converted for old reports)' -ForEach $managedDatasets { + [regex]::Matches($hubFiles.v1_2, "database\('Ingestion'\)\.$($_)_final_v1_4\b").Count | Should -Be 1 -Because "data ingested under the v1_4 schema must remain visible to consumers still on the v1_2 functions" + } + } + + Context 'Provider column rename round trip (D3)' { + + # FOCUS 1.3 renamed ProviderName -> HostProviderName and repurposed PublisherName -> + # ServiceProviderName. Down-conversion (v1_4 data through v1_0/v1_2 functions) must map back: + # ProviderName = HostProviderName, PublisherName = ServiceProviderName. + It 'HubSetup_.kql down-converts the provider columns in its v1_4 arm' -ForEach @( + @{ Key = 'v1_0' } + @{ Key = 'v1_2' } + ) { + $hubFiles[$Key] | Should -Match 'ProviderName\s*=\s*HostProviderName' -Because 'the deprecated ProviderName maps back from HostProviderName' + $hubFiles[$Key] | Should -Match 'PublisherName\s*=\s*ServiceProviderName' -Because 'the deprecated PublisherName maps back from ServiceProviderName' + } + + # The inverse guard is scoped to DIRECT single assignments (HostProviderName = PublisherName / + # ServiceProviderName = ProviderName) because the up-conversion in IngestionSetup_v1_4.kql and + # HubSetup_v1_4.kql legitimately uses the deprecated ProviderName as a *later fallback* inside + # case()/iff() expressions when deriving ServiceProviderName. A direct assignment is only ever + # the swapped-mapping defect this test exists to catch (PR #2126 review finding C1). + It ' does not invert the provider column mapping' -ForEach $v14SetupFiles { + $content = Get-Content -Path $FullName -Raw + $content | Should -Not -Match '(?() to <_>_v1_4() (latest GA)' -ForEach $allDatasets { + $hubFiles.Latest | Should -Match "(?ms)$($_)\(\)\s*\{\s*$($_)_v1_4\(\)\s*\}" + } + + It 'Does NOT alias to singular ContractCommitment' { + $hubFiles.Latest | Should -Not -Match '(?m)^ContractCommitment\(\)' + } + + It 'Does NOT alias to v1_2 or older' { + $hubFiles.Latest | Should -Not -Match '_v1_2\(\)' + $hubFiles.Latest | Should -Not -Match '_v1_0\(\)' + } + + It 'Points all 8 unversioned functions at _v1_4()' { + [regex]::Matches($hubFiles.Latest, '_v1_4\(\)').Count | Should -Be 8 -Because 'every unversioned dataset function aliases exactly one _v1_4 function' + } + } +}