[Hubs] FinOps hub apps

[flanakin]

📝 Scenario

As an engineer, I need to extend FinOps hubs safely in order to add new or change existing features to meet specific requirements at scale in a way that can be deployed across many hub instances

💎 Solution

Goal

Build an extensibility model in FinOps hubs that allows customers and partners to extend FinOps hubs without risk of upgrades breaking them.

Summary

The FinOps hubs extensibility model embraces traditional infrastructure as code (IaC) patterns and is enabled via bicep (and eventually terraform) modules. Use of the hub modules is not explicitly required but can save time and avoid conflicts in the future as FinOps hubs evolves.

Open issues

1. How can we guarantee that apps maintain the same version of the hub instance?
2. Should events follow .NET standards or simplify a bit?
   1. Microsoft.FinOpsToolkit.Hubs.IngestionManifest.Adding/Added (.NET)
   2. Microsoft.FinOpsToolkit.Hubs.AddIngestionManifest.Before/After (simpler code, more readable)
   3. Microsoft.FinOpsToolkit.Hubs.IngestionManifest.Add.Before/After (simpler code, less readable)

Tasks

• Create reusable modules
  • finops-hub-app to register an app with the hub instance
    • Parameters:
      • hubId string
      • publisher string
      • namespace string
      • appName string
      • displayName string
      • features string array = [ Pipeline, Storage, Vault ]
      • events string array
      • TODO: How do we identify when ADF/storage resources need to be created?
      • TODO: Declare required permissions
    • Create a telemetry deployment for the app
    • Create app manifest in storage: config/apps/{publisher}/{app-name}.json
    • Create storage account, if registered
    • Create Data Factory instance, if registered
      • If storage is used, create an ADF linked service for the storage account
      • TODO: If events are specified, create pipeline to publish events back to the main factory
    • TODO: Grant Data Factory managed identity permissions
    • Outputs:
      • dataFactoryId resource ID
      • storageId resource ID
  • finops-hub-storage to create containers and upload files
    • Parameters:
      • container string
    • Create container
    • Upload files
    • Enforce retention
    • Outputs:
      • resourceId resource ID
      • resourceName resource name
  • finops-hub-event-trigger to create ADF triggers that publish events
    • Parameters:
      • name string
      • storagePath string
      • frequency string (Daily, Monthly)
      • startTime string
      • days number array
      • beforeEvent string
      • afterEvent string
    • Create trigger based on the storagePath
    • Create pipeline that runs the before and after events
  • finops-hub-pipeline to create ADF pipelines that subscribe to events
    • Parameters:
      • name string
      • properties object
      • event string
    • Create pipeline
    • Register the pipeline as an event listener in config/events.json
  • finops-hub-vault to save secrets
    • Parameters:
      • secrets object array (?)
    • Create secrets
    • Outputs:
      • vaultId resource ID
  • finops-hub-database to run database scripts
    • Parameters:
      • name string
      • scripts object (key-value pair of file names/contents)
    • Create the database
    • Create scripts
    • Outputs:
      • clusterUri
      • databaseName
• Extract Microsoft.FinOpsToolkit.Hubs.Core basics from hub.bicep
  • Use finops-hub-app to register the app, track telemetry, and create resources (ADF, Storage)
  • Use finops-hub-storage to create a config container
  • Create/update settings.json
  • Use the finops-hub-container module to create the ingestions container
  • Use the finops-hub-event-trigger module to create the ingestion_ManifestAdded trigger (IngestionManifestAdded event)
• Implement storage retention
  • Update the finops-hub-container module to support daily and monthly data retention
• Add eventing
  • Create the apps_PublishEvent pipeline to run subscribers
  • Create/update events.json to track who's subscribed to which events
• Microsoft.FinOpsToolkit.Hubs.Core managed exports infrastructure
  • Use finops-hub-event-trigger to create the config_SettingsUpdated trigger
  • Use finops-hub-pipeline to create the config_DetectChanges pipeline that identifies scope changes
    • Listens to the Settings.Updating event
    • Triggers the Scopes.Adding/ed and Scopes.Removing/ed events
  • Use the finops-hub-event-trigger module to create the config_DailyDataRefresh trigger (DataRefreshStarted event)
  • Use the finops-hub-event-trigger module to create the config_MonthlyDataRefresh trigger (DataRefreshStarted event)
  • Use the finops-hub-pipeline module to create the config_RunBackfill pipeline
    • Triggers the Backfill.Started event
    • TODO: Should this be the same as DataRefresh.Starting/ed?
    • TODO: Should this use finops-hub-event-trigger with an option for manual triggers?
• Microsoft.FinOpsToolkit.Hubs.RemoteHub
  • Use finops-hub-app to register the remote hub app, track telemetry, and create resources (Key Vault)
    • Depends on Microsoft.FinOpsToolkit.Hubs.Core
  • Use the finops-hub-vault module to store the remote hub URI and key
  • Use the finops-hub-pipeline module to create the ingestion_ETL_remoteHub pipeline
    • Copy individual files to the remote hub ingestion container
    • Copy the manifest to the remote hub ingestion container
  • Use the finops-hub-pipeline module to create the ingestion_PublishToRemoteHub pipeline
    • Listens to the IngestionManifest.Added event
    • Runs the ingestion_ETL_remoteHub pipeline
• Microsoft.FinOpsToolkit.Hubs.Analytics
  • Use finops-hub-app to register the analytics app and track telemetry
    • Depends on Microsoft.FinOpsToolkit.Hubs.Core
  • Use the finops-hub-database module to create the Ingestion database and run scripts
  • Use the finops-hub-database module to create the Hub database and run scripts
  • Use the finops-hub-pipeline module to create the config_InitializeDataExplorer pipeline
    • Listes to the Settings.Updated event
    • Setup the open data tables
    • Update the HubSettingsLog table
  • Use the finops-hub-pipeline module to create the ingestion_ETL_dataExplorer pipeline
    • Same as current implementation
  • Use the finops-hub-pipeline module to create the ingestion_ExecuteETL pipeline
    • Listens to the IngestionManifest.Added event
    • Same as current implementation
    • Runs the ingestion_ETL_dataExplorer pipeline
    • TODO: Consider renaming to be more generic
• Microsoft.CostManagement.Exports / Microsoft.FinOpsToolkit.CostManagement app
  • Use finops-hub-app to register the CM exports app, track telemetry, and create resources (ADF, storage)
    • Depends on Microsoft.FinOpsToolkit.Hubs.Core
  • Use finops-hub-storage to upload schema files to the schemas container
  • Use finops-hub-storage to create a msexports container
  • Use the finops-hub-event-trigger module to create the msexports_ManifestAdded trigger (ExportManifestAdded event)
  • Use the finops-hub-pipeline module to create the msexports_ETL_ingestion pipeline
    • Same as current implementation
  • Use the finops-hub-pipeline module to create the msexports_ExecuteETL pipeline
    • Listens to the ExportManifestAdded event
    • Same as current implementation
    • Runs the ingestion_ETL_dataExplorer pipeline
  • Use the finops-hub-pipeline module to create the config_AddExports pipeline
    • Listens to the Scopes.ScopesAdded event
    • Create new exports
  • Use the finops-hub-pipeline module to create the config_AddExports pipeline
    • Listens to the Scopes.ScopesRemoved event
    • Delete existing exports
  • Use the finops-hub-pipeline module to create the config_RunBackfillJob pipeline
    • Same as current implementation
    • TODO: Consider renaming to dataRefresh_RunBackfillJob
    • TODO: Consider merging with config_RunExportJobs
  • Use the finops-hub-pipeline module to create the config_StartBackfillProcess pipeline
    • Listens to the Microsoft.FinOpsToolkit.Hubs.BackfillStarted event
    • Same as current implementation
    • Runs config_RunBackfillJob
    • TODO: Consider merging with config_StartExportProcess
  • Use the finops-hub-pipeline module to create the config_RunExportJobs pipeline
    • Same as current implementation
    • TODO: Consider renaming to dataRefresh_RunExportJobs
  • Use the finops-hub-pipeline module to create the config_StartExportProcess pipeline
    • Listens to the Microsoft.FinOpsToolkit.Hubs.DataRefreshStarted event
    • Same as current implementation
    • Runs config_RunExportJobs
• Demo IaC setup
  • Use the finops-hub-instance module to enable supported apps:
    • HubCore (implicitly required)
    • RemoteHub
    • Analytics (Data Explorer)
    • CM Exports

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.