[ADF] msexports_ETL_ingestion fails with PathNotFound for reservation recommendation exports

[RolandKrummenacher]

Bug description

The msexports_ETL_ingestion pipeline fails with a PathNotFound error when processing reservation recommendation exports. The pipeline tries to read Parquet files from a path like:

ingestion/Recommendations/<year>/<month>/providers/microsoft.billing/billingaccounts/<id>/billingprofiles/<id>-reservationrecommendations-daily

The path does not exist in the storage account, causing the Get Existing Parquet Files activity to fail with:

ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'.
ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'

This cascades into msexports_ExecuteETL and subsequently ingestion_ExecuteETL / ingestion_ETL_dataExplorer failures.

Steps to reproduce

1. Deploy a FinOps hub with Fabric and recommendations enabled
2. Have a Cost Management export configured for reservation recommendations
3. Wait for the daily trigger to run the msexports_ExecuteETL pipeline

Expected behavior

The pipeline should handle the case where the ingestion path does not yet exist (e.g., first run or no data from the export) gracefully, instead of failing with PathNotFound.

Environment

• FinOps toolkit version: 12 (deployed), 14.0-dev (source)
• Analytics backend: Microsoft Fabric

🤖 Generated with Claude Code

[microsoft-github-policy-service]

Thank you for submitting this issue! We try to respond within 3 business days. In the meantime:

• 📖 Check the troubleshooting guide for common issues
• 💬 Ask questions in Discussions
• 📅 Join our biweekly office hours for live help from the team

[RolandKrummenacher]

Investigation Results

Root Cause

The "Get Existing Parquet Files" GetMetadata activity in the msexports_ETL_ingestion pipeline does not handle the case where the destination folder doesn't exist yet. On first ingestion (or when no prior data exists), the folder path hasn't been created, causing a PathNotFound error.

Affected Code

Primary location: src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep

1. Lines ~1090-1122 — The GetMetadata activity reads from @pipeline().parameters.destinationFolder with retry: 0 and no error handling for missing paths.

2. Lines ~875-880 — Destination folder construction in msexports_ExecuteETL. For Recommendations, the path includes the exportName, adding an extra segment:

   Recommendations/<year>/<month>/<scope>/<exportName>/
   

   Other export types use: <DatasetName>/<year>/<month>/<scope>/

3. Lines ~1123-1147 — The downstream "Filter Out Current Exports" activity uses dependencyConditions: ['Completed'], but the pipeline still fails because the GetMetadata activity error propagates.

Also affected

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep (lines ~1667-1706) has an identical GetMetadata activity with the same vulnerability.

Why Recommendations are most affected

Recommendations exports add an exportName segment to the path hierarchy, making it more likely that the full path doesn't exist on first run. However, any export type could fail if the year/month/scope folder structure hasn't been created yet (e.g., initial deployment).

Possible Fix Approaches

1. Add a path existence check before the GetMetadata call (e.g., a GetMetadata on the parent folder or a Web activity to check the path via the ADLS REST API)
2. Wrap the GetMetadata in an If Condition that handles the "empty folder" case by returning an empty list instead of failing
3. Add error handling to the GetMetadata activity so that a PathNotFound error is treated as "no existing files" (empty result)

🤖 Generated with Claude Code