Wire Foundry IQ defaults into deployment parameters

.gitmodules

@@ -1,6 +1,5 @@
 [submodule "infra"]
 	path = infra
 	url = https://github.com/Azure/bicep-ptn-aiml-landing-zone.git
-	branch = v2.1.4
+	branch = v2.1.5
 	ignore = dirty
-

CHANGELOG.md

@@ -1,5 +1,66 @@
 # Changelog
 
+## [v3.0.8] - 2026-06-30
+
+### User and operator impact
+
+Fresh Zero Trust deployments with Foundry IQ native Blob now provision and retrieve documents end to end. The release consumes AI Landing Zone `v2.1.5`, which fixes the primary application Search service private-link ordering for Foundry/OpenAI/Cognitive Services, and carries the GPT-RAG post-provision fixes that stamp and consume the native Blob settings required for Knowledge Source and Knowledge Base creation.
+
+### Changed
+
+- **AI Landing Zone Bicep module pin bumped to [`v2.1.5`](https://github.com/Azure/bicep-ptn-aiml-landing-zone/releases/tag/v2.1.5):** `manifest.json`, `.gitmodules`, and the `infra` submodule HEAD are aligned on the validated landing-zone release.
+- **Fresh Foundry IQ deployments now carry native Blob settings into infrastructure:** `main.parameters.json` passes `RETRIEVAL_BACKEND=foundry_iq` and the native `FOUNDRY_IQ_*` defaults through azd/Bicep so fresh Zero Trust provisions stamp App Configuration for Foundry IQ `azureBlob` instead of silently falling back to direct Azure AI Search.
+
+### Fixed
+
+- **Fresh Foundry IQ standard extraction setup now seeds required search and AI Services settings:** Post-provision search setup derives `SEARCH_RAG_INDEX_NAME` before rendering Foundry IQ knowledge resources and supplies the `aiServices.uri` / chat model resource URI required by Azure AI Search when `FOUNDRY_IQ_CONTENT_EXTRACTION_MODE=standard`.
+- **Generated Foundry IQ Blob indexers run privately under network isolation:** Search setup enforces `parameters.configuration.executionEnvironment = Private` on service-generated Blob indexers when `NETWORK_ISOLATION=true`.
+- **Regional preflight blocks unsupported Content Understanding regions:** The preflight gate rejects regions where Foundry IQ standard extraction cannot run before provisioning starts.
+
+### Validation
+
+- Full Zero Trust validation passed in Australia East with `NETWORK_ISOLATION=true`, Foundry IQ native Blob, Azure Firewall, Bastion, jumpbox, NAT Gateway, and ACR task agent pool enabled.
+- Provisioning and jumpbox post-provision completed successfully.
+- App Configuration contained `NETWORK_ISOLATION`; the Foundry IQ Knowledge Source and Knowledge Base were created; the generated Blob indexer used private execution.
+- The requested PDF was indexed through the private Blob path, the Knowledge Base index contained documents, and Knowledge Base retrieval returned references from the indexed PDF.
+
+The following component versions are pinned for this release:
+
+| Component | Version |
+| --- | --- |
+| gpt-rag-ui | v2.3.13 |
+| gpt-rag-orchestrator | v3.0.4 |
+| gpt-rag-ingestion | v2.4.14 |
+| bicep-ptn-aiml-landing-zone | v2.1.5 |
+
+## [v3.0.7] - 2026-06-29
+
+### User and operator impact
+
+NL2SQL deployments now consume the orchestrator patch that strengthens the
+read-only enforcement before SQL execution. This umbrella release pins
+`gpt-rag-orchestrator` to `v3.0.4`, which rejects stacked SQL batches and
+non-read-only SQL before datasource access. Operators should still keep every
+SQL Server, Azure SQL, or Fabric SQL datasource principal least-privilege and
+read-only.
+
+### Changed
+
+- **Orchestrator pin bumped to [`v3.0.4`](https://github.com/Azure/gpt-rag-orchestrator/releases/tag/v3.0.4):** consumes the NL2SQL read-only enforcement hardening from [Azure/gpt-rag-orchestrator#256](https://github.com/Azure/gpt-rag-orchestrator/pull/256). UI, ingestion, and AI Landing Zone pins are unchanged from `v3.0.6`.
+
+### Validation
+
+- Release metadata validation passed for `manifest.json` and `CHANGELOG.md`.
+
+The following component versions are pinned for this release:
+
+| Component | Version |
+| --- | --- |
+| gpt-rag-ui | v2.3.13 |
+| gpt-rag-orchestrator | v3.0.4 |
+| gpt-rag-ingestion | v2.4.14 |
+| bicep-ptn-aiml-landing-zone | v2.1.4 |
+
 ## [v3.0.6] - 2026-06-28
 
 ### User and operator impact

README.md

@@ -35,9 +35,12 @@ For unattended provisioning, set `AZURE_SKIP_NETWORK_ISOLATION_WARNING=true` to
 
 ## Retrieval backend configuration
 
-GPT-RAG supports Azure AI Search as the direct retrieval backend and is adding
-Foundry IQ as a first-class backend for v3 deployments. Existing deployments
-should keep `RETRIEVAL_BACKEND=ai_search` until they intentionally migrate.
+GPT-RAG supports Azure AI Search as the direct retrieval backend and Foundry IQ
+as the default backend for fresh v3 deployments. Native Foundry IQ Blob
+Knowledge Sources use `RETRIEVAL_BACKEND=foundry_iq`,
+`FOUNDRY_IQ_PATTERN=azureBlob`, and `FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND=azureBlob`.
+Existing deployments can keep `RETRIEVAL_BACKEND=ai_search` until they
+intentionally migrate.
 
 For the operator guide, including Pattern A vs Pattern B, security modes,
 `knowledgeRetrieval` billing, rollback, and known limitations, see the

config/search/search.j2

@@ -1,3 +1,6 @@
+{% set is_foundry_iq_standard_blob = RETRIEVAL_BACKEND == 'foundry_iq' and FOUNDRY_IQ_PATTERN != 'searchIndex' and FOUNDRY_IQ_CONTENT_EXTRACTION_MODE == 'standard' %}
+{% set content_understanding_chat_models = ['gpt-4.1', 'gpt-4.1-mini', 'gpt-4.1-nano', 'gpt-4o', 'gpt-4o-mini', 'gpt-5.2'] %}
+{% set is_content_understanding_chat_model_supported = GPT_MODEL_INFO and GPT_MODEL_INFO.model_name in content_understanding_chat_models %}
 {
   "indexes": [
     {
@@ -357,7 +360,14 @@
         "ingestionParameters": {
           "identity": null,
           "disableImageVerbalization": false,
-          "chatCompletionModel": null,
+          "chatCompletionModel": {% if is_foundry_iq_standard_blob and is_content_understanding_chat_model_supported %}{
+            "kind": "azureOpenAI",
+            "azureOpenAIParameters": {
+              "resourceUri": "{{FOUNDRY_IQ_AI_SERVICES_ENDPOINT}}",
+              "deploymentId": "{{GPT_MODEL_INFO.deployment_name}}",
+              "modelName": "{{GPT_MODEL_INFO.model_name}}"
+            }
+          }{% else %}null{% endif %},
           "embeddingModel": {% if EMBEDDING_MODEL_INFO %}{
             "kind": "azureOpenAI",
             "azureOpenAIParameters": {
@@ -368,7 +378,10 @@
           }{% else %}null{% endif %},
           "contentExtractionMode": "{{FOUNDRY_IQ_CONTENT_EXTRACTION_MODE}}",
           "ingestionSchedule": null,
-          "ingestionPermissionOptions": {{FOUNDRY_IQ_INGESTION_PERMISSION_OPTIONS | tojson}}
+          "ingestionPermissionOptions": {{FOUNDRY_IQ_INGESTION_PERMISSION_OPTIONS | tojson}}{% if is_foundry_iq_standard_blob %},
+          "aiServices": {
+            "uri": "{{FOUNDRY_IQ_AI_SERVICES_ENDPOINT}}"
+          }{% endif %}
         }
       }
     }

config/search/search.settings.j2

@@ -1,30 +1,40 @@
+{% set search_rag_index_name = SEARCH_RAG_INDEX_NAME | default('ragindex-' ~ RESOURCE_TOKEN, true) %}
+{% set foundry_iq_pattern = FOUNDRY_IQ_PATTERN | default('azureBlob', true) %}
+{% set foundry_iq_ai_services_endpoint = FOUNDRY_IQ_AI_SERVICES_ENDPOINT | default('', true) %}
+{% if not foundry_iq_ai_services_endpoint and AI_FOUNDRY_PROJECT_ENDPOINT is defined and AI_FOUNDRY_PROJECT_ENDPOINT %}
+{% set foundry_iq_ai_services_endpoint = AI_FOUNDRY_PROJECT_ENDPOINT.split('/api/projects/')[0].rstrip('/') ~ '/' %}
+{% endif %}
+{% if not foundry_iq_ai_services_endpoint and AI_FOUNDRY_ACCOUNT_NAME is defined and AI_FOUNDRY_ACCOUNT_NAME %}
+{% set foundry_iq_ai_services_endpoint = 'https://' ~ AI_FOUNDRY_ACCOUNT_NAME ~ '.services.ai.azure.com/' %}
+{% endif %}
 {
     "SEARCH_API_VERSION": "2025-05-01-preview",
     "SEARCH_ANALYZER_NAME": "standard.lucene",
     "EMBEDDINGS_VECTOR_DIMENSIONS": "3072",
-    "SEARCH_RAG_INDEX_NAME": "ragindex-{{RESOURCE_TOKEN}}",
+    "SEARCH_RAG_INDEX_NAME": "{{search_rag_index_name}}",
     "SEARCH_QUERIES_INDEX_NAME": "nl2sql-{{RESOURCE_TOKEN}}-queries",
     "SEARCH_TABLES_INDEX_NAME": "nl2sql-{{RESOURCE_TOKEN}}-tables",
     "SEARCH_MEASURES_INDEX_NAME": "nl2sql-{{RESOURCE_TOKEN}}-measures",
     "ENABLE_AGENTIC_RETRIEVAL": "{{ENABLE_AGENTIC_RETRIEVAL | default('false')}}",
     "RETRIEVAL_BACKEND": "{{RETRIEVAL_BACKEND | default('ai_search')}}",
-    "FOUNDRY_IQ_PATTERN": "{{FOUNDRY_IQ_PATTERN | default('azureBlob')}}",
-    "KNOWLEDGE_BASE_NAME": "{{KNOWLEDGE_BASE_NAME | default(SEARCH_RAG_INDEX_NAME ~ '-rag-kb')}}",
+    "FOUNDRY_IQ_PATTERN": "{{foundry_iq_pattern}}",
+    "KNOWLEDGE_BASE_NAME": "{{KNOWLEDGE_BASE_NAME | default(search_rag_index_name ~ '-rag-kb', true)}}",
     "KNOWLEDGE_BASE_ENDPOINT": "{{KNOWLEDGE_BASE_ENDPOINT | default(SEARCH_SERVICE_QUERY_ENDPOINT)}}",
     "KNOWLEDGE_BASE_CONNECTION_ID": "{{KNOWLEDGE_BASE_CONNECTION_ID | default('')}}",
     "FOUNDRY_IQ_API_VERSION": "{{FOUNDRY_IQ_API_VERSION | default('2026-05-01-preview')}}",
     "FOUNDRY_IQ_KNOWLEDGE_RETRIEVAL_BILLING_PLAN": "{{FOUNDRY_IQ_KNOWLEDGE_RETRIEVAL_BILLING_PLAN | default('free')}}",
-    "FOUNDRY_IQ_KNOWLEDGE_SOURCE_NAME": "{{FOUNDRY_IQ_KNOWLEDGE_SOURCE_NAME | default(SEARCH_RAG_INDEX_NAME ~ ('-rag-ks' if (FOUNDRY_IQ_PATTERN | default('azureBlob')) == 'searchIndex' else '-blob-ks'))}}",
-    "FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND": "{{FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND | default('searchIndex' if (FOUNDRY_IQ_PATTERN | default('azureBlob')) == 'searchIndex' else 'azureBlob')}}",
+    "FOUNDRY_IQ_KNOWLEDGE_SOURCE_NAME": "{{FOUNDRY_IQ_KNOWLEDGE_SOURCE_NAME | default(search_rag_index_name ~ ('-rag-ks' if foundry_iq_pattern == 'searchIndex' else '-blob-ks'), true)}}",
+    "FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND": "{{FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND | default('searchIndex' if foundry_iq_pattern == 'searchIndex' else 'azureBlob', true)}}",
     "FOUNDRY_IQ_STORAGE_CONTAINER_NAME": "{{FOUNDRY_IQ_STORAGE_CONTAINER_NAME | default('documents')}}",
     "FOUNDRY_IQ_STORAGE_FOLDER_PATH": "{{FOUNDRY_IQ_STORAGE_FOLDER_PATH | default('')}}",
     "FOUNDRY_IQ_IS_ADLS_GEN2": "{{FOUNDRY_IQ_IS_ADLS_GEN2 | default('false')}}",
-    "FOUNDRY_IQ_CONTENT_EXTRACTION_MODE": "{{FOUNDRY_IQ_CONTENT_EXTRACTION_MODE | default('minimal')}}",
+    "FOUNDRY_IQ_CONTENT_EXTRACTION_MODE": "{{FOUNDRY_IQ_CONTENT_EXTRACTION_MODE | default('standard')}}",
+    "FOUNDRY_IQ_AI_SERVICES_ENDPOINT": "{{foundry_iq_ai_services_endpoint}}",
     "FOUNDRY_IQ_INGESTION_PERMISSION_OPTIONS": {{FOUNDRY_IQ_INGESTION_PERMISSION_OPTIONS | default(['rbacScope']) | tojson}},
     "FOUNDRY_IQ_FILTER_ADD_ON_ENABLED": "{{FOUNDRY_IQ_FILTER_ADD_ON_ENABLED | default('false')}}",
     "FOUNDRY_IQ_SECURITY_FIELD_NAME": "{{FOUNDRY_IQ_SECURITY_FIELD_NAME | default('metadata_security_id')}}",
     "FOUNDRY_IQ_MAX_OUTPUT_DOCUMENTS": "{{FOUNDRY_IQ_MAX_OUTPUT_DOCUMENTS | default('')}}",
-    {% if GPT_MODEL_INFO %}
+    {% if GPT_MODEL_INFO | default({}) %}
     "GPT_MODEL_DEPLOYMENT_NAME": "{{GPT_MODEL_INFO.deployment_name}}",
     "GPT_MODEL_NAME": "{{GPT_MODEL_INFO.model_name}}"
     {% else %}

config/search/setup.py

@@ -76,6 +76,18 @@ def is_truthy_setting(value: Any) -> bool:
     return str(value).strip().lower() in {"1", "true", "t", "yes", "y"}
 
 
+def strip_odata_metadata(value: Any) -> Any:
+    if isinstance(value, dict):
+        return {
+            key: strip_odata_metadata(item)
+            for key, item in value.items()
+            if not key.startswith("@odata.")
+        }
+    if isinstance(value, list):
+        return [strip_odata_metadata(item) for item in value]
+    return value
+
+
 def normalize_foundry_iq_settings(settings: Dict[str, Any]) -> Dict[str, Any]:
     kind = str(settings.get("FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND") or "").lower()
     is_adls_gen2 = is_truthy_setting(settings.get("FOUNDRY_IQ_IS_ADLS_GEN2"))
@@ -89,6 +101,51 @@ def normalize_foundry_iq_settings(settings: Dict[str, Any]) -> Dict[str, Any]:
     return settings
 
 
+def normalize_endpoint_uri(value: Any) -> str:
+    endpoint = str(value or "").strip()
+    if not endpoint:
+        return ""
+    return endpoint.rstrip("/") + "/"
+
+
+def derive_foundry_iq_ai_services_endpoint(settings: Dict[str, Any]) -> str:
+    endpoint = normalize_endpoint_uri(settings.get("FOUNDRY_IQ_AI_SERVICES_ENDPOINT"))
+    if endpoint:
+        return endpoint
+
+    project_endpoint = str(settings.get("AI_FOUNDRY_PROJECT_ENDPOINT") or "").strip()
+    if "/api/projects/" in project_endpoint:
+        return normalize_endpoint_uri(project_endpoint.split("/api/projects/", 1)[0])
+
+    account_name = str(settings.get("AI_FOUNDRY_ACCOUNT_NAME") or "").strip()
+    if account_name:
+        return f"https://{account_name}.services.ai.azure.com/"
+
+    return ""
+
+
+def is_foundry_iq_standard_blob(settings: Dict[str, Any]) -> bool:
+    retrieval_backend = str(settings.get("RETRIEVAL_BACKEND") or "").lower()
+    pattern = str(settings.get("FOUNDRY_IQ_PATTERN") or "").lower()
+    kind = str(settings.get("FOUNDRY_IQ_KNOWLEDGE_SOURCE_KIND") or "").lower()
+    mode = str(settings.get("FOUNDRY_IQ_CONTENT_EXTRACTION_MODE") or "").lower()
+    return (
+        retrieval_backend == "foundry_iq"
+        and pattern != "searchindex"
+        and kind == "azureblob"
+        and mode == "standard"
+    )
+
+
+def validate_foundry_iq_settings(settings: Dict[str, Any]) -> None:
+    if is_foundry_iq_standard_blob(settings) and not settings.get("FOUNDRY_IQ_AI_SERVICES_ENDPOINT"):
+        raise ValueError(
+            "FOUNDRY_IQ_CONTENT_EXTRACTION_MODE is set to 'standard', but no AI Services endpoint "
+            "could be derived. Set FOUNDRY_IQ_AI_SERVICES_ENDPOINT to the Foundry resource endpoint "
+            "on services.ai.azure.com, for example 'https://<foundry-resource>.services.ai.azure.com/'."
+        )
+
+
 def load_appconfig_settings(ac_client: AzureAppConfigurationClient, label_filter: Optional[str] = None) -> Dict[str, Any]:
     """
     Reads all settings from App Configuration under given label_filter (or None for no label).
@@ -171,10 +228,10 @@ def extract_gpt_model_info(ctx: dict) -> dict:
                     model_obj = model.get("model")
                     if isinstance(model_obj, dict):
                         model_name = model_obj.get("name")
-                        model_format = model_obj.get("format")
+                        model_format = model_obj.get("format") or model.get("modelFormat")
                     else:
                         model_name = model_obj
-                        model_format = None
+                        model_format = model.get("modelFormat")
 
                     gpt_info = {
                         "deployment_name": model.get("name"),
@@ -254,6 +311,15 @@ def render_and_parse_json(template_name_inner: str, ctx: dict) -> Optional[dict]
         if vars_dict:
             vars_dict = normalize_foundry_iq_settings(normalize_json_like_settings(vars_dict))
             context.update(vars_dict)
+            ai_services_endpoint = derive_foundry_iq_ai_services_endpoint(context)
+            if ai_services_endpoint:
+                context["FOUNDRY_IQ_AI_SERVICES_ENDPOINT"] = ai_services_endpoint
+                vars_dict["FOUNDRY_IQ_AI_SERVICES_ENDPOINT"] = ai_services_endpoint
+            try:
+                validate_foundry_iq_settings(context)
+            except ValueError as ve:
+                logging.error(str(ve))
+                return None, context
             for key, val in vars_dict.items():
                 if isinstance(val, (dict, list)):
                     final_val = json.dumps(val)
@@ -416,6 +482,47 @@ def provision_knowledge_sources(defs: dict, context: dict, cred: ChainedTokenCre
     return success_count == len(knowledge_sources)
 
 
+def enforce_private_execution_for_generated_indexers(defs: dict, context: dict, cred: ChainedTokenCredential, search_endpoint: str, api_version: str):
+    if not is_truthy_setting(context.get("NETWORK_ISOLATION")):
+        return
+
+    generated_indexers = []
+    for ks in defs.get("knowledgeSources", []):
+        if ks.get("kind") != "azureBlob" or not ks.get("name"):
+            continue
+        generated_indexers.append(
+            ks.get("azureBlobParameters", {}).get("createdResources", {}).get("indexer")
+            or f"{ks['name']}-indexer"
+        )
+    generated_indexers = [name for name in generated_indexers if name]
+    if not generated_indexers:
+        return
+
+    token = cred.get_token("https://search.azure.com/.default").token
+    headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
+    for indexer_name in generated_indexers:
+        url = f"{search_endpoint}/indexers/{indexer_name}?api-version={api_version}"
+        resp = requests.get(url, headers=headers)
+        if resp.status_code == 404:
+            logging.info(f"ℹ️ Generated indexer '{indexer_name}' not found yet; skipping private execution update.")
+            continue
+        if resp.status_code >= 400:
+            logging.warning(f"❗️ GET indexers/{indexer_name} failed {resp.status_code}: {resp.text}")
+            continue
+
+        body = strip_odata_metadata(resp.json())
+        if not isinstance(body.get("parameters"), dict):
+            body["parameters"] = {}
+        if not isinstance(body["parameters"].get("configuration"), dict):
+            body["parameters"]["configuration"] = {}
+        body["parameters"]["configuration"]["executionEnvironment"] = "Private"
+        update_resp = requests.put(url, headers=headers, json=body)
+        if update_resp.status_code >= 400:
+            logging.warning(f"❗️ PUT indexers/{indexer_name} failed {update_resp.status_code}: {update_resp.text}")
+            continue
+        logging.info(f"✅ Set generated indexer '{indexer_name}' executionEnvironment to Private")
+
+
 def provision_knowledge_bases(defs: dict, context: dict, cred: ChainedTokenCredential, search_endpoint: str):
     """Create or update Foundry IQ knowledge bases.
 
@@ -445,8 +552,7 @@ def provision_knowledge_bases(defs: dict, context: dict, cred: ChainedTokenCrede
 # ── Main Provisioning to AI Search elements (datasources, indexes, skillset and indexers) ─────────────────────
 def execute_setup(defs: Optional[dict], context: dict):
     if defs is None:
-        logging.error("No search definitions to provision. Skipping setup.")
-        return
+        raise RuntimeError("No search definitions were rendered; aborting Azure Search setup")
     cred = ChainedTokenCredential(AzureCliCredential(),ManagedIdentityCredential())
     indexers = defs.get("indexers", [])
     ds_to_indexers = {}
@@ -473,6 +579,7 @@ def execute_setup(defs: Optional[dict], context: dict):
 
     # Step 3: Provision knowledge base resources (KS -> KB)
     knowledge_sources_ok = provision_knowledge_sources(defs, context, cred, search_endpoint)
+    enforce_private_execution_for_generated_indexers(defs, context, cred, search_endpoint, api_version)
     knowledge_bases_ok = provision_knowledge_bases(defs, context, cred, search_endpoint)
     if context.get("RETRIEVAL_BACKEND") == "foundry_iq" and (not knowledge_sources_ok or not knowledge_bases_ok):
         raise RuntimeError("Foundry IQ knowledge source/base provisioning failed")