fix: skip resources with empty IDs from conditional env vars in config processing

[Elbehery]

What does this PR do?

This PR fixes config processing to skip registered resources when their ID fields resolve to empty/None from conditional environment variable syntax (e.g., ${env.VAR:+value}).

Changes

• Added RESOURCE_ID_FIELDS constant listing all resource ID fields: model_id, shield_id, dataset_id, scoring_fn_id, benchmark_id, toolgroup_id
• Modified replace_env_vars() to check if any resource ID field resolves to empty/None and skip the entire resource entry early during config processing
• Added unit tests for the new functionality

Why is this needed?

When using conditional env var syntax like ${env.ENABLE_BENCHMARK:+my-benchmark}, if the env var is not set, the benchmark_id (or other resource ID) resolves to empty/None. Previously this would cause validation errors. Now these entries are gracefully skipped during config processing.

Fixes #4453

[Elbehery]

cc @leseb @saichandrapandraju

[leseb]

need to backport in 0.4.x branch

[Elbehery]

sure, shall we merge this first ?

[leseb]

the fix needs to be in replace_env_vars not after the processing, also we need to not only handle benchmarks but other registerable ressources. this diff worked for me

diff --git a/src/llama_stack/core/stack.py b/src/llama_stack/core/stack.py
index 3ea2e8996..d573fba64 100644
--- a/src/llama_stack/core/stack.py
+++ b/src/llama_stack/core/stack.py
@@ -110,6 +110,10 @@ REGISTRY_REFRESH_INTERVAL_SECONDS = 300
 REGISTRY_REFRESH_TASK = None
 TEST_RECORDING_CONTEXT = None

+# ID fields for registered resources that should trigger skipping
+# when they resolve to empty/None (from conditional env vars like :+)
+RESOURCE_ID_FIELDS = ["model_id", "shield_id", "dataset_id", "scoring_fn_id", "benchmark_id", "toolgroup_id"]
+

 def is_request_model(t: Any) -> bool:
     """Check if a type is a request model (Pydantic BaseModel).
@@ -346,15 +350,31 @@ def replace_env_vars(config: Any, path: str = "") -> Any:
                             logger.debug(
                                 f"Skipping config env variable expansion for disabled provider: {v.get('provider_id', '')}"
                             )
-                            # Create a copy with resolved provider_id but original config
-                            disabled_provider = v.copy()
-                            disabled_provider["provider_id"] = resolved_provider_id
                             continue
                     except EnvVarError:
                         # If we can't resolve the provider_id, continue with normal processing
                         pass

-                # Normal processing for non-disabled providers
+                # Special handling for registered resources: check if ID field resolves to empty/None
+                # from conditional env vars (e.g., ${env.VAR:+value}) and skip the entry if so
+                if isinstance(v, dict):
+                    should_skip = False
+                    for id_field in RESOURCE_ID_FIELDS:
+                        if id_field in v:
+                            try:
+                                resolved_id = replace_env_vars(v[id_field], f"{path}[{i}].{id_field}")
+                                if resolved_id is None or resolved_id == "":
+                                    logger.debug(
+                                        f"Skipping resource with empty {id_field} (conditional env var not set)"
+                                    )
+                                    should_skip = True
+                                    break
+                            except EnvVarError:
+                                pass
+                    if should_skip:
+                        continue
+
+                # Normal processing
                 result.append(replace_env_vars(v, f"{path}[{i}]"))
             except EnvVarError as e:
                 raise EnvVarError(e.var_name, e.path) from None

[leseb]

please revise your commit message and PR title/content

[Elbehery]

updated

[skamenan7]

Please log without swallowing the exception :)

[skamenan7]

this log will not be visible in default log level of info. consider info log level so users know about this skip.

[skamenan7]

Also, consider adding resource type or path information in the log to know if that id is model, shield or dataset.

[skamenan7]

Is there a test when ID field uses ${env.VAR} syntax (without := or :+ operators) and the env var is not set. Feels like it is missing.