[Evaluation] Fix AOAI evaluation to preserve list values instead of stringifying them

[aprilk-ms]

Fixes

1. Preserve list values instead of stringifying them in AOAI evaluation

Problem: The _convert_value helper in _get_data_source converted list values to strings via str(), turning [] into the string '[]'. The AOAI API rejected these with:

Error validating file against schema: '[]' is not of type 'array'

Fix: Moved list from the stringify branch to the pass-through branch alongside dict, since both are structured JSON types that should be preserved as native objects for proper JSON serialization.

2. Infer correct schema types for list/dict columns

Problem: Schema generation defaulted all columns/leaves to "string" type in both flat and nested modes, causing schema-data mismatches when data contained lists or dicts.

Fix:

• Flat mode: Samples rows (skipping None, NaN, pd.NA, NaT, etc.) to emit the correct JSON Schema type (array, object, or string).
• Nested mode: Added leaf_type_map parameter to _build_schema_tree_from_paths so per-leaf types can be inferred from the DataFrame, overriding the default force_leaf_type="string" for paths whose data contains lists or dicts.

3. Fix pass_threshold propagation and zero-threshold logging

Problem: Three related issues with threshold handling:

• _build_internal_log_attributes used a truthiness check (if event_data.get("threshold")) which silently dropped threshold=0.
• _pass_threshold from evaluator config was not propagated into testing criteria metadata.
• PythonGrader/code evaluators that don't emit a threshold column had no way to get one injected from config.

Fix:

• Use is not None check so threshold=0 is correctly logged.
• Propagate _pass_threshold from evaluator_config into testing_criteria_metadata.
• Inject pass_threshold into metric results when the evaluator doesn't emit one, using _is_none_or_nan to also handle NaN threshold values, without overwriting valid evaluator-provided thresholds.

Tests

• test_aoai_data_source.py (33 tests pass): Added tests for list/dict value preservation, flat schema type inference, None/NaN/pd.NA null-skipping, nested schema leaf type inference, and leaf_type_map in _build_schema_tree_from_paths; updated 3 existing assertions; added integration test for schema-data alignment.
• test_evaluate.py (13 new tests pass): TestBuildInternalLogAttributesThreshold (5 tests), TestExtractTestingCriteriaMetadataPassThreshold (3 tests), TestProcessCriteriaMetricsThresholdInjection (5 tests including NaN injection).

[Copilot]

Pull request overview

Fixes Azure OpenAI (AOAI) evaluation payload/schema mismatches by preserving structured JSON values (lists/dicts) in the datasource payload, inferring correct JSON Schema types in flat schema mode, and improving pass_threshold/threshold propagation and logging.

Changes:

• Preserve list and dict values in AOAI datasource JSON (stop stringifying them).
• Infer "array" / "object" types for list/dict columns when generating flat datasource schemas (skipping null-like samples).
• Ensure threshold=0 is logged, propagate _pass_threshold into testing-criteria metadata, and inject missing thresholds from config into metric results.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate_aoai.py	Updates AOAI datasource value conversion and flat schema type inference for list/dict columns.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py	Fixes threshold logging for zero, propagates pass_threshold metadata, and injects thresholds into results when missing.
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_aoai_data_source.py	Adds/updates unit tests validating list/dict preservation and flat schema inference behavior.
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_evaluate.py	Adds unit tests for threshold logging, pass_threshold propagation, and threshold injection behavior.

---

You can also share your feedback on Copilot code review. Take the survey.