Python: Foundry Evals integration for Python

[alliscode]

Add evaluation framework with local and Foundry-hosted evaluator support:

• EvalItem/EvalResult core types with conversation splitting strategies
• @evaluator decorator for defining custom evaluation functions
• LocalEvaluator for running evaluations locally
• FoundryEvals provider for Azure AI Foundry hosted evaluations
• evaluate_agent() orchestration with expected values support
• evaluate_workflow() for multi-agent workflow evaluation
• Comprehensive test suite and evaluation samples

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/azure-ai/agent_framework_azure_ai
_foundry_evals.py	251	3	98%	452, 457, 672
packages/core/agent_framework
_evaluation.py	645	77	88%	228, 243, 460, 462, 570, 573, 652–654, 659, 696–699, 755–756, 759, 765–767, 771, 804–806, 860, 895, 907–909, 914, 938–943, 1034, 1112–1113, 1115–1119, 1125, 1163, 1508, 1510, 1518, 1528, 1532, 1577, 1595–1596, 1628–1631, 1707, 1713, 1728, 1732–1734, 1764, 1770–1774, 1808, 1856–1857, 1859, 1884–1885, 1890
TOTAL	28908	3471	87%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
5660	20 💤	0 ❌	0 🔥	1m 26s ⏱️

[Copilot]

Pull request overview

This PR adds a provider-agnostic evaluation framework to the Python Agent Framework, with both local (no-API) evaluators and an Azure AI Foundry-backed provider, plus end-to-end samples that demonstrate agent and workflow evaluation patterns.

Changes:

• Introduces core evaluation types and orchestration (EvalItem, EvalResults, evaluate_agent(), evaluate_workflow()) plus local checks (LocalEvaluator, @evaluator).
• Adds Azure AI Foundry provider integration (FoundryEvals) and trace/target evaluation helpers.
• Adds/updates evaluation samples (Foundry evals patterns + self-reflection groundedness) and expands test coverage for local evaluation.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
python/samples/05-end-to-end/evaluation/self_reflection/self_reflection.py	Migrates groundedness scoring to FoundryEvals and updates batch runner.
python/samples/05-end-to-end/evaluation/self_reflection/README.md	Updates self-reflection sample documentation for Foundry Evals usage and env vars.
python/samples/05-end-to-end/evaluation/self_reflection/.env.example	Updates env var example to FOUNDRY_PROJECT_ENDPOINT.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_workflow_sample.py	New sample: evaluate multi-agent workflows with Foundry evaluators.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py	New sample: evaluate existing responses / traces via Foundry.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_multiturn_sample.py	New sample: demonstrate conversation split strategies for eval.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_mixed_sample.py	New sample: mix LocalEvaluator with Foundry evaluators in one call.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_all_patterns_sample.py	New “kitchen sink” sample covering all evaluation patterns.
python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py	New sample: evaluate_agent patterns + direct FoundryEvals.evaluate().
python/samples/05-end-to-end/evaluation/foundry_evals/README.md	New README describing Foundry eval samples and entry points.
python/samples/05-end-to-end/evaluation/foundry_evals/.env.example	New env example for Foundry eval samples.
python/samples/03-workflows/evaluation/evaluate_workflow.py	New workflow evaluation sample using local checks.
python/samples/02-agents/evaluation/evaluate_with_expected.py	New sample demonstrating expected outputs/tool call expectations.
python/samples/02-agents/evaluation/evaluate_agent.py	New sample demonstrating basic local evaluation for agents.
python/packages/core/tests/core/test_observability.py	Adjusts OTLP exporter-related test skipping.
python/packages/core/tests/core/test_local_eval.py	Adds a comprehensive test suite for local eval framework behaviors.
python/packages/core/agent_framework/_evaluation.py	Adds the provider-agnostic evaluation framework implementation.
python/packages/core/agent_framework/init.py	Re-exports evaluation APIs/types from the package root.
python/packages/azure-ai/agent_framework_azure_ai/_foundry_evals.py	Adds the Foundry-backed FoundryEvals provider + trace/target helpers.
python/packages/azure-ai/agent_framework_azure_ai/init.py	Exposes FoundryEvals and helper functions from the azure-ai package.

[TaoChenOSU]

nit: should we also check if the name is actually one of the builtins?

[alliscode]

Copilot on behalf of alliscode: Good call. We considered strict validation but decided to pass through any builtin.* name as-is since Foundry may add new evaluators that aren't in our local mapping yet. Only unqualified shorthand names (relevance, coherence, etc.) are validated against the known set. Added a comment explaining this rationale — see bb5f546.

[TaoChenOSU]

nit: is there a model defined somewhere with all the parameters documented and typed? The dictionary is hard to reason about and creates potential maintenance overhead in the future.

[alliscode]

Copilot on behalf of alliscode: The testing_criteria dict structure is dictated by the OpenAI SDK's evals.create() API — it accepts dict[str, Any] with no typed model yet. We could introduce a TypedDict for internal representation, but it would just wrap the same dict. For now, added an inline comment with a link to the API reference docs. If the SDK adds typed models in the future, we'll adopt them. See bb5f546.

[TaoChenOSU]

{{item.query_messages}}

Are these placeholders?

[alliscode]

Copilot on behalf of alliscode: Yes, these are Mustache-style template placeholders used by the OpenAI evals API data_mapping feature. They reference fields in the JSONL data items (e.g. {{item.query_messages}} maps to the query_messages field in each item). Added an inline comment explaining this — see bb5f546.

[TaoChenOSU]

If remaining is less than the poll interval, should we still go for another round? Also, there is the 60 second implicit upper bound on the poll interval, which may throw people off.

[alliscode]

Copilot on behalf of alliscode: Yes — if remaining < poll_interval, the min(poll_interval, remaining, 60.0) clamp ensures we still sleep for just the remaining time and do one final check. Added a comment documenting the 60s upper bound rationale: it prevents a single long sleep from consuming the entire timeout, allowing the loop to check status more frequently. See bb5f546.

[TaoChenOSU]

nit: is it possible to narrow down the type of run?

[alliscode]

Copilot on behalf of alliscode: Updated — removed the | Any union entirely. This is a brand new feature so no backward compat needed. The type is now just RunRetrieveResponse. See d05d032.

[TaoChenOSU]

If we can tighten the type, we can get rid of a lot of the getattr in the code.

[alliscode]

Copilot on behalf of alliscode: Agreed — done. Replaced most getattr calls with direct field access using the typed RunRetrieveResponse and PerTestingCriteriaResult types from the OpenAI SDK. Still using getattr for result_counts since the return value is a Pydantic model that needs extraction, but the individual fields (errored/failed/passed/total) are now accessed by known name. See bb5f546.

[TaoChenOSU]

Question: are all the Foundry evaluators supported by a bare openai client?

[alliscode]

Copilot on behalf of alliscode: Great point — updated. FoundryEvals now accepts client: FoundryChatClient instead of openai_client: AsyncOpenAI. Since the builtin.* evaluators require a Foundry endpoint, the type should reflect that. The project_client fallback is retained for the standalone functions (evaluate_traces, evaluate_foundry_target). All samples updated. See 669d67c.

[TaoChenOSU]

Does this name matter if the user supplies an openai client?

[alliscode]

Copilot on behalf of alliscode: self.name is used in EvalResults.provider to identify which evaluator produced the results (e.g. in multi-evaluator workflows). It's a display/identification name, not functionally significant to the API. It defaults to Microsoft Foundry because the evaluator uses Foundry's built-in evaluators. A user-supplied OpenAI client still talks to a Foundry endpoint for evals, so the name is accurate regardless.

[TaoChenOSU]

Why are there two empty lists?

[alliscode]

Copilot on behalf of alliscode: They were placeholder empty lists for the two queries. Since no tools are expected, it's cleaner to just omit expected_tool_calls entirely (it's optional). Removed in bb5f546.