Python: Foundry Evals integration for Python#4750

Merged

alliscode merged 52 commits intomicrosoft:mainfrom

alliscode:af-foundry-evals-python

Mar 31, 2026

Commits on Mar 27, 2026

Foundry Evals integration for Python

alliscode
and
Copilot
committed
fix: resolve mypy redundant-cast errors while keeping pyright happy

alliscode
and
Copilot
committed
fix: CI failures — pyupgrade, evaluator overloads, sample API, reset attr

alliscode
and
Copilot
committed
fix: skip gRPC-dependent observability test

alliscode
and
Copilot
committed
fix: add nosec B101 for bandit assert check

alliscode
and
Copilot
committed
style: align eval samples with repo conventions

alliscode
and
Copilot
committed
Address PR review feedback: async fixes, sample bugs, deprecation warnings

alliscode
and
Copilot
committed
Add test coverage for review feedback items

alliscode
and
Copilot
committed
Fix ruff ISC004 lint error and apply formatter

alliscode
and
Copilot
committed
Remove core type changes (extracted to fix/workflow-stale-session branch)

alliscode
and
Copilot
committed
Address PR review round 2: bugs, tests, and architecture

alliscode
and
Copilot
committed
Fix CI: revert test_full_conversation, fix pyright errors

alliscode
and
Copilot
committed
Address PR review round 3: reliability, test gaps, cleanup

alliscode
and
Copilot
committed
Fix CI: ruff S101 assert, pyright and mypy arg-type errors

alliscode
and
Copilot
committed
Address PR review round 4: bugs, reliability, test fixes

alliscode
and
Copilot
committed
Address PR review round 5: reliability, docs, test coverage

alliscode
and
Copilot
committed
Fix error message: evaluate_responses() → evaluate_traces(response_ids=...)

alliscode
and
Copilot
committed
Remove dead to_eval_data() method, fix docstring claims

alliscode
and
Copilot
committed
Reduce default eval timeout from 600s to 180s (3 minutes)

alliscode
and
Copilot
committed
Remove dead _evaluate_via_responses method from FoundryEvals

alliscode
and
Copilot
committed
Revert unrelated formatting changes to get-started samples

alliscode
and
Copilot
committed
Fix pyright: remove phantom FoundryMemoryProvider import, apply ruff format

alliscode
and
Copilot
committed
Fix eval samples: use FoundryChatClient for Agent()

alliscode
and
Copilot
committed
Simplify self-reflection sample using FoundryEvals

alliscode
and
Copilot
committed
Update eval samples to FoundryChatClient and FOUNDRY_PROJECT_ENDPOINT

alliscode
and
Copilot
committed
Fix lint errors in eval samples (E501, ASYNC240, formatting)

alliscode
and
Copilot
committed
Remove evaluate_all_patterns_sample.py (redundant with focused samples)

alliscode
and
Copilot
committed
Fix async credential mismatch: use azure.identity.aio for async AIProjectClient

alliscode
and
Copilot
committed
Revert test_observability.py to upstream/main (not our test)

alliscode
and
Copilot
committed
Address moonbox3 review: sphinx docstrings, pagination, isinstance check

alliscode
and
Copilot
committed
Fix test failures and address remaining moonbox3 review comments

alliscode
and
Copilot
committed
Address Tao's PR review comments on Foundry Evals

alliscode
and
Copilot
committed
Remove unnecessary Any union from run type annotations

alliscode
and
Copilot
committed
Accept FoundryChatClient instead of raw AsyncOpenAI

alliscode
and
Copilot
committed
Remove implicit 60s upper bound on poll interval

alliscode
and
Copilot
committed
Remove 1s floor on poll interval — let the developer control it

alliscode
and
Copilot
committed
Update python/samples/05-end-to-end/evaluation/foundry_evals/.env.example

authored and
alliscode
committed
Update python/samples/02-agents/evaluation/evaluate_agent.py

authored and
alliscode
committed
Address eavanvalkenburg review (round 2) on Python eval PR

alliscode
and
Copilot
committed
Move FoundryEvals to foundry package, split tool eval sample

alliscode
and
Copilot
committed
Auto-create FoundryChatClient from env vars when no client provided

alliscode
and
Copilot
committed
Fix pyright errors: remove dead _normalize_queries, suppress EvalAPIError check

alliscode
and
Copilot
committed

Commits on Mar 30, 2026

Support multimodal image content in eval pipeline

alliscode
and
Copilot
committed
Address remaining review comments

alliscode
and
Copilot
committed
Fix ruff lint errors (E501, SIM108, SIM102)

alliscode
and
Copilot
committed
Fix pyright errors: type-narrow dict to dict[str, Any], add ignore comments

alliscode
and
Copilot
committed
Replace ConversationSplitter type alias with Protocol

alliscode
and
Copilot
committed
Standardize on AZURE_AI_MODEL_DEPLOYMENT_NAME and fix Unicode in samples

alliscode
and
Copilot
committed

Commits on Mar 31, 2026