Decouple ATIF schemas from core and make nvidia-nat-eval lightweight by default

[AnuradhaKaruppiah]

Description

Summary
Standalone ATIF custom evaluation should not require the full core runtime stack. This split preserves full config-driven nat eval behavior while enabling a lightweight dependency path for ATIF-native evaluation workflows.

Details

• Introduces nvidia-nat-atif to own ATIF schema models previously sourced from nvidia-nat-core.
• Makes base nvidia-nat-eval lightweight by removing hard dependencies on nvidia-nat-core, datasets, and openpyxl.
• Adds nvidia-nat-eval[full] for full nat eval runtime paths, with actionable guidance when full deps are missing.
• Updates docs/notebooks to reflect standalone ATIF vs full eval installation and usage modes.

Notes

• Core now consumes ATIF models via nvidia-nat-atif.
• "uv sync --all-groups --extra most" now includes nvidia-nat-eval[full].
• Full-path users should install nvidia-nat-eval[full] for config-driven nat eval workflows.
• Standalone runtime graph remains lightweight: nvidia-nat-eval + nvidia-nat-atif + pydantic.

Complete Runtime Graph (rooted at nvidia-nat-eval)

• nvidia-nat-eval
  • nvidia-nat-atif
    • pydantic==2.12.5
  • pydantic==2.12.5
    • annotated-types==0.7.0
    • pydantic-core==2.41.5
      • typing-extensions==4.15.0
    • typing-extensions==4.15.0
    • typing-inspection==0.4.2
      • typing-extensions==4.15.0

By Submitting this PR I confirm:

• I am familiar with the Contributing Guidelines.
• We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  • Any contribution which contains commits that are not Signed-Off will not be accepted.
• When the PR is ready for review, new or existing tests cover these changes.
• When the PR is ready for review, the documentation is up to date with these changes.

Summary by CodeRabbit

• Documentation

  • Clarified installation choices (standalone ATIF vs full config-driven evaluation), added explicit install commands, actionable error guidance, and updated CLI, migration guides, notebooks, and prerequisites to reflect the new extras and install modes.

• Refactor

  • Harmonized evaluator/ATIF usage across examples and tests and loosened evaluator output typing to improve interoperability while preserving runtime behavior.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Evaluator I/O moved from concrete Pydantic models to runtime-checkable Protocols; core and front-end type annotations were relaxed to protocol/BaseModel types. ATIF and evaluator import paths were updated across examples/tests/notebooks. Documentation now distinguishes base vs full nvidia-nat-eval installation modes and adds nvidia-nat-atif as a core dependency.

Changes

Cohort / File(s)	Summary
API: evaluator data models → Protocols packages/nvidia_nat_core/src/nat/data_models/evaluator.py	Replaced Pydantic EvalOutputItem/EvalOutput with @runtime_checkable Protocols EvalOutputItemLike/EvalOutputLike; removed concrete Pydantic field/schema definitions.
Core typing updates packages/nvidia_nat_core/src/nat/builder/evaluator.py, packages/nvidia_nat_core/src/nat/data_models/evaluate_runtime.py, packages/nvidia_nat_core/src/nat/finetuning/interfaces/trajectory_builder.py	Loosened type annotations to use EvalOutputLike / EvalOutputItemLike or BaseModel instead of concrete EvalOutput/EvalOutputItem.
Front-end response typing packages/nvidia_nat_core/src/nat/front_ends/fastapi/fastapi_front_end_config.py	Changed EvaluateItemResponse.result type from `EvalOutputItem
Packaging: core dependency & uv source packages/nvidia_nat_core/pyproject.toml	Added nvidia-nat-atif to core dependencies and added an editable local uv source mapping to ../nvidia_nat_atif.
Docs: installation, CLI, migration, and evaluator registration guidance docs/source/extend/custom-components/custom-evaluator.md, docs/source/improve-workflows/evaluate.md, docs/source/get-started/installation.md, docs/source/reference/cli.md, docs/source/resources/migration-guide.md, docs/source/run-workflows/observe/observe-workflow-with-langsmith.md	Clarified installation modes: base nvidia-nat-eval / nvidia-nat[eval-base] for standalone ATIF/custom-evaluator usage vs nvidia-nat-eval[full] / nvidia-nat[eval-full] for full config-driven nat eval; updated examples and evaluator registration docs to reference nat.plugins.eval.data_models.evaluator_io.
Notebooks & examples – import path and install updates examples/notebooks/eval_atif_custom_evaluator.ipynb, examples/notebooks/eval_atif_standalone.ipynb, examples/notebooks/eval_harbor_atif_interop.ipynb, examples/evaluation_and_profiling/simple_web_query_eval/tests/test_atif_only_evaluator_register.py	Updated imports: ATIFTrajectory now from nat.atif; EvalOutputItem import switched to nat.plugins.eval.data_models.evaluator_io; one notebook changed package install to editable local install.
Tests – import adjustments packages/nvidia_nat_core/tests/nat/finetuning/interfaces/test_trajectory_builder.py, examples/.../test_atif_only_evaluator_register.py	Adjusted test imports to new module locations (nat.atif, nat.plugins.eval.data_models.evaluator_io); test logic unchanged.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main changes: decoupling ATIF schemas from core and making nvidia-nat-eval lightweight by default, matching the PR objectives.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[AnuradhaKaruppiah]

/ok to test 8a90e77

[coderabbitai]

🧹 Nitpick comments (1)

docs/source/resources/migration-guide.md (1)

61-63: Clarify that these are alternative install options (choose one).

These bullets read like cumulative steps; please label them as equivalent alternatives to avoid users installing all three paths.

As per coding guidelines, "Documentation must be clear and comprehensive."

Also applies to: 96-99

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/source/resources/migration-guide.md` around lines 61 - 63, The three pip
install bullets showing `pip install "nvidia-nat[eval]" nvidia-nat-langchain`,
`pip install "nvidia-nat[eval-full]" nvidia-nat-langchain`, and `pip install
"nvidia-nat-eval[full]" nvidia-nat-langchain` should be explicitly labeled as
mutually exclusive alternatives; change the surrounding text to introduce them
as "Choose one of the following install options:" (or prepend each bullet with
"Option 1/2/3 — choose one:") so readers do not interpret them as cumulative
steps, and apply the same clarification to the similar group at lines showing
the other set (the bullets referenced in the comment at 96-99).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@docs/source/resources/migration-guide.md`:
- Around line 61-63: The three pip install bullets showing `pip install
"nvidia-nat[eval]" nvidia-nat-langchain`, `pip install "nvidia-nat[eval-full]"
nvidia-nat-langchain`, and `pip install "nvidia-nat-eval[full]"
nvidia-nat-langchain` should be explicitly labeled as mutually exclusive
alternatives; change the surrounding text to introduce them as "Choose one of
the following install options:" (or prepend each bullet with "Option 1/2/3 —
choose one:") so readers do not interpret them as cumulative steps, and apply
the same clarification to the similar group at lines showing the other set (the
bullets referenced in the comment at 96-99).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 259445f5-9bf2-42f2-bd8a-37b3d845309d

📥 Commits

Reviewing files that changed from the base of the PR and between 1715109 and 2bf8b3e.

📒 Files selected for processing (4)

• docs/source/get-started/installation.md
• docs/source/reference/cli.md
• docs/source/resources/migration-guide.md
• pyproject.toml

✅ Files skipped from review due to trivial changes (1)

• docs/source/get-started/installation.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/source/reference/cli.md

[AnuradhaKaruppiah]

/ok to test 66e8b99

[willkill07]

Conditionally approving based on documentation update(s).

[dagardner-nv]

Approving for deps

[AnuradhaKaruppiah]

/ok to test 7fd7ade

[AnuradhaKaruppiah]

/ok to test b73421d

[AnuradhaKaruppiah]

/ok to test f3bdc49

[AnuradhaKaruppiah]

/ok to test 5589f7b

[AnuradhaKaruppiah]

/ok to test 703a6fd

[AnuradhaKaruppiah]

/ok to test f6905d3

[AnuradhaKaruppiah]

/merge