Skip to content

test(evaluator): run plugin skill evals in NeMo Platform - #1063

Open
ngoncharenko wants to merge 3 commits into
ngoncharenko/aalgo-550-eval-plugin-skillsfrom
ngoncharenko/aalgo-550-eval-plugin-skills-runtime
Open

test(evaluator): run plugin skill evals in NeMo Platform#1063
ngoncharenko wants to merge 3 commits into
ngoncharenko/aalgo-550-eval-plugin-skillsfrom
ngoncharenko/aalgo-550-eval-plugin-skills-runtime

Conversation

@ngoncharenko

Copy link
Copy Markdown
Contributor

Summary

  • add a native Harbor evaluation environment derived from the pinned nmp-api:0.3.0 image
  • start a minimal NeMo Platform with authentication disabled and the subprocess/default execution profile
  • replace the help-only evaluator skill fixture with a live submit, wait, and aggregate-result evaluation
  • add focused contract tests for the Harbor manifest, platform configuration, bootstrap script, and Docker context

Why

The evaluator plugin skill was evaluated in the default SkillEvaluator container, which did not have NeMo Platform running. That only exercised CLI guidance and could not verify the platform submission workflow.

The new harness starts NeMo Platform inside the evaluation container, allows up to 240 seconds for readiness, and submits the bundled two-row exact-match evaluation. It does not use Docker-in-Docker: platform jobs execute through the in-container subprocess runtime.

Validation

  • uv run --frozen pytest plugins/nemo-evaluator/tests/test_skill_examples.py -q — 22 passed
  • Ruff lint and formatting checks passed
  • SkillEvaluator 0.9.2 Harbor contract validation — 10 checks passed, 2 expected warnings
  • derived Docker image built successfully from the pinned NMP image
  • live job nemo-evaluator-hwhtqurn completed with exact-match.exact-match mean 0.5
  • confirmed the evaluation container had no Docker socket and created no child containers

Stack

This PR is stacked on #1055 so its diff contains only the runtime-evaluation work. Retarget it to main after #1055 merges.

Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
@github-actions github-actions Bot added the test conventional-commit type label Aug 4, 2026
@ngoncharenko ngoncharenko self-assigned this Aug 4, 2026
@ngoncharenko
ngoncharenko marked this pull request as ready for review August 4, 2026 16:27
@ngoncharenko
ngoncharenko requested review from a team as code owners August 4, 2026 16:27
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test conventional-commit type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant