Skip to content

test(evaluator): run plugin skill evals in NeMo Platform #1063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngoncharenko wants to merge 1 commit into
base: main
Choose a base branch
from
Open

test(evaluator): run plugin skill evals in NeMo Platform #1063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions skills/nemo-evaluator-plugin/evals/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 1
harbor:
task_source: native_harbor
custom_dockerfile_mode: preserve
base_image_mode: disabled
skill_workspace:
mode: isolated
grading:
mode: default
25 changes: 25 additions & 0 deletions skills/nemo-evaluator-plugin/evals/environment/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

FROM nvcr.io/nvidia/nemo-platform/nmp-api:0.3.0@sha256:852758b994aaca5a9646a163fffaf46a20938b7e0aa10928c275e0d4a32d0723

USER 0

RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends jq procps ripgrep \
&& rm -rf /var/lib/apt/lists/*

COPY config/nmp-eval-config.yaml /etc/nmp/eval.yaml
COPY scripts/nmp-eval-bootstrap /usr/local/bin/nmp-eval-bootstrap

RUN chmod 0555 /usr/local/bin/nmp-eval-bootstrap \
&& chmod 0444 /etc/nmp/eval.yaml

ENV PATH="/app/.venv/bin:${PATH}"

WORKDIR /workspace

# Astra supplies the sandbox keepalive command. Do not inherit the NMP image's
# `nemo services run` entrypoint or its default arguments.
ENTRYPOINT []
CMD []
33 changes: 33 additions & 0 deletions skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

platform:
runtime: none
base_url: http://127.0.0.1:8080

jobs:
executors:
- provider: subprocess
profile: default
backend: subprocess
config:
working_directory: /workspace/.nemo/subprocess-jobs
cleanup_completed_jobs_immediately: false
ttl_seconds_before_active: 60
ttl_seconds_active: 3600
ttl_seconds_after_finished: 300
executor_defaults:
subprocess:
working_directory: /workspace/.nemo/subprocess-jobs
cleanup_completed_jobs_immediately: false
ttl_seconds_before_active: 60
ttl_seconds_active: 3600
ttl_seconds_after_finished: 300

secrets:
allow_key_creation: true

files:
default_storage_config:
type: local
path: /workspace/.nemo/files
83 changes: 83 additions & 0 deletions skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

readonly NMP_EVAL_BASE_URL="${NMP_BASE_URL:-http://127.0.0.1:8080}"
readonly NMP_EVAL_DATA_DIR="${NMP_DATA_DIR:-/workspace/.nemo}"
readonly NMP_EVAL_CONFIG_PATH="${NMP_CONFIG_FILE_PATH:-/etc/nmp/eval.yaml}"
readonly NMP_EVAL_LOG_PATH="${NMP_EVAL_DATA_DIR}/platform.log"
readonly NMP_EVAL_PID_PATH="${NMP_EVAL_DATA_DIR}/platform.pid"
readonly NMP_EVAL_STARTUP_TIMEOUT_SECONDS="${NMP_EVAL_STARTUP_TIMEOUT_SECONDS:-240}"

export NMP_AUTH_ENABLED=false
export NMP_BASE_URL="${NMP_EVAL_BASE_URL}"
export NMP_CONFIG_FILE_PATH="${NMP_EVAL_CONFIG_PATH}"
export NMP_DATA_DIR="${NMP_EVAL_DATA_DIR}"
export PATH="/app/.venv/bin:${PATH}"

mkdir -p "${NMP_EVAL_DATA_DIR}/files" "${NMP_EVAL_DATA_DIR}/subprocess-jobs"

profile_is_ready() {
curl --fail --silent \
"${NMP_EVAL_BASE_URL%/}/apis/jobs/v2/execution-profiles" \
| jq --exit-status 'any(.[]; .provider == "subprocess" and .profile == "default")' >/dev/null
}

platform_is_ready() {
curl --fail --silent "${NMP_EVAL_BASE_URL%/}/health/ready" >/dev/null \
&& profile_is_ready
Comment on lines +22 to +30

@coderabbitai coderabbitai Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Bound each readiness request.

curl has no connection or transfer timeout. A stalled local endpoint can block past startup_deadline and prevent the failure log from running. Add short --connect-timeout and --max-time values to both requests.

Proposed fix 
-    curl --fail --silent \
+    curl --fail --silent --connect-timeout 2 --max-time 5 \
@@
-    curl --fail --silent "${NMP_EVAL_BASE_URL%/}/health/ready" >/dev/null \
+    curl --fail --silent --connect-timeout 2 --max-time 5 \
+        "${NMP_EVAL_BASE_URL%/}/health/ready" >/dev/null \
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
profile_is_ready() {
curl --fail --silent \
"${NMP_EVAL_BASE_URL%/}/apis/jobs/v2/execution-profiles" \
| jq --exit-status 'any(.[]; .provider == "subprocess" and .profile == "default")' >/dev/null
}
platform_is_ready() {
curl --fail --silent "${NMP_EVAL_BASE_URL%/}/health/ready" >/dev/null \
&& profile_is_ready
profile_is_ready() {
curl --fail --silent --connect-timeout 2 --max-time 5 \
"${NMP_EVAL_BASE_URL%/}/apis/jobs/v2/execution-profiles" \
| jq --exit-status 'any(.[]; .provider == "subprocess" and .profile == "default")' >/dev/null
}
platform_is_ready() {
curl --fail --silent --connect-timeout 2 --max-time 5 \
"${NMP_EVAL_BASE_URL%/}/health/ready" >/dev/null \
&& profile_is_ready
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap`
around lines 22 - 30, Update the curl invocations in profile_is_ready and
platform_is_ready to include short --connect-timeout and --max-time bounds on
both readiness requests, ensuring stalled endpoints return before
startup_deadline handling is reached.

}

platform_pid() {
if [[ -s "${NMP_EVAL_PID_PATH}" ]]; then
tr -d '[:space:]' <"${NMP_EVAL_PID_PATH}"
fi
}

platform_is_running() {
local pid
pid="$(platform_pid)"
[[ "${pid}" =~ ^[0-9]+$ ]] && kill -0 "${pid}" 2>/dev/null
}

print_failure_log() {
if [[ -f "${NMP_EVAL_LOG_PATH}" ]]; then
echo "NeMo Platform startup log (last 200 lines):" >&2
tail -n 200 "${NMP_EVAL_LOG_PATH}" >&2
fi
}

if platform_is_ready; then
exit 0
fi

if ! platform_is_running; then
rm -f "${NMP_EVAL_PID_PATH}"
nohup /app/.venv/bin/nemo services run \
--services entities,jobs,files,secrets,evaluator \
--controllers jobs \
--config "${NMP_EVAL_CONFIG_PATH}" \
--host 127.0.0.1 \
--port 8080 \
>"${NMP_EVAL_LOG_PATH}" 2>&1 </dev/null &
echo "$!" >"${NMP_EVAL_PID_PATH}"
fi

readonly startup_deadline=$((SECONDS + NMP_EVAL_STARTUP_TIMEOUT_SECONDS))
while ((SECONDS < startup_deadline)); do
if platform_is_ready; then
exit 0
fi
if ! platform_is_running; then
echo "NeMo Platform exited before becoming ready." >&2
print_failure_log
exit 1
fi
sleep 2
done

echo "NeMo Platform did not become ready within ${NMP_EVAL_STARTUP_TIMEOUT_SECONDS} seconds." >&2
print_failure_log
exit 1
37 changes: 21 additions & 16 deletions skills/nemo-evaluator-plugin/evals/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,21 @@
[
{
"id": "nemo-evaluator-plugin-001",
"question": "I need help with the nemo-evaluator-plugin. How do I run an inline exact-match evaluation using the nemo CLI?",
"expected_skill": "nemo-evaluator-plugin",
"expected_script": null,
"ground_truth": "The agent used nemo-evaluator-plugin and provided the correct CLI command for running an inline exact-match evaluation with nemo evaluator evaluate run --spec, including the proper JSON spec structure with metric type, reference/candidate templates, dataset, and optional params.",
"expected_behavior": [
"The agent read the nemo-evaluator-plugin SKILL.md before responding",
"The agent provided the exact CLI command syntax for nemo evaluator evaluate run --spec with the exact-match metric configuration",
"The agent included the JSON spec structure showing metric type, reference template, candidate template, and dataset fields",
"The agent mentioned activating the Python virtual environment as a prerequisite",
"The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
]
}
]
{
"skill_name": "nemo-evaluator-plugin",
"evals": [
{
"id": "nemo-evaluator-plugin-001",
"prompt": "Use the nemo-evaluator-plugin skill and the running NeMo Platform to submit the bundled two-row offline exact-match evaluation. Wait for the job to finish, retrieve its aggregate scores, and report the job name, terminal status, metric name, and mean. Do not use the standalone nemo evaluator evaluate run path.",
"expected_skill": "nemo-evaluator-plugin",
"expected_script": null,
"expected_output": "The agent read nemo-evaluator-plugin, resolved its bundled assets/specs/exact_match_metric.json, submitted it to the running platform with nemo evaluator evaluate submit --spec-file, retained the returned job name, waited for the job to reach completed, downloaded the aggregate-scores result, and reported exact-match.exact-match with a mean of 0.5.",
"assertions": [
"The agent read the nemo-evaluator-plugin SKILL.md before responding",
"The agent resolved and used the bundled assets/specs/exact_match_metric.json two-row offline evaluation spec",
"The agent submitted the evaluation to the running platform with nemo evaluator evaluate submit --spec-file and did not use nemo evaluator evaluate run",
"The agent retained the returned platform job name and waited until nemo jobs get-status reported the terminal completed status",
"The agent listed or downloaded the aggregate-scores result and reported the exact-match.exact-match mean as 0.5",
"The agent reported observable evidence including the job name, completed status, metric name, and mean rather than inventing a result",
"The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
]
}
]
}
9 changes: 9 additions & 0 deletions skills/nemo-evaluator-plugin/evals/harbor/dataset.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[dataset]
name = "nvidia/nemo-evaluator-plugin"
description = "Live NeMo Platform evaluator plugin skill evaluation"

[[tasks]]
name = "nvidia/nemo-evaluator-plugin-001"
1 change: 1 addition & 0 deletions skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Use the nemo-evaluator-plugin skill and the running NeMo Platform to submit the bundled two-row offline exact-match evaluation. Wait for the job to finish, retrieve its aggregate scores, and report the job name, terminal status, metric name, and mean. Do not use the standalone nemo evaluator evaluate run path.
35 changes: 35 additions & 0 deletions skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version = "1.3"

[task]
name = "nvidia/nemo-evaluator-plugin-001"
description = "Submit an offline exact-match evaluation to a live NeMo Platform."

[agent]
timeout_sec = 300.0

[verifier]
timeout_sec = 180.0

[environment]
cpus = 2
memory_mb = 4096
storage_mb = 10240
network_mode = "public"

@coderabbitai coderabbitai Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- local instructions ---'
find .. -name AGENTS.local.md -print
printf '%s\n' '--- task file ---'
sed -n '1,120p' skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
printf '%s\n' '--- Harbor/network references ---'
rg -n -i --glob '!*.lock' 'network_mode|network mode|public|isolat|loopback|127\.0\.0\.1|localhost' skills/nemo-evaluator-plugin/evals skills/nemo-evaluator-plugin README.md Makefile 2>/dev/null | head -250

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 3465

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- evaluator configuration ---'
sed -n '1,100p' skills/nemo-evaluator-plugin/evals/config.yml
printf '%s\n' '--- bootstrap script ---'
sed -n '1,140p' skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap
printf '%s\n' '--- runtime configuration ---'
sed -n '1,100p' skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yaml
printf '%s\n' '--- task references ---'
rg -n -C 3 -i 'offline|network|internet|download|curl|wget|pip|npm|localhost|127\.0\.0\.1' skills/nemo-evaluator-plugin/evals/harbor skills/nemo-evaluator-plugin/evals/environment skills/nemo-evaluator-plugin/references

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 18614

🌐 Web query:

Harbor benchmark task.toml network_mode public isolated semantics loopback

💡 Result:

In the context of the Harbor framework, network configuration in task.toml is managed through the network_mode field within environment and phase configurations [1][2]. The framework supports three primary network modes: public, no-network, and allowlist [1][3]. The terms "public," "isolated," and "loopback" represent different concepts related to network accessibility and isolation in containerized environments, rather than being equivalent values for the network_mode field: 1. Harbor Network Modes: - public: Provides full network access [1][3]. This is the default when network_mode is omitted [2]. - no-network: Provides no network access; typically used for maximum isolation where only the loopback interface is available [1][2]. - allowlist: Restricts network access to specific hosts defined in allowed_hosts [1][3]. 2. Technical Context: - Public: In Harbor, this refers to unrestricted egress/ingress [1]. In standard Docker terminology, it usually implies a bridged network with outbound internet access [4]. - Isolated: This typically describes configurations like no-network or internal-only bridge networks, where a container is cut off from the outside internet or external host services [4][5]. - Loopback: This refers to the local network interface (127.0.0.1) [5][6]. "Loopback isolation" is often discussed in technical benchmarks or security configurations to ensure processes within a container or shared namespace can communicate via 127.0.0.1 without exposing those services on the host's public interfaces [6][7]. In summary, for a Harbor benchmark task.toml, you should use one of the supported modes (public, no-network, or allowlist) [1]. If you require "isolated" or "loopback" semantics, the no-network mode is the intended setting to restrict external traffic while maintaining local loopback capabilities [1][2]. "Isolated" and "loopback" are not valid network_mode values in the Harbor schema [3].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re

task = Path("skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml").read_text()
bootstrap = Path("skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap").read_text()
config = Path("skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yaml").read_text()
instruction = Path("skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.md").read_text()

network = re.search(r'(?m)^\s*network_mode\s*=\s*"([^"]+)"', task)
urls = sorted(set(re.findall(r'https?://[^"\s`\\]+', "\n".join((task, bootstrap, config)))))
print({"network_mode": network.group(1) if network else None})
print({"urls": urls})
print({"uses_loopback": any("127.0.0.1" in u or "localhost" in u for u in urls)})
print({"instruction_is_offline": "offline" in instruction.lower()})
print({"external_dependency_markers": sorted(set(re.findall(
    r'(?i)\b(?:curl|wget|pip|npm|apt|apk|docker|git clone)\b', bootstrap + config
)))})
PY

Repository: NVIDIA-NeMo/nemo-platform

Length of output: 351

Security Misconfiguration (CWE-668)

Reachability: External · Exploitability: Moderate

Set network_mode = "no-network" for this offline task. Harbor public mode permits unrestricted egress. The bootstrap and evaluation use only loopback NeMo Platform services and bundled data, so no-network preserves the required runtime path while blocking external endpoints.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml`
at line 20, Update the task configuration’s network_mode setting from "public"
to "no-network" so the offline bootstrap and evaluation use only loopback
services and bundled data without external network access.

skills_dir = "/workspace/skills"
build_timeout_sec = 2400.0

[environment.env]
NMP_AUTH_ENABLED = "false"
NMP_BASE_URL = "http://127.0.0.1:8080"
NMP_CONFIG_FILE_PATH = "/etc/nmp/eval.yaml"
NMP_DATA_DIR = "/workspace/.nemo"
NMP_EVAL_STARTUP_TIMEOUT_SECONDS = "240"

[environment.healthcheck]
command = "/usr/local/bin/nmp-eval-bootstrap"
interval_sec = 5.0
timeout_sec = 245.0
retries = 1
Loading