feat(skills): add Experimentalist workflow - #1088
Open
gaiadilorenzo wants to merge 6 commits into
Open
Conversation
Contributor
|
gaiadilorenzo
commented
Aug 5, 2026
| with each run so another operator can reproduce it. If the agent or its spec | ||
| does not exist yet, use `nemo-explore` to design it and `nemo-spec` to create | ||
| the spec before returning here. | ||
|
|
Contributor
Author
There was a problem hiding this comment.
Once the pr with the harbor dataset conversion, it should be referenced here
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
gaiadilorenzo
force-pushed
the
experimentalist-plugin-skill/gdilorenzo
branch
2 times, most recently
from
August 5, 2026 12:24
7031bca to
1cd0253
Compare
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
gaiadilorenzo
marked this pull request as ready for review
August 5, 2026 13:41
Contributor
📝 WalkthroughWalkthroughAdded the ChangesExperimentalist skill
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md`:
- Around line 276-277: Insert a blank line between the closing fenced code block
and the following “For a Git source, append a ref such as” paragraph in
SKILL.md, preserving the existing content and formatting otherwise.
- Line 5: Update the SKILL.md content around the fenced command block in the
nemo-experimentalist skill so there is a blank line after the closing fence
before the following prose. Keep the existing command block content unchanged
and only adjust the Markdown spacing immediately after the fenced block.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: fc3cd401-5fca-4255-a512-317199222dc2
📒 Files selected for processing (2)
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/tests.json
| name: nemo-experimentalist | ||
| description: Improve an existing NeMo agent from an Insight or explicit Harbor-compatible evaluation datasets. Run the Experimentalist, validate candidate changes against a held-out split, and optionally publish a changed winner as a draft PR or MR. | ||
| triggers: | ||
| - run experimentalist |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- target skill ---'
cat -n packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md | sed -n '1,40p;260,285p'
printf '%s\n' '--- skill trigger definitions ---'
rg -n --hidden -S 'run experimentalist|experimentalist|trigger|built-in|builtin' \
packages/nemo_platform_ext/src/nemo_platform_ext/skills \
packages/nemo_platform_ext/src/nemo_platform_ext \
2>/dev/null | head -300
printf '%s\n' '--- relevant skill files ---'
find packages/nemo_platform_ext/src/nemo_platform_ext/skills -name SKILL.md -print | sortRepository: NVIDIA-NeMo/nemo-platform
Length of output: 22379
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- target skill ---'
cat -n packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md | sed -n '1,40p;260,285p'
printf '%s\n' '--- trigger definitions ---'
rg -n --hidden -S 'run experimentalist|experimentalist|trigger|built-in|builtin' \
packages/nemo_platform_ext/src/nemo_platform_ext/skills \
packages/nemo_platform_ext/src/nemo_platform_ext \
2>/dev/null | head -300
printf '%s\n' '--- skill files ---'
find packages/nemo_platform_ext/src/nemo_platform_ext/skills -name SKILL.md -print | sortRepository: NVIDIA-NeMo/nemo-platform
Length of output: 22364
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- skill registry and matcher references ---'
cat -n packages/nemo_platform_ext/src/nemo_platform_ext/skills/__init__.py | sed -n '1,220p'
rg -n --hidden -S 'triggers|user-invocable|skill.*match|match.*skill|load.*skill|discover.*skill' \
--glob '*.py' --glob '*.md' --glob '*.json' . 2>/dev/null | head -400
printf '%s\n' '--- every declared trigger ---'
python3 - <<'PY'
from pathlib import Path
import re
for p in sorted(Path("packages/nemo_platform_ext/src/nemo_platform_ext/skills").glob("**/SKILL.md")):
lines=p.read_text().splitlines()
in_triggers=False
vals=[]
for line in lines:
if line == "triggers:":
in_triggers=True
continue
if in_triggers:
m=re.match(r"\s+-\s+(.+)$", line)
if m:
vals.append(m.group(1))
elif line and not line.startswith(" "):
break
print(f"{p}: {vals}")
PY
printf '%s\n' '--- target formatting context ---'
sed -n '250,282p' packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md
printf '%s\n' '--- markdown configuration and lint references ---'
find . -maxdepth 3 -type f \( -iname '*markdown*' -o -name '.markdownlint*' -o -name 'pyproject.toml' \) -print
rg -n --hidden -S 'MD031|fenced|markdownlint|pre-commit' . \
--glob '.markdownlint*' --glob '.pre-commit-config.yaml' --glob 'pyproject.toml' --glob 'Makefile' 2>/dev/null | head -200Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- skill-test matcher ---'
cat -n scripts/skill-test.py | sed -n '1,180p'
printf '%s\n' '--- experimentalist skill tests ---'
cat -n packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/tests.json
printf '%s\n' '--- all trigger blocks ---'
python3 - <<'PY'
from pathlib import Path
for p in sorted(Path("packages/nemo_platform_ext/src/nemo_platform_ext/skills").glob("**/SKILL.md")):
lines=p.read_text().splitlines()
start=next((i for i,x in enumerate(lines) if x=="triggers:"),None)
if start is None:
continue
print(f"\n{p}")
for x in lines[start:start+12]:
print(x)
if x and not x.startswith(("triggers:", " ")):
break
PY
printf '%s\n' '--- fence-spacing check ---'
python3 - <<'PY'
from pathlib import Path
p=Path("packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md")
lines=p.read_text().splitlines()
for i,line in enumerate(lines):
if line.strip()=="```":
before=lines[i-1] if i else ""
after=lines[i+1] if i+1<len(lines) else ""
print(f"line {i+1}: before={before!r}, after={after!r}")
PY
printf '%s\n' '--- markdown lint configuration ---'
for f in .markdownlint.json .markdownlint.yaml .markdownlint.yml .markdownlint-cli2.json .markdownlint-cli2.yaml .markdownlint-cli2.mjs .pre-commit-config.yaml; do
[ -f "$f" ] && { echo "### $f"; cat "$f"; }
doneRepository: NVIDIA-NeMo/nemo-platform
Length of output: 18187
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import re
paths = sorted(
set(Path(".").glob("**/SKILL.md"))
- {p for p in Path("node_modules").glob("**/SKILL.md")}
)
triggers = []
for path in paths:
lines = path.read_text(errors="replace").splitlines()
try:
start = lines.index("triggers:") + 1
except ValueError:
continue
for line in lines[start:]:
if line and not line.startswith((" ", "\t")):
break
m = re.match(r"\s*-\s+(.+?)\s*$", line)
if m:
triggers.append((path, m.group(1).strip().lower()))
print("--- triggers containing standalone 'run' ---")
for path, trigger in triggers:
words = re.findall(r"[a-z][a-z0-9_-]+", trigger)
if "run" in words:
print(path, repr(trigger))
print("--- exact trigger duplicates ---")
seen = {}
for path, trigger in triggers:
seen.setdefault(trigger, []).append(path)
for trigger, owners in sorted(seen.items()):
if len(owners) > 1:
print(repr(trigger), owners)
print("--- target fence spacing violations ---")
path = Path("packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md")
lines = path.read_text().splitlines()
for i, line in enumerate(lines):
if line.strip() == "```":
if i + 1 < len(lines) and lines[i + 1].strip():
print(f"closing fence at line {i + 1} is not followed by a blank line")
PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 21162
Add a blank line after the fenced command block.
Line 276 is followed immediately by prose, which violates Markdown fence spacing.
🧰 Tools
🪛 SkillSpector (2.5.1)
[warning] 1: [TR2] Shadow Command Trigger: Shadow Command Trigger: 'run experimentalist' conflicts with built-in command 'run'
Remediation: Choose triggers that do not conflict with built-in commands or other skills. Prefix with a unique namespace if necessary.
(Trigger Abuse (TR2))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md`
at line 5, Update the SKILL.md content around the fenced command block in the
nemo-experimentalist skill so there is a blank line after the closing fence
before the following prose. Keep the existing command block content unchanged
and only adjust the Markdown spacing immediately after the fenced block.
Source: Linters/SAST tools
Comment on lines
+276
to
+277
| ``` | ||
| For a Git source, append a ref such as |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a blank line after the fenced block.
Markdownlint requires a blank line between the closing fence and the following paragraph.
Proposed fix
--config path/to/experimentalist.yamlFor a Git source, append a ref such as
</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
--config path/to/experimentalist.yaml
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 276-276: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
🪛 SkillSpector (2.5.1)
[warning] 1: [TR2] Shadow Command Trigger: Shadow Command Trigger: 'run experimentalist' conflicts with built-in command 'run'
Remediation: Choose triggers that do not conflict with built-in commands or other skills. Prefix with a unique namespace if necessary.
(Trigger Abuse (TR2))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experimentalist/SKILL.md`
around lines 276 - 277, Insert a blank line between the closing fenced code
block and the following “For a Git source, append a ref such as” paragraph in
SKILL.md, preserving the existing content and formatting otherwise.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemo-experimentalistskillValidation
uv run --frozen --python <arm64-python> pytest packages/nemo_platform_ext/tests/cli/commands/skills/test_skill_content.py -q(12 passed)Summary by CodeRabbit
New Features
nemo-experimentalistskill with guidance for selecting optimization modes, configuring environments, preparing datasets, running evaluations, and validating results.Tests