feat(studio): new example agent, eval modal supports dataset-driven, per-row results view - #1059
Queued
nv-odrulea wants to merge 12 commits into
Queued
feat(studio): new example agent, eval modal supports dataset-driven, per-row results view#1059nv-odrulea wants to merge 12 commits into
nv-odrulea wants to merge 12 commits into
Conversation
nv-odrulea
force-pushed
the
od/studio-common-updates
branch
from
August 4, 2026 18:06
10e25f5 to
0aa0870
Compare
nv-odrulea
marked this pull request as ready for review
August 4, 2026 18:34
nv-odrulea
force-pushed
the
od/studio-dataset-eval-feature
branch
from
August 4, 2026 20:57
89e54c2 to
ba750d7
Compare
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds task-driven and dataset-driven evaluation support to Studio. The change adds reusable email-security samples, submission APIs, deployment validation, evaluator job handling, aggregate and row-level result views, score normalization, and cancellation controls. ChangesEvaluation workflow
Sequence Diagram(s)sequenceDiagram
participant User
participant SubmitEvaluationModal
participant EvaluatorAPI
participant EvaluationResultDetailsRoute
participant useDatasetEvalResults
User->>SubmitEvaluationModal: select evaluation sample
SubmitEvaluationModal->>EvaluatorAPI: submit task or dataset evaluation
EvaluatorAPI-->>SubmitEvaluationModal: return job
SubmitEvaluationModal->>EvaluationResultDetailsRoute: navigate to result route
EvaluationResultDetailsRoute->>useDatasetEvalResults: load aggregate and row artifacts
useDatasetEvalResults-->>EvaluationResultDetailsRoute: return scores and rows
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 11
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsx (1)
77-79: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winStop the live timer after terminal status.
useLiveSecondsretainslockedStartDatewhenstartDatebecomesundefined. Both call sites leaveenabledtrue, so completed jobs continue to create one-second updates.
web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsx#L77-L79: passenabled: !isTerminalStatus.web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsx#L95-L97: passenabled: !isJobTerminal.Based on supplied
useLiveSecondscontext, disabling the hook clears its interval.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsx` around lines 77 - 79, Stop live timer updates for terminal jobs by passing enabled: !isTerminalStatus to useLiveSeconds in DetailsPanel.tsx and enabled: !isJobTerminal to useLiveSeconds in AgentEvaluationDetailRoute.tsx.
🧹 Nitpick comments (8)
web/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.ts (1)
40-100: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDeclare the hook result contract.
useDatasetEvalResultsis an exported complex API. Define a result interface and add an explicit return type.As per coding guidelines, use explicit return types for public APIs and complex functions in TypeScript.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.ts` around lines 40 - 100, The exported useDatasetEvalResults hook lacks an explicit result contract. Define a named interface describing its returned scores, rows, status flags, loading flags, and error fields, then annotate useDatasetEvalResults with that interface as its return type while preserving the existing return behavior.Source: Coding guidelines
web/packages/studio/src/components/evaluation/MetricScoreChip.tsx (1)
8-11: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMark immutable component inputs as
readonly.These interfaces describe React component inputs. Mark their properties as
readonly. Use readonly array types where the component does not mutate the collection.
web/packages/studio/src/components/evaluation/MetricScoreChip.tsx#L8-L11: marklabelandvalueasreadonly.web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx#L22-L25: markscoresandemptyMessageasreadonly.web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx#L24-L26: markrowsasreadonly.web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsx#L12-L14: markscoresasreadonly.web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsx#L18-L20: marktasksasreadonly.As per coding guidelines, use
readonlyfor immutable properties in TypeScript interfaces and types.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/MetricScoreChip.tsx` around lines 8 - 11, Mark all listed React component prop interfaces as immutable: in web/packages/studio/src/components/evaluation/MetricScoreChip.tsx lines 8-11, mark label and value readonly; in web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx lines 22-25, mark scores and emptyMessage readonly; in web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx lines 24-26, mark rows readonly; in web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsx lines 12-14, mark scores readonly; and in web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsx lines 18-20, mark tasks readonly. Use readonly array types for collection props.Source: Coding guidelines
web/packages/studio/src/components/evaluation/submitEvaluationJob.ts (1)
20-26: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueEmpty base yields a leading dash. If
toValidFilesetName(filesetName)reduces to only dashes,basebecomes''and the job name starts with-, which most name validators reject. Fall back to a fixed prefix whenbaseis empty.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/submitEvaluationJob.ts` around lines 20 - 26, Update buildEvalJobName so that when the normalized and trimmed base is empty, it uses a fixed valid prefix before appending the suffix; preserve the existing truncation and suffix behavior for non-empty bases.web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.json (1)
554-556: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
number-checklowercases a numeric string.left_templateapplies| lowerto the parsed index. The filter is inert for digits but signals a copy from the string metrics. Drop| lowerfor clarity.Also applies to: 607-609
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.json` around lines 554 - 556, Remove the unnecessary `| lower` filter from the numeric `left_template` expressions comparing against `item.reference.phish_index`, including both the shown template and the additional occurrence noted in the review. Preserve the existing split, trim, and numeric comparison behavior.web/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonl (1)
1-40: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winRows are ordered by label. Lines 1-20 are all
phishing, lines 21-40 allbenign. Any partial run, row limit, or early-stop yields a single-class score. Interleave the labels.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonl` around lines 1 - 40, The dataset labels are grouped into contiguous phishing and benign blocks, making partial evaluation produce a single-class sample. Reorder the records in the dataset so phishing and benign examples are interleaved throughout, while preserving each record’s content and label.web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts (1)
146-165: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNo coverage for the dataset-driven path.
parseEvalConfiggained adatasetbranch with two new validation errors, andisDatasetEvalSpec/buildDatasetEvalRequestBodyroute submissions to a different endpoint. None of that is tested here. Add cases for: a valid dataset config, a dataset config with emptymetrics, a dataset config with noprompt_template, and thefield_mappingnull default.Want me to draft them?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts` around lines 146 - 165, Extend the parseEvalConfig tests with dataset-driven cases: verify a valid dataset config parses, empty dataset metrics and a missing prompt_template are rejected, and an omitted field_mapping defaults to null. Also cover isDatasetEvalSpec and buildDatasetEvalRequestBody routing so dataset submissions use the dataset endpoint and request shape.web/packages/studio/src/constants/sampleAgents.ts (1)
56-70: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the sample registry immutable.
EvalConfigSampleandEVAL_CONFIG_SAMPLESexpose mutable config paths. An importer can change later evaluation submissions. Mark the fields and exported array asreadonly. Verify that no consumer mutates this registry first.As per coding guidelines, use
readonlyfor immutable properties in TypeScript interfaces and types.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/constants/sampleAgents.ts` around lines 56 - 70, Make EvalConfigSample immutable by marking its configuration fields readonly, including key, displayName, description, configPath, datasetPath, and readmePath. Declare EVAL_CONFIG_SAMPLES as a readonly array so consumers cannot mutate the registry, and verify existing consumers do not rely on modifying these values.Source: Coding guidelines
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md (1)
1-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSplit explanation content from editing instructions.
Both READMEs are EXPLANATION pages that contain HOW-TO content. They also omit prerequisites, Next Steps, and tested Python SDK and CLI tab sets.
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md#L1-L17: keep the evaluation concepts as an EXPLANATION and add prerequisites at the top.web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md#L73-L85: move editing steps to a dedicated HOW-TO with tested Python SDK and CLI tabs.web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md#L1-L30: keep task-driven concepts as an EXPLANATION and add prerequisites at the top.web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md#L69-L75: move editing steps to a dedicated HOW-TO with tested Python SDK and CLI tabs. Add Next Steps cross-links to both pages.As per coding guidelines, each documentation page must use one Diataxis quadrant, list prerequisites first, end with Next Steps, and provide Python SDK and CLI examples in tab sets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md` around lines 1 - 17, Update web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md lines 1-17 to remain an Explanation page with prerequisites first; move the editing instructions from lines 73-85 into a dedicated How-To page section with tested Python SDK and CLI tab sets, and end with Next Steps. Apply the same structure to web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md lines 1-30 and 69-75, preserving its task-driven concepts and adding Next Steps cross-links between both evaluation pages.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/packages/studio/public/sample-agents/email-security-analyst/agent.yml`:
- Line 12: Update the comment in the email-security-analyst agent configuration
to reference the dataset-driven config file, eval-config.dataset-driven.json,
instead of the outdated “email-phishing sample” name.
In
`@web/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonl`:
- Line 26: Update the dataset rows at the referenced email entries so their
labels and content align with the intended BEC ground truth: mark the
lookalike-domain banking request and changed-wire-instructions message as
phishing/BEC, or revise them to use a known sender domain and remove requests to
provide or change banking details while retaining benign labels.
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.json`:
- Line 192: Update all 10 template expressions in the eval configuration,
including the one containing candidate and the expressions at the referenced
locations, to use an escaped newline sequence in the JSON source (`\\n`) rather
than `\n`. Preserve the existing split, trim, and lower behavior.
In `@web/packages/studio/src/api/evaluation/evaluator-jobs.ts`:
- Around line 23-41: Update the evaluator-job pagination loop around
jobsListJobs to enforce a hard maximum page count, stopping before requesting
pages beyond that ceiling while preserving the existing batch-size and
shouldStop termination checks.
In `@web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsx`:
- Around line 37-47: Update the referenceText helper’s JSON.stringify return
path to coalesce an undefined result to null, preserving its declared string |
null return type while leaving the primitive-value handling unchanged.
In `@web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx`:
- Around line 127-139: Update the StudioDataView configuration in
EvalAggregateScoresTable so DataViewRoot.data contains only the current page by
slicing scores with dataViewState.pagination.state page index and page size.
Keep totalCount based on the full scores array so pagination reflects all
results while rendering only the selected page.
In
`@web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx`:
- Around line 70-85: Update the fallback row index in the DatasetEvalRow results
column and the corresponding modal/fallback row ID logic to use pageIndex *
pageSize + row.index whenever row.original.row_index is absent. Preserve the
existing row_index value when it is provided so labels, titles, and IDs remain
globally unique across pages.
- Around line 30-36: Update the expectedValue helper to preserve numeric and
boolean values from EXPECTED_FIELDS by converting supported scalar values to
strings, while continuing to ignore nullish, empty-string, and unsupported
values and fall back across fields as before.
In
`@web/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.ts`:
- Around line 40-48: Update useDatasetEvalResults so an undefined status and
every nonterminal status keep enabled false while the job is unresolved or
running. Only enable artifact requests when status is the successful terminal
state, while preserving the existing workspace and jobName requirements.
In `@web/packages/studio/src/components/evaluation/SubmitEvaluationModal.tsx`:
- Around line 217-219: Update the upload error handling around filesUploadFile
in SubmitEvaluationModal so a filesDeleteFileset failure is reported instead of
swallowed. Preserve the original upload error while surfacing the rollback
failure, including the affected fileset name in the error context, and retain
cleanup behavior when rollback succeeds.
- Around line 279-292: Update the deployment verification flow around
useAgentsListDeployments, hasRunningDeployment, noDeploymentError, and the
Submit control so pending, query-error, and successful verification states are
distinct. Keep Submit disabled while deployment verification is loading or has
failed, surface a verification failure for query errors rather than
NO_DEPLOYMENT_MESSAGE, and enable Submit only after a successful check confirms
a running deployment.
---
Outside diff comments:
In `@web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsx`:
- Around line 77-79: Stop live timer updates for terminal jobs by passing
enabled: !isTerminalStatus to useLiveSeconds in DetailsPanel.tsx and enabled:
!isJobTerminal to useLiveSeconds in AgentEvaluationDetailRoute.tsx.
---
Nitpick comments:
In
`@web/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonl`:
- Around line 1-40: The dataset labels are grouped into contiguous phishing and
benign blocks, making partial evaluation produce a single-class sample. Reorder
the records in the dataset so phishing and benign examples are interleaved
throughout, while preserving each record’s content and label.
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md`:
- Around line 1-17: Update
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md
lines 1-17 to remain an Explanation page with prerequisites first; move the
editing instructions from lines 73-85 into a dedicated How-To page section with
tested Python SDK and CLI tab sets, and end with Next Steps. Apply the same
structure to
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md
lines 1-30 and 69-75, preserving its task-driven concepts and adding Next Steps
cross-links between both evaluation pages.
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.json`:
- Around line 554-556: Remove the unnecessary `| lower` filter from the numeric
`left_template` expressions comparing against `item.reference.phish_index`,
including both the shown template and the additional occurrence noted in the
review. Preserve the existing split, trim, and numeric comparison behavior.
In
`@web/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.ts`:
- Around line 40-100: The exported useDatasetEvalResults hook lacks an explicit
result contract. Define a named interface describing its returned scores, rows,
status flags, loading flags, and error fields, then annotate
useDatasetEvalResults with that interface as its return type while preserving
the existing return behavior.
In `@web/packages/studio/src/components/evaluation/MetricScoreChip.tsx`:
- Around line 8-11: Mark all listed React component prop interfaces as
immutable: in web/packages/studio/src/components/evaluation/MetricScoreChip.tsx
lines 8-11, mark label and value readonly; in
web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx lines
22-25, mark scores and emptyMessage readonly; in
web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx
lines 24-26, mark rows readonly; in
web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsx
lines 12-14, mark scores readonly; and in
web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsx
lines 18-20, mark tasks readonly. Use readonly array types for collection props.
In `@web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts`:
- Around line 146-165: Extend the parseEvalConfig tests with dataset-driven
cases: verify a valid dataset config parses, empty dataset metrics and a missing
prompt_template are rejected, and an omitted field_mapping defaults to null.
Also cover isDatasetEvalSpec and buildDatasetEvalRequestBody routing so dataset
submissions use the dataset endpoint and request shape.
In `@web/packages/studio/src/components/evaluation/submitEvaluationJob.ts`:
- Around line 20-26: Update buildEvalJobName so that when the normalized and
trimmed base is empty, it uses a fixed valid prefix before appending the suffix;
preserve the existing truncation and suffix behavior for non-empty bases.
In `@web/packages/studio/src/constants/sampleAgents.ts`:
- Around line 56-70: Make EvalConfigSample immutable by marking its
configuration fields readonly, including key, displayName, description,
configPath, datasetPath, and readmePath. Declare EVAL_CONFIG_SAMPLES as a
readonly array so consumers cannot mutate the registry, and verify existing
consumers do not rely on modifying these values.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c4a17702-7d96-4056-ad67-f235a7ce7b4a
📒 Files selected for processing (36)
web/packages/studio/public/sample-agents/email-security-analyst/agent.ymlweb/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonlweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.mdweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.jsonweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.mdweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.jsonweb/packages/studio/src/api/evaluation/agent-evaluations.tsweb/packages/studio/src/api/evaluation/eval-config-fileset.tsweb/packages/studio/src/api/evaluation/evaluator-jobs.tsweb/packages/studio/src/api/evaluation/utils.tsweb/packages/studio/src/components/dataViews/AgentEvaluationsDataView/index.tsxweb/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.test.tsxweb/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsxweb/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsxweb/packages/studio/src/components/evaluation/Jobs/DetailsPanel.test.tsxweb/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/ResultsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.tsweb/packages/studio/src/components/evaluation/MetricScoreChip.tsxweb/packages/studio/src/components/evaluation/SubmitEvaluationModal.tsxweb/packages/studio/src/components/evaluation/submitEvaluationJob.test.tsweb/packages/studio/src/components/evaluation/submitEvaluationJob.tsweb/packages/studio/src/components/evaluation/utils.tsweb/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/index.tsxweb/packages/studio/src/constants/links.tsweb/packages/studio/src/constants/sampleAgents.tsweb/packages/studio/src/routes/agents/AgentDetailRoute/index.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/LabeledSection.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.tsweb/packages/studio/src/routes/evaluation/EvaluationResultDetailsRoute/index.tsx
💤 Files with no reviewable changes (6)
- web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.test.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/LabeledSection.tsx
- web/packages/studio/src/components/evaluation/Jobs/ResultsPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
Contributor
|
nv-odrulea
force-pushed
the
od/studio-dataset-eval-feature
branch
from
August 4, 2026 23:36
7818d45 to
004b6db
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (6)
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md (1)
1-75: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSeparate the documentation types and add navigation sections.
This page mixes EXPLANATION content with HOW-TO/REFERENCE content. Keep this page as one Diataxis type, or move the configuration rules under
## Editing this configinto a separate page. Add a prerequisites section before## What is being evaluatedand a## Next Stepssection with links to the dataset-driven configuration and result documentation.As per coding guidelines, each documentation page must use one Diataxis quadrant, list prerequisites first, and include a
Next Stepssection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md` around lines 1 - 75, Restructure this README as a single Diataxis documentation type by moving the configuration rules currently under “Editing this config” to a separate page, if needed. Add a prerequisites section before “What is being evaluated,” and append a “Next Steps” section linking to the dataset-driven configuration and result documentation.Source: Coding guidelines
web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md (4)
81-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a
Next Stepssection.End the page with
## Next Stepsand cross-links to the task-driven sample README and the Run Evaluation/result documentation.As per coding guidelines, documentation pages must end with Next Steps and cross-links.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md` around lines 81 - 85, Add a final “Next Steps” section to the dataset-driven sample README, including links to the task-driven sample README and the Run Evaluation/results documentation. Ensure this section is the last content on the page and follows the repository’s existing documentation link conventions.Source: Coding guidelines
69-71: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove troubleshooting details behind progressive disclosure.
The near-zero-score diagnosis is troubleshooting content. Put these lines in a dropdown or separate troubleshooting page. Keep
Reading the resultsfocused on result interpretation.As per coding guidelines, use dropdowns for troubleshooting details.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md` around lines 69 - 71, Move the near-zero-score diagnosis currently under “Reading the results” into a troubleshooting dropdown, using the document’s existing progressive-disclosure pattern if available. Keep “Reading the results” focused on interpreting evaluation results while preserving the troubleshooting guidance unchanged.Source: Coding guidelines
7-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftKeep this page in one Diátaxis quadrant.
This page combines EXPLANATION, REFERENCE, and HOW-TO content. Keep it as one type. Move the field reference and editing procedure to separate pages, then link to them.
As per coding guidelines, each documentation page must fit one Diátaxis quadrant and use cross-links.
Also applies to: 19-59, 73-85
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md` around lines 7 - 17, Keep the dataset-driven README focused on explanatory content about the evaluation’s purpose and scope. Move field definitions and configuration-editing instructions into separate documentation pages, then add links from this README to those pages; apply the same separation to the sections covered by lines 19-59 and 73-85.Source: Coding guidelines
1-5: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd prerequisites before the overview.
Add
## Prerequisitesafter the title. List the required seeded fileset, co-located config and dataset, and supported Run Evaluation flow.As per coding guidelines, prerequisites must appear before other content.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md` around lines 1 - 5, Add a “Prerequisites” section immediately after the title in the README, before the existing overview text. Document the required seeded fileset, the co-located evaluation config and dataset, and the supported Run Evaluation flow, while preserving the remaining content.Source: Coding guidelines
web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx (1)
22-25: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake evaluation result props read-only.
Both components only read their input arrays.
web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx#L22-L25: markscoresandemptyMessagereadonly, and usereadonly EvalAggregateScoreRow[].web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx#L24-L26: markrowsreadonly, and usereadonly DatasetEvalRow[].Confirm callers do not mutate these arrays through the prop types. As per coding guidelines, “Use
readonlyfor immutable properties in TypeScript interfaces and types.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx` around lines 22 - 25, Update the props interfaces for EvalAggregateScoresTable in web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx:22-25 and DatasetEvalRowResultsPanel in web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx:24-26 to mark all input properties readonly and change scores and rows to readonly arrays of their existing row types; verify callers do not mutate these arrays through the prop types.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md`:
- Around line 63-67: Update the baseline metrics paragraph to include the run
date, dataset/config revision, and evaluator version alongside the 0.63 score
and error counts, using concrete reproducibility metadata; if those values are
unavailable, remove the fixed baseline metrics instead of leaving them
undocumented.
- Around line 3-5: Update the README’s lifecycle description, including the
seeding guidance around the dataset reference, to state that the eval config
artifact’s dataset reference is rewritten during seeding while the dataset and
README edits are not regenerated. Explicitly clarify which edits survive
reseeding and when the rewrite occurs.
In
`@web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx`:
- Around line 16-21: Validate and normalize the metrics data at the
parseRowScores boundary before it reaches scoreCells. Ensure every metric value
is an array of score objects, removing or rejecting invalid non-array values so
scoreCells can safely call .map without runtime errors.
---
Nitpick comments:
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md`:
- Around line 81-85: Add a final “Next Steps” section to the dataset-driven
sample README, including links to the task-driven sample README and the Run
Evaluation/results documentation. Ensure this section is the last content on the
page and follows the repository’s existing documentation link conventions.
- Around line 69-71: Move the near-zero-score diagnosis currently under “Reading
the results” into a troubleshooting dropdown, using the document’s existing
progressive-disclosure pattern if available. Keep “Reading the results” focused
on interpreting evaluation results while preserving the troubleshooting guidance
unchanged.
- Around line 7-17: Keep the dataset-driven README focused on explanatory
content about the evaluation’s purpose and scope. Move field definitions and
configuration-editing instructions into separate documentation pages, then add
links from this README to those pages; apply the same separation to the sections
covered by lines 19-59 and 73-85.
- Around line 1-5: Add a “Prerequisites” section immediately after the title in
the README, before the existing overview text. Document the required seeded
fileset, the co-located evaluation config and dataset, and the supported Run
Evaluation flow, while preserving the remaining content.
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md`:
- Around line 1-75: Restructure this README as a single Diataxis documentation
type by moving the configuration rules currently under “Editing this config” to
a separate page, if needed. Add a prerequisites section before “What is being
evaluated,” and append a “Next Steps” section linking to the dataset-driven
configuration and result documentation.
In `@web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx`:
- Around line 22-25: Update the props interfaces for EvalAggregateScoresTable in
web/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsx:22-25
and DatasetEvalRowResultsPanel in
web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx:24-26
to mark all input properties readonly and change scores and rows to readonly
arrays of their existing row types; verify callers do not mutate these arrays
through the prop types.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 73dbe452-29e7-4eb6-bfd2-75eec319adfe
📒 Files selected for processing (36)
web/packages/studio/public/sample-agents/email-security-analyst/agent.ymlweb/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonlweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.mdweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.jsonweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.mdweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.jsonweb/packages/studio/src/api/evaluation/agent-evaluations.tsweb/packages/studio/src/api/evaluation/eval-config-fileset.tsweb/packages/studio/src/api/evaluation/evaluator-jobs.tsweb/packages/studio/src/api/evaluation/utils.tsweb/packages/studio/src/components/dataViews/AgentEvaluationsDataView/index.tsxweb/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.test.tsxweb/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsxweb/packages/studio/src/components/evaluation/EvalAggregateScoresTable.tsxweb/packages/studio/src/components/evaluation/Jobs/DetailsPanel.test.tsxweb/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/ResultsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsxweb/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.tsweb/packages/studio/src/components/evaluation/MetricScoreChip.tsxweb/packages/studio/src/components/evaluation/SubmitEvaluationModal.tsxweb/packages/studio/src/components/evaluation/submitEvaluationJob.test.tsweb/packages/studio/src/components/evaluation/submitEvaluationJob.tsweb/packages/studio/src/components/evaluation/utils.tsweb/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/index.tsxweb/packages/studio/src/constants/links.tsweb/packages/studio/src/constants/sampleAgents.tsweb/packages/studio/src/routes/agents/AgentDetailRoute/index.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/LabeledSection.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.tsweb/packages/studio/src/routes/evaluation/EvaluationResultDetailsRoute/index.tsx
💤 Files with no reviewable changes (6)
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/LabeledSection.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsx
- web/packages/studio/src/components/evaluation/Jobs/ResultsPanel.tsx
- web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.test.tsx
🚧 Files skipped from review as they are similar to previous changes (26)
- web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.test.tsx
- web/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/index.tsx
- web/packages/studio/src/api/evaluation/evaluator-jobs.ts
- web/packages/studio/src/api/evaluation/eval-config-fileset.ts
- web/packages/studio/src/constants/sampleAgents.ts
- web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts
- web/packages/studio/src/components/evaluation/Jobs/datasetEval/useDatasetEvalResults.ts
- web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalScoresPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsx
- web/packages/studio/src/constants/links.ts
- web/packages/studio/src/components/evaluation/AgentEvalTaskResultsPanel.tsx
- web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.json
- web/packages/studio/src/components/dataViews/AgentEvaluationsDataView/index.tsx
- web/packages/studio/src/routes/evaluation/EvaluationResultDetailsRoute/index.tsx
- web/packages/studio/src/routes/agents/AgentDetailRoute/index.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsx
- web/packages/studio/src/components/evaluation/SubmitEvaluationModal.tsx
- web/packages/studio/src/api/evaluation/agent-evaluations.ts
- web/packages/studio/src/components/evaluation/Jobs/DetailsPanel.tsx
- web/packages/studio/public/sample-agents/email-security-analyst/dataset.jsonl
- web/packages/studio/src/components/evaluation/utils.ts
- web/packages/studio/src/api/evaluation/utils.ts
- web/packages/studio/src/components/evaluation/submitEvaluationJob.ts
- web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.json
- web/packages/studio/src/components/evaluation/MetricScoreChip.tsx
- web/packages/studio/public/sample-agents/email-security-analyst/agent.yml
nv-odrulea
changed the title
nv-odrulea
changed the title
…ests - CardSelect: new common component for card-style option selection used by the eval config picker in SubmitEvaluationModal - StudioDataViewToolbar: adds support for row actions - entityName.test.ts: removes stale secrets SDK assertions after the secrets schema dropped named regex/max exports Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Studio can now submit and display both evaluation paradigms against a deployed agent. Previously only task-driven (/agent-evaluate/jobs) was reachable; this adds dataset-driven (/evaluate/jobs). - SubmitEvaluationModal: detects eval config shape (tasks[] vs dataset+metrics) and routes to the correct evaluator endpoint. Fileset creation is now atomic (409 → user-facing collision error) with upload rollback on failure. - EvaluationResultDetailsRoute: new route for dataset-driven results showing aggregate scores and per-row results panels. - AgentEvaluationDetailRoute: Task-Driven chip on header; single cancel path. - DatasetEvalRowResultsPanel, DatasetEvalScoresPanel, useDatasetEvalResults: new components for dataset eval result rendering with failure/pending states. - eval-config-fileset.ts, evaluator-jobs.ts: unified job fetcher and hardened fileset helpers shared by both paradigms. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
…uation SubmitEvaluationModal, AgentEvalTaskResultsPanel, and its test were moved to src/components/evaluation/; delete the stale copies under routes/agents/AgentEvaluationsRoute/components/. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
…ation Test was renamed to submitEvaluationJob.test.ts and moved to components/evaluation/; delete the copy at the old route path. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
nv-odrulea
force-pushed
the
od/studio-dataset-eval-feature
branch
from
August 5, 2026 20:40
ddd95df to
a73d855
Compare
steramae-nvidia
approved these changes
Aug 5, 2026
nv-odrulea
added this pull request to the merge queue
Aug 5, 2026
Any commits made after this event will not be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Studio can now submit and display both evaluation paradigms against a deployed agent. Previously only task-driven (
/agent-evaluate/jobs) was reachable; this adds dataset-driven (/evaluate/jobs).components/evaluation/): detects eval config shape (tasks[]vsdataset+metrics) and routes to the correct evaluator endpoint. Fileset creation is now atomic (409 → collision error) with upload rollback on failure.You can now choose which type of eval
Eval job Detail screen (for dataset-driven) Loading
Eval job Detail screen (for dataset-driven) Done
Summary by CodeRabbit
New Features
Improvements