Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 35 additions & 8 deletions docs/evaluator/manage-tasks-tasksets.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,29 +72,56 @@ Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `nam
the task's workspace). The service returns the stored `Task`.

```python
from nemo_evaluator.api.schemas import MetadataItem, MetricRef, TaskInput, TaskInputs
from nemo_evaluator.api.schemas import AgentEvalTaskDefinitionInput, MetadataItem, MetricRef, TaskInput, TaskInputs

task = TaskInput(
intent="Answer the user's geography question with the capital city.",
inputs=TaskInputs(instruction="What is the capital of France?"),
metrics=[MetricRef("default/answer-exact-match")],
spec=AgentEvalTaskDefinitionInput(
intent="Answer the user's geography question with the capital city.",
inputs=TaskInputs(instruction="What is the capital of France?"),
metrics=[MetricRef("default/answer-exact-match")],
),
metadata=[MetadataItem(key="suite", value="geography")],
)

stored = tasks.create("capital-of-france", task=task)
print(stored.id, stored.metrics)
print(stored.id, stored.spec.metrics)
```

### `TaskInput` fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `spec` | `TaskSpecInput` | Yes | The task's content, discriminated by `kind` — see below. |
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |

### Task kinds

A task is an evaluation unit; its `kind` says which runner executes it. Both kinds are stored as the
same record type, so a taskset can group them and you manage every evaluation unit in one place.

`AgentEvalTaskDefinitionInput` (`kind="agent_eval"`) — scored by platform metrics:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `intent` | `str` | Yes | Human-readable description of the desired agent behavior. |
| `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. |
| `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. |
| `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. |
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |

`HarborTaskDefinition` (`kind="harbor"`) — a Harbor task, scored by Harbor's own reward:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `archive_ref` | `str` | Yes | Files reference to the task's packaged directory (`workspace/fileset#path`). One fileset per task, so a task shared by several tasksets is stored once. |
| `archive_digest` | `str` | Yes | Content hash Harbor computed over the task directory. |
| `instruction` | `str` | No | The task's instruction text, when it has one. |
| `config` | `dict` | No | Harbor's own task configuration (verifier, agent, environment, steps), stored as published. |

<Note>
A run has one target, so it executes one kind. A taskset may group both, but submitting it against a
target whose runner cannot execute a member is rejected with `422` before the run starts.
</Note>

<Note>
A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored
Expand All @@ -107,7 +134,7 @@ why `stored.metrics` always comes back as a list of `MetricRef` references.
```python
# Retrieve one task by name (its current content)
task = tasks.retrieve("capital-of-france")
print(task.revision, task.tags) # e.g. 1 {'latest': 1}
print(task.spec.kind, task.revision, task.tags) # e.g. agent_eval 1 {'latest': 1}

# List tasks in the workspace (paginated)
page = tasks.list(page=1, page_size=100, sort="-created_at")
Expand Down
216 changes: 167 additions & 49 deletions plugins/nemo-evaluator/openapi/openapi.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading