feat(evaluator): make a stored task runner-polymorphic via kind - #1071
Draft
SandyChapman wants to merge 1 commit into
Draft
feat(evaluator): make a stored task runner-polymorphic via kind#1071SandyChapman wants to merge 1 commit into
SandyChapman wants to merge 1 commit into
Conversation
Contributor
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest. It is a projection
of `task.toml`, which lives inside the archive, so a real change already
moves `archive_digest`; hashing the projection too would make our revision
history sensitive to Harbor's serialization.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a member whose kind the target cannot run, rather than
projecting it onto an agent-eval DTO. A Harbor task's content is a directory of
files, not fields — a pure projection would silently produce an empty task.
Mixed tasksets remain storable; the mismatch surfaces at submit as a 422.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
harbor-task-kind/schapman
branch
from
August 4, 2026 17:48
49d82b5 to
59c4f39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A task is an evaluation unit; how it runs is a property of the task, not a different kind of record. The target side already models this —
AgentRunnerTargetis akind-discriminated union ofcodex | fabric | harbor, with a comment saying "widen with more members as runners land."This makes the stored side match, so a user manages every evaluation unit in one place regardless of which runner executes it. It's the foundation for publishing Harbor tasks into NeMo.
What
Task content moves under a discriminated
spec:agent_evalintent,inputs,metrics,viewsharborarchive_ref(fileset),archive_digest,instruction, opaque HarborconfigNested rather than flattened with nullable per-kind fields. Flat would make every field optional with nothing enforcing a coherent set; nesting keeps each variant's required fields required, and the revision digest then covers the spec as a unit — so two kinds with coincidentally similar metadata can't collide on content. There's a test for that.
Named
…Definition, not…Spec.AgentEvalTaskSpecalready names the runtime task DTO injobs.agent_spec; two models sharing a class name across modules break OpenAPI generation with a schema-name collision. (Found by regenerating the spec — see below.)One fileset per Harbor task, so a task shared by several tasksets is stored once.
archive_refis shape-validated, so a malformed reference is rejected at publish rather than surfacing as a download failure mid-run — matching how a metric reference is checked when a task is stored.Taskset expansion is now runner-aware
resolve_taskset_refpreviously projected every member onto an agent-eval DTO. A Harbor task's content is a directory of files, not fields, so that projection would silently produce a task with no intent and no metrics — an evaluation that runs and scores nothing.It now raises
UnsupportedTaskKindError, which the job submit path surfaces as a 422 before the run starts. Mixed tasksets remain storable; the mismatch is caught at submit, where the target is known.Deliberate, with a note in the code
Harbor's
configis inside the revision digest — it's task content, and two tasks differing only in verifier or environment are different tasks. The cost is that the digest is sensitive to Harbor's serialization: if a Harbor release reorders keys, republishing an unchanged task cuts a new revision. Accepted for now; documented at the field with the trigger for revisiting.Verification
RUN_AGENT_EVAL_INTEGRATION=1)tools/lint/lint-all.sh— 13/13make refresh-openapi— regenerated;make docs-check— 0 errorsDocs updated: examples, the
TaskInputfield table, and a new task-kinds section.Note for reviewers
An earlier revision of this branch passed unit tests but broke the integration suite and failed OpenAPI generation. Both are fixed; flagging it because the schema-name collision in particular is the kind of thing that only shows up when the generator runs.
🤖 Generated with Claude Code