feat(evaluator): add content-addressed revisions for tasks and tasksets

docs/evaluator/manage-tasks-tasksets.mdx

@@ -18,15 +18,21 @@ metrics inline every time.
 | Concept | What it is | Members |
 |---------|------------|---------|
 | **Task** | A reusable agent-eval unit: `intent`, `inputs`, and the `metrics` that score it. | References the metrics that score it. |
-| **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`. Membership is a **set** — order is not significant and duplicate references are rejected. |
+| **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`, each pinned to an exact revision. Membership is a **set** — order is not significant and duplicate references are rejected. |
+| **Revision** | An immutable published snapshot of a task's or taskset's content, addressed by a content digest. | Belongs to the task or taskset it snapshots. |
 
 Both are addressed by `workspace/name`. Names are unique within a workspace, limited to 255
 characters, and must match `^[\w\-\.]+$`.
 
+Every stored task and taskset is **versioned**. Creating one publishes revision 1; replacing its
+content publishes the next revision. Earlier revisions stay readable for as long as the task or
+taskset exists — deleting it removes its revisions with it — which is what lets an evaluation be
+re-run against exactly the content it ran against the first time.
+
 <Note>
-Tasks and tasksets support **create, retrieve, list, and delete** — there is no update. To change a
-stored task or taskset, delete it and create a new one, or store a new version under a different
-name.
+Publishing is **idempotent**. Replacing a task with content identical to its current revision
+publishes nothing and returns the existing revision — so a pipeline can re-submit the same
+definition freely without accumulating versions.
 </Note>
 
 ## Initialize the SDK
@@ -89,6 +95,7 @@ print(stored.id, stored.metrics)
 | `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. |
 | `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. |
 | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
+| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |
 
 <Note>
 A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored
@@ -99,26 +106,107 @@ why `stored.metrics` always comes back as a list of `MetricRef` references.
 ### Retrieve, list, and delete
 
 ```python
-# Retrieve one task by name
+# Retrieve one task by name (its current content)
 task = tasks.retrieve("capital-of-france")
+print(task.revision, task.tags)  # e.g. 1 {'latest': 1}
 
 # List tasks in the workspace (paginated)
 page = tasks.list(page=1, page_size=100, sort="-created_at")
 for item in page.data:
     print(item.name, item.intent)
 
-# Delete a task
+# Delete a task (this also removes all of its revisions)
 tasks.delete("capital-of-france")
 ```
 
 `sort` accepts `name`, `created_at`, or `updated_at`, each optionally prefixed with `-` for
 descending order.
 
+## Revisions
+
+### Publish a new revision
+
+Use `replace` to publish new content. It creates the task if it does not exist, so a publisher needs
+no existence check.
+
+```python
+revised_task = TaskInput(
+    intent="Answer the user's geography question with the capital city.",
+    inputs=TaskInputs(instruction="Name the capital city of France."),
+    metrics=[MetricRef("default/answer-exact-match")],
+    metadata=[MetadataItem(key="suite", value="geography")],
+)
+
+updated = tasks.replace("capital-of-france", task=revised_task)
+print(updated.revision)  # 2
+```
+
+Submitting content identical to the current revision publishes nothing and returns the existing
+revision — but any tags in the body are still applied, which is how you tag a revision after the
+fact.
+
+### List revisions
+
+Each entry carries the `content_hash` used to pin a reference.
+
+```python
+page = tasks.list_revisions("capital-of-france")
+for revision in page.data:      # newest first
+    print(revision.revision, revision.content_hash, revision.tags)
+```
+
+### Read a specific revision
+
+Pass a content digest or a tag. This returns the content **as published**, not the current content.
+
+```python
+# Revisions come back newest-first and paginated, so index by ordinal rather than by position —
+# `data[-1]` is only the oldest entry on the page you happen to have fetched.
+page = tasks.list_revisions("capital-of-france")
+digest = next(revision.content_hash for revision in page.data if revision.revision == 1)
+
+original = tasks.retrieve("capital-of-france", revision=digest)  # revision 1, as published
+current = tasks.retrieve("capital-of-france")                    # revision 2, the current content
+
+assert original.revision == 1 and current.revision == 2
+assert original.inputs.instruction != current.inputs.instruction
+```
+
+### Tag a revision
+
+A tag is a mutable pointer to a revision — useful for marking one as reviewed or approved after it
+has been evaluated. Read it back with `tag=`, the counterpart to `revision=`:
+
+```python
+tasks.tag("capital-of-france", tag="blessed", revision=digest)
+
+blessed = tasks.retrieve("capital-of-france", tag="blessed")
+```
+
+`revision=` takes a content digest and `tag=` takes a tag name. They select the same thing two ways,
+so pass one or the other — passing both raises `ValueError`.
+
+<Note>
+A tag names exactly one revision. Re-tagging moves the pointer rather than adding a second one, so
+`retrieve(tag=...)` always resolves to a single revision — there is no way for two revisions to
+share a tag.
+
+`latest` is managed automatically and always names the most recently published revision; it cannot be
+moved by hand. A tag name may not be empty, and may not look like a content digest (64 hexadecimal
+characters) — such a tag could be stored but never resolved, because a digest-shaped reference is
+looked up as a digest rather than as a tag.
+</Note>
+
 ## Manage Tasksets
 
 A taskset references existing tasks by `workspace/name`. All referenced tasks must already exist when
 the taskset is created; a missing or duplicate reference is rejected.
 
+Member references are **resolved to an exact revision when the taskset is stored**. You may submit a
+bare name (`capital-of-france`), a tag (`capital-of-france#latest`), or a digest — what gets stored
+is always `workspace/name#<digest>`. This is why a stored taskset keeps naming the same content even
+after a member task publishes something new, and it is what makes a suite reproducible.
+
 ```python
 from nemo_evaluator.api.schemas import TaskRef, TasksetInput
 
@@ -132,15 +220,17 @@ taskset = TasksetInput(
 
 stored = tasksets.create("geography-suite", taskset=taskset)
 print(stored.tasks)
+# ['default/capital-of-france#a1b2...', 'default/capital-of-japan#c3d4...']
 ```
 
 ### `TasksetInput` fields
 
 | Field | Type | Required | Description |
 |-------|------|----------|-------------|
 | `description` | `str` | No | Human-readable description of the grouping. |
-| `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace). Set semantics — duplicates rejected. |
+| `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace), optionally pinned with `#<tag-or-digest>`. Each is resolved to an exact digest when stored. Set semantics — duplicates rejected. |
 | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
+| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |
 
 ### Retrieve, list, and delete
 
@@ -154,6 +244,26 @@ for item in page.data:
 tasksets.delete("geography-suite")
 ```
 
+Tasksets carry the same revision surface as tasks — `replace`, `list_revisions`, `tag`, and
+`retrieve(revision=...)` / `retrieve(tag=...)`:
+
+```python
+# Re-resolving membership after a member task published new content cuts a new revision.
+tasksets.replace("geography-suite", taskset=taskset)
+
+for revision in tasksets.list_revisions("geography-suite").data:
+    print(revision.revision, revision.content_hash)
+```
+
+<Note>
+Re-submitting the *same* member names can still publish a new revision. Members are re-resolved on
+every write, so if a member task published in the meantime the grouping now names different content
+and genuinely differs. A taskset's identity is the exact revisions it names, not the names alone.
+
+Member *order*, by contrast, is not part of that identity: membership is a set, so it is stored in a
+canonical order and reordering the same members publishes nothing.
+</Note>
+
 Deleting a taskset does not delete its member tasks — a taskset only holds references.
 
 ## Run an evaluation over a taskset
@@ -178,15 +288,44 @@ input_spec = AgentEvalInputSpec(
 )
 ```
 
-When the job runs, the taskset reference is resolved: its member tasks are loaded, and each task's
-stored metric references are hydrated into runnable metrics — exactly as if you had inlined them. The
-same spec is submitted as the agent-evaluate job input; see
+When the job runs, the taskset reference is resolved like this:
+
+- Each member is loaded at the **revision pinned in the taskset**, not the task's current tip.
+- Metric references on those members are hydrated into runnable metrics, the same as for inline tasks.
+- Re-running the same taskset therefore evaluates the same content, even if a member has been
+  republished since.
+
+```text
+# What is stored on the taskset (digests fixed at create/replace time)
+geography-suite@1
+  └─ tasks:
+       - default/capital-of-france#aaa111...   ← revision 1 content
+       - default/capital-of-japan#bbb222...
+
+# Later: the member task publishes new content
+capital-of-france tip → revision 2 (#ccc333...)
+
+# Re-grade still uses the taskset pins, not the tip
+JobRun A  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...
+JobRun B  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...   ← same content
+```
+
+To evaluate the *new* task content, publish a new taskset revision (for example
+`tasksets.replace(...)`) so membership is re-resolved to the newer digests.
+
+Submit this spec as the agent-evaluate job input; see
 [Agent Evaluation](/documentation/evaluate-models/agent-eval) for the full run, target, and
 results flow.
 
 The inline form remains available for one-off tasks — swap `tasks=TasksetRef(...)` for
 `tasks=[AgentEvalTaskInput(...), ...]`.
 
+<Note>
+A `TasksetRef` names the taskset's current revision; it cannot yet carry a `#<tag-or-digest>`
+fragment of its own. Member content is pinned, so a re-run always grades the same task content — but
+if the taskset itself is replaced, a re-submitted spec expands the new membership.
+</Note>
+
 <Note>
 Stored tasks carry no grader-only `reference` (held-out ground truth): that field lives only on inline
 `AgentEvalTaskInput`. Taskset-driven tasks therefore run with an empty `reference`, so use a taskset
@@ -227,17 +366,42 @@ The SDK resources are a thin client over the Evaluator plugin REST API, mounted
 | Method | Path | Description |
 |--------|------|-------------|
 | `GET` | `/tasks` | List tasks (paginated). |
-| `POST` | `/tasks/{name}` | Create a task. |
-| `GET` | `/tasks/{name}` | Retrieve a task. |
-| `DELETE` | `/tasks/{name}` | Delete a task. |
+| `POST` | `/tasks/{name}` | Create a task and publish revision 1. |
+| `PUT` | `/tasks/{name}` | Replace a task's content and publish; creates it if absent. |
+| `GET` | `/tasks/{name}` | Retrieve a task's current content. |
+| `GET` | `/tasks/{name}/revisions` | List published revisions (paginated, newest first). |
+| `GET` | `/tasks/{name}/revisions/{revision}` | Retrieve content as of a digest or tag. |
+| `PUT` | `/tasks/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. |
+| `DELETE` | `/tasks/{name}` | Delete a task and all of its revisions. |
 | `GET` | `/tasksets` | List tasksets (paginated). |
-| `POST` | `/tasksets/{name}` | Create a taskset. |
-| `GET` | `/tasksets/{name}` | Retrieve a taskset. |
-| `DELETE` | `/tasksets/{name}` | Delete a taskset. |
-
-Creating a name that already exists returns `409`. An invalid metric reference (task) or a missing or
-duplicate task reference (taskset) returns `422`. Retrieving or deleting a name that does not exist
-returns `404`.
+| `POST` | `/tasksets/{name}` | Create a taskset and publish revision 1. |
+| `PUT` | `/tasksets/{name}` | Replace a taskset's membership and publish; creates it if absent. |
+| `GET` | `/tasksets/{name}` | Retrieve a taskset's current membership. |
+| `GET` | `/tasksets/{name}/revisions` | List published revisions (paginated, newest first). |
+| `GET` | `/tasksets/{name}/revisions/{revision}` | Retrieve membership as of a digest or tag. |
+| `PUT` | `/tasksets/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. |
+| `DELETE` | `/tasksets/{name}` | Delete a taskset and all of its revisions. |
+
+### Edge Cases
+
+`PUT` distinguishes its two outcomes by status: **`201`** when a new revision was published, and
+**`200`** when the submitted content was already the current revision and nothing was cut. The rest:
+
+| Case | Status | Behavior |
+|------|--------|----------|
+| `PUT` with new content | `201` | A revision is published and `latest` moves to it. |
+| `PUT` with content identical to the current revision | `200` | Nothing is published; any tags in the body are still applied. |
+| `PUT` with taskset members reordered | `200` | Membership is a set, so a reordering is not a content change. |
+| `PUT` reverting to *older* content | `201` | The record genuinely changed, so it publishes a new ordinal rather than reusing the old one. |
+| `PUT` re-resolving a member that has since republished | `201` | A taskset's identity is the exact revisions it names. |
+| Two identical `PUT`s racing | `201` + `200` | One publishes; the other adopts its revision rather than cutting a duplicate. |
+| `POST` on an existing name | `409` | Create is strict; use `PUT` to upsert. |
+| `PUT` losing a race with a concurrent write | `409` | Retry against the current state. |
+| Invalid metric reference (task) | `422` | Rejected at validation. |
+| Missing or duplicate task reference (taskset) | `422` | Members must exist, and must resolve to distinct tasks. |
+| Reserved or malformed tag name | `422` | `latest` cannot be moved by hand; a digest-shaped tag is refused. |
+| Retrieving or deleting an unknown name or revision | `404` | Applies to both records and revisions. |
+| `DELETE` on a task a taskset pins | `204` | Not prevented; the taskset's reference dangles and fails on read. |
 
 ## Related Topics