diff --git a/docs/agents/insight-driven-optimization.mdx b/docs/agents/insight-driven-optimization.mdx index e902cbc3dc..d6fe1cc21e 100644 --- a/docs/agents/insight-driven-optimization.mdx +++ b/docs/agents/insight-driven-optimization.mdx @@ -1,7 +1,8 @@ --- title: "Insight-Driven Optimization" -description: "" +description: "Turn agent telemetry into evidence-backed Insights, experiments, evaluation suites, and validated changes." --- + Insight-driven optimization automates the agent improvement loop. It turns @@ -19,13 +20,13 @@ The system introduces one first-class entity and a family of agents that operate on it: - The **Insight** is a named, persistent description of a recurring problem in - the agent under test (AUT), backed by the traces that evidence it. The + the agent under test (AUT), backed by the traces that provide evidence. The Insight is the unit of action across the whole loop. An experiment or an evaluation-suite change can always be traced back to the Insight that motivated it. - The **Analyst** turns raw traces into actionable Insights. - The **Experimenter** turns a single Insight into an empirically validated - pull request. + pull request (PR). - The **Eval Author** builds and extends evaluation suites in response to real-world usage. @@ -40,18 +41,18 @@ want to automate portions of the agent improvement loop. - A NeMo Platform deployment (local or remote) with Intake enabled, so the Analyst has traces to read. -- The optimization plugins installed in the same Python environment as the - NeMo CLI. From a source checkout, `uv sync` installs all three through the - default `enabled-plugins` group. The Experimenter and Eval Author - plugins require Python earlier than 3.14. +- The optimization plugins installed in the same Python environment as the NeMo command-line + interface (CLI). From a source checkout, `uv sync` installs all three through the default + `enabled-plugins` group. The Experimenter and Eval Author plugins require Python earlier than 3.14. - Access to the code base for the agent under test. The Experimenter needs a change surface (source, prompts, tool definitions, model selection, or - runtime config) and a way to run or evaluate the agent reproducibly. + runtime configuration) and a way to run or evaluate the agent reproducibly. - Traces in Intake for the agent under test. The Analyst diagnoses only what it can observe, so an agent with no telemetry cannot be analyzed. -- Model access for the optimization agents themselves. The Analyst and the - Experimenter each drive their own LLMs, which is separate from the model - access your AUT needs at runtime. You can run any LLM you want here. +- Model access for the optimization agents themselves. The Analyst uses Claude Opus 4.8 through the + NVIDIA Inference Gateway. The Experimenter and Eval Author use configurable large language models + (LLMs) through an OpenAI-compatible endpoint. This access is separate from the model access that + your AUT needs at runtime. - Train and validation datasets in a Harbor-compatible layout, plus a task template and an output directory for artifacts. This is what the Eval Author works from and what the Experimenter uses to validate changes. @@ -59,8 +60,8 @@ want to automate portions of the agent improvement loop. To install only one side of the loop, use the convenience groups: ```bash -uv sync --group insights # Analyst only -uv sync --group experimentalist # Experimenter and Eval Author +uv sync --only-group insights # Analyst only +uv sync --only-group experimentalist # Experimenter and Eval Author ``` ### Dependencies @@ -83,9 +84,9 @@ dependencies: - All Insights service endpoints are workspace-scoped under `/apis/insights/v2/workspaces/{workspace}/` and depend on platform auth. The service checks workspace access before reading or writing. -- The plugins register themselves with the platform through entry points - (CLI, service, controller, job, SDK, skills). For a fully local platform, - restart `nemo services run` after installing so the platform discovers them. +- The plugins register themselves with the platform through CLI, service, controller, job, software + development kit (SDK), and skill entry points. For a fully local platform, restart + `nemo services run` after installation so the platform discovers them. ### The Shared Profile @@ -140,34 +141,34 @@ Use the optimization agents when you want to: - **Keep evaluation suites relevant.** Grow datasets and metrics in response to the failures your agent actually exhibits. -Production traffic is the best fuel. New agents can start with staged replays -or generated cases and improve coverage as real traffic lands. +Production traffic provides useful evidence. New agents can start with staged replays +or generated cases and improve coverage as production traffic arrives. ### Core Concepts and Data Model -**The Insight.** +#### The Insight -The first-class entity of the loop. An Insight is a -persistent, named description of a recurring problem, stored in the platform -entity store with these fields: +An Insight is the first-class entity of the loop. It is a persistent, named description of a +recurring problem that is stored in the platform entity store with these fields: | Field | Type | Meaning | |-------|------|---------| | `title` | string | A short, human-readable sentence naming the core issue common to the linked traces. | | `description` | string | A paragraph describing the problem statement and the situations in which the failure pattern is observed. | | `agent` | string | The registered agent name the Insight is about. | -| `status` | enum | `open` (default), `resolved`, or `deleted`. An Insight starts open; you resolve it when fixed, or delete it if it isn't a real problem. | -| `trace_refs` | list[string] | Intake trace IDs the Analyst cited as evidence. Drives the evidence view in the UI and lets the loop find similar traces. | +| `status` | enum | `open` (default), `resolved`, or `deleted`. An Insight starts open; you resolve it when fixed, or delete it if it is not a real problem. | +| `trace_refs` | list[string] | Intake trace identifiers (IDs) the Analyst cited as evidence. Drives the evidence view in the user interface (UI) and lets the loop find similar traces. | The store assigns `id`, `created_at`, and `updated_at`. The Analyst aims for at least three representative traces as evidence before filing a new Insight, and when it finds more evidence for an existing Insight it *appends* trace refs rather than restating the problem. -**Insight persistence: platform versus local file.** +#### Insight Persistence: Platform Versus Local File When no profile is -discovered, the Analyst reads and writes Insights through the Insights API. +discovered, the Analyst reads and writes Insights through the Insights application programming +interface (API). When a profile governs the run, the Analyst reads and writes the shared local file at `/.nemo-optimizer/insights.yaml` instead, which is the same default the Experimenter reads. Pass `--insights-file-output ` @@ -177,16 +178,16 @@ deployments that host Intake data but do not have the Insights plugin installed. Each run merges into the file (de-duplicating trace refs) rather than overwriting it. -**Telemetry hierarchy.** +#### Telemetry Hierarchy The loop inherits Intake's model: a span is one timed operation (LLM call, tool call, and so on), a trace is one end-to-end run, and a session groups related traces. Insight evidence is cited at the trace level. -**Experiment entities.** +#### Experiment Entities An optimization run is tracked as an `ExperimentRun` -(agent, insight, config snapshot, status, rounds completed, winner, summary). +(agent, insight, configuration snapshot, status, rounds completed, winner, summary). ## The Agents @@ -195,16 +196,16 @@ An optimization run is tracked as an `ExperimentRun` The Analyst reads telemetry from Intake and emits Insights. It runs as a single reasoning agent with a set of read-only tools over Intake: -- `fetch_spans` — survey spans, either grouped (for example by `session_id`, +- `fetch_spans`: Survey spans, either grouped (for example by `session_id`, to fan out across many runs) or flat (to drill into one session). Filters include agent, status, span kind, model, provider, tool name, dataset, and time range. -- `get_span` — fetch a single span by ID. -- `fetch_scores` — read evaluator results (verifier and judge outputs) +- `get_span`: Fetch a single span by ID. +- `fetch_scores`: Read evaluator results (verifier and judge outputs) attached to a span. -- `fetch_annotations` and `get_annotation` — read feedback, labels, notes, and +- `fetch_annotations` and `get_annotation`: Read feedback, labels, notes, and metadata. Negative feedback is the strongest starting signal. -- `list_insights` — read existing Insights so findings are de-duplicated +- `list_insights`: Read existing Insights so findings are de-duplicated against what is already filed. Its method is to survey sessions broadly, gather evidence (starting from @@ -219,21 +220,21 @@ intended behavior. The Experimenter turns an Insight into an empirically validated candidate. Internally it is an evolutionary optimization loop that runs in rounds: -1. **Baseline** — build the baseline agent (`agent-0`) and evaluate it on the +1. **Baseline**: Build the baseline agent (`agent-0`) and evaluate it on the validation split; build an initial goal tree (a weighted capability rubric used for trajectory scoring). -2. **Analyze** — read the target Insight and perform root cause analysis. -3. **Propose** — generate a small number of candidate improvements targeting +2. **Analyze**: Read the target Insight and perform root cause analysis. +3. **Propose**: Generate a small number of candidate improvements targeting those root causes, each tagged with an optimization type. -4. **Implement** — a coding agent applies each proposed change to a copy of - the agent and runs an integration smoke test with a bounded repair loop. -5. **Validate** — score new candidates on the held-out validation split, +4. **Implement**: A coding agent applies each proposed change to a copy of + the agent and runs a basic integration test with a bounded repair loop. +5. **Validate**: Score new candidates on the held-out validation split, optionally adding a qualitative trajectory score against the goal tree. -6. **Select and continue** — keep a diverse Pareto front of survivors and +6. **Select and continue**: Keep a diverse Pareto front of survivors and iterate until a budget or convergence condition is met, then pick the winner. -**Insight mode.** +#### Insight Mode The Experimenter starts from a single Insight. By default it reads the local `.nemo-optimizer/insights.yaml` beside the profile; @@ -243,12 +244,12 @@ title, or zero-based index. The agent referenced by the Insight is used unless `--agent` overrides it. The Eval Author step builds an Insight-specific evaluation suite before optimization begins, which requires a task template. -**Dataset mode.** +#### Dataset Mode Pass `--no-insight` to bypass both an explicit Insight and the profile-local default and optimize directly against a dataset. -**Train and validation isolation.** +#### Train and Validation Isolation The validation split is *hidden* during candidate generation. At run start the validation data is moved into a @@ -257,7 +258,7 @@ blocks reads of that path. Candidates therefore cannot be tuned against the data they are later scored on. Validation data is temporarily restored only when validation scoring runs. -**Evaluation and rewards.** +#### Evaluation and Rewards Candidates are scored with the Harbor evaluator. Each trial yields metrics from the verifier, and the aggregate reward is the @@ -266,22 +267,26 @@ to `[0.0, 1.0]`, where 1.0 is perfect. Live and production datasets often carry no verifiable reward, so validation on those relies on curated metrics and trajectory scoring rather than a ground-truth verifier. -**Output.** +#### Output -Everything lands under `/eval-and-optimize/`: the +Output is written under `/eval-and-optimize/`: the run record, per-candidate agent code and metadata, per-round analysis and goal -trees, and evaluator results. When configured with a git source, the -Experimenter can archive candidate branches and open a draft PR or MR for -the winning candidate against the baseline ref. +trees, and evaluator results. When configured with a Git source, the +Experimenter can archive candidate branches and open a draft PR or merge request (MR) for +the winning candidate against the baseline reference. ### Eval Author -The Eval Author builds and maintains the evaluation suites the loop depends -on. Given an Insight and its evidence traces, it creates an Insight-specific -evaluation suite that can be used to validate optimized candidates aimed at -resolving that Insight. It runs as a library-only plugin that the -Experimenter invokes in Insight mode; it is configured through the -`eval_author` section of the experiment config. +The Eval Author builds and maintains the evaluation suites that the loop uses. +Given an Insight and its evidence traces, it creates an Insight-specific +evaluation suite for validating candidates that address the Insight. The +Experimenter invokes the Eval Author workflow in Insight mode and reads the +`eval_author` section of the experiment configuration. + +The plugin also exposes the canonical `nemo agents eval-author` command +namespace with `discover`, `audit`, `propose`, `run`, and `doctor` verbs. These +standalone verbs are currently scaffolding and exit with a nonzero status until +their implementations are available. ## Get Started @@ -294,12 +299,14 @@ deployment. ```bash export NMP_BASE_URL=http://localhost:8080 export WORKSPACE=default -export AGENT= # Must have traces in Intake +export NMP_STUDIO_URL="$NMP_BASE_URL/studio" +export NMP_ACCESS_TOKEN="$(nemo auth token)" +export AGENT=your-agent-name # This agent must have traces in Intake. -# The Analyst's model reads a gateway virtual key: +# The Analyst uses Claude Opus 4.8 through NVIDIA Inference Gateway. export INFERENCE_API_KEY=sk-... -# The Experimenter and Eval Author drive their own endpoint: +# The Experimenter and Eval Author use an OpenAI-compatible endpoint. export NEMO_EXPERIMENTALIST_API_BASE=https://inference-api.nvidia.com/v1 export NEMO_EXPERIMENTALIST_API_KEY=sk-... export NEMO_EXPERIMENTALIST_MODELS_SMART=openai/openai/openai/gpt-5.6-sol @@ -307,33 +314,38 @@ export NEMO_EXPERIMENTALIST_MODELS_MID=openai/openai/openai/gpt-5.6-terra export NEMO_EXPERIMENTALIST_MODELS_FAST=openai/openai/openai/gpt-5.6-luna ``` +Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path. +For an authenticated remote deployment, log in against that base URL before +requesting the token. For local development with authentication disabled, omit +the `NMP_ACCESS_TOKEN` command and the `Authorization` header in the API example. + Confirm the plugins are installed and discoverable: ```bash -nemo insights --help -nemo experimentalist --help +nemo agents analyst --help +nemo agents experimentalist --help +nemo agents eval-author --help ``` From an agent directory with an `optimizer.yaml` profile, check that the effective inputs and credentials resolve: ```bash -nemo insights doctor -nemo experimentalist doctor +nemo agents analyst doctor +nemo agents experimentalist doctor ``` -**Write traces to Intake.** +### Send Traces to Intake -The Analyst depends on Intake as its -observability store, so your agent must have traces in Intake before it can -generate Insights. +The Analyst depends on Intake as its observability store. Your agent must have traces in Intake +before it can generate Insights. ### Generate Insights Run the Analyst against the target agent's traces: ```bash -nemo insights analyze \ +nemo agents analyst run \ --agent "$AGENT" \ --workspace "$WORKSPACE" \ --base-url "$NMP_BASE_URL" @@ -341,37 +353,49 @@ nemo insights analyze \ Useful flags: -- `--agent-spec AGENT-SPEC.md` — append a spec so the Analyst can flag +- `--agent-spec AGENT-SPEC.md`: Append a spec so the Analyst can flag divergence from intended behavior. -- `--insights-file-output tmp/insights.yaml` — read and write Insights from a +- `--insights-file-output tmp/insights.yaml`: Read and write Insights from a specific local YAML file. Trace reads still hit `--base-url`. -- `--verbose` or `-v` — stream the Analyst's tool calls and reasoning to - stderr. +- `--verbose` or `-v`: Stream the Analyst's tool calls and reasoning to + standard error. -**Confirm it worked.** +### Confirm It Worked -List the Insights the Analyst filed through the API: +The persistence target depends on whether the Analyst discovered an +`optimizer.yaml` profile. + +For a profile-based run, confirm that the local Insights file exists and +contains the generated records: ```bash -curl "$NMP_BASE_URL/apis/insights/v2/workspaces/$WORKSPACE/insights?agent=$AGENT&page=1&page_size=20" +test -s .nemo-optimizer/insights.yaml +sed -n '1,120p' .nemo-optimizer/insights.yaml ``` -Or view them in Studio at -`http://localhost:8080/studio/workspaces/default/optimizer`. +If no profile was discovered and you did not pass `--insights-file-output`, +list the Insights stored through the API: -**What good looks like:** +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/insights/v2/workspaces/$WORKSPACE/insights?agent=$AGENT&page=1&page_size=20" +``` -at least one Insight appears for the agent, each with -a clear `title`, an actionable `description`, and `trace_refs` pointing at real -Intake traces. +API-stored Insights also appear in Studio at +`$NMP_STUDIO_URL/workspaces/$WORKSPACE/optimizer`. Profile-based Insights do +not appear there because they are stored in the local YAML file. + +A successful result contains at least one Insight for the agent. Each Insight +has a clear `title`, an actionable `description`, and `trace_refs` that point +to Intake traces. ### Run an Experiment -**Convert evaluations to Harbor.** +#### Convert Evaluations to Harbor -The Experimenter validates its work by -running evaluations against your agent so that it only keeps empirically -validated optimizations. The only supported evaluation backend today is +The Experimenter validates its work by running evaluations against your agent. It only keeps +empirically validated optimizations. The only supported evaluation backend today is [Harbor](https://www.harborframework.com/). Before you can run the Experimenter, express your agent's evaluation as @@ -382,13 +406,13 @@ this: ```text / - task.toml # resources, timeouts, and env passthrough (e.g. INFERENCE_API_KEY) - instruction.md # the prompt given to the agent, incl. where to write output + task.toml # resources, timeouts, and environment variable passthrough + instruction.md # the prompt given to the agent, including where to write output environment/ Dockerfile # a container that can run your agent (and the verifier) tests/ - test.sh # verifier entry point -> writes /logs/verifier/reward.json - score.py # your scorer(s) — e.g. LLM-as-judge + test.sh # verifier entry point that writes /logs/verifier/reward.json + score.py # one or more scorers, such as an LLM-based judge ``` The Experimenter relies on a classic train/test split for validation, so @@ -401,7 +425,7 @@ Two things to know about the Harbor integration: `WrappedAgent(BaseAgent)`. Its `setup()` uploads your agent directory to `/app` and runs `uv sync` on it. Its `run()` executes the agent with the task instruction as `--prompt`. -- *How eval scores reach Intake.* The Analyst and the Experimenter both +- *How evaluation scores reach Intake.* The Analyst and the Experimenter both read scores from Intake, but neither Harbor nor the verifier pushes scores there; the verifier's reward is written to local disk by default. The Experimenter uploads scores to Intake after the Harbor run completes, and @@ -409,40 +433,40 @@ Two things to know about the Harbor integration: the trace in Intake. Pushing results to Intake requires a platform client, so pass `--base-url` and `--workspace`. -**Kick off the Experimenter.** +#### Start the Experimenter Hand it a single Insight: ```bash -nemo experimentalist run \ - --insight \ - --agent \ +nemo agents experimentalist run \ + --insight insight-id \ + --agent ./path-to-agent \ --train-dataset ./harbor_eval/dataset/train \ --validation-dataset ./harbor_eval/dataset/validation \ --task-template ./task-template \ - --config \ + --config ./experiment-config.yaml \ --experiment-dir tmp/experiment \ - --workspace default \ - --base-url http://localhost:8080 + --workspace "$WORKSPACE" \ + --base-url "$NMP_BASE_URL" ``` From an agent directory with an `optimizer.yaml` profile, most of those flags come from the profile: ```bash -nemo experimentalist run +nemo agents experimentalist run ``` The Experimenter's own models are configured through environment variables -(see [Models](#models)). Train and validation datasets are required, and for -the Harbor evaluator they must be local paths. The loop runs a baseline eval, +(refer to [Models](#models)). Train and validation datasets are required, and they must be local +paths for the Harbor evaluator. The loop runs a baseline evaluation, performs root cause analysis, implements changes, then validates and picks a winner. -**Review the results.** +#### Review the Results View the experiment in Studio at -`http://localhost:8080/studio/workspaces/default/experiment`. There you can see +`$NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment`. There you can see the originating Insight and compare evaluation runs across all the candidates. The Experimenter also writes intermediate artifacts along the way. Those @@ -450,18 +474,18 @@ land under `/eval-and-optimize/`, which defaults to `/.nemo-optimizer/experiments/` when a profile governs the run, and `./tmp` otherwise: -- `OPTIMIZATION.md` — the report with per-agent breakdowns and round-by-round +- `OPTIMIZATION.md`: The report with per-agent breakdowns and round-by-round root causes. -- `agents/agent-0` — the baseline, with `agents/agent-N` as the candidates. +- `agents/agent-0`: The baseline, with `agents/agent-N` as the candidates. Each holds full agent source. -- `results/` — the per-agent evaluation outputs. +- `results/`: The per-agent evaluation outputs. ## Additional Workflows ### Opt an Agent into Periodic Analysis -Instead of running `analyze` by hand, opt an agent in and let the platform run -the Analyst on a schedule: +Instead of running `nemo agents analyst run` manually, opt an agent in and let +the platform run the Analyst on a schedule: ```bash nemo insights analysis enable --agent "$AGENT" --workspace "$WORKSPACE" @@ -470,15 +494,16 @@ nemo insights analysis disable --agent "$AGENT" --workspace "$WORKSPACE" ``` A framework controller (`insights-analysis`) reconciles on a fixed 60-second -loop, finds every agent opted in across workspaces, and submits one analyze job +loop, finds every agent opted in across workspaces, and submits one Analyst job per agent that is *due* and has enough new telemetry. Runs are incremental: each successful run records a cursor (`last_successful_run_at`), and subsequent runs only consider traces newer than that cursor. ### Configure the Analysis Schedule -The global schedule lives in the Insights plugin config (or environment). -Defaults shown: +The global schedule lives in the Insights plugin configuration or environment. The `timezone` +setting uses an Internet Assigned Numbers Authority (IANA) time zone name and defaults to +Coordinated Universal Time (UTC). ```yaml insights: @@ -487,34 +512,36 @@ insights: frequency: daily # daily (default) or weekly run_at_hour: 0 # local hour-of-day, 0-23 run_on_weekday: monday # only used when frequency: weekly - timezone: UTC # IANA name; e.g. America/Denver + timezone: UTC # IANA name, such as America/Denver job_profile: default # jobs execution profile for scheduled runs ``` -`run_at_hour` is interpreted in `timezone` (an IANA name) and converted to the +`run_at_hour` is interpreted in `timezone` and converted to the server clock at evaluation time, so runs fire at the intended local hour even across daylight-saving transitions. The controller still reconciles every 60 -seconds and submits a run once the scheduled daily or weekly window is reached. +seconds and submits a run after the scheduled daily or weekly window is reached. ## Command Reference -The Analyst lives under `nemo insights` and the Experimenter lives under -`nemo experimentalist`. They are separate command namespaces that share the -`optimizer.yaml` profile, and each validates its own part of it. +The Analyst lives under `nemo agents analyst`, and the Experimenter lives under +`nemo agents experimentalist`. The Eval Author command surface lives under +`nemo agents eval-author`. These command namespaces share the `optimizer.yaml` +profile, and each validates its own configuration. Scheduled analysis remains +under `nemo insights analysis`. -### `nemo insights analyze` +### `nemo agents analyst run` -Run the Analyst once against an agent's traces. +Run the Analyst for one analysis of an agent's traces. | Flag | Required | Default | Description | |------|----------|---------|-------------| | `--agent` | yes, unless a profile supplies it | profile `agent` | Agent under test the Analyst should focus on. | -| `--agent-spec` | no | profile `agent_spec`, else `AGENT-SPEC.md` or `README.md` beside the profile | Path to a markdown spec for the AUT. | +| `--agent-spec` | no | profile `agent_spec`, else `AGENT-SPEC.md` or `README.md` beside the profile | Path to a Markdown spec for the AUT. | | `--workspace` | no | profile `workspace`, else `default` | Workspace to operate in. | | `--base-url` | no | `NMP_BASE_URL`, else `http://localhost:8080` | Running platform instance the Analyst's tools call. | -| `--profile` | no | discovered by walking up from cwd | Path to `optimizer.yaml`. | +| `--profile` | no | discovered by walking up from the current working directory | Path to `optimizer.yaml`. | | `--insights-file-output` | no | `/.nemo-optimizer/insights.yaml` when a profile is found, else the Insights API | Read and write Insights from a local YAML file. | -| `--verbose` / `-v` | no | off | Stream tool calls and reasoning to stderr. | +| `--verbose` / `-v` | no | off | Stream tool calls and reasoning to standard error. | ### `nemo insights analysis enable | disable | status` @@ -522,82 +549,90 @@ Manage per-agent opt-in for periodic analysis. `enable` and `disable` require `--agent`; `status` takes an optional `--agent` (omit it to list all configs in the workspace). All three accept `--workspace` and `--base-url`. -### `nemo insights doctor` +### `nemo agents analyst doctor` Check whether the current profile is ready for analysis. Exits non-zero when a required check fails. -### `nemo experimentalist run` +### `nemo agents experimentalist run` Run the local Experimenter loop. | Flag | Required | Default | Description | |------|----------|---------|-------------| | `--insight` | no | local `.nemo-optimizer/insights.yaml` | The Insight to optimize against: a local Insight file or a platform Insight ID. | -| `--insight-id` | no | — | Select an exact ID, exact title, or zero-based index from a local multi-Insight file. | +| `--insight-id` | no | Not applicable | Select an exact ID, exact title, or zero-based index from a local multi-Insight file. | | `--no-insight` | no | off | Run against a dataset directly rather than guided by an Insight. | | `--agent` | no | profile `agent_source` | Baseline agent override: a local directory or a git URL with optional ref (`...repo.git@main`). A git source records provenance and enables opening a draft PR for the winner. | -| `--agent-spec` | no | profile `agent_spec` | URI of a markdown file describing the AUT. | +| `--agent-spec` | no | profile `agent_spec` | Location of a Markdown file describing the AUT. | | `--train-dataset` | yes, unless the profile supplies it | profile `datasets.train` | Train dataset. Local path for the Harbor evaluator. | | `--validation-dataset` | yes, unless the profile supplies it | profile `datasets.validation` | Validation dataset. Local path for the Harbor evaluator. | | `--task-template` | required with an Insight | profile `task_template` | Evaluator-specific task-template URI, used to build the Insight-specific evaluation suite. | | `--experiment-dir` / `-o` | no | `/.nemo-optimizer/experiments/`, else `./tmp` | Local experiment directory; writes `eval-and-optimize/` here. | | `--framework-skills` | no | profile `framework_skills` | Directory of framework skills to load into the optimization agents. Repeatable. | -| `--mode` | no | `local` | `local` or `remote`. Only `local` is implemented today. | -| `--profile` | no | discovered by walking up from cwd | Path to `optimizer.yaml`. | +| `--profile` | no | discovered by walking up from the current working directory | Path to `optimizer.yaml`. | | `--workspace` | no | profile `workspace` | Workspace for traces and run/candidate metadata. | | `--base-url` | no | `NMP_BASE_URL`, else `http://localhost:8080` | Running platform instance. | -| `--config` | no | profile `experiment_config` | YAML or JSON configuration for the run. | +| `--config` | no | profile `experiment_config` | YAML or JavaScript Object Notation (JSON) configuration for the run. | -### `nemo experimentalist doctor` +### `nemo agents experimentalist doctor` Diagnose the Experimenter setup: profile, credentials, Insight resolution, datasets, and the experiment plan. +### `nemo agents eval-author` + +Discover the current Eval Author command surface with `--help`. The +`discover`, `audit`, `propose`, `run`, and `doctor` verbs are placeholders and +exit with a nonzero status until their implementations are available. + ### Models -The optimization agents drive their own LLMs, configured by environment -variable: +The optimization agents use the following model configuration: | Variable | Used by | Default and notes | |----------|---------|-------------------| -| `INFERENCE_API_KEY` | Analyst | Required. API key for the Analyst model, reached through the NVIDIA Inference Gateway. | +| `INFERENCE_API_KEY` | Analyst | Required. API key for Claude Opus 4.8 through the NVIDIA Inference Gateway. The Analyst model and endpoint are fixed. | | `NEMO_EXPERIMENTALIST_API_BASE` | Experimenter, Eval Author | OpenAI-compatible model API base URL. `run` and `doctor` fall back to the NVIDIA Inference Gateway when it is unset, so it is only required against another endpoint. Also settable as `api_base` under the `experimentalist:` config section; the environment wins. | | `NEMO_EXPERIMENTALIST_API_KEY` | Experimenter, Eval Author | Required. Model API key. On the gateway, `INFERENCE_API_KEY` fills this. | | `NEMO_EXPERIMENTALIST_MODELS_SMART` | Experimenter, Eval Author | Required. High-capability model for analysis, proposing, coding, and curation. No default: a model name is only meaningful against a specific endpoint. | -| `NEMO_EXPERIMENTALIST_MODELS_MID` | Experimenter, Eval Author | Required. Mid-tier model, used by the Coder's `apply_change` among others. No default. | +| `NEMO_EXPERIMENTALIST_MODELS_MID` | Experimenter, Eval Author | Required. Mid-tier model used for trajectory scoring and architecture documentation. No default. | | `NEMO_EXPERIMENTALIST_MODELS_FAST` | Experimenter, Eval Author | Required. Low-latency model for lightweight steps such as termination checks and summarization. No default. | +| `AUTHOR_API_BASE` | Eval Author | Optional override for the OpenAI-compatible model API base URL. Falls back to `NEMO_EXPERIMENTALIST_API_BASE`. | +| `AUTHOR_API_KEY` | Eval Author | Optional override for the model API key. Falls back to `NEMO_EXPERIMENTALIST_API_KEY`, or to `INFERENCE_API_KEY` for the NVIDIA Inference Gateway. | +| `AUTHOR_SMART_MODEL_NAME` | Eval Author | Optional high-capability model override. | +| `AUTHOR_MID_MODEL_NAME` | Eval Author | Optional mid-tier model override for Experimentalist helpers used by Eval Author. | +| `AUTHOR_FAST_MODEL_NAME` | Eval Author | Optional low-latency model override. | ## Troubleshooting -**`analyze` returns no Insights or aborts on the trace floor.** +### Analyst Returns No Insights or Aborts on the Trace Floor -The agent has -too few (or no) traces in Intake for the target workspace. Confirm telemetry is +The agent has too few or no traces in Intake for the target workspace. Confirm that telemetry is flowing and that you are filtering by the right `agent` name. -**Periodic analysis never fires.** +### Periodic Analysis Does Not Start Check that the agent is enabled (`nemo insights analysis status`), that `insights.analyst.enabled` is true, that the scheduled window has passed in the configured `timezone`, and that at least 10 new sessions have landed since the last run. -**Insights are not in the platform.** +### Insights Are Not in the Platform A discovered `optimizer.yaml` profile routes Insights to `.nemo-optimizer/insights.yaml` instead of the Insights API, as does an explicit `--insights-file-output`. Check that file, or run without a profile to write through the API. -**The experiment cannot score, or rewards are zero.** +### The Experiment Cannot Score or Rewards Are Zero Confirm datasets are local Harbor-compatible paths. Live and production datasets may lack verifiable rewards by design; rely on curated metrics and trajectory scoring in that case. -**No PR was opened for the winner.** +### No Pull Request Was Opened for the Winner Publishing requires a git agent source -(`--agent @`) and publishing enabled in the run config. A +(`--agent @`) and publishing enabled in the run configuration. A local-directory source cannot open a PR. diff --git a/docs/agents/observability.mdx b/docs/agents/observability.mdx index e5a9aa9d81..1c018e933d 100644 --- a/docs/agents/observability.mdx +++ b/docs/agents/observability.mdx @@ -1,12 +1,13 @@ --- title: "Observe Agents" -description: "Ingest, store, and query agent telemetry with NeMo Intake — OTLP, chat-completion, and ATIF ingest paths, annotations, and evaluator results, reviewable in NeMo Studio." +description: "Ingest, store, and query agent telemetry with NeMo Intake, including supported ingest paths, annotations, evaluator results, and review in NeMo Studio." --- Observability on NeMo Platform is provided by **NeMo Intake**, the trace ingestion and query service -for agent telemetry, and optionally by [NeMo Studio](/documentation/studio), the web UI for browsing +for agent telemetry, and optionally by [NeMo Studio](/documentation/studio), the web user interface +(UI) for browsing that telemetry. Observability helps humans and agents understand what happened in an agent system: which models were called, which tools were used, what inputs and outputs were produced, where failures occurred, and what feedback or evaluation scores were attached. @@ -23,12 +24,13 @@ diagnose failures from traces, in a production system or in offline evaluation f - A reachable ClickHouse database. - A NeMo Platform environment with the `intake` service running. -- A telemetry source: an OpenTelemetry/OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow, - NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or ATIF trajectories. -- (Optional) [NeMo Studio](/documentation/studio) for the UI review flow. +- A telemetry source, such as an OpenTelemetry or OpenInference exporter, NeMo Agent Toolkit (NAT), + NeMo Flow, NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or Agent + Trajectory Interchange Format (ATIF) trajectories. +- Optional: [NeMo Studio](/documentation/studio) for the UI review flow. -You do not need to deploy your agents on NeMo Platform. The lightweight path is to run the platform, -post one representative interaction, and confirm it appears. +You do not need to deploy your agents on NeMo Platform. To start, run the platform, post one +representative interaction, and confirm that it appears. ## How It Works @@ -54,14 +56,14 @@ Use Intake when you need to: - Standardize telemetry across teams so everyone shares one vocabulary, data model, and review surface. -Production traffic is the best fuel for optimization. New agents can start with staged traffic or -generated cases, then improve coverage as real traffic arrives. +Production traffic provides useful optimization evidence. New agents can start with staged traffic +or generated cases, then improve coverage as production traffic arrives. Intake accepts three standardized ingest formats depending on your instrumentation: | Format | Example use cases | Endpoint | |--------|-------------------|----------| -| OTLP/HTTP protobuf (OTel GenAI or OpenInference semantic conventions) | NeMo Relay, LangChain Deep Agents | `/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces` | +| OpenTelemetry Protocol (OTLP) over HTTP using Protocol Buffers (OpenTelemetry generative AI or OpenInference semantic conventions) | NeMo Relay, LangChain Deep Agents | `/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces` | | Chat completions | Importing raw logs, instrumenting a proxy server, custom logging | `/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions` | | ATIF | Running Harbor evaluations | `/apis/intake/v2/workspaces/{workspace}/ingest/atif` | @@ -69,8 +71,8 @@ Intake accepts three standardized ingest formats depending on your instrumentati Telemetry uses a three-level hierarchy: -- A **span** is one timed operation: an LLM call, tool invocation, retrieval, guardrail, evaluator, or - chain step. +- A **span** is one timed operation, such as a large language model (LLM) call, tool invocation, + retrieval, guardrail, evaluator, or chain step. - A **trace** is one end-to-end agent run, made of spans that share a trace ID. - A **session** groups related traces, such as a multi-turn conversation or a multi-system evaluation. @@ -80,7 +82,7 @@ Additional records attach signal to the same session or span: - **Evaluator results** store numeric, boolean, categorical, or text scores for a span. - **Experiments** and **Evaluations** organize evaluation runs into leaderboard rollups for comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment - rolls related Evaluations into one leaderboard. See + rolls related Evaluations into one leaderboard. Refer to [Experiments](/documentation/evaluate-models/experiments). The most useful instrumentation logs granular steps: every model call, tool call, final response, and @@ -93,32 +95,40 @@ error. A practical test: the trace can answer _where did the system go wrong, an The examples assume a running NeMo Platform reachable at `$NMP_BASE_URL`. Point at whatever you have, whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`: -```shell +```bash export NMP_BASE_URL=http://127.0.0.1:8080 export WORKSPACE=default +export NMP_STUDIO_URL="$NMP_BASE_URL/studio" +export NMP_ACCESS_TOKEN="$(nemo auth token)" ``` +Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path. +The examples use bearer-token authentication and stop on HTTP errors. For a local development +deployment with authentication disabled, omit the `Authorization` header and the +`NMP_ACCESS_TOKEN` command. Also omit `OTEL_EXPORTER_OTLP_HEADERS` from the OTLP example. + If you are bringing the pieces up yourself from a repository checkout, start them in this order. 1. **ClickHouse** (the telemetry datastore): - ```shell + ```bash services/intake/scripts/spans/run_clickhouse.sh ``` 1. **Backend services**: `intake` plus its `auth` (access checks) and `entities` (entity store) - dependencies. `--port` defaults to `8080`; drop `uv run` if you installed the `nemo` CLI: + dependencies. `--port` defaults to `8080`; omit `uv run` if you installed the `nemo` + command-line interface (CLI): - ```shell + ```bash uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 ``` 1. **Studio** (optional, for the UI review flow). From the `web/` workspace, with the intake feature flag on and pointed at the backend: - ```shell + ```bash VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 ``` @@ -127,8 +137,10 @@ If you are bringing the pieces up yourself from a repository checkout, start the Confirm the Intake read path can reach ClickHouse: -```shell -curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1" +```bash +curl --fail-with-body -i \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1" ``` A `200` response with an empty list is healthy. A `503` response means the Intake service is running @@ -136,12 +148,14 @@ but cannot reach ClickHouse. ### First Workflow -Send one captured chat-completion interaction. This is the lowest-friction smoke test — it needs no +Send one captured chat-completion interaction. This basic verification test does not require an OpenTelemetry exporter: -```shell +```bash export SESSION_ID="demo-session-001" -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \ +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "session_id": "'"$SESSION_ID"'", @@ -165,22 +179,24 @@ curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-com Read the span back from Intake: -```shell -curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10" ``` -Then open Studio and navigate to `/workspaces/default/intake/traces`. +Then open `$NMP_STUDIO_URL/workspaces/$WORKSPACE/intake/traces`. -**What good looks like:** +A successful result has the following characteristics: - The API call returns your interaction: one span with `session_id` `demo-session-001`, model `example-model`, and the prompt and response you sent as its input and output. - Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and response in the span tree. -That is the full loop: something you posted is now queryable through the API and reviewable in the UI. -The fields Intake captured here — session, trace, span, request, response, status, error, plus any -feedback or evaluator results — are the vocabulary the rest of these docs build on. +The interaction is now queryable through the application programming interface (API) and reviewable +in the UI. The captured fields include session, trace, span, request, response, status, error, +feedback, and evaluator results. The remaining documentation uses this vocabulary. ## Common Workflows @@ -188,37 +204,42 @@ feedback or evaluator results — are the vocabulary the rest of these docs buil Use OTLP when your framework or collector already emits OpenTelemetry traces: -```shell +```bash export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces" export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$NMP_ACCESS_TOKEN" ``` -Then run the instrumented agent. Intake maps OpenInference and OTel GenAI semantic attributes onto -span fields so model, tool, status, token, and error data are queryable. +Then run the instrumented agent. Intake maps OpenInference and OpenTelemetry generative AI semantic +attributes onto span fields. You can then query model, tool, status, token, and error data. ### Send ATIF Trajectories -Use ATIF when your source system exports complete agent trajectories — steps, agent metadata, or final -metrics: +Use ATIF when your source system exports complete agent trajectories, including steps, agent +metadata, or final metrics: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ --data-binary @trajectory.json ``` Each trajectory is stored as a structured session of spans. If the trajectory's top-level `extra.verifier_result.rewards` is populated, Intake writes those as evaluator results automatically. -Stock Harbor output does not populate it — Harbor's rewards live in a separate `reward.json` — so it -must be enriched first. See [Capture Evaluator Results](#capture-evaluator-results) for the automatic -and explicit paths. +Stock Harbor output does not populate it. Harbor rewards are stored in a separate `reward.json`, so +you must enrich the output first. Refer to +[Capture Evaluator Results](#capture-evaluator-results) for the automatic and explicit paths. ### Add Feedback and Labels Use annotations to attach review signal after a trace lands: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }' ``` @@ -230,16 +251,16 @@ Use `kind: "note"` for reviewer notes, `kind: "label"` for categorical or numeri -Evaluator results — a judge's rating, a pass/fail check, a similarity score — attach to a span. There -are two ways to get them into Intake. +Evaluator results attach a judge rating, pass or fail check, or similarity score to a span. You can +send them to Intake in two ways. **Automatically, from ATIF.** Put a `verifier_result.rewards` object in the trajectory's **top-level** `extra`, keyed by criterion (`{criterion: score}`). On ingest, Intake synthesizes an evaluator span named `harbor.verifier` and writes one evaluator result per key onto it; a bare scalar `extra.verifier_result.score` instead becomes a single `reward` result. Nothing else to send. -A stock Harbor `trajectory.json` does **not** carry this — Harbor writes rewards to a separate -`reward.json` — so you must copy them into the ATIF `extra` before ingest, or use the explicit path +A stock Harbor `trajectory.json` does **not** carry this value. Harbor writes rewards to a separate +`reward.json`, so you must copy them into the ATIF `extra` before ingestion or use the explicit path below. For example: ```json @@ -259,8 +280,10 @@ post. `data_type` picks which field carries the score: For example, a numeric score: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "span_id": "", @@ -274,18 +297,26 @@ curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-resul Read them back for one span, or list and filter across the workspace by evaluator name, data type, or value range: -```shell -curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans//evaluator-results" -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8" +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans//evaluator-results" +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8" ``` ### Find Recurring Failures Query traces or spans with filters, then group spans by session or trace to find repeated errors: -```shell -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=ERROR&page=1&page_size=20" -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=ERROR" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=error&page=1&page_size=20" +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=error" ``` In Studio, start from the trace list, filter to negative feedback or error status, then open trace @@ -294,7 +325,7 @@ detail to inspect the span tree. ### Turn Evaluation Telemetry into a Leaderboard Once you are ingesting evaluation runs, group them into an **Experiment** to compare them side by -side. See [Experiments](/documentation/evaluate-models/experiments), or use the agent-assisted +side. Refer to [Experiments](/documentation/evaluate-models/experiments), or use the agent-assisted `nemo-experiments-upload` skill to go from zero to a populated leaderboard. ## Operations @@ -303,7 +334,7 @@ side. See [Experiments](/documentation/evaluate-models/experiments), or use the - OTLP request bodies are capped at 5 MiB by default. - Read endpoints default to a 30-day lookback when no time filter is supplied. -- Page size is capped at 1000 records. +- Page size is capped at 1,000 records. - Span data and trace-index data expire after 90 days. Raise the OTLP body cap with `NMP_INTAKE_OTLP_MAX_BODY_BYTES` when larger batches are required. For @@ -316,9 +347,10 @@ depends on platform auth and checks workspace access before ingesting or reading -Do not send secrets, credentials, raw PII, or regulated data unless your deployment, retention, and -access-control policy permits it. Intake preserves request and response payloads so reviewers can -diagnose behavior — useful for debugging, and important for data governance. +Do not send secrets, credentials, raw personally identifiable information (PII), or regulated data +unless your deployment, retention, and access-control policy permits it. Intake preserves request +and response payloads so reviewers can +diagnose behavior. This data is useful for debugging and important for data governance. @@ -327,8 +359,8 @@ diagnose behavior — useful for debugging, and important for data governance. ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the tables on first use. -Span and trace-index tables have a 90-day TTL. Annotations and evaluator results are retained in their -ClickHouse tables without that 90-day span TTL. +Span and trace-index tables have a 90-day time-to-live (TTL) setting. Annotations and evaluator +results are retained in their ClickHouse tables without that 90-day span TTL. For production storage sizing, estimate span volume across the retained window, then size ClickHouse for interactive reads over recent traces and periodic aggregate queries. diff --git a/docs/evaluator/experiments.mdx b/docs/evaluator/experiments.mdx index 024765f83b..a7a938dc42 100644 --- a/docs/evaluator/experiments.mdx +++ b/docs/evaluator/experiments.mdx @@ -1,15 +1,14 @@ --- title: "Experiments" -description: "Compare evaluation runs on one leaderboard with NeMo Experiments — cost, latency, token, and evaluator rollups computed at read time from NeMo Intake telemetry." +description: "Compare evaluation runs on a NeMo Experiments leaderboard with cost, latency, token, and evaluator rollups computed from NeMo Intake telemetry." --- NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact -of changes to a harness, infrastructure, tools, or agent code. It gives you one place to visualize and -compare evaluation results from the runner of your choice — Harbor, the NeMo Optimizer, NeMo -Evaluator, or your own — ranked on the metrics that matter, such as cost, latency, and evaluator -scores. +of changes to a harness, infrastructure, tools, or agent code. It provides one place to visualize and +compare evaluation results from Harbor, the NeMo Optimizer, NeMo Evaluator, or your own runner. You +can rank these results by cost, latency, evaluator scores, and other metrics. Those metrics are derived from [NeMo Intake](/documentation/agents/observe-agents) observability data, the same traces and evaluator results your runs already produce, so any runner that lands telemetry in @@ -30,10 +29,11 @@ decide what to promote. - The platform entity store (Postgres). - A reachable ClickHouse database. - A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or - evaluation framework such as Harbor, or a direct API integration. -- (Recommended) [NeMo Studio](/documentation/studio) for the full UI — trace comparison views, a - customizable leaderboard, and Pareto charts. Everything is also available through the API, but - Studio is where the experience really lives. The Experiments UI is gated by the + evaluation framework such as Harbor, or a direct application programming interface (API) + integration. +- Recommended: [NeMo Studio](/documentation/studio) for the complete web user interface (UI), + including trace comparison views, a customizable leaderboard, and Pareto charts. The API provides + the same data. The Experiments UI is gated by the `VITE_FF_EXPERIMENT` feature flag, which is off by default. ## How It Works @@ -45,9 +45,9 @@ rolls up cost, latency, tokens, and per-evaluator scores from the underlying tel and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects current telemetry, with no denormalized score table to maintain. -You group runs however makes sense — an insight to investigate, a model bake-off, a benchmark -leaderboard, or the top runs promoted from several groups — and a single run can belong to more than -one group. Sorting, filtering, and pinning within a group surface the runs that matter. +You can group runs by an insight to investigate, a model comparison, a benchmark leaderboard, or +promoted runs from several groups. A single run can belong to more than one group. Sorting, +filtering, and pinning within a group surface the relevant runs. ### When to Use It @@ -62,7 +62,7 @@ Use Experiments when you need to: - **Standardize comparison across a team**: everyone reads the same metrics, in the same place, with the same vocabulary. -Experiments are most useful once real evaluation telemetry exists. A new group can start empty and +Experiments are most useful after real evaluation telemetry exists. A new group can start empty and fill in as runs land. ### Core Concepts and Data Model @@ -74,7 +74,7 @@ Experiments sit on top of Intake's telemetry hierarchy (span → trace → sessi a `pareto` (default X/Y metrics for the Pareto view). - An **Evaluation** is one run and one leaderboard row. It records producer-supplied fields (`dataset_name`, `dataset_version`, `source_link`, `metadata`, `description`, `status`, - `root_cause`), the `experiment_ids` it belongs to (at least one — an Evaluation can live in more + `root_cause`), the `experiment_ids` it belongs to (at least one; an Evaluation can live in more than one Experiment), and an optional `parent_evaluation_id` linking a variant back to the run it was derived from. - A **Session** is one test-case execution within an Evaluation, a single ingested run made of spans. @@ -84,7 +84,7 @@ At read time, each Evaluation is enriched with rollups derived from its sessions | Rollup | Meaning | |--------|---------| -| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_id` values). Sessions with no `test_case_id` don't count toward it or the rollups. | +| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_id` values). Sessions with no `test_case_id` do not count toward it or the rollups. | | `cost_usd` | Cost aggregate across the Evaluation's sessions. | | `latency_ms` | Latency aggregate across the Evaluation's sessions. | | `tokens` | Average total tokens (input + output) per test case. | @@ -93,16 +93,17 @@ At read time, each Evaluation is enriched with rollups derived from its sessions Metric aggregates expose these statistics: `sum`, `mean`, `median`, `p90`, `p95`, `p99`, `count`. A **metric path** is therefore `test_case_count`, `cost_usd.`, `latency_ms.`, -`tokens.`, or `evaluators..` — the same grammar used for sorting and filtering below. +`tokens.`, or `evaluators..`. Sorting and filtering use the same grammar. Two things about evaluator rollups specifically: -- **Response shape.** `evaluators..` is the sort/filter *query* grammar. In the Evaluation - JSON the same scores come back under `aggregate_scores` — a map keyed by evaluator name, each value - carrying the stats above — alongside an `evaluator_names` list. There is no `evaluators` field in +- **Response shape.** `evaluators..` is the sort/filter *query* grammar. The Evaluation + JSON response returns the same scores under `aggregate_scores`, a map keyed by evaluator name. + Each value contains the statistics above, and an `evaluator_names` list appears alongside it. + There is no `evaluators` field in the response; `evaluators..` (query) reads `aggregate_scores[].` (response). - **Missing-value semantics.** Evaluator rollups are test-case-weighted: each stat is computed over - `test_case_count`, the full set of test cases, with a test case that didn't report a given evaluator + `test_case_count`, the full set of test cases, with a test case that did not report a given evaluator counted as `0` rather than dropped. Two Experiment-level behaviors are worth knowing: @@ -110,7 +111,7 @@ Two Experiment-level behaviors are worth knowing: - **Default sort.** An Experiment stores a `default_sort` (a sort-param string such as `-evaluators.solved.mean`) so its leaderboard opens ordered by the metric the team cares about. It defaults to `-created_at` (newest first). -- **Pinning.** Any Evaluation can be pinned to the top of its Experiment — for example, the current +- **Pinning.** Any Evaluation can be pinned to the top of its Experiment, such as the current baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of the active sort. @@ -122,11 +123,18 @@ Experiments are part of the `intake` service, so any running NeMo Platform alrea at whatever you have, whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`: -```shell +```bash export NMP_BASE_URL=http://127.0.0.1:8080 export WORKSPACE=default +export NMP_STUDIO_URL="$NMP_BASE_URL/studio" +export NMP_ACCESS_TOKEN="$(nemo auth token)" ``` +Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path. +The examples use bearer-token authentication and stop on HTTP errors. For a local development +deployment with authentication disabled, omit the `Authorization` header and the +`NMP_ACCESS_TOKEN` command. + ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read time. @@ -136,35 +144,41 @@ If you are bringing the pieces up yourself from a repository checkout, start the 1. **ClickHouse** (required for the rollups): - ```shell + ```bash services/intake/scripts/spans/run_clickhouse.sh ``` 1. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to - `8080`; drop `uv run` if you installed the `nemo` CLI: + `8080`; omit `uv run` if you installed the `nemo` command-line interface (CLI): - ```shell + ```bash uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 ``` 1. **Studio** (optional, for the leaderboard and drill-down UI). From the `web/` workspace, with the Experiments feature flag on and intake enabled for trace drill-down: - ```shell + ```bash VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 ``` -Confirm the Experiments read path is reachable: +Confirm that the entity read path and ClickHouse rollups are reachable: -```shell -curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?page=1&page_size=1" +```bash +curl --fail-with-body -i \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?sort=-created_at&page=1&page_size=1" +curl --fail-with-body -i \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?sort=-cost_usd.mean&page=1&page_size=1" ``` -A `200` with a (possibly empty) list is healthy. A `503` means the service is running but cannot reach -ClickHouse; reads that need metric rollups will fail until it recovers. +The first request checks the entity store and can return `200` while ClickHouse is unavailable. The +second request explicitly sorts by a rollup metric. A `200` response from both requests is healthy. +A `503` response from the second request means that Intake cannot compute ClickHouse rollups. ### First Workflow @@ -172,19 +186,22 @@ Create an Experiment, add an Evaluation, send it telemetry, then see it in Studi **1. Create the Experiment** (the leaderboard container): -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", - "description": "Iterating on the support-bench RAG agent'\''s reranker and system prompt." + "description": "Iterating on the support-bench retrieval-augmented generation agent'\''s reranker and system prompt." }' ``` Capture its `id` for the next step: -```shell -export EXPERIMENT_ID=$(curl -sf \ +```bash +export EXPERIMENT_ID=$(curl --fail-with-body --silent \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') echo "experiment id: $EXPERIMENT_ID" @@ -193,8 +210,10 @@ echo "experiment id: $EXPERIMENT_ID" **2. Add an Evaluation** to it. `experiment_ids` is a list, so an Evaluation belongs to one or more existing Experiments: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-add-cross-encoder", @@ -205,20 +224,22 @@ curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ }' ``` -An Evaluation is a durable record on its own; its leaderboard metrics appear once evaluation telemetry -for it lands in Intake. +An Evaluation is a durable record. Its leaderboard metrics appear after its evaluation telemetry +lands in Intake. -**3. Send it telemetry.** Send the run's telemetry to Intake just as you would for any agent run, +**3. Send it telemetry.** Send the run's telemetry to Intake in the same way as other agent runs, whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each session with the Evaluation's identity: -- For ATIF and chat-completions, add a **top-level** `evaluation_context` object to the ingest payload +- For Agent Trajectory Interchange Format (ATIF) and chat-completions, add a **top-level** + `evaluation_context` object to the ingest payload carrying `evaluation_id` (the Evaluation's **name**) and `test_case_id`. -- For OTLP, set the `nemo.experiment.id` and `nemo.test_case.id` root-span attributes. +- For OpenTelemetry Protocol (OTLP), set the `nemo.experiment.id` and `nemo.test_case.id` root-span + attributes. The per-evaluator scores on the leaderboard come from **evaluator results** captured on those sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results -endpoint. See +endpoint. Refer to [Capture Evaluator Results](/documentation/agents/observe-agents#capture-evaluator-results) for both paths, and [Observe Agents](/documentation/agents/observe-agents) for the ingestion paths themselves. The `nemo-experiments-upload` skill walks this through end to end. @@ -226,9 +247,9 @@ The `nemo-experiments-upload` skill walks this through end to end. **`test_case_id` is required for a populated leaderboard.** A session tagged with only `evaluation_id` -still ingests and appears in the Evaluation's session list, but it doesn't count toward -`test_case_count` or any rollup — so the row reads as all zeros, with tokens, model, and agent blank -too. Always send `test_case_id` alongside `evaluation_id`. +still ingests and appears in the Evaluation's session list, but it does not count toward +`test_case_count` or any rollup. The row therefore reads as all zeros, with tokens, model, and agent +blank too. Always send `test_case_id` alongside `evaluation_id`. @@ -236,14 +257,16 @@ too. Always send `test_case_id` alongside `evaluation_id`. List the Experiment's Evaluations and confirm the new row is present: -```shell -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20" ``` -Then open Studio and navigate to `/workspaces/default/experiment`, then open +Then open `$NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment`, then open **reranker-prompt-iteration**. -**What good looks like:** the Experiment opens in Studio, your Evaluation appears as a row, and once +A successful result opens the Experiment in Studio and shows your Evaluation as a row. After its sessions are ingested the row shows non-zero `test_case_count`, cost, latency, tokens, and evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its individual test cases. @@ -255,8 +278,10 @@ individual test cases. This is the everyday operation: an Experiment already exists and you want to record another run in it. Create the Evaluation with the Experiment's `id` in `experiment_ids`: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ +```bash +curl --fail-with-body -X POST \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-bge-large", @@ -270,13 +295,15 @@ curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ To **move or re-scope** an existing Evaluation's membership, `PATCH` its `experiment_ids`, which must stay non-empty: -```shell -curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ +```bash +curl --fail-with-body -X PATCH \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }' ``` -An Evaluation can belong to several Experiments at once, which is useful when the same run should +An Evaluation can belong to several Experiments simultaneously, which is useful when the same run should appear on both a per-project board and a cross-project benchmark. ### Update an Evaluation @@ -284,21 +311,25 @@ appear on both a per-project board and a cross-project benchmark. Use `PATCH` for partial updates: only the fields you send change. `name`, `dataset_name`, and `dataset_version` are immutable: -```shell -curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ +```bash +curl --fail-with-body -X PATCH \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }' ``` -`PUT` does a full replace of the mutable fields — omitted fields reset — so prefer `PATCH` for one-off +`PUT` does a full replacement of the mutable fields. Omitted fields reset, so prefer `PATCH` for individual edits. ### Rank an Experiment by the Metric That Matters List an Experiment's Evaluations sorted by a metric. Prefix the field with `-` for descending: -```shell -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20" ``` You can sort by an entity column (`name`, `created_at`) or any rollup metric (`test_case_count`, @@ -308,16 +339,18 @@ way. ### Filter to the Evaluations You Care About -Filter by a metric range to narrow the leaderboard — for example, only Evaluations whose average cost +Filter by a metric range to narrow the leaderboard. For example, select only Evaluations whose average cost is under $0.50: -```shell -curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5" ``` Metric filters use the same grammar as sort, such as `filter[test_case_count][\$gte]=5` or `filter[evaluators.groundedness.mean][\$gte]=0.8`. You can also filter by -`filter[metadata.]=`, `filter[status]=…`, `filter[is_pinned]=true`, and +`filter[metadata.]=`, `filter[status]=`, `filter[is_pinned]=true`, and `created_at`/`updated_at` ranges. In Studio, use the column filters on cost, latency, test-case count, and evaluator columns. @@ -325,8 +358,10 @@ and evaluator columns. Store the ordering the team should see first so no one has to re-sort each visit: -```shell -curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ +```bash +curl --fail-with-body -X PUT \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }' ``` @@ -337,14 +372,18 @@ In Studio, set it from the Experiment's Edit dialog. Keep the current baseline, or any reference run, at the top of the Experiment for everyone: -```shell -curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" +```bash +curl --fail-with-body -X POST \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" # Unpin: -curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" +curl --fail-with-body -X DELETE \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" ``` -Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List just the +Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List only the pinned set with `filter[is_pinned]=true`. ### Drill into an Evaluation's Test Cases @@ -352,8 +391,10 @@ pinned set with `filter[is_pinned]=true`. Open one Evaluation's sessions to see per-test-case behavior: status, latency, cost, and evaluator scores: -```shell -curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20" +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20" ``` Use this to move from "this candidate scores lower" to "these specific cases regressed." In Studio, @@ -361,13 +402,15 @@ open the Evaluation row to reach its sessions, then follow a session into its tr ### Configure the Pareto View -Each Experiment stores a `pareto` config: the default X/Y metrics for its Pareto (trade-off) chart in +Each Experiment stores a `pareto` configuration: the default X/Y metrics for its Pareto chart in Studio, defaulting to cost vs. latency. The axes are `x_metric` and `y_metric`, and each takes a base -metric id — `cost_usd`, `latency_ms`, or `evaluators.`, with no `.` suffix. Set it on +metric ID: `cost_usd`, `latency_ms`, or `evaluators.`, with no `.` suffix. Set it on create or update: -```shell -curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ +```bash +curl --fail-with-body -X PUT \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }' @@ -375,23 +418,36 @@ curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/rera ### Delete an Experiment or Evaluation -Deleting an Experiment soft-deletes it and cascades to its Evaluations; deleting an Evaluation -soft-deletes just that row. Soft-deleted records are hidden from list and get operations unless -explicitly requested with `filter[is_deleted]=true`: +Deleting an Experiment soft-deletes it. An Evaluation that belongs only to that Experiment is also +soft-deleted. A shared Evaluation remains active, and the deleted Experiment is removed from its +`experiment_ids` membership. Deleting an Evaluation directly soft-deletes only that Evaluation. + +```bash +curl --fail-with-body -X DELETE \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" +curl --fail-with-body -X DELETE \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" +``` + +Soft deletion renames the stored record. A `GET` request that uses the original name does not return +the deleted record. To audit deleted records, use `filter[is_deleted]=true` on a list operation: -```shell -curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" -curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" +```bash +curl -g --fail-with-body \ + -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[is_deleted]=true&page=1&page_size=20" ``` ### Where Evaluations Come From -You usually don't create Evaluations by hand. Common producers: +You typically do not create Evaluations manually. Common producers include: - **Optimizer**: records each run as an Experiment and its candidates as Evaluations, so results land - automatically. See [Optimize Agents](/documentation/agents/optimize-agents). + automatically. Refer to [Optimize Agents](/documentation/agents/optimize-agents). - **Evaluation framework**: a framework like Harbor sends complete trajectories through Intake's ATIF - ingest, and each run becomes an Evaluation with its final metrics as evaluator results. See + ingest, and each run becomes an Evaluation with its final metrics as evaluator results. Refer to [Observe Agents](/documentation/agents/observe-agents). - **Direct API**: create the Experiment and Evaluations, then ingest their sessions, for a custom evaluation pipeline. @@ -432,20 +488,23 @@ regulated data unless your deployment's policy permits it. ### Retention and Storage Experiment and Evaluation metadata lives in the platform entity store and persists until you delete -it; it does not expire on the telemetry TTL. Rollups are derived from ClickHouse at read time, so -leaderboard metrics reflect whatever telemetry is still retained: span and trace-index data expire -after 90 days, while evaluator results are retained without that span TTL. An Experiment older than -the span window keeps its records and evaluator-based scores, but cost, latency, and run-count rollups -reflect only the retained telemetry window. +it. This metadata does not expire with telemetry. Span and trace-index records in ClickHouse expire +after 90 days. Evaluator-result rows do not use that 90-day time-to-live (TTL) setting. + +All leaderboard rollups depend on the retained trace index. Evaluator score rollups join the retained +trace index to evaluator-result rows at read time. After a trace-index entry expires, its evaluator +scores no longer appear on the leaderboard even if the evaluator-result rows remain. Cost, latency, +token, test-case count, and evaluator score rollups therefore reflect the retained 90-day trace +window. ### Troubleshooting | Symptom | Cause and fix | |---------|---------------| -| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_id`, so they don't count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, that sessions carry `test_case_id`, and that the read path returns `200` rather than `503`. | -| **A metric sort or filter returns `503`** | Rollups can't be computed because ClickHouse is down. Retry once the read path is healthy, or fall back to an entity-column sort. | +| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_id`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_id`, and run the explicit metric-sort health check. | +| **A metric sort or filter returns `503`** | Rollups cannot be computed because ClickHouse is down. Retry after the read path is healthy, or use an entity-column sort. | | **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. | -| **An Evaluation isn't in the Experiment** | Confirm you created it with the correct Experiment `id` in `experiment_ids` and are querying the right workspace (`filter[experiment_id]=`). | +| **An Evaluation is not in the Experiment** | Confirm that you created it with the correct Experiment `id` in `experiment_ids` and that you are querying the correct workspace (`filter[experiment_id]=`). | | **`422` on create Evaluation** | An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings. | ## Related Topics