diff --git a/docs/modules/graph/observability.md b/docs/modules/graph/observability.md index 41aa7ec..ea67e7b 100644 --- a/docs/modules/graph/observability.md +++ b/docs/modules/graph/observability.md @@ -313,14 +313,17 @@ The batch maps the graph run onto Langfuse's trace/observation model: - one `trace-create` (id defaults to the run's `root_run_id`; name defaults to the `graph_id`; session defaults to the run's `thread_id`) with the `GraphHealthSummary` folded into trace metadata +- one structural `span-create` for the graph run (`{trace}:run:{run_id}`, named + for the `graph_id`), parented to the trace, bracketing the whole run - one timed `span-create` per superstep (`{trace}:step:{n}`), parented to the - trace + graph-run span - one timed `span-create` per node handler (`{trace}:node:{name}:{step}`), parented to its superstep span; `node.failed` promotes the span to `ERROR` level with the rendered error as `statusMessage` -- one timed `span-create` per embedded subgraph +- one timed `span-create` per embedded subgraph, parented to the graph-run span - an `event-create` for every remaining observation (routes, checkpoints, - interrupts, custom writes, run lifecycle), with `run.failed` mapped to `ERROR` + interrupts, custom writes, run lifecycle), parented to the graph-run span, + with `run.failed` mapped to `ERROR` Still-running work (a `node.started` with no terminal event) is exported as an open span with a start time but no end time. @@ -334,6 +337,14 @@ every graph step, node, model generation, and tool call under one Langfuse trace. This is what makes full end-to-end telemetry — including tool health and tool timing — visible in a single trace tree. +The nesting is exact, not just co-located: the graph-run span +(`{trace}:run:{run_id}`) uses the same id scheme the harness exporter parents +its agent run spans to. A sub-agent a node spawns carries `parent_run_id` equal +to the graph run id, so its harness-exported run span resolves to +`{trace}:run:{graph_run_id}` and nests directly under the graph-run span rather +than floating at the trace root — the graph, its nodes, and their agents form +one contiguous tree. + ## Debug Payloads Debug streams are for executor introspection. They should be structured enough diff --git a/docs/modules/harness/observability.md b/docs/modules/harness/observability.md index fbd436c..6d35e7e 100644 --- a/docs/modules/harness/observability.md +++ b/docs/modules/harness/observability.md @@ -343,12 +343,42 @@ replays them verbatim. Downstream, the Langfuse exporter reads these fields to populate the Input/Output panels of a generation (`generation-create`) and a tool -observation (`tool-create`). With capture disabled the fields are absent and the +observation (`span-create`). With capture disabled the fields are absent and the observation renders with ids, timing, and usage only. The harness exporter also defaults the trace-level metadata from the run lineage (root/first run ids and parent), mirroring the graph exporter, so a trace is correlatable even when the caller passes no metadata. +## Langfuse Observation Tree + +The harness exporter mirrors LangChain's callback **run tree** rather than a +flat event list. For each distinct `run_id` in a batch it emits one +`span-create` (`{trace_id}:run:{run_id}`, named `agent` for the top-level run +and `sub-agent` for any spawned child), folding the `RunStarted`/`RunCompleted`/ +`RunFailed` events into that span's start/end window and ERROR status. Every +model generation, tool span, and remaining event then nests under its run span +via `parentObservationId`. + +Because run-span ids are deterministic and trace-namespaced, the tree spans +**separately exported batches**: a sub-agent run whose observations are shipped +in their own batch carries `parent_run_id`, so its run span resolves to +`{trace_id}:run:{parent_run_id}` and nests under the parent agent's span in the +same trace. The graph exporter emits a matching `{trace_id}:run:{run_id}` span +for the graph run, so a graph run and the agent runs its nodes spawn +reconstruct one unified nested tree. + +## Scores + +`LangfuseClient::create_score` (and the offline `build_score_batch`) mirrors +Langfuse's `createScore` — the way a LangChain/LangGraph trace is graded after +the fact (human ratings, LLM-as-judge, regression metrics). A `LangfuseScore` +is numeric, categorical, or boolean; scoped to a `trace_id` and optionally to a +single observation id; and carries an optional comment. Score ids default to a +deterministic value derived from the target and metric name, so re-scoring the +same target upserts rather than duplicating. Point the score's `trace_id` at an +exported trace (recall both exporters default the trace id to the run's +`root_run_id`) to attach the evaluation to that run. + ## Graph Events State-graph event payloads should be rich enough for a UI to render run diff --git a/src/graph/observability/README.md b/src/graph/observability/README.md index 2bd2108..46cd611 100644 --- a/src/graph/observability/README.md +++ b/src/graph/observability/README.md @@ -43,10 +43,12 @@ so existing runs are unchanged. `from_status`). - `GraphLangfuseExporter` (`langfuse/`) — exports a run's observations to Langfuse, turning supersteps and nodes into timed spans (failures promoted - to `ERROR`) and attaching the health summary to the trace. It shares the - harness `LangfuseClient` transport and defaults its `traceId` to the run's - `root_run_id`, so a graph run and the agent/tool runs its nodes spawn land - under one trace. + to `ERROR`) and attaching the health summary to the trace. It emits a + structural graph-run span (`{trace}:run:{run_id}`) that steps, subgraphs, and + point events nest under. It shares the harness `LangfuseClient` transport and + defaults its `traceId` to the run's `root_run_id`; because the graph-run span + uses the same id scheme the harness exporter parents its agent run spans to, + a graph run and the agent/tool runs its nodes spawn nest into one trace tree. ## Persistence bridge diff --git a/src/graph/observability/langfuse.rs b/src/graph/observability/langfuse.rs index 8421229..89c8402 100644 --- a/src/graph/observability/langfuse.rs +++ b/src/graph/observability/langfuse.rs @@ -128,18 +128,29 @@ impl GraphLangfuseExporter { let trace_ts = iso_ms(first.ts_ms); let health = GraphHealthSummary::from_observations(observations); - let mut batch = Vec::with_capacity(observations.len() + 1); + let mut batch = Vec::with_capacity(observations.len() + 2); batch.push(trace_create(&trace_id, &trace_ts, &trace, first, &health)); + // A structural span for the graph run itself, sitting directly under the + // trace. Every step, subgraph, and point event nests under it, and — key + // for a unified trace — its id follows the same `{trace}:run:{run_id}` + // scheme the harness exporter parents its run spans to. A sub-agent that + // a graph node spawns carries `parent_run_id == graph run id`, so its + // harness-exported run span nests directly under this graph-run span + // rather than floating at the trace root. + let run_parent = run_span_id(&trace_id, first.root_run_id.as_str()); + batch.push(graph_run_span(&trace_id, &run_parent, observations, first)); + let mut consumed = vec![false; observations.len()]; push_span_events( &trace_id, + &run_parent, observations, &mut consumed, &mut batch, self.span_metadata_fn.as_deref(), ); - push_point_events(&trace_id, observations, &consumed, &mut batch); + push_point_events(&trace_id, &run_parent, observations, &consumed, &mut batch); Ok(json!({ "batch": batch })) } @@ -155,6 +166,59 @@ impl GraphLangfuseExporter { } } +/// The stable Langfuse observation id for a run's span, shared with the harness +/// exporter (`{trace_id}:run:{run_id}`) so a graph run and the harness agent +/// runs its nodes spawn resolve to one nested tree under the same trace. +fn run_span_id(trace_id: &str, run_id: &str) -> String { + format!("{trace_id}:run:{run_id}") +} + +/// Builds the structural `span-create` for the graph run, parented to the trace. +/// +/// The span brackets the run: it starts at the first observation and ends at the +/// last (or at a terminal `RunCompleted`/`RunFailed` when present). It is a plain +/// structural anchor — run failure is still surfaced by the dedicated +/// `run.failed` point event — so its only job is to give steps, subgraphs, and +/// any child agent runs a single parent to nest under. +fn graph_run_span( + trace_id: &str, + run_span_id: &str, + observations: &[GraphObservation], + first: &GraphObservation, +) -> Value { + let start_ts = first.ts_ms; + // Prefer a terminal run event's timestamp; else close at the last observation. + let end_ts = observations + .iter() + .rev() + .find(|o| { + matches!( + o.event, + GraphEvent::RunCompleted { .. } | GraphEvent::RunFailed { .. } + ) + }) + .or_else(|| observations.last()) + .map(|o| o.ts_ms) + .unwrap_or(start_ts); + json!({ + "id": run_span_id, + "timestamp": iso_ms(end_ts), + "type": "span-create", + "body": clean_nulls(json!({ + "id": run_span_id, + "traceId": trace_id, + "name": first.graph_id.as_str(), + "startTime": iso_ms(start_ts), + "endTime": iso_ms(end_ts), + "metadata": json!({ + "run_id": first.run_id.as_str(), + "root_run_id": first.root_run_id.as_str(), + "graph_id": first.graph_id.as_str(), + }), + })), + }) +} + /// Resolves the Langfuse trace id: the configured id when set, else the first /// observation's root run id so it aligns with the harness agent exporter. fn resolve_trace_id(trace: &LangfuseTraceConfig, observations: &[GraphObservation]) -> String { @@ -225,6 +289,7 @@ fn trace_create( /// re-emitted as point events. Unpaired starts become open spans (start only). fn push_span_events( trace_id: &str, + run_parent: &str, observations: &[GraphObservation], consumed: &mut [bool], batch: &mut Vec, @@ -249,6 +314,7 @@ fn push_span_events( consumed[idx] = true; batch.push(step_span( trace_id, + run_parent, *step, start_ts, Some(obs.ts_ms), @@ -309,6 +375,7 @@ fn push_span_events( consumed[idx] = true; batch.push(subgraph_span( trace_id, + run_parent, node, namespace, start_ts, @@ -327,7 +394,7 @@ fn push_span_events( while let Some((start_idx, start_ts)) = queue.pop_front() { let obs = &observations[start_idx]; batch.push(step_span( - trace_id, step, start_ts, None, obs, obs, injector, + trace_id, run_parent, step, start_ts, None, obs, obs, injector, )); } } @@ -349,6 +416,7 @@ fn push_span_events( while let Some((start_idx, start_ts)) = queue.pop_front() { batch.push(subgraph_span( trace_id, + run_parent, &node, &namespace, start_ts, @@ -364,6 +432,7 @@ fn push_span_events( /// span, mapping `run.failed` to `ERROR` level with the rendered error. fn push_point_events( trace_id: &str, + run_parent: &str, observations: &[GraphObservation], consumed: &[bool], batch: &mut Vec, @@ -388,6 +457,7 @@ fn push_point_events( "body": clean_nulls(json!({ "id": obs.event_id.as_str(), "traceId": trace_id, + "parentObservationId": run_parent, "name": obs.event.kind(), "startTime": ts, "level": level, @@ -402,6 +472,7 @@ fn push_point_events( #[allow(clippy::too_many_arguments)] fn step_span( trace_id: &str, + run_parent: &str, step: usize, start_ts: u64, end_ts: Option, @@ -413,7 +484,7 @@ fn step_span( span_event( trace_id, &format!("{trace_id}:step:{step}"), - None, + Some(run_parent.to_string()), &format!("step {step}"), start_ts, end_ts, @@ -454,8 +525,10 @@ fn node_span( } /// Builds a `span-create` for an embedded subgraph, parented to the trace. +#[allow(clippy::too_many_arguments)] fn subgraph_span( trace_id: &str, + run_parent: &str, node: &NodeId, namespace: &[String], start_ts: u64, @@ -467,7 +540,7 @@ fn subgraph_span( span_event( trace_id, &format!("{trace_id}:subgraph:{}", namespace.join("/")), - None, + Some(run_parent.to_string()), &format!("subgraph {}", node.as_str()), start_ts, end_ts, diff --git a/src/graph/observability/langfuse/test.rs b/src/graph/observability/langfuse/test.rs index 790ddfb..3adf595 100644 --- a/src/graph/observability/langfuse/test.rs +++ b/src/graph/observability/langfuse/test.rs @@ -172,13 +172,22 @@ fn steps_and_nodes_become_timed_spans() { .unwrap(); let events = batch["batch"].as_array().unwrap(); + // The graph run itself is a structural span under the trace, named for the + // graph, that every step nests beneath. Its id uses the shared + // `{trace}:run:{run}` scheme so harness sub-agent runs (parent_run_id = the + // graph run) nest under it too, unifying graph + agent into one trace tree. + let run_span = find(events, "span-create", "root-1:run:root-1").expect("graph run span"); + assert_eq!(run_span["body"]["name"], "demo-graph"); + assert!(run_span["body"].get("parentObservationId").is_none()); + assert_eq!(run_span["body"]["startTime"], "2024-01-01T00:00:01.000Z"); + // Superstep span with real start/end times. let step = find(events, "span-create", "root-1:step:1").expect("step span"); assert_eq!(step["body"]["name"], "step 1"); assert_eq!(step["body"]["startTime"], "2024-01-01T00:00:01.010Z"); assert_eq!(step["body"]["endTime"], "2024-01-01T00:00:01.090Z"); - // Steps parent to the trace, not another span. - assert!(step["body"].get("parentObservationId").is_none()); + // Steps parent to the graph-run span, not straight to the trace. + assert_eq!(step["body"]["parentObservationId"], "root-1:run:root-1"); // Node a completed cleanly and is parented to its step span. let node_a = find(events, "span-create", "root-1:node:a:1").expect("node a span"); diff --git a/src/harness/observability/README.md b/src/harness/observability/README.md index b993ec3..30e55a1 100644 --- a/src/harness/observability/README.md +++ b/src/harness/observability/README.md @@ -40,7 +40,14 @@ surface and operational constraints. `AgentLatencyMetrics::from_observations(&[AgentObservation])`. - `LangfuseAuth`, `LangfuseClient`, `LangfuseTraceConfig` (re-exported from the private `langfuse` submodule) — the Langfuse exporter used to ship harness - (and, via shared helpers, graph) traces to Langfuse. + (and, via shared helpers, graph) traces to Langfuse. The exporter emits a + per-run span (`{trace}:run:{run_id}`, `agent`/`sub-agent`) and nests every + generation, tool span, and event under it, so a run — and sub-agent recursion + across separately-exported batches — surfaces as a tree, matching LangChain's + callback run hierarchy. +- `LangfuseScore` / `LangfuseScoreValue` — an evaluation score (numeric, + categorical, or boolean; trace- or observation-scoped) attached to an exported + trace via `LangfuseClient::create_score`, mirroring Langfuse's `createScore`. ## Persistence bridge diff --git a/src/harness/observability/langfuse/mod.rs b/src/harness/observability/langfuse/mod.rs index 22ea180..849d7c3 100644 --- a/src/harness/observability/langfuse/mod.rs +++ b/src/harness/observability/langfuse/mod.rs @@ -16,7 +16,9 @@ use crate::harness::usage::Usage; mod types; -pub use types::{LangfuseAuth, LangfuseClient, LangfuseTraceConfig}; +pub use types::{ + LangfuseAuth, LangfuseClient, LangfuseScore, LangfuseScoreValue, LangfuseTraceConfig, +}; impl LangfuseClient { /// Creates a client from a base URL and auth mode. @@ -118,7 +120,7 @@ impl LangfuseClient { // generation's cost is $0. let call_models = collect_call_models(observations); - let mut batch = Vec::with_capacity(observations.len() + 1); + let mut batch = Vec::with_capacity(observations.len() + 2); batch.push(json!({ "id": format!("{}:trace", trace_id), "timestamp": timestamp, @@ -137,13 +139,67 @@ impl LangfuseClient { })), })); + // Emit one span per run so the batch surfaces a **run tree** rather than + // a flat list, mirroring how LangChain's callback handler nests LLM + // generations and tool spans under the chain/agent run that produced + // them. Each run span is parented (cross-batch, via a deterministic id) + // to its `parent_run_id`'s span, so a sub-agent run exported in its own + // batch nests under the parent agent's span in the same trace. + for run in run_spans(&trace_id, observations) { + batch.push(run); + } + for obs in observations { + // Run-lifecycle events are consumed into the run span (its start, + // end, and error/status); re-emitting them as standalone events + // would duplicate the run boundary the span already represents. + if is_run_lifecycle(&obs.event) { + continue; + } batch.push(observation_event(&trace_id, obs, &call_models)); } Ok(json!({ "batch": batch })) } + /// Builds a `score-create` ingestion batch for `score` without sending it. + /// + /// The score id defaults to a deterministic value derived from the trace, + /// observation, and name so re-scoring the same target is idempotent + /// (Langfuse upserts scores by id). Pair with a `trace_id` that matches an + /// exported trace — recall the harness/graph exporters default their trace + /// id to the run's `root_run_id` — to attach an evaluation to a run. + pub fn build_score_batch(&self, score: LangfuseScore) -> Value { + let timestamp = iso_ms(now_ms()); + let score_id = score.id.clone().unwrap_or_else(|| default_score_id(&score)); + let event_id = format!("{score_id}:score"); + json!({ + "batch": [json!({ + "id": event_id, + "timestamp": timestamp, + "type": "score-create", + "body": clean_nulls(json!({ + "id": score_id, + "traceId": score.trace_id, + "observationId": score.observation_id, + "name": score.name, + "value": score.value.to_value(), + "dataType": score.value.data_type(), + "comment": score.comment, + })), + })] + }) + } + + /// Attaches an evaluation `score` to an already-exported trace (or one of + /// its observations) by sending a `score-create` ingestion batch. Mirrors + /// Langfuse's `createScore` — the way LangChain/LangGraph traces are graded + /// after the fact (human ratings, LLM-as-judge, regression metrics). + pub async fn create_score(&self, score: LangfuseScore) -> Result { + let payload = self.build_score_batch(score); + self.send_batch(payload).await + } + /// Sends observations as one Langfuse ingestion batch. pub async fn send_observations( &self, @@ -278,6 +334,121 @@ fn trace_metadata(trace: &LangfuseTraceConfig, observations: &[AgentObservation] } } +/// Whether an event marks a run boundary that the run span already represents. +/// +/// These events are folded into the per-run span (its start time, end time, and +/// error/status) rather than emitted as standalone observations. +fn is_run_lifecycle(event: &AgentEvent) -> bool { + matches!( + event, + AgentEvent::RunStarted { .. } + | AgentEvent::RunCompleted { .. } + | AgentEvent::RunFailed { .. } + ) +} + +/// The stable, cross-batch Langfuse observation id for a run's span. Prefixing +/// with the (globally-unique) `trace_id` keeps it unique across turns/threads +/// while staying deterministic, so a child run exported in a *separate* batch +/// can reference its parent run's span by the same id and nest under it. +fn run_span_id(trace_id: &str, run_id: &str) -> String { + format!("{trace_id}:run:{run_id}") +} + +/// Accumulates a single run's timing/lineage while scanning the batch so the +/// exporter can emit one span per run. +struct RunAcc<'a> { + parent_run_id: Option<&'a str>, + root_run_id: &'a str, + first_ts: u64, + last_ts: u64, + /// Terminal timestamp from a `RunCompleted`/`RunFailed`, when the run ended + /// within this batch. `None` leaves the span open (still running). + end_ts: Option, + error: Option<&'a str>, +} + +impl<'a> RunAcc<'a> { + fn new(obs: &'a AgentObservation) -> Self { + Self { + parent_run_id: obs.parent_run_id.as_ref().map(|id| id.as_str()), + root_run_id: obs.root_run_id.as_str(), + first_ts: obs.ts_ms, + last_ts: obs.ts_ms, + end_ts: None, + error: None, + } + } + + fn observe(&mut self, obs: &'a AgentObservation) { + self.first_ts = self.first_ts.min(obs.ts_ms); + self.last_ts = self.last_ts.max(obs.ts_ms); + match &obs.event { + AgentEvent::RunCompleted { .. } => self.end_ts = Some(obs.ts_ms), + AgentEvent::RunFailed { error, .. } => { + self.end_ts = Some(obs.ts_ms); + self.error = Some(error.as_str()); + } + _ => {} + } + } +} + +/// Builds a `span-create` per distinct run in the batch, in first-seen order. +/// +/// A run whose `run_id == root_run_id` is the top-level agent (`"agent"`); any +/// other run was spawned by a parent and is labelled `"sub-agent"`. The span is +/// parented to its `parent_run_id`'s run span (which may live in another batch) +/// so the trace renders the full recursion tree. +fn run_spans(trace_id: &str, observations: &[AgentObservation]) -> Vec { + let mut order: Vec<&str> = Vec::new(); + let mut runs: BTreeMap<&str, RunAcc> = BTreeMap::new(); + for obs in observations { + let rid = obs.run_id.as_str(); + runs.entry(rid) + .and_modify(|acc| acc.observe(obs)) + .or_insert_with(|| { + order.push(rid); + let mut acc = RunAcc::new(obs); + acc.observe(obs); + acc + }); + } + + order + .into_iter() + .map(|rid| { + let acc = &runs[rid]; + let span_id = run_span_id(trace_id, rid); + let is_root = rid == acc.root_run_id; + let name = if is_root { "agent" } else { "sub-agent" }; + // Close the span at the terminal event when present, else at the + // last observation seen (a real window for a completed export). + let end_iso = iso_ms(acc.end_ts.unwrap_or(acc.last_ts)); + json!({ + "id": span_id, + "timestamp": end_iso, + "type": "span-create", + "body": clean_nulls(json!({ + "id": span_id, + "traceId": trace_id, + "parentObservationId": acc.parent_run_id.map(|p| run_span_id(trace_id, p)), + "name": name, + "startTime": iso_ms(acc.first_ts), + "endTime": end_iso, + "level": acc.error.map(|_| "ERROR"), + "statusMessage": acc.error, + "metadata": json!({ + "run_id": rid, + "root_run_id": acc.root_run_id, + "parent_run_id": acc.parent_run_id, + }), + })), + }) + }) + .collect() +} + /// Collect the model each `ModelStarted` announced, keyed by its `call_id`. /// /// The per-call `generation-create` is built from `ModelCompleted`, which does @@ -300,6 +471,9 @@ fn observation_event( call_models: &BTreeMap<&str, &str>, ) -> Value { let timestamp = iso_ms(obs.ts_ms); + // Every per-call observation nests under its run's span so the trace renders + // as a tree (agent → generation/tool), matching LangChain's run hierarchy. + let parent = run_span_id(trace_id, obs.run_id.as_str()); // Attach only run lineage, offset, and the event *kind* — not the full event // payload. The event's meaningful fields (input/output/usage/name) are // already lifted into the observation `body`; embedding the whole event here @@ -340,6 +514,7 @@ fn observation_event( "body": clean_nulls(json!({ "id": scoped_observation_id(trace_id, call_id.as_str()), "traceId": trace_id, + "parentObservationId": parent, "name": "model", // The model the matching `ModelStarted` announced for this // call. Langfuse maps its pricing table off this field, so @@ -396,6 +571,7 @@ fn observation_event( "body": clean_nulls(json!({ "id": scoped_observation_id(trace_id, call_id.as_str()), "traceId": trace_id, + "parentObservationId": parent, "name": tool_name, "startTime": started_at_ms.map(iso_ms).unwrap_or_else(|| timestamp.clone()), "endTime": end_time, @@ -407,20 +583,8 @@ fn observation_event( })), }) } - AgentEvent::RunFailed { error, .. } => json!({ - "id": obs.event_id.as_str(), - "timestamp": timestamp, - "type": "event-create", - "body": clean_nulls(json!({ - "id": obs.event_id.as_str(), - "traceId": trace_id, - "name": obs.event.kind(), - "startTime": timestamp, - "level": "ERROR", - "statusMessage": error, - "metadata": metadata, - })), - }), + // Run-lifecycle events (RunStarted/RunCompleted/RunFailed) never reach + // here — they are consumed into the run span by `is_run_lifecycle`. _ => json!({ "id": obs.event_id.as_str(), "timestamp": timestamp, @@ -428,6 +592,7 @@ fn observation_event( "body": clean_nulls(json!({ "id": obs.event_id.as_str(), "traceId": trace_id, + "parentObservationId": parent, "name": obs.event.kind(), "startTime": timestamp, "metadata": metadata, @@ -436,6 +601,16 @@ fn observation_event( } } +/// Derives a stable score id from its target and name so re-scoring the same +/// (trace, observation, metric) upserts the existing score rather than +/// duplicating it. A trace-level score omits the observation segment. +fn default_score_id(score: &LangfuseScore) -> String { + match &score.observation_id { + Some(obs) => format!("{}:{}:score:{}", score.trace_id, obs, score.name), + None => format!("{}:score:{}", score.trace_id, score.name), + } +} + /// Build a globally-unique-yet-stable Langfuse observation id for a call-scoped /// observation (model generation / tool span). Langfuse keys observations by /// `id` across the whole project and upserts on collision, so we prefix the diff --git a/src/harness/observability/langfuse/test.rs b/src/harness/observability/langfuse/test.rs index db8049b..fbc68d0 100644 --- a/src/harness/observability/langfuse/test.rs +++ b/src/harness/observability/langfuse/test.rs @@ -177,28 +177,47 @@ fn populates_generation_and_tool_io_when_captured() { .unwrap(); let events = batch["batch"].as_array().unwrap(); - assert_eq!(events[1]["type"], "generation-create"); - assert_eq!(events[1]["body"]["input"][0]["content"], "hi"); - assert_eq!(events[1]["body"]["output"]["content"], "hello there"); - assert_eq!(events[2]["type"], "span-create"); - assert_eq!(events[2]["body"]["input"]["query"], "weather"); - assert_eq!(events[2]["body"]["output"], "sunny"); + let generation = events + .iter() + .find(|e| e["type"] == "generation-create") + .expect("a generation-create observation"); + let tool = events + .iter() + .find(|e| e["type"] == "span-create" && e["body"]["name"] == "lookup") + .expect("the tool span"); + assert_eq!(generation["body"]["input"][0]["content"], "hi"); + assert_eq!(generation["body"]["output"]["content"], "hello there"); + assert_eq!(tool["body"]["input"]["query"], "weather"); + assert_eq!(tool["body"]["output"], "sunny"); // The loop-captured start time gives each observation a real duration // (start < end) instead of the zero-width start == end point. - assert_eq!(events[1]["body"]["startTime"], iso_ms(1_704_067_199_000)); - assert_eq!(events[1]["body"]["endTime"], iso_ms(1_704_067_200_000)); - assert_eq!(events[2]["body"]["startTime"], iso_ms(1_704_067_199_500)); + assert_eq!(generation["body"]["startTime"], iso_ms(1_704_067_199_000)); + assert_eq!(generation["body"]["endTime"], iso_ms(1_704_067_200_000)); + assert_eq!(tool["body"]["startTime"], iso_ms(1_704_067_199_500)); // The tool span's end time is now start + the event's own duration_ms // (1_704_067_199_500 + 250), a real execution window rather than the // journal-append timestamp. - assert_eq!(events[2]["body"]["endTime"], iso_ms(1_704_067_199_750)); + assert_eq!(tool["body"]["endTime"], iso_ms(1_704_067_199_750)); // Result size rides metadata even though this fixture also captured output. - assert_eq!(events[2]["body"]["metadata"]["output_bytes"], 5); + assert_eq!(tool["body"]["metadata"]["output_bytes"], 5); + + // Both the generation and the tool span nest under the run's span, which + // itself sits directly under the trace — the LangChain run-tree shape. + assert_eq!( + generation["body"]["parentObservationId"], + "root-1:run:run-1" + ); + assert_eq!(tool["body"]["parentObservationId"], "root-1:run:run-1"); + let run_span = events + .iter() + .find(|e| e["body"]["id"] == "root-1:run:run-1") + .expect("a run span"); + assert_eq!(run_span["type"], "span-create"); // Observation metadata carries only lineage + event kind, not the whole // event payload (which would duplicate input/output already in `body`). - let gen_meta = &events[1]["body"]["metadata"]; + let gen_meta = &generation["body"]["metadata"]; assert_eq!(gen_meta["event_kind"], "model.completed"); assert!( gen_meta.get("event").is_none(), @@ -269,7 +288,11 @@ fn call_scoped_observation_ids_are_unique_per_trace() { .as_str() .unwrap() .to_string(); - let span_id = events.iter().find(|e| e["type"] == "span-create").unwrap()["body"]["id"] + // The tool span, not the run span (which is also a `span-create`). + let span_id = events + .iter() + .find(|e| e["type"] == "span-create" && e["body"]["name"] == "lookup") + .unwrap()["body"]["id"] .as_str() .unwrap() .to_string(); @@ -292,6 +315,109 @@ fn call_scoped_observation_ids_are_unique_per_trace() { assert_eq!(span_b, "thread-B:turn-2:agent_turn-tool-1"); } +#[test] +fn sub_agent_run_nests_under_its_parent_run_span() { + // A child run (distinct run_id, parented to the top-level run) exported in + // its own batch must reference its parent's run span so the two batches + // reconstruct one recursion tree under a shared trace. + let client = + LangfuseClient::proxy("https://backend.test/telemetry/langfuse/ingestion", "t").unwrap(); + + let child = AgentObservation { + event_id: EventId::new("evt-child"), + run_id: RunId::new("child-run"), + parent_run_id: Some(RunId::new("run-1")), + root_run_id: RunId::new("root-1"), + offset: 0, + ts_ms: 1_704_067_200_500, + event: AgentEvent::ModelCompleted { + call_id: CallId::new("child-model"), + started_at_ms: None, + usage: None, + input: None, + output: None, + }, + }; + + let batch = client + .build_ingestion_batch( + LangfuseTraceConfig { + trace_id: Some("root-1".to_string()), + ..Default::default() + }, + &[child], + ) + .unwrap(); + let events = batch["batch"].as_array().unwrap(); + + // The child run gets a "sub-agent" span parented to the parent run's span, + // and its generation nests under that sub-agent span. + let run_span = events + .iter() + .find(|e| e["type"] == "span-create" && e["body"]["name"] == "sub-agent") + .expect("a sub-agent run span"); + assert_eq!(run_span["body"]["id"], "root-1:run:child-run"); + assert_eq!( + run_span["body"]["parentObservationId"], "root-1:run:run-1", + "the sub-agent span nests under its parent run's span (from the parent batch)" + ); + let generation = events + .iter() + .find(|e| e["type"] == "generation-create") + .expect("the child generation"); + assert_eq!( + generation["body"]["parentObservationId"], + "root-1:run:child-run" + ); +} + +#[test] +fn run_span_carries_run_error_and_window() { + // RunStarted/RunFailed are folded into the run span rather than emitted as + // standalone events: the span carries the start/end window and ERROR status. + let client = + LangfuseClient::proxy("https://backend.test/telemetry/langfuse/ingestion", "t").unwrap(); + let batch = client + .build_ingestion_batch( + LangfuseTraceConfig::default(), + &[ + obs( + 0, + AgentEvent::RunStarted { + run_id: RunId::new("run-1"), + thread_id: None, + }, + ), + obs( + 1, + AgentEvent::RunFailed { + run_id: RunId::new("run-1"), + error: "boom".to_string(), + }, + ), + ], + ) + .unwrap(); + let events = batch["batch"].as_array().unwrap(); + + // No standalone run.started / run.failed events survive. + assert!( + !events + .iter() + .any(|e| e["body"]["name"] == "run.started" || e["body"]["name"] == "run.failed"), + "run-lifecycle events are consumed into the run span" + ); + let run_span = events + .iter() + .find(|e| e["body"]["id"] == "root-1:run:run-1") + .expect("a run span"); + assert_eq!(run_span["type"], "span-create"); + assert_eq!(run_span["body"]["level"], "ERROR"); + assert_eq!(run_span["body"]["statusMessage"], "boom"); + assert_eq!(run_span["body"]["startTime"], iso_ms(1_704_067_200_000)); + assert_eq!(run_span["body"]["endTime"], iso_ms(1_704_067_200_001)); +} + #[test] fn merges_caller_trace_metadata_over_defaults() { let client = @@ -318,3 +444,58 @@ fn merges_caller_trace_metadata_over_defaults() { assert_eq!(meta["root_run_id"], "override"); assert_eq!(meta["run_id"], "run-1"); } + +#[test] +fn builds_numeric_trace_score() { + let client = + LangfuseClient::proxy("https://backend.test/telemetry/langfuse/ingestion", "t").unwrap(); + let batch = client.build_score_batch( + LangfuseScore::numeric("trace-1", "helpfulness", 0.9).with_comment("solid answer"), + ); + let event = &batch["batch"].as_array().unwrap()[0]; + assert_eq!(event["type"], "score-create"); + assert_eq!(event["body"]["traceId"], "trace-1"); + assert_eq!(event["body"]["name"], "helpfulness"); + assert_eq!(event["body"]["value"], 0.9); + assert_eq!(event["body"]["dataType"], "NUMERIC"); + assert_eq!(event["body"]["comment"], "solid answer"); + // A trace-level score derives a deterministic, observation-free id. + assert_eq!(event["body"]["id"], "trace-1:score:helpfulness"); + assert!(event["body"].get("observationId").is_none()); +} + +#[test] +fn builds_categorical_and_boolean_scores() { + let client = + LangfuseClient::proxy("https://backend.test/telemetry/langfuse/ingestion", "t").unwrap(); + + let categorical = + client.build_score_batch(LangfuseScore::categorical("trace-1", "verdict", "correct")); + let cat_body = &categorical["batch"].as_array().unwrap()[0]["body"]; + assert_eq!(cat_body["value"], "correct"); + assert_eq!(cat_body["dataType"], "CATEGORICAL"); + + let boolean = client.build_score_batch(LangfuseScore::boolean("trace-1", "passed", true)); + let bool_body = &boolean["batch"].as_array().unwrap()[0]["body"]; + // Boolean scores serialize as 1/0 with a BOOLEAN data type. + assert_eq!(bool_body["value"], 1); + assert_eq!(bool_body["dataType"], "BOOLEAN"); +} + +#[test] +fn score_can_target_a_single_observation() { + let client = + LangfuseClient::proxy("https://backend.test/telemetry/langfuse/ingestion", "t").unwrap(); + let batch = client.build_score_batch( + LangfuseScore::numeric("trace-1", "faithfulness", 1.0) + .on_observation("trace-1:agent_turn-model-1"), + ); + let body = &batch["batch"].as_array().unwrap()[0]["body"]; + assert_eq!(body["observationId"], "trace-1:agent_turn-model-1"); + // The id namespaces the observation so per-generation scores don't collide + // with the trace-level score of the same name. + assert_eq!( + body["id"], + "trace-1:trace-1:agent_turn-model-1:score:faithfulness" + ); +} diff --git a/src/harness/observability/langfuse/types.rs b/src/harness/observability/langfuse/types.rs index 61cdef1..2263f1d 100644 --- a/src/harness/observability/langfuse/types.rs +++ b/src/harness/observability/langfuse/types.rs @@ -51,6 +51,130 @@ pub struct LangfuseTraceConfig { pub metadata: Value, } +/// The value of a [`LangfuseScore`], typed the way Langfuse classifies scores. +/// +/// Langfuse stores three score data types; the variant selected here maps to +/// the `dataType` and the shape of the `value` field in the `score-create` +/// ingestion event (a number for numeric/boolean, a string for categorical). +#[derive(Clone, Debug, PartialEq)] +pub enum LangfuseScoreValue { + /// A continuous numeric score (`dataType: "NUMERIC"`). + Numeric(f64), + /// A discrete label (`dataType: "CATEGORICAL"`), e.g. `"correct"`. + Categorical(String), + /// A pass/fail score (`dataType: "BOOLEAN"`), serialized as `1`/`0`. + Boolean(bool), +} + +impl LangfuseScoreValue { + /// Returns the Langfuse `dataType` string for this value. + pub fn data_type(&self) -> &'static str { + match self { + LangfuseScoreValue::Numeric(_) => "NUMERIC", + LangfuseScoreValue::Categorical(_) => "CATEGORICAL", + LangfuseScoreValue::Boolean(_) => "BOOLEAN", + } + } + + /// Returns the JSON `value` payload: a number for numeric/boolean scores, a + /// string for categorical ones. + pub fn to_value(&self) -> Value { + match self { + LangfuseScoreValue::Numeric(n) => Value::from(*n), + LangfuseScoreValue::Categorical(s) => Value::from(s.clone()), + LangfuseScoreValue::Boolean(b) => Value::from(if *b { 1 } else { 0 }), + } + } +} + +/// An evaluation score attached to a Langfuse trace or a single observation. +/// +/// Mirrors Langfuse's `createScore` / `score-create` ingestion event: a named, +/// typed value scoped to a `trace_id` and optionally narrowed to one +/// `observation_id` (a specific generation or span), with an optional free-text +/// comment. This is how post-hoc evaluations — human ratings, automated +/// LLM-as-judge checks, regression metrics — are correlated back to the run +/// that produced them. +#[derive(Clone, Debug, PartialEq)] +pub struct LangfuseScore { + /// The trace the score is attached to. + pub trace_id: String, + /// A specific observation (generation/span) to scope the score to, when the + /// score grades one step rather than the whole trace. + pub observation_id: Option, + /// The score name (its metric key), e.g. `"helpfulness"`. + pub name: String, + /// The typed score value. + pub value: LangfuseScoreValue, + /// Optional free-text rationale stored alongside the score. + pub comment: Option, + /// Stable score id for idempotent re-ingestion. Defaults (in + /// [`LangfuseClient::build_score_batch`]) to a deterministic id derived from + /// the trace, observation, and name, so re-scoring the same target updates + /// the existing score instead of creating a duplicate. + pub id: Option, +} + +impl LangfuseScore { + /// Builds a trace-level numeric score. + pub fn numeric(trace_id: impl Into, name: impl Into, value: f64) -> Self { + Self { + trace_id: trace_id.into(), + observation_id: None, + name: name.into(), + value: LangfuseScoreValue::Numeric(value), + comment: None, + id: None, + } + } + + /// Builds a trace-level categorical score. + pub fn categorical( + trace_id: impl Into, + name: impl Into, + value: impl Into, + ) -> Self { + Self { + trace_id: trace_id.into(), + observation_id: None, + name: name.into(), + value: LangfuseScoreValue::Categorical(value.into()), + comment: None, + id: None, + } + } + + /// Builds a trace-level boolean score. + pub fn boolean(trace_id: impl Into, name: impl Into, value: bool) -> Self { + Self { + trace_id: trace_id.into(), + observation_id: None, + name: name.into(), + value: LangfuseScoreValue::Boolean(value), + comment: None, + id: None, + } + } + + /// Scopes the score to a single observation (generation/span) id. + pub fn on_observation(mut self, observation_id: impl Into) -> Self { + self.observation_id = Some(observation_id.into()); + self + } + + /// Attaches a free-text comment to the score. + pub fn with_comment(mut self, comment: impl Into) -> Self { + self.comment = Some(comment.into()); + self + } + + /// Overrides the auto-derived score id (for a caller-controlled identity). + pub fn with_id(mut self, id: impl Into) -> Self { + self.id = Some(id.into()); + self + } +} + /// Async Langfuse ingestion client. #[derive(Clone, Debug)] pub struct LangfuseClient { diff --git a/src/harness/observability/mod.rs b/src/harness/observability/mod.rs index d7492c4..477f154 100644 --- a/src/harness/observability/mod.rs +++ b/src/harness/observability/mod.rs @@ -30,7 +30,9 @@ mod worker; pub(crate) use worker::{AppendWorker, DEFAULT_DRAIN_CAPACITY}; -pub use langfuse::{LangfuseAuth, LangfuseClient, LangfuseTraceConfig}; +pub use langfuse::{ + LangfuseAuth, LangfuseClient, LangfuseScore, LangfuseScoreValue, LangfuseTraceConfig, +}; // Shared Langfuse payload helpers reused by the graph observability exporter so // ISO-8601 timestamp formatting and null-field pruning live in one place. pub(crate) use langfuse::{clean_nulls, iso_ms}; diff --git a/src/lib.rs b/src/lib.rs index 3299563..178b940 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -147,7 +147,9 @@ pub use harness::observability::{ HarnessStatusStore, InMemoryEventJournal, InMemoryStatusStore, JournalSink, JsonlSink, RedactingSink, StoreEventJournal, }; -pub use harness::observability::{LangfuseAuth, LangfuseClient, LangfuseTraceConfig}; +pub use harness::observability::{ + LangfuseAuth, LangfuseClient, LangfuseScore, LangfuseScoreValue, LangfuseTraceConfig, +}; // --- Graph: durable execution model (LangGraph-style) --- // Re-exported with explicit names so the durable API is discoverable at the diff --git a/tests/e2e_observability.rs b/tests/e2e_observability.rs index 678e3e7..1e7a693 100644 --- a/tests/e2e_observability.rs +++ b/tests/e2e_observability.rs @@ -272,12 +272,29 @@ async fn harness_run_with_capture_exports_generation_and_tool_io() { // Tool observations are exported as `span-create` (a valid Langfuse // ingestion type); `tool-create` is rejected by older/self-hosted Langfuse. + // Select by input payload since the per-run "agent" span is also a + // `span-create` (the run tree's root), sitting above the tool span. let tool = events .iter() - .find(|e| e["type"] == "span-create") + .find(|e| e["type"] == "span-create" && e["body"].get("input").is_some()) .expect("a tool observation"); assert_eq!(tool["body"]["input"]["q"], "weather"); assert_eq!(tool["body"]["output"], "tool-output"); + + // Every generation and the tool span nest under the run's span, so the + // exported trace renders as a tree rather than a flat event list. + let run_span = events + .iter() + .find(|e| e["type"] == "span-create" && e["body"]["name"] == "agent") + .expect("a run span"); + let run_span_id = run_span["body"]["id"].as_str().unwrap(); + assert_eq!(tool["body"]["parentObservationId"], run_span_id); + assert!( + generations + .iter() + .all(|g| g["body"]["parentObservationId"] == run_span_id), + "every generation nests under the run span" + ); } /// A two-node line graph over `i32` with overwrite semantics: `a -> b`. @@ -404,4 +421,24 @@ async fn graph_observations_export_to_langfuse_trace_and_spans() { .any(|e| e["type"] == "span-create" && e["body"]["level"] == "ERROR"), "healthy run has no ERROR spans" ); + + // The graph run is a structural span under the trace, and every step nests + // beneath it. Its id follows the `{trace}:run:{run}` scheme the harness + // exporter parents its agent run spans to, so a graph run and the sub-agent + // runs its nodes spawn reconstruct a single nested tree under one trace. + let graph_run_span = format!("{run_id}:run:{run_id}"); + let run_span = events + .iter() + .find(|e| e["body"]["id"] == graph_run_span) + .expect("graph run span"); + assert_eq!(run_span["type"], "span-create"); + assert!(run_span["body"].get("parentObservationId").is_none()); + let step_1 = events + .iter() + .find(|e| e["body"]["id"] == format!("{run_id}:step:1")) + .expect("step 1 span"); + assert_eq!(step_1["body"]["parentObservationId"], graph_run_span); + // A harness agent run spawned by a graph node carries parent_run_id == the + // graph run id, so its harness run span resolves to exactly this parent id + // and nests under the graph run rather than floating at the trace root. }