harness/graph(langfuse): nest observations into a run tree + add scores API#53
Conversation
The harness Langfuse exporter emitted every model generation and tool span as a flat child of the trace, so a run — and especially sub-agent recursion — never surfaced as a tree the way LangChain's callback handler renders it. Emit one span per run (agent / sub-agent), fold the run-lifecycle events (started/completed/failed) into it as the start/end window and ERROR status, and parent every generation, tool span, and event to its run's span. Run spans carry deterministic, trace-namespaced ids, so a sub-agent run exported in its own batch nests under its parent run's span within the same trace.
Mirror Langfuse's createScore / score-create ingestion event so a run's exported trace (or a single generation/span within it) can be graded after the fact — human ratings, LLM-as-judge, regression metrics. Adds LangfuseScore (numeric/categorical/boolean, trace- or observation-scoped, with optional comment) and LangfuseClient::create_score / build_score_batch. Score ids are derived deterministically from the target and metric name so re-scoring upserts rather than duplicating.
…runs
Add a structural graph-run span (`{trace}:run:{run_id}`) directly under the
trace that supersteps, subgraphs, and point events parent to. The span reuses
the same id scheme the harness exporter parents its agent run spans to, so a
sub-agent a graph node spawns (whose `parent_run_id` is the graph run id)
nests directly under the graph-run span instead of floating at the trace root —
the graph, its nodes, and their agents now reconstruct one contiguous tree.
Also document the harness run tree and the new scores API in the module docs
and READMEs.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (12)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02739a297c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // a graph node spawns carries `parent_run_id == graph run id`, so its | ||
| // harness-exported run span nests directly under this graph-run span | ||
| // rather than floating at the trace root. | ||
| let run_parent = run_span_id(&trace_id, first.root_run_id.as_str()); |
There was a problem hiding this comment.
When graph observations are stamped with lineage, for example a graph run whose run_id is child-graph and whose root_run_id is the outer run, this creates the structural span as {trace}:run:{root_run_id} instead of {trace}:run:{run_id}. That collides with the root run span and, more importantly, harness runs spawned by the graph parent themselves to {trace}:run:{child-graph}, which this batch never creates, so nested graph/agent traces lose the tree promised by the exporter. Build the graph span from first.run_id and parent it via first.parent_run_id when present.
Useful? React with 👍 / 👎.
| /// exported trace — recall the harness/graph exporters default their trace | ||
| /// id to the run's `root_run_id` — to attach an evaluation to a run. | ||
| pub fn build_score_batch(&self, score: LangfuseScore) -> Value { | ||
| let timestamp = iso_ms(now_ms()); |
There was a problem hiding this comment.
Langfuse identifies a score update by id, name, and the calendar date of timestamp (see https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk#preventing-duplicate-scores), so a stable id alone is not enough. Because build_score_batch always stamps now_ms() and LangfuseScore has no timestamp override, rescoring the same trace/observation on a later UTC date produces a second score instead of updating the previous one, despite the idempotency guarantee in this API.
Useful? React with 👍 / 👎.
1 participant
Summary
Our Langfuse integration surfaced observations as a flat list under the trace rather than the nested run tree LangChain's callback handler produces — so a run, and especially sub-agent recursion, never rendered as a graph. This PR maps the LangGraph/LangChain → Langfuse touch points onto our exporters.
1. Harness run tree
The harness exporter emitted every model generation and tool span as a direct child of the trace. Now it emits one
span-createper run ({trace}:run:{run_id}, namedagent/sub-agent), folds theRunStarted/RunCompleted/RunFailedevents into that span's start/end window and ERROR status, and nests every generation, tool span, and event under it viaparentObservationId. Run-span ids are deterministic and trace-namespaced, so a sub-agent run exported in its own batch nests under its parent's span in the same trace — matching LangChain's cross-run hierarchy.2. Scores API (
createScore)Adds
LangfuseScore(numeric/categorical/boolean; trace- or observation-scoped; optional comment) andLangfuseClient::create_score/build_score_batch, mirroring Langfuse'screateScore— the way LangGraph traces are graded after the fact (human ratings, LLM-as-judge, regression metrics). Score ids default deterministically so re-scoring upserts rather than duplicating.3. Graph ↔ agent unification
The graph exporter now emits a structural graph-run span (
{trace}:run:{run_id}) that supersteps, subgraphs, and point events parent to, reusing the same id scheme the harness parents its agent run spans to. A sub-agent a node spawns carriesparent_run_id == graph run id, so its harness-exported run span resolves to{trace}:run:{graph_run_id}and nests directly under the graph-run span — the graph, its nodes, and their agents form one contiguous tree.Out of scope
Prompt management/linking (
createPrompt/getPrompt, generation→prompt version). This repo has no prompt store, so wiring it would be speculative dead code — it's a separate product feature, not an observability gap.Commands run locally
cargo fmt --check— cleancargo clippy --all-targets -- -D warnings— cleancargo test— all pass (1044 lib + integration tests)API changes
LangfuseScore,LangfuseScoreValue(viaharness::observabilityand crate root)LangfuseClient::create_score,LangfuseClient::build_score_batchspan-creates andparentObservationIdon generations/tools; graph batches include a graph-runspan-createand re-parent steps/subgraphs/point-events to it. Docs and tests updated accordingly.https://claude.ai/code/session_01SLevgujKgUiKbektueRSND