Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,66 @@ def my_auth_callback(request):
context.metadata.trace_user_email = user_info.get("email")
```

## OTel-Based Exporter

Weave also supports an [OTel-based integration](https://weave-docs.wandb.ai/guides/tracking/otel) that sends traces via OpenTelemetry. This uses the `opentelemetry` plugin instead of the `weave` plugin, which means it does not require installing `weave` as a dependency — only the `opentelemetry` extra is needed.

### Install the OTel plugin

::::{tab-set}
:sync-group: install-tool

:::{tab-item} source
:selected:
:sync: source

```bash
uv pip install -e ".[opentelemetry]"
```

:::

:::{tab-item} package
:sync: package

```bash
uv pip install "nvidia-nat[opentelemetry]"
```

:::

::::

### Configuration

Update your workflow configuration to use the `weave_otel` exporter type:

```yaml
general:
telemetry:
tracing:
weave_otel:
_type: weave_otel
project: "nat-demo"
entity: "your-team-name"
```

| Parameter | Description | Example |
|-----------|-------------|---------|
| `project` | The name of your W&B Weave project | `"nat-demo"` |
| `entity` | Your W&B username or team name | `"your-team-name"` |
| `endpoint` | The W&B Weave OTel endpoint (optional) | `"https://trace.wandb.ai/otel/v1/traces"` |
| `api_key` | The W&B API key (optional, falls back to `WANDB_API_KEY` env var) | — |

### Run the Workflow

```bash
nat run --config_file examples/observability/simple_calculator_observability/configs/config-weave-otel.yml --input "What's the sum of 7 and 8?"
```

## Resources

- Learn more about tracing [here](https://weave-docs.wandb.ai/guides/tracking/tracing).
- Learn more about how to navigate the logged traces [here](https://weave-docs.wandb.ai/guides/tracking/trace-tree).
- Learn more about PII redaction [here](https://weave-docs.wandb.ai/guides/tracking/redact-pii).
- Learn more about the OTel-based integration [here](https://weave-docs.wandb.ai/guides/tracking/otel).
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,22 @@ For simple local development and debugging, you can export traces directly to a

For detailed Weave setup instructions, refer to the [Fine-grained Tracing with Weave](../../../docs/source/run-workflows/observe/observe-workflow-with-weave.md) guide.

### Weave OTel Integration

Weave also supports an [OTel-based integration](https://weave-docs.wandb.ai/guides/tracking/otel) that sends traces via OpenTelemetry. This approach uses the `opentelemetry` plugin instead of the `weave` plugin, which means it does not require installing `weave` as a dependency.

1. Set your Weights & Biases API key:

```bash
export WANDB_API_KEY=<your_api_key>
```

2. Run the workflow:

```bash
nat run --config_file examples/observability/simple_calculator_observability/configs/config-weave-otel.yml --input "What's the sum of 7 and 8?"
```

### AI Safety Monitoring with Patronus

[Patronus](https://patronus.ai/) enables AI safety monitoring and compliance tracking.
Expand Down Expand Up @@ -339,6 +355,7 @@ The example includes multiple configuration files for different observability pl
| `config-langfuse.yml` | Langfuse | Langfuse monitoring and analytics |
| `config-langsmith.yml` | LangSmith | LangChain/LangGraph ecosystem integration |
| `config-weave.yml` | Weave | Workflow-focused tracking |
| `config-weave-otel.yml` | Weave (OTel) | W&B Weave OTel tracing and monitoring integration |
| `config-patronus.yml` | Patronus | AI safety and compliance monitoring |
| `config-catalyst.yml` | Catalyst | RagaAI Catalyst integration |
| `config-galileo.yml` | Galileo | Galileo integration |
Expand Down
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You added this configuration, but the README wasn't updated.

examples/observability/simple_calculator_observability/README.md

The documentation should also be updated accordingly to reflect that W&B Weave has an OTel exporter.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it! It should be updated now. I put the OTel guide in the existing Weave docs but happy to place it into a separate one if that's preferable

Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


general:
telemetry:
tracing:
weave_otel:
_type: weave_otel
project: "nat-demo"
entity: "your-team-name"

function_groups:
calculator:
_type: calculator

functions:
current_datetime:
_type: current_datetime

llms:
nim_llm:
_type: nim
model_name: meta/llama-3.1-70b-instruct
temperature: 0.0
max_tokens: 1024

workflow:
_type: react_agent
tool_names: [calculator, current_datetime]
llm_name: nim_llm
verbose: true
parse_agent_response_max_retries: 3
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import logging
import os
from collections.abc import AsyncGenerator

from pydantic import Field

Expand Down Expand Up @@ -209,6 +210,49 @@ async def galileo_telemetry_exporter(config: GalileoTelemetryExporter, builder:
)


class WeaveOtelTelemetryExporter(BatchConfigMixin, TelemetryExporterBaseConfig, name="weave_otel"):
"""A telemetry exporter to transmit traces to Weights & Biases Weave via OTel."""

endpoint: str = Field(
description="The W&B Weave OTel endpoint",
default="https://trace.wandb.ai/otel/v1/traces",
)
api_key: SerializableSecretStr = Field(description="The W&B API key",
default_factory=lambda: SerializableSecretStr(""))
project: str = Field(description="The W&B project name.")
entity: str = Field(description="The W&B username or team name.")


@register_telemetry_exporter(config_type=WeaveOtelTelemetryExporter)
async def weave_otel_telemetry_exporter(
config: WeaveOtelTelemetryExporter, builder: Builder,
) -> AsyncGenerator["OTLPSpanAdapterExporter", None]:
"""Create a Weave OTel telemetry exporter."""
Comment on lines +227 to +230
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the reported lint error on this file.
ruff check --select F821 packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 648


🏁 Script executed:

#!/bin/bash
# Find where OTLPSpanAdapterExporter is defined
rg -n "class OTLPSpanAdapterExporter" --type py

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 237


🏁 Script executed:

#!/bin/bash
# Check the current imports and structure of the file around the problematic section
head -230 packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py | tail -50

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 2318


🏁 Script executed:

#!/bin/bash
# Check the imports at the top of the register.py file
head -50 packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 2392


🏁 Script executed:

#!/bin/bash
# Verify the full function context and check if there are any existing TYPE_CHECKING imports
grep -n "TYPE_CHECKING\|from typing\|from collections" packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py | head -20

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 114


🏁 Script executed:

#!/bin/bash
# Check if there are TYPE_CHECKING imports elsewhere in the codebase for similar patterns
rg -n "TYPE_CHECKING" packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/ --type py

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 51


🏁 Script executed:

#!/bin/bash
# Verify the exact line numbers in the function to understand the context better
sed -n '225,260p' packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py

Repository: NVIDIA/NeMo-Agent-Toolkit

Length of output: 1484


Add TYPE_CHECKING import to resolve Ruff F821 for forward reference in return annotation.

Line 229 has a forward reference "OTLPSpanAdapterExporter" in the return type annotation that triggers Ruff F821 because the name is not in scope at type-checking time. The local import on line 231 executes at runtime, not during type checking.

Add a module-level import guard to satisfy the type checker:

Suggested fix
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from nat.plugins.opentelemetry import OTLPSpanAdapterExporter
+
 async def weave_otel_telemetry_exporter(
     config: WeaveOtelTelemetryExporter, builder: Builder,
 ) -> AsyncGenerator["OTLPSpanAdapterExporter", None]:

The local import inside the function (line 231) can remain for backward compatibility with runtime behavior.

🧰 Tools
🪛 Ruff (0.15.7)

[error] 229-229: Undefined name OTLPSpanAdapterExporter

(F821)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/nvidia_nat_opentelemetry/src/nat/plugins/opentelemetry/register.py`
around lines 227 - 230, The forward reference "OTLPSpanAdapterExporter" used in
the return annotation of weave_otel_telemetry_exporter triggers Ruff F821
because the symbol isn't visible at type-check time; add a module-level guarded
import under if TYPE_CHECKING: from <module> import OTLPSpanAdapterExporter (and
import TYPE_CHECKING from typing) so the type checker sees the name, while
keeping the existing local runtime import inside weave_otel_telemetry_exporter
for backward compatibility. Ensure you reference the exact class name
OTLPSpanAdapterExporter and the function weave_otel_telemetry_exporter when
making the change.


from nat.plugins.opentelemetry import OTLPSpanAdapterExporter

api_key = get_secret_value(config.api_key) if config.api_key else os.environ.get("WANDB_API_KEY")
if not api_key:
raise ValueError("API key is required for Weave (set api_key or WANDB_API_KEY env var)")

headers = {"wandb-api-key": api_key}
resource_attributes = {
"wandb.project": config.project,
"wandb.entity": config.entity,
}

yield OTLPSpanAdapterExporter(
endpoint=config.endpoint,
headers=headers,
resource_attributes=resource_attributes,
batch_size=config.batch_size,
flush_interval=config.flush_interval,
max_queue_size=config.max_queue_size,
drop_on_overflow=config.drop_on_overflow,
shutdown_timeout=config.shutdown_timeout,
)


class DBNLTelemetryExporter(BatchConfigMixin, TelemetryExporterBaseConfig, name="dbnl"):
"""A telemetry exporter to transmit traces to DBNL."""

Expand Down