Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions docs/adapters/frameworks-agno.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,24 +54,27 @@ sink.close()

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First `run` per agent. |
| `agent.input` | L1 | Beginning of every `run` / `arun`. |
| `agent.output` | L1 | End of every `run` / `arun`. |
| `agent.action` | L4a | Per intermediate reasoning step. |
| `agent.handoff` | L4a | When a team agent delegates to a sub-agent. |
| `agent.state.change` | cross-cutting | Memory mutations. |
| `tool.call` | L5a | Per tool invocation. |
| `model.invoke` | L3 | Per LLM call. |
| `environment.config` | L4a | First `run` per agent (`lifecycle.py:462`). |
| `agent.input` | L1 | Beginning of every `run` / `arun` (`lifecycle.py:289`). |
| `agent.output` | L1 | End of every `run` / `arun` (`lifecycle.py:324`). |
| `agent.handoff` | L4a | When a team agent delegates to a sub-agent (`lifecycle.py:267,398`). |
| `agent.state.change` | cross-cutting | Run completion / failure (`lifecycle.py:325`). |
| `tool.call` | L5a | Per tool invocation (`lifecycle.py:246,358`). |
| `model.invoke` | L3 | Per LLM call (`lifecycle.py:216,388`). |
| `cost.record` | cross-cutting | Per LLM call when `result.metrics` / `result.usage` are populated (`lifecycle.py:228`). |

## Agno specifics

- **Teams**: Agno supports multi-agent teams via `Team(agents=[...])`.
Each team member must be instrumented individually with
`adapter.instrument_agent(team_member)` — or call
`instrument_agent(team)` and the convenience helper recurses.
- **Reasoning agents**: when `reasoning=True` is set on an Agent, the
intermediate reasoning steps emit `agent.action` events with a
`step_index` field.
- **Reasoning agents**: Agno's `reasoning=True` mode is not specially
instrumented — intermediate reasoning steps are not surfaced as
separate events. The reasoning chain is observable only via the
inputs/outputs already captured on `agent.input` / `agent.output` and
any inner `model.invoke` / `tool.call` events the underlying agent
emits.
- **Storage backends**: Agno session storage (Postgres, sqlite, Redis,
etc.) emits `agent.state.change` on every save.

Expand All @@ -83,7 +86,8 @@ from layerlens.instrument.adapters._base import CaptureConfig
# Recommended.
adapter = AgnoAdapter(capture_config=CaptureConfig.standard())

# Heavy: include reasoning steps as agent.code (the chain-of-thought).
# Heavy: enable all capture layers (L2 agent.code currently has no
# Agno-specific emission sites — reasoning is not specially instrumented).
adapter = AgnoAdapter(
capture_config=CaptureConfig(
l1_agent_io=True,
Expand Down
15 changes: 8 additions & 7 deletions docs/adapters/frameworks-bedrock_agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ provided `bedrock-agent-runtime` client:
emits `agent.input` and `environment.config` on first agent encounter.
- `after-call.bedrock-agent-runtime.InvokeAgent` — fires after the response
comes back. Walks the `trace` blocks in the streamed events and emits
`model.invoke` / `tool.call` / `agent.action` per trace step.
`model.invoke` / `tool.call` / `agent.handoff` per trace step
(`lifecycle.py:200-213`).

`disconnect()` unregisters both hooks.

Expand All @@ -67,12 +68,12 @@ provided `bedrock-agent-runtime` client:
| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First `InvokeAgent` per `agentId`. |
| `agent.input` | L1 | Beginning of every `InvokeAgent`. |
| `agent.output` | L1 | End of every `InvokeAgent` (after stream consumption). |
| `agent.action` | L4a | Per `orchestrationTrace.modelInvocationInput` block. |
| `agent.handoff` | L4a | Per cross-agent collaboration step. |
| `tool.call` | L5a | Per `actionGroupInvocationInput` / `knowledgeBaseLookupInput` block. |
| `model.invoke` | L3 | Per `modelInvocationOutput` block (with token usage). |
| `agent.input` | L1 | Beginning of every `InvokeAgent` (`lifecycle.py:289`). |
| `agent.output` | L1 | End of every `InvokeAgent` after stream consumption (`lifecycle.py:323`). |
| `agent.handoff` | L4a | Per `AGENT_COLLABORATOR` trace step (`lifecycle.py:269,386`). |
| `tool.call` | L5a | Per `ACTION_GROUP` / `KNOWLEDGE_BASE` trace step (`lifecycle.py:217,230,348`). |
| `model.invoke` | L3 | Per `MODEL_INVOCATION` trace step with token usage (`lifecycle.py:254,377`). |
| `cost.record` | cross-cutting | Per `MODEL_INVOCATION` when `usage` is present (`lifecycle.py:256`). |

## Bedrock Agents specifics

Expand Down
35 changes: 19 additions & 16 deletions docs/adapters/frameworks-llama_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,15 +46,15 @@ dispatches.
## What's wrapped

`adapter.instrument_workflow(...)` registers a `BaseEventHandler` with
`llama_index.core.instrumentation.get_dispatcher()`. The handler observes:
`llama_index.core.instrumentation.get_dispatcher()`. The handler observes
(`lifecycle.py:185-198`):

- LLM events (`LLMChatStartEvent`, `LLMChatEndEvent`,
`LLMCompletionStartEvent`, `LLMCompletionEndEvent`)
- Tool events (`AgentToolCallEvent`)
- Agent events (`AgentRunStepStartEvent`, `AgentRunStepEndEvent`,
`AgentChatWithStepStartEvent`, `AgentChatWithStepEndEvent`)
- Retrieval events (`RetrievalStartEvent`, `RetrievalEndEvent`)
- Embedding events (`EmbeddingStartEvent`, `EmbeddingEndEvent`)
- LLM events: `LLMChatStartEvent`, `LLMStartEvent` (start);
`LLMChatEndEvent`, `LLMCompletionEndEvent` (end)
- Tool events: `ToolCallEvent`
- Agent events: `AgentRunStepStartEvent`, `AgentRunStepEndEvent`
- Retrieval / query events: `RetrievalStartEvent`, `QueryStartEvent`
(start); `RetrievalEndEvent`, `QueryEndEvent` (end)

`disconnect()` removes the handler from the dispatcher's
`event_handlers` list, restoring the original behaviour.
Expand All @@ -63,20 +63,23 @@ dispatches.

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First agent / workflow event per process. |
| `agent.input` | L1 | `AgentChatWithStepStartEvent` / agent step start. |
| `agent.output` | L1 | `AgentChatWithStepEndEvent` / agent step end. |
| `agent.action` | L4a | Per `AgentRunStepEndEvent`. |
| `tool.call` | L5a | Per `AgentToolCallEvent`. |
| `model.invoke` | L3 | Per LLM start/end pair. |
| `environment.config` | L4a | First `AgentRunStepStartEvent` per agent (`lifecycle.py:261,428`). |
| `agent.input` | L1 | Per `AgentRunStepStartEvent` (`lifecycle.py:266`). |
| `agent.output` | L1 | Per `AgentRunStepEndEvent` (`lifecycle.py:283`). |
| `tool.call` | L5a | Per `ToolCallEvent` and per `RetrievalEndEvent` / `QueryEndEvent` (tool_name = `"retrieval"`, tool_type = `"retrieval"`) (`lifecycle.py:231,246`). |
| `model.invoke` | L3 | Per LLM end event (`LLMChatEndEvent` / `LLMCompletionEndEvent`) (`lifecycle.py:216`). |
| `cost.record` | cross-cutting | Per LLM end event when token counts are present (`lifecycle.py:218`). |
| `agent.handoff` | L4a | Exposed via the `on_handoff` lifecycle hook (`lifecycle.py:400`) — not auto-emitted from the dispatcher event stream. |

## LlamaIndex specifics

- **Workflows**: the new `Workflow` class emits dispatcher events the same
way; the same handler captures both classic agents (`ReActAgent`,
`OpenAIAgent`) and workflow `@step` runs.
- **RAG retrievers**: retrieval events are surfaced as `tool.call` with
`tool_name="retriever"` and the resolved chunk count.
- **RAG retrievers**: `RetrievalEndEvent` and `QueryEndEvent` are surfaced
as `tool.call` with `tool_name="retrieval"`, `tool_type="retrieval"`,
and the resolved chunk count in `result_count`
(`lifecycle.py:244-257`).
- **Streaming**: streamed LLM responses fire one `LLMChatEndEvent` after
the final chunk; the adapter emits one consolidated `model.invoke`.
- **Span propagation**: LlamaIndex span IDs propagate into the event
Expand Down
21 changes: 12 additions & 9 deletions docs/adapters/frameworks-ms_agent_framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,14 @@ filters. `disconnect()` restores the originals.

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First wrap of each chat. |
| `agent.input` | L1 | Beginning of every `invoke` / `invoke_stream`. |
| `agent.output` | L1 | End of every invocation (per response). |
| `agent.action` | L4a | Per intermediate step. |
| `agent.handoff` | L4a | Per `AgentGroupChat` speaker turn. |
| `tool.call` | L5a | Per plugin function invocation. |
| `model.invoke` | L3 | Per LLM call. |
| `environment.config` | L4a | First wrap of each chat (`lifecycle.py:481`). |
| `agent.input` | L1 | Beginning of every `invoke` / `invoke_stream` (`lifecycle.py:299`). |
| `agent.output` | L1 | End of every invocation (per response) (`lifecycle.py:334`). |
| `agent.state.change` | cross-cutting | Per `invoke` / `invoke_stream` end (`lifecycle.py:335`). |
| `agent.handoff` | L4a | Per `AgentGroupChat` speaker turn (`lifecycle.py:223,408`). |
| `tool.call` | L5a | Per plugin function invocation (`lifecycle.py:237,247,368`). |
| `model.invoke` | L3 | Per LLM call (`lifecycle.py:262,398`). |
| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:272`). |

## MS Agent Framework specifics

Expand All @@ -83,8 +84,10 @@ filters. `disconnect()` restores the originals.
on each speaker turn.
- **Plugins**: Semantic Kernel plugin functions surface as `tool.call` —
the plugin name + function name combine into `tool_name`.
- **Multi-agent terminations**: configurable termination strategies
emit `agent.action` with `terminate_reason` when a group chat ends.
- **Multi-agent terminations**: configurable termination strategies are
not separately instrumented. When a group chat ends, the outer
`agent.output` carries the final response and the run completion fires
one `agent.state.change` (`lifecycle.py:335`).
- **Streaming**: `invoke_stream` emits one consolidated `model.invoke`
on stream completion; per-chunk text is accumulated.

Expand Down
17 changes: 9 additions & 8 deletions docs/adapters/frameworks-openai_agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,15 @@ events.

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First agent span observed. |
| `agent.input` | L1 | Per agent span start. |
| `agent.output` | L1 | Per agent span end. |
| `agent.action` | L4a | Per `response_span` (model call decision). |
| `agent.handoff` | L4a | Per `handoff_span`. |
| `tool.call` | L5a | Per `function_span`. |
| `model.invoke` | L3 | Per `generation_span` (model call). |
| `policy.violation` | cross-cutting | Per `guardrail_span` that fails. |
| `environment.config` | L4a | First agent span observed (`lifecycle.py:497`). |
| `agent.input` | L1 | Per agent span start (`lifecycle.py:256,358`). |
| `agent.output` | L1 | Per agent span end (`lifecycle.py:269,392`). |
| `agent.state.change` | cross-cutting | On agent span start/end (`lifecycle.py:193,211`). |
| `model.invoke` | L3 | Per `response_span` / `generation_span` model call (`lifecycle.py:293,446`). |
| `cost.record` | cross-cutting | Per model call when token counts are present (`lifecycle.py:295`). |
| `tool.call` | L5a | Per `function_span` (`lifecycle.py:308,417`). |
| `agent.handoff` | L4a | Per `handoff_span` (`lifecycle.py:325,463`). |
| `policy.violation` | cross-cutting | Per `guardrail_span` that fails (`lifecycle.py:338`). |

## OpenAI Agents specifics

Expand Down
20 changes: 12 additions & 8 deletions docs/adapters/frameworks-pydantic_ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,14 @@ sink.close()

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First wrap of each agent. |
| `agent.input` | L1 | Beginning of every `run` / `run_sync`. |
| `agent.output` | L1 | End of every `run` / `run_sync`. |
| `agent.action` | L4a | Per intermediate model step (multi-step runs). |
| `tool.call` | L5a | Per registered tool invocation. |
| `model.invoke` | L3 | Per LLM call (one per model step). |
| `environment.config` | L4a | First wrap of each agent (`lifecycle.py:407`). |
| `agent.input` | L1 | Beginning of every `run` / `run_sync` (`lifecycle.py:248`). |
| `agent.output` | L1 | End of every `run` / `run_sync` (`lifecycle.py:282`). |
| `agent.state.change` | cross-cutting | On run end (`lifecycle.py:283`). |
| `tool.call` | L5a | Per registered tool invocation (`lifecycle.py:227,315`). |
| `model.invoke` | L3 | Per LLM call (one per model step) (`lifecycle.py:218,344`). |
| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:203`). |
| `agent.handoff` | L4a | Cross-agent delegation (`lifecycle.py:353`). |

The `model.invoke` payload includes the model name (parsed from the
PydanticAI model spec like `openai:gpt-4o-mini`), token usage from
Expand All @@ -70,8 +72,10 @@ PydanticAI model spec like `openai:gpt-4o-mini`), token usage from

- **Structured results**: when an agent declares `result_type=MyModel`, the
validated Pydantic model is included in `agent.output` (subject to
`CaptureConfig.capture_content`). Validation errors emit
`policy.violation`.
`CaptureConfig.capture_content`). Validation errors are surfaced on
the `agent.output` payload (via the wrapping `try/except` in
`_create_traced_run`) — the adapter does not emit a separate
`policy.violation` event for these.
- **Model spec parsing**: PydanticAI accepts model spec strings like
`"openai:gpt-4o-mini"` or `"anthropic:claude-3-5-sonnet"`. The adapter
splits these into `provider` + `model` for downstream cost lookups.
Expand Down
15 changes: 7 additions & 8 deletions docs/adapters/frameworks-semantic_kernel.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,13 @@ and the kernel returns to its original behaviour.

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First plugin invocation per kernel. |
| `agent.input` | L1 | Function invocation start. |
| `agent.output` | L1 | Function invocation end (success or error). |
| `agent.code` | L2 | Per plugin function when `l2_agent_code` is true. |
| `agent.action` | L4a | Per planner step. |
| `agent.state.change` | cross-cutting | Memory store reads/writes. |
| `tool.call` | L5a | Per `auto_function_invocation` (model-selected plugin). |
| `model.invoke` | L3 | Per LLM call inside the kernel. |
| `environment.config` | L4a | First plugin invocation per kernel and per-plugin discovery (`lifecycle.py:203,442`). |
| `agent.input` | L1 | Kernel invoke start (`lifecycle.py:397`). |
| `agent.output` | L1 | Kernel invoke end — success or error (`lifecycle.py:423`). |
| `agent.code` | L2 | Per plugin function / prompt render (`lifecycle.py:271,355`). |
| `tool.call` | L5a | Per `auto_function_invocation` and per memory-store operation (`lifecycle.py:247,390`). |
| `model.invoke` | L3 | Per LLM call inside the kernel (`lifecycle.py:306`). |
| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:310`). |

## Semantic Kernel specifics

Expand Down
24 changes: 13 additions & 11 deletions docs/adapters/frameworks-strands.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,13 @@ hooks. `disconnect()` restores the originals.

| Event | Layer | When |
|---|---|---|
| `environment.config` | L4a | First wrap of each agent. |
| `agent.input` | L1 | Beginning of every `__call__` / `invoke`. |
| `agent.output` | L1 | End of every `__call__` / `invoke`. |
| `agent.action` | L4a | Per intermediate reasoning loop iteration. |
| `agent.handoff` | L4a | Multi-agent collaboration handoffs. |
| `tool.call` | L5a | Per Strands tool invocation. |
| `model.invoke` | L3 | Per LLM call (Strands routes these through Bedrock). |
| `environment.config` | L4a | First wrap of each agent (`lifecycle.py:430`). |
| `agent.input` | L1 | Beginning of every `__call__` / `invoke` (`lifecycle.py:272`). |
| `agent.output` | L1 | End of every `__call__` / `invoke` (`lifecycle.py:307`). |
| `agent.state.change` | cross-cutting | Mid-run state mutations and run completion (`lifecycle.py:249,308`). |
| `tool.call` | L5a | Per Strands tool invocation (`lifecycle.py:223,341`). |
| `model.invoke` | L3 | Per LLM call — Strands routes through Bedrock (`lifecycle.py:188,371`). |
| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:209`). |

## Strands specifics

Expand All @@ -72,10 +72,12 @@ hooks. `disconnect()` restores the originals.
- **Tools**: Strands tools registered via the `@tool` decorator surface
their function name and JSON schema in `tool.call.tool_schema`.
- **Loops**: Strands runs a reasoning loop (think → act → observe). Each
loop iteration emits an `agent.action` with `loop_index` and a copy of
the conversation state.
- **Multi-agent**: Strands supports orchestrator/worker patterns; cross-agent
delegation emits `agent.handoff` with `source_agent` + `target_agent`.
iteration is observable via the inner `model.invoke` and `tool.call`
events the adapter captures from Strands' callback hooks — there is no
separate per-iteration loop event.
- **State changes**: mid-run state mutations emit `agent.state.change`
(`lifecycle.py:249`) and run completion emits a terminal
`agent.state.change` alongside `agent.output` (`lifecycle.py:308`).

## Capture config

Expand Down