diff --git a/docs/adapters/frameworks-agno.md b/docs/adapters/frameworks-agno.md index fd7e36be..74049d69 100644 --- a/docs/adapters/frameworks-agno.md +++ b/docs/adapters/frameworks-agno.md @@ -54,14 +54,14 @@ sink.close() | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First `run` per agent. | -| `agent.input` | L1 | Beginning of every `run` / `arun`. | -| `agent.output` | L1 | End of every `run` / `arun`. | -| `agent.action` | L4a | Per intermediate reasoning step. | -| `agent.handoff` | L4a | When a team agent delegates to a sub-agent. | -| `agent.state.change` | cross-cutting | Memory mutations. | -| `tool.call` | L5a | Per tool invocation. | -| `model.invoke` | L3 | Per LLM call. | +| `environment.config` | L4a | First `run` per agent (`lifecycle.py:462`). | +| `agent.input` | L1 | Beginning of every `run` / `arun` (`lifecycle.py:289`). | +| `agent.output` | L1 | End of every `run` / `arun` (`lifecycle.py:324`). | +| `agent.handoff` | L4a | When a team agent delegates to a sub-agent (`lifecycle.py:267,398`). | +| `agent.state.change` | cross-cutting | Run completion / failure (`lifecycle.py:325`). | +| `tool.call` | L5a | Per tool invocation (`lifecycle.py:246,358`). | +| `model.invoke` | L3 | Per LLM call (`lifecycle.py:216,388`). | +| `cost.record` | cross-cutting | Per LLM call when `result.metrics` / `result.usage` are populated (`lifecycle.py:228`). | ## Agno specifics @@ -69,9 +69,12 @@ sink.close() Each team member must be instrumented individually with `adapter.instrument_agent(team_member)` — or call `instrument_agent(team)` and the convenience helper recurses. -- **Reasoning agents**: when `reasoning=True` is set on an Agent, the - intermediate reasoning steps emit `agent.action` events with a - `step_index` field. +- **Reasoning agents**: Agno's `reasoning=True` mode is not specially + instrumented — intermediate reasoning steps are not surfaced as + separate events. The reasoning chain is observable only via the + inputs/outputs already captured on `agent.input` / `agent.output` and + any inner `model.invoke` / `tool.call` events the underlying agent + emits. - **Storage backends**: Agno session storage (Postgres, sqlite, Redis, etc.) emits `agent.state.change` on every save. @@ -83,7 +86,8 @@ from layerlens.instrument.adapters._base import CaptureConfig # Recommended. adapter = AgnoAdapter(capture_config=CaptureConfig.standard()) -# Heavy: include reasoning steps as agent.code (the chain-of-thought). +# Heavy: enable all capture layers (L2 agent.code currently has no +# Agno-specific emission sites — reasoning is not specially instrumented). adapter = AgnoAdapter( capture_config=CaptureConfig( l1_agent_io=True, diff --git a/docs/adapters/frameworks-bedrock_agents.md b/docs/adapters/frameworks-bedrock_agents.md index 99dddf14..06db65c7 100644 --- a/docs/adapters/frameworks-bedrock_agents.md +++ b/docs/adapters/frameworks-bedrock_agents.md @@ -58,7 +58,8 @@ provided `bedrock-agent-runtime` client: emits `agent.input` and `environment.config` on first agent encounter. - `after-call.bedrock-agent-runtime.InvokeAgent` — fires after the response comes back. Walks the `trace` blocks in the streamed events and emits - `model.invoke` / `tool.call` / `agent.action` per trace step. + `model.invoke` / `tool.call` / `agent.handoff` per trace step + (`lifecycle.py:200-213`). `disconnect()` unregisters both hooks. @@ -67,12 +68,12 @@ provided `bedrock-agent-runtime` client: | Event | Layer | When | |---|---|---| | `environment.config` | L4a | First `InvokeAgent` per `agentId`. | -| `agent.input` | L1 | Beginning of every `InvokeAgent`. | -| `agent.output` | L1 | End of every `InvokeAgent` (after stream consumption). | -| `agent.action` | L4a | Per `orchestrationTrace.modelInvocationInput` block. | -| `agent.handoff` | L4a | Per cross-agent collaboration step. | -| `tool.call` | L5a | Per `actionGroupInvocationInput` / `knowledgeBaseLookupInput` block. | -| `model.invoke` | L3 | Per `modelInvocationOutput` block (with token usage). | +| `agent.input` | L1 | Beginning of every `InvokeAgent` (`lifecycle.py:289`). | +| `agent.output` | L1 | End of every `InvokeAgent` after stream consumption (`lifecycle.py:323`). | +| `agent.handoff` | L4a | Per `AGENT_COLLABORATOR` trace step (`lifecycle.py:269,386`). | +| `tool.call` | L5a | Per `ACTION_GROUP` / `KNOWLEDGE_BASE` trace step (`lifecycle.py:217,230,348`). | +| `model.invoke` | L3 | Per `MODEL_INVOCATION` trace step with token usage (`lifecycle.py:254,377`). | +| `cost.record` | cross-cutting | Per `MODEL_INVOCATION` when `usage` is present (`lifecycle.py:256`). | ## Bedrock Agents specifics diff --git a/docs/adapters/frameworks-llama_index.md b/docs/adapters/frameworks-llama_index.md index 76d04b25..176e0599 100644 --- a/docs/adapters/frameworks-llama_index.md +++ b/docs/adapters/frameworks-llama_index.md @@ -46,15 +46,15 @@ dispatches. ## What's wrapped `adapter.instrument_workflow(...)` registers a `BaseEventHandler` with -`llama_index.core.instrumentation.get_dispatcher()`. The handler observes: +`llama_index.core.instrumentation.get_dispatcher()`. The handler observes +(`lifecycle.py:185-198`): -- LLM events (`LLMChatStartEvent`, `LLMChatEndEvent`, - `LLMCompletionStartEvent`, `LLMCompletionEndEvent`) -- Tool events (`AgentToolCallEvent`) -- Agent events (`AgentRunStepStartEvent`, `AgentRunStepEndEvent`, - `AgentChatWithStepStartEvent`, `AgentChatWithStepEndEvent`) -- Retrieval events (`RetrievalStartEvent`, `RetrievalEndEvent`) -- Embedding events (`EmbeddingStartEvent`, `EmbeddingEndEvent`) +- LLM events: `LLMChatStartEvent`, `LLMStartEvent` (start); + `LLMChatEndEvent`, `LLMCompletionEndEvent` (end) +- Tool events: `ToolCallEvent` +- Agent events: `AgentRunStepStartEvent`, `AgentRunStepEndEvent` +- Retrieval / query events: `RetrievalStartEvent`, `QueryStartEvent` + (start); `RetrievalEndEvent`, `QueryEndEvent` (end) `disconnect()` removes the handler from the dispatcher's `event_handlers` list, restoring the original behaviour. @@ -63,20 +63,23 @@ dispatches. | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First agent / workflow event per process. | -| `agent.input` | L1 | `AgentChatWithStepStartEvent` / agent step start. | -| `agent.output` | L1 | `AgentChatWithStepEndEvent` / agent step end. | -| `agent.action` | L4a | Per `AgentRunStepEndEvent`. | -| `tool.call` | L5a | Per `AgentToolCallEvent`. | -| `model.invoke` | L3 | Per LLM start/end pair. | +| `environment.config` | L4a | First `AgentRunStepStartEvent` per agent (`lifecycle.py:261,428`). | +| `agent.input` | L1 | Per `AgentRunStepStartEvent` (`lifecycle.py:266`). | +| `agent.output` | L1 | Per `AgentRunStepEndEvent` (`lifecycle.py:283`). | +| `tool.call` | L5a | Per `ToolCallEvent` and per `RetrievalEndEvent` / `QueryEndEvent` (tool_name = `"retrieval"`, tool_type = `"retrieval"`) (`lifecycle.py:231,246`). | +| `model.invoke` | L3 | Per LLM end event (`LLMChatEndEvent` / `LLMCompletionEndEvent`) (`lifecycle.py:216`). | +| `cost.record` | cross-cutting | Per LLM end event when token counts are present (`lifecycle.py:218`). | +| `agent.handoff` | L4a | Exposed via the `on_handoff` lifecycle hook (`lifecycle.py:400`) — not auto-emitted from the dispatcher event stream. | ## LlamaIndex specifics - **Workflows**: the new `Workflow` class emits dispatcher events the same way; the same handler captures both classic agents (`ReActAgent`, `OpenAIAgent`) and workflow `@step` runs. -- **RAG retrievers**: retrieval events are surfaced as `tool.call` with - `tool_name="retriever"` and the resolved chunk count. +- **RAG retrievers**: `RetrievalEndEvent` and `QueryEndEvent` are surfaced + as `tool.call` with `tool_name="retrieval"`, `tool_type="retrieval"`, + and the resolved chunk count in `result_count` + (`lifecycle.py:244-257`). - **Streaming**: streamed LLM responses fire one `LLMChatEndEvent` after the final chunk; the adapter emits one consolidated `model.invoke`. - **Span propagation**: LlamaIndex span IDs propagate into the event diff --git a/docs/adapters/frameworks-ms_agent_framework.md b/docs/adapters/frameworks-ms_agent_framework.md index 295f2b9b..d5cfc770 100644 --- a/docs/adapters/frameworks-ms_agent_framework.md +++ b/docs/adapters/frameworks-ms_agent_framework.md @@ -68,13 +68,14 @@ filters. `disconnect()` restores the originals. | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First wrap of each chat. | -| `agent.input` | L1 | Beginning of every `invoke` / `invoke_stream`. | -| `agent.output` | L1 | End of every invocation (per response). | -| `agent.action` | L4a | Per intermediate step. | -| `agent.handoff` | L4a | Per `AgentGroupChat` speaker turn. | -| `tool.call` | L5a | Per plugin function invocation. | -| `model.invoke` | L3 | Per LLM call. | +| `environment.config` | L4a | First wrap of each chat (`lifecycle.py:481`). | +| `agent.input` | L1 | Beginning of every `invoke` / `invoke_stream` (`lifecycle.py:299`). | +| `agent.output` | L1 | End of every invocation (per response) (`lifecycle.py:334`). | +| `agent.state.change` | cross-cutting | Per `invoke` / `invoke_stream` end (`lifecycle.py:335`). | +| `agent.handoff` | L4a | Per `AgentGroupChat` speaker turn (`lifecycle.py:223,408`). | +| `tool.call` | L5a | Per plugin function invocation (`lifecycle.py:237,247,368`). | +| `model.invoke` | L3 | Per LLM call (`lifecycle.py:262,398`). | +| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:272`). | ## MS Agent Framework specifics @@ -83,8 +84,10 @@ filters. `disconnect()` restores the originals. on each speaker turn. - **Plugins**: Semantic Kernel plugin functions surface as `tool.call` — the plugin name + function name combine into `tool_name`. -- **Multi-agent terminations**: configurable termination strategies - emit `agent.action` with `terminate_reason` when a group chat ends. +- **Multi-agent terminations**: configurable termination strategies are + not separately instrumented. When a group chat ends, the outer + `agent.output` carries the final response and the run completion fires + one `agent.state.change` (`lifecycle.py:335`). - **Streaming**: `invoke_stream` emits one consolidated `model.invoke` on stream completion; per-chunk text is accumulated. diff --git a/docs/adapters/frameworks-openai_agents.md b/docs/adapters/frameworks-openai_agents.md index f9d983b2..64b083ba 100644 --- a/docs/adapters/frameworks-openai_agents.md +++ b/docs/adapters/frameworks-openai_agents.md @@ -59,14 +59,15 @@ events. | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First agent span observed. | -| `agent.input` | L1 | Per agent span start. | -| `agent.output` | L1 | Per agent span end. | -| `agent.action` | L4a | Per `response_span` (model call decision). | -| `agent.handoff` | L4a | Per `handoff_span`. | -| `tool.call` | L5a | Per `function_span`. | -| `model.invoke` | L3 | Per `generation_span` (model call). | -| `policy.violation` | cross-cutting | Per `guardrail_span` that fails. | +| `environment.config` | L4a | First agent span observed (`lifecycle.py:497`). | +| `agent.input` | L1 | Per agent span start (`lifecycle.py:256,358`). | +| `agent.output` | L1 | Per agent span end (`lifecycle.py:269,392`). | +| `agent.state.change` | cross-cutting | On agent span start/end (`lifecycle.py:193,211`). | +| `model.invoke` | L3 | Per `response_span` / `generation_span` model call (`lifecycle.py:293,446`). | +| `cost.record` | cross-cutting | Per model call when token counts are present (`lifecycle.py:295`). | +| `tool.call` | L5a | Per `function_span` (`lifecycle.py:308,417`). | +| `agent.handoff` | L4a | Per `handoff_span` (`lifecycle.py:325,463`). | +| `policy.violation` | cross-cutting | Per `guardrail_span` that fails (`lifecycle.py:338`). | ## OpenAI Agents specifics diff --git a/docs/adapters/frameworks-pydantic_ai.md b/docs/adapters/frameworks-pydantic_ai.md index d2b5865a..9cc28cc9 100644 --- a/docs/adapters/frameworks-pydantic_ai.md +++ b/docs/adapters/frameworks-pydantic_ai.md @@ -55,12 +55,14 @@ sink.close() | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First wrap of each agent. | -| `agent.input` | L1 | Beginning of every `run` / `run_sync`. | -| `agent.output` | L1 | End of every `run` / `run_sync`. | -| `agent.action` | L4a | Per intermediate model step (multi-step runs). | -| `tool.call` | L5a | Per registered tool invocation. | -| `model.invoke` | L3 | Per LLM call (one per model step). | +| `environment.config` | L4a | First wrap of each agent (`lifecycle.py:407`). | +| `agent.input` | L1 | Beginning of every `run` / `run_sync` (`lifecycle.py:248`). | +| `agent.output` | L1 | End of every `run` / `run_sync` (`lifecycle.py:282`). | +| `agent.state.change` | cross-cutting | On run end (`lifecycle.py:283`). | +| `tool.call` | L5a | Per registered tool invocation (`lifecycle.py:227,315`). | +| `model.invoke` | L3 | Per LLM call (one per model step) (`lifecycle.py:218,344`). | +| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:203`). | +| `agent.handoff` | L4a | Cross-agent delegation (`lifecycle.py:353`). | The `model.invoke` payload includes the model name (parsed from the PydanticAI model spec like `openai:gpt-4o-mini`), token usage from @@ -70,8 +72,10 @@ PydanticAI model spec like `openai:gpt-4o-mini`), token usage from - **Structured results**: when an agent declares `result_type=MyModel`, the validated Pydantic model is included in `agent.output` (subject to - `CaptureConfig.capture_content`). Validation errors emit - `policy.violation`. + `CaptureConfig.capture_content`). Validation errors are surfaced on + the `agent.output` payload (via the wrapping `try/except` in + `_create_traced_run`) — the adapter does not emit a separate + `policy.violation` event for these. - **Model spec parsing**: PydanticAI accepts model spec strings like `"openai:gpt-4o-mini"` or `"anthropic:claude-3-5-sonnet"`. The adapter splits these into `provider` + `model` for downstream cost lookups. diff --git a/docs/adapters/frameworks-semantic_kernel.md b/docs/adapters/frameworks-semantic_kernel.md index b29e16b9..8599ed8f 100644 --- a/docs/adapters/frameworks-semantic_kernel.md +++ b/docs/adapters/frameworks-semantic_kernel.md @@ -60,14 +60,13 @@ and the kernel returns to its original behaviour. | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First plugin invocation per kernel. | -| `agent.input` | L1 | Function invocation start. | -| `agent.output` | L1 | Function invocation end (success or error). | -| `agent.code` | L2 | Per plugin function when `l2_agent_code` is true. | -| `agent.action` | L4a | Per planner step. | -| `agent.state.change` | cross-cutting | Memory store reads/writes. | -| `tool.call` | L5a | Per `auto_function_invocation` (model-selected plugin). | -| `model.invoke` | L3 | Per LLM call inside the kernel. | +| `environment.config` | L4a | First plugin invocation per kernel and per-plugin discovery (`lifecycle.py:203,442`). | +| `agent.input` | L1 | Kernel invoke start (`lifecycle.py:397`). | +| `agent.output` | L1 | Kernel invoke end — success or error (`lifecycle.py:423`). | +| `agent.code` | L2 | Per plugin function / prompt render (`lifecycle.py:271,355`). | +| `tool.call` | L5a | Per `auto_function_invocation` and per memory-store operation (`lifecycle.py:247,390`). | +| `model.invoke` | L3 | Per LLM call inside the kernel (`lifecycle.py:306`). | +| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:310`). | ## Semantic Kernel specifics diff --git a/docs/adapters/frameworks-strands.md b/docs/adapters/frameworks-strands.md index a8ff2517..e452049b 100644 --- a/docs/adapters/frameworks-strands.md +++ b/docs/adapters/frameworks-strands.md @@ -56,13 +56,13 @@ hooks. `disconnect()` restores the originals. | Event | Layer | When | |---|---|---| -| `environment.config` | L4a | First wrap of each agent. | -| `agent.input` | L1 | Beginning of every `__call__` / `invoke`. | -| `agent.output` | L1 | End of every `__call__` / `invoke`. | -| `agent.action` | L4a | Per intermediate reasoning loop iteration. | -| `agent.handoff` | L4a | Multi-agent collaboration handoffs. | -| `tool.call` | L5a | Per Strands tool invocation. | -| `model.invoke` | L3 | Per LLM call (Strands routes these through Bedrock). | +| `environment.config` | L4a | First wrap of each agent (`lifecycle.py:430`). | +| `agent.input` | L1 | Beginning of every `__call__` / `invoke` (`lifecycle.py:272`). | +| `agent.output` | L1 | End of every `__call__` / `invoke` (`lifecycle.py:307`). | +| `agent.state.change` | cross-cutting | Mid-run state mutations and run completion (`lifecycle.py:249,308`). | +| `tool.call` | L5a | Per Strands tool invocation (`lifecycle.py:223,341`). | +| `model.invoke` | L3 | Per LLM call — Strands routes through Bedrock (`lifecycle.py:188,371`). | +| `cost.record` | cross-cutting | Per LLM call when token usage is present (`lifecycle.py:209`). | ## Strands specifics @@ -72,10 +72,12 @@ hooks. `disconnect()` restores the originals. - **Tools**: Strands tools registered via the `@tool` decorator surface their function name and JSON schema in `tool.call.tool_schema`. - **Loops**: Strands runs a reasoning loop (think → act → observe). Each - loop iteration emits an `agent.action` with `loop_index` and a copy of - the conversation state. -- **Multi-agent**: Strands supports orchestrator/worker patterns; cross-agent - delegation emits `agent.handoff` with `source_agent` + `target_agent`. + iteration is observable via the inner `model.invoke` and `tool.call` + events the adapter captures from Strands' callback hooks — there is no + separate per-iteration loop event. +- **State changes**: mid-run state mutations emit `agent.state.change` + (`lifecycle.py:249`) and run completion emits a terminal + `agent.state.change` alongside `agent.output` (`lifecycle.py:308`). ## Capture config