docs(adapters): fix framework user-doc inaccuracies on agents branch (PR-2a + PR-2b, ~18 issues)#157
Closed
mmercuri wants to merge 2 commits into
Closed
Conversation
…rks-*.md docs (per audit) Fixes the systemic `agent.action` claim in 7 framework adapter user docs on the `feat/instrument-frameworks-agents` branch. Grep across the entire framework adapter source tree confirms zero `"agent.action"` literals are emitted — the doc claims were copied across adapters without a source check. Per `A:/tmp/adapter-user-docs-audit.md` (2026-05-10), each table is now rewritten against what the lifecycle source actually emits, with source file:line citations on every row. Also addresses the related per-doc findings the audit listed alongside the agent.action row: - agno: remove `agent.action`; rewrite "Reasoning agents" bullet to note reasoning is not specially instrumented (lifecycle.py: no match for reasoning-specific emit). Add `cost.record` row (lifecycle.py:228). - bedrock_agents: remove `agent.action`; correct the "What's wrapped" trace-walk description; add `cost.record` (lifecycle.py:256). - ms_agent_framework: remove `agent.action`; rewrite the termination bullet (no per-strategy emission in source); add `agent.state.change` (lifecycle.py:335) and `cost.record` (lifecycle.py:272). - openai_agents: remove `agent.action`; map `response_span` to `model.invoke` (lifecycle.py:293,446); add `agent.state.change` (lifecycle.py:193,211) and `cost.record` (lifecycle.py:295). - pydantic_ai: remove `agent.action`; remove `policy.violation` claim (no such literal in pydantic_ai source) — validation errors surface on agent.output instead; add `cost.record`, `agent.state.change`, `agent.handoff`. - semantic_kernel: remove `agent.action`; remove `agent.state.change` (no literal in source — memory operations emit `tool.call` at lifecycle.py:390); add `cost.record` (lifecycle.py:310). - strands: remove `agent.action`; remove `agent.handoff` (no literal in source); rewrite "Loops" and "Multi-agent" bullets; add `agent.state.change` (lifecycle.py:249,308) and `cost.record` (lifecycle.py:209). All replacement claims trace to a specific lifecycle.py line; verified by re-grepping after edit. Zero `agent.action` strings remain in any adapter doc.
…d cost.record (per audit) Per `A:/tmp/adapter-user-docs-audit.md` (2026-05-10) — heavy rewrite of the llama_index doc's event-type taxonomy. The previous doc claimed event classes the source never observes and omitted events the source actually emits. Every replacement now cites a lifecycle.py line. Fixes (lifecycle.py reference points are the routed handlers at L179-198 and the emit sites that follow): 1. `LLMCompletionStartEvent` does not exist in source — the start branch handles `LLMChatStartEvent` and `LLMStartEvent` (`lifecycle.py:185`). Replaced. 2. `AgentToolCallEvent` does not exist in source — the routing handles `ToolCallEvent` (`lifecycle.py:189`). Renamed in the wrapped-events list and the events-emitted table. 3. `AgentChatWithStepStartEvent` / `AgentChatWithStepEndEvent` are not observed — only `AgentRunStepStartEvent` / `AgentRunStepEndEvent` are routed (`lifecycle.py:195-198`). Removed from the bullet list and corrected in the events-emitted table. 4. `EmbeddingStartEvent` / `EmbeddingEndEvent` — no embedding-event routing in source. The "Embedding events" bullet has been removed from the wrapped-events list. 5. `agent.action` per `AgentRunStepEndEvent` was wrong — `_on_agent_step_end` emits `agent.output` (`lifecycle.py:283`). Row removed. 6. `cost.record` was missing from the events-emitted table — the source emits one per LLM end event when token counts are present (`lifecycle.py:218`). Added. 7. "retrieval events surfaced as tool.call with tool_name='retriever'" was a wording mismatch — the actual emit uses tool_name="retrieval" and tool_type="retrieval" (`lifecycle.py:246-257`). Corrected and `RetrievalEndEvent` / `QueryEndEvent` are now explicit on the tool.call row. Also added an `agent.handoff` row noting the source exposes a lifecycle hook (`lifecycle.py:400`) that is not auto-emitted from the dispatcher stream — flagged for honesty so customers don't expect handoff events without calling `on_handoff` themselves. Heavy rewrites for `frameworks-benchmark_import.md` and `frameworks-embedding.md` are NOT included in this commit — they are already in PR #150 (`docs/fix-frameworks-user-doc-inaccuracies`) and double-committing would conflict.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes 17 of the 18 inaccuracies the audit attributed to the agents branch in
A:/tmp/adapter-user-docs-audit.md(2026-05-10). One inaccuracy (the embedding event-name fix) and one heavy rewrite (benchmark_import) are already addressed by PR #150 and have been deliberately excluded here to avoid double-committing.Two themed commits:
agno,bedrock_agents,ms_agent_framework,openai_agents,pydantic_ai,semantic_kernel,strands. Grep confirmed zero"agent.action"literals in the entire framework adapter source tree. Per-adapter event tables rewritten against whatlifecycle.pyactually emits, withlifecycle.py:<line>citations on every row. Also fixes the related per-doc findings the audit listed alongside eachagent.actionrow (reasoning bullet on agno, termination bullet on ms_agent_framework,policy.violationon pydantic_ai,agent.state.changeon semantic_kernel,agent.handoffon strands).LLMCompletionStartEvent→LLMStartEvent,AgentToolCallEvent→ToolCallEvent; removedAgentChatWithStep*events and the Embedding events bullet (no source match); replacedagent.actionrow with the actualagent.output(lifecycle.py:283); added the previously-omittedcost.recordrow (lifecycle.py:218); corrected RAG retriever wording (tool_name="retrieval",tool_type="retrieval",lifecycle.py:244-257).Per-doc fix breakdown
frameworks-agno.mdagent.actionrow removed; reasoning bullet rewritten;cost.recordaddedlifecycle.py:228(noagent.actionliteral anywhere)frameworks-bedrock_agents.mdagent.actionrow removed; "What's wrapped" trace-walk corrected;cost.recordaddedlifecycle.py:206-213(trace-step routing),:256(cost)frameworks-ms_agent_framework.mdagent.actionrow removed; termination bullet rewritten;agent.state.change+cost.recordaddedlifecycle.py:335,272frameworks-openai_agents.mdagent.actionrow removed;response_spanmapped tomodel.invoke;agent.state.change+cost.recordaddedlifecycle.py:293,446,193,211,295frameworks-pydantic_ai.mdagent.actionremoved;policy.violationremoved (no source emission — validation errors surface onagent.output);cost.record/agent.state.change/agent.handoffaddedlifecycle.py:203,283,353(nopolicy.violationliteral)frameworks-semantic_kernel.mdagent.actionremoved;agent.state.changeremoved (memory ops emittool.callat:390);cost.recordaddedlifecycle.py:390,310(noagent.state.changeliteral)frameworks-strands.mdagent.actionremoved;agent.handoffremoved (no literal); Loops/Multi-agent bullets rewritten;agent.state.change+cost.recordaddedlifecycle.py:249,308,209(noagent.handoffliteral)frameworks-llama_index.mdcost.recordadd + retriever wording fixlifecycle.py:185,189,195-198,283,218,244-257Overlap with PR #150 (skipped on purpose)
frameworks-embedding.mdevent-name fix (vector_store.query→retrieval.query) — already done in PR docs(adapters): fix user-doc inaccuracies (closes spec PR #148 audit) #150.frameworks-benchmark_import.mdheavy rewrite (heuristic claim, persistence API,_benchmarkscache,BaseAdaptercarve-out) — already done in PR docs(adapters): fix user-doc inaccuracies (closes spec PR #148 audit) #150.These were verified by reading
gh pr diff 150before starting; no changes to those two files in this PR.Hard constraints (CLAUDE.md)
lifecycle.py:<line>.--no-verify, no escape-hatch flags.Test plan
feat/instrument-frameworks-agents.grep -rn 'agent.action' docs/adapters/frameworks-*.mdreturns 0 results (confirmed locally before push).Audit report:
A:/tmp/adapter-user-docs-audit.md(sections L79-170, lines covering agno through strands plus llama_index). Recommended PR scoping #2 on L243-251.