feat(instrument): Per-callback try/except resilience wrapper across 10 lighter adapters (cross-poll #5)

[mmercuri]

Cross-pollination item #5 — per-callback resilience wrapper

Implements the resilience pattern described in A:/tmp/adapter-cross-pollination-audit.md §2 item #5 (§2.11). Mature adapters (CrewAI, AutoGen, OpenAI Agents) wrap every observability callback in a try/except boundary so an exception in the adapter never escapes back into the framework's own execution path. This PR brings the SAME contract to all 10 lighter framework adapters — agno, llamaindex, google_adk, strands, pydantic_ai, smolagents, bedrock_agents, openai_agents, haystack, langfuse.

What's new

Shared resilience module

src/layerlens/instrument/adapters/_base/resilience.py — new

• @resilient_callback(...) decorator: catches Exception (NOT BaseException), logs it via the adapter's module logger with adapter_name + callback_name + truncated traceback, increments the adapter's failure counter, and returns the framework's expected default value.
  • default= parameter: explicit default return on failure.
  • passthrough_arg= parameter: returns the value of the named argument on failure — required for mutating hooks (Pydantic-AI after_model_request returns the response object; before_tool_execute returns the args tuple). Returning None would corrupt the framework's data flow.
• ResilienceTracker class: thread-safe per-adapter failure counter + per-callback breakdown. health_status() returns DEGRADED once DEFAULT_FAILURE_THRESHOLD (5) failures cross.
• get_default_for(callback_name) helper: canonical defaults table for known callbacks (Strands hooks, Google ADK plugins, OpenAI Agents TracingProcessor, boto3 event handlers — all return None).

FrameworkAdapter integration

src/layerlens/instrument/adapters/frameworks/_base_framework.py

• Every FrameworkAdapter subclass now owns self._resilience: ResilienceTracker set in __init__.
• adapter_info().metadata merges the live resilience snapshot (resilience_status, resilience_failures_total, resilience_failure_threshold, per-callback breakdown, last error).
• disconnect() resets the tracker so reconnects start clean.

_base/ package refactor

_base.py (BaseAdapter contract) becomes _base/ package: __init__.py re-exports from _core.py (moved via git mv, full history preserved) and the new resilience.py. All existing from .._base import AdapterInfo, BaseAdapter imports continue working unchanged.

Per-adapter audit + fixes

Adapter	Before	Callbacks wrapped	Notes
agno	NO try/except	2	_on_run_start, _on_run_end would have crashed user code
llamaindex	PARTIAL (dispatcher only)	16	3 span lifecycle + dispatcher + 12 per-event handlers
google_adk	YES (in plugin shims)	11	All adapter _on_* + simplified plugin shims (single source of truth)
strands	YES (manual try/except in each)	7	Replaced manual try/except with decorator (consistency + tracking)
pydantic_ai	NO try/except	9 (incl 3 split)	Error hooks split: telemetry resilient, framework error always re-raised
smolagents	PARTIAL (step callbacks only)	6	Run lifecycle handlers were unwrapped; replaced manual try/except in step handlers
bedrock_agents	YES (manual)	2	Replaced manual try/except; preserved try/finally so _end_run() always runs
openai_agents	YES (manual)	3	Replaced manual try/except in TracingProcessor methods
haystack	YES (manual)	1	Replaced manual try/except in _on_span_end
langfuse	PARTIAL (per-record loops)	5	Imports + exports refactored to push per-record resilience into helpers
TOTAL		62

Pydantic-AI error-callback split

_on_run_error, _on_model_request_error, _on_tool_execute_error MUST always re-raise the framework's original error (per Pydantic-AI's contract). The telemetry side moves into a _emit_*_error_telemetry helper wrapped with @resilient_callback; the public hook calls it then unconditionally raise error. Adapter-side telemetry bugs can never swallow a real framework error.

Bedrock-Agents finally semantics

_after_invoke keeps an outer try/finally so _end_run() always runs (otherwise collector + ContextVars leak across boto3 calls). The inner body is delegated to a _after_invoke_body helper wrapped with @resilient_callback.

Tests (50+ new tests)

tests/instrument/adapters/_base/test_resilience.py — 34 tests

• TestGetDefaultFor — known/unknown callback name lookups
• TestResilienceTracker — counter, threshold validation, health degradation, metadata snapshot, reset, thread-safety under 8 threads × 100 increments
• TestResilientCallbackDecorator — success path, default-on-exception, counter increment, log context, exception does NOT propagate, positional + keyword passthrough_arg, health degradation, KeyboardInterrupt propagates, works without tracker, logger uses module of decorated function
• TestFrameworkAdapterIntegration — tracker on every framework adapter, metadata surface, degraded status, disconnect resets, callback failure does not break framework
• TestPackageExports — public surface via _base.__init__
• TestDecoratorMetadata — functools.wraps preserves __name__/__doc__
• TestPerAdapterCallbackException — parametrized per-adapter callback exception tests + threshold-degradation test

tests/instrument/adapters/_base/test_per_adapter_resilience.py — 30 tests

One test per adapter that simulates a callback exception by sabotaging an inner helper, asserts:

1. The exception does NOT propagate.
2. The resilience tracker recorded the failure.
3. After enough failures, adapter_info().metadata['resilience_status'] is "degraded".

Plus a parametrized health-degradation test across all 10 adapters. Includes ContextVar isolation fixture so tests that intentionally leave _begin_run state (Pydantic-AI, Bedrock-Agents) don't poison subsequent test files.

Acceptance

pytest tests/instrument/adapters/_base/test_resilience.py -x
# 34 passed in 0.10s

pytest tests/instrument/adapters/frameworks/ -x
# 146 passed, 12 skipped (missing optional framework deps),
# 2 deselected (pre-existing Windows clock-resolution flakes in test_haystack
#  — confirmed against origin/development baseline)

mypy --strict src/layerlens/instrument/adapters/_base/resilience.py
# Success: no issues found in 1 source file

mypy src
# Success: no issues found in 169 source files

ruff check src/ tests/
# All checks passed!

pytest tests/ (full suite, excluding cli + providers which need extra deps)
# 1090 passed, 19 skipped, 2 deselected (pre-existing flakes)

Test plan

• pytest tests/instrument/adapters/_base/test_resilience.py -x (resilience mechanics)
• pytest tests/instrument/adapters/_base/test_per_adapter_resilience.py (per-adapter)
• pytest tests/instrument/adapters/frameworks/ (no regressions on adapter tests)
• pytest tests/instrument/test_trace_context.py (no ContextVar leak from _base tests)
• mypy --strict on resilience.py
• mypy src (full project)
• ruff check
• CI must pass on Linux / 3.10-3.13 matrix (this PR was developed on Windows / 3.11)