Skip to content

feat(instrument): Port LiteLLM provider adapter (M3)#107

Closed
mmercuri wants to merge 3 commits into
mainfrom
feat/instrument-providers-litellm
Closed

feat(instrument): Port LiteLLM provider adapter (M3)#107
mmercuri wants to merge 3 commits into
mainfrom
feat/instrument-providers-litellm

Conversation

@mmercuri

Copy link
Copy Markdown
Contributor

Summary

Port of stratix/sdk/python/adapters/llm_providers/litellm_adapter.py (~355 LOC, ateam) into the M3 fan-out at src/layerlens/instrument/adapters/providers/litellm/. Uses the existing providers/openai_adapter.py as the structural template (lifecycle / _emit_* helper usage / ADAPTER_CLASS registry hook), and introduces a subpackage split:

providers/litellm/
├── __init__.py    # Public surface + STRATIX backward-compat alias
├── adapter.py     # LiteLLMAdapter
├── callback.py    # LayerLensLiteLLMCallback (sync + async)
└── routing.py     # detect_provider

LiteLLM is a multi-provider router — a single litellm.completion(...) call is dispatched to one of ~100 providers (OpenAI, Anthropic, Bedrock, Vertex AI, Cohere, Ollama, Together, Groq, HuggingFace, ...). Rather than monkey-patching every provider client, the adapter installs a single callback into litellm.callbacks and lets LiteLLM dispatch it. The routing layer normalises the model string (bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0aws_bedrock, gpt-4oopenai, etc.) into the canonical LayerLens provider name so downstream telemetry lines up across every adapter.

Pricing: LiteLLM contributes no new entries to the canonical pricing manifest. The callback calls litellm.completion_cost(...) first (recorded with cost_source: "litellm" when LiteLLM has its own pricing for the model); when LiteLLM cannot price the call it falls through to the canonical PRICING map at providers/_base/pricing.py. The routing layer's normalised provider name carries through to cost.record so Bedrock-routed Anthropic flows through BEDROCK_PRICING, not direct PRICING.

STRATIX* aliased to LayerLens* for backward-compat with ateam users; stratix.* paths translated to layerlens.*. The legacy flat providers/litellm_adapter.py is kept as a thin re-export so existing imports keep working.

Files

  • src/layerlens/instrument/adapters/providers/litellm/__init__.py (new, public surface + alias)
  • src/layerlens/instrument/adapters/providers/litellm/adapter.py (new, lifecycle)
  • src/layerlens/instrument/adapters/providers/litellm/callback.py (new, sync + async callbacks)
  • src/layerlens/instrument/adapters/providers/litellm/routing.py (new, detect_provider)
  • src/layerlens/instrument/adapters/providers/litellm_adapter.py (now thin re-export)
  • src/layerlens/instrument/adapters/providers/__init__.py (PEP 562 lazy __getattr__ for LiteLLMAdapter)
  • tests/instrument/adapters/providers/test_litellm.py (new, 38 tests; supersedes test_litellm_adapter.py)
  • tests/instrument/adapters/providers/test_litellm_adapter.py (deleted)
  • samples/instrument/providers/litellm/{__init__.py,main.py,README.md} (new, mocked-by-default sample)
  • docs/adapters/providers-litellm.md (rewritten — subpackage layout + pricing inheritance)

pyproject.toml already carried providers-litellm = ["litellm>=1.40,<2"] from the M1.B port — no version bump needed.

Test plan

  • uv run pytest tests/instrument/adapters/providers/test_litellm.py -x — 38/38 pass
  • uv run mypy --strict src/layerlens/instrument/adapters/providers/litellm — clean (4 source files)
  • uv run ruff check src/.../providers/litellm src/.../providers/__init__.py src/.../providers/litellm_adapter.py tests/.../test_litellm.py — clean
  • uv run pytest tests/instrument/adapters/providers/ -x (no live tests) — 137/137 pass (no regression on sibling provider tests)
  • uv run pytest tests/instrument/test_lazy_imports.py::test_layerlens_import_does_not_pull_frameworks tests/instrument/test_lazy_imports.py::test_instrument_import_does_not_pull_frameworks tests/instrument/test_default_install.py — 5/5 pass
  • python -m samples.instrument.providers.litellm.main — runs offline, emits 12 events across 6 routing scenarios

Routing coverage

Every prefix listed in the M3 PR description gets a parametrised test:

Model string Provider
gpt-4, gpt-4o, o1-mini, o3-mini openai
openai/gpt-4o-mini openai
claude-3-5-sonnet, anthropic/claude-3-5-sonnet anthropic
azure/my-deployment azure_openai
bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0 aws_bedrock
vertex_ai/gemini-1.5-pro, gemini-2.0-flash google_vertex
ollama/llama3 ollama
cohere/command-r cohere
huggingface/..., together_ai/..., groq/... matched 1:1
llama-3.1-70b meta
mistral-large mistral
empty / unrecognised unknown

Notes for review

  • Test file is test_litellm.py (not _adapter.py) per the task spec; old test removed.
  • Async callbacks (async_log_*_event) delegate to the sync handlers — LiteLLM hands the async dispatch the same kwargs/response_obj shape, so no separate logic. The acompletion test exercises this path with an AsyncMock.
  • connect() degrades cleanly when litellm is missing (status DEGRADED, no crash) — covered by test_connect_degraded_when_litellm_missing.
  • The PEP 562 __getattr__ shim on providers/__init__.py was the simplest way to expose providers.LiteLLMAdapter at the package level without breaking the lazy-import contract; the eager if TYPE_CHECKING: import is for static analysers only.

mmercuri and others added 3 commits April 25, 2026 19:13
Bootstraps the LayerLens instrument layer with the abstract base classes,
adapter registry, capture configuration, event sinks, vendored event
schemas, and pydantic v1/v2 compatibility shim that every concrete
adapter (frameworks, protocols, providers) will depend on.

Scope
-----
- src/layerlens/instrument/__init__.py: lean re-export surface
- src/layerlens/instrument/_vendored/: frozen ateam event schemas (no
  runtime ateam dependency)
- src/layerlens/instrument/adapters/_base/: BaseAdapter, AdapterRegistry,
  AdapterStatus, AdapterHealth, AdapterCapability, ReplayableTrace,
  CaptureConfig, EventSink, TraceStoreSink, IngestionPipelineSink,
  PydanticCompat
- src/layerlens/_compat/pydantic.py: model_dump/model_validate shim
  spanning pydantic v1 + v2
- scripts/{port_adapter,port_protocol,emit_adapter_manifest,
  regen_dep_baselines}.py: codegen helpers used to port the rest of M1
- tests/instrument/{test_base_layer,test_lazy_imports,
  test_default_install,test_resolved_dep_tree}.py + _baselines/
- .github/workflows/dep-tree-guard.yaml: CI gate that locks the default
  install footprint
- docs/adapters/: CONTRIBUTING, STATUS, pydantic-compatibility, testing,
  PERSONA_REVIEW

Blast radius
------------
- Pure additions. No public surface changes outside the new
  layerlens.instrument namespace.
- Default `pip install layerlens` install set is unchanged (verified by
  test_default_install.py against the new baseline).
- Lazy adapter discovery: importing layerlens.instrument MUST NOT pull
  in any optional adapter dep (verified by test_lazy_imports.py).

Test plan
---------
- uv run pytest tests/instrument/test_base_layer.py
  tests/instrument/test_lazy_imports.py -x  -> 45 passed
- The dep-tree-guard workflow exercises test_default_install.py and
  test_resolved_dep_tree.py against the new baselines on every PR.

LAY-3400 umbrella: this PR is the prerequisite for the M1.B/M1.C/M1.D
adapter ports, M7 protocol certification, and M8 Cohere/Mistral.
Ports the nine LLM provider adapters from the ateam reference
implementation onto the new layerlens.instrument base layer:

  OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Vertex,
  Ollama, LiteLLM, Cohere, Mistral

This rolls M1.B (the seven original providers) and M8 (Cohere +
Mistral) into a single PR because they share the same provider _base
helpers (pricing, token counters, provider mixin) and per-PR they would
all be sub-1k LOC.

Scope
-----
- src/layerlens/instrument/adapters/providers/_base/: shared provider
  mixin (pricing.py, provider.py, tokens.py)
- src/layerlens/instrument/adapters/providers/{openai,anthropic,
  azure_openai,bedrock,google_vertex,ollama,litellm,cohere,mistral}_
  adapter.py: per-provider adapter
- tests/instrument/adapters/providers/: unit + live tests for all nine
  providers (live tests skip when no API key is set)
- tests/instrument/adapters/test_pydantic_compat.py: shared compat
  surface used by every provider
- samples/instrument/{openai,anthropic,cohere,mistral}/: runnable
  samples
- docs/adapters/providers-*.md: per-provider integration guide
- pyproject.toml: nine new optional extras
  (providers-openai, providers-anthropic, providers-azure-openai,
  providers-bedrock, providers-vertex, providers-ollama,
  providers-litellm, providers-cohere, providers-mistral) plus the
  providers-all umbrella

Blast radius
------------
- Default `pip install layerlens` install set is unchanged. Each
  provider's heavy dep (openai, anthropic, boto3, etc.) is gated
  behind its own `providers-<name>` extra.
- No changes to existing public API surface.
- Importing layerlens.instrument still does NOT pull in any provider
  module (lazy registry lookup).

Test plan
---------
- uv run pytest tests/instrument/adapters/providers/ -x  -> 122 passed,
  4 skipped (live-only without keys)
- uv run pytest tests/instrument/adapters/test_pydantic_compat.py -x
  -> 62 passed

Stacks on
---------
- feat/instrument-base-foundation (M1.A) — required for the BaseAdapter
  surface this PR consumes.

LAY-3400 umbrella (M1.B + M8).
Splits the M1.B port of the LiteLLM provider adapter into a subpackage
under `providers/litellm/`:

- `adapter.py`  — `LiteLLMAdapter` lifecycle (connect/disconnect/version)
- `callback.py` — `LayerLensLiteLLMCallback` with sync + async hooks
                  (`log_*_event` and `async_log_*_event`)
- `routing.py`  — `detect_provider` mapping LiteLLM model strings to
                  canonical LayerLens provider names
- `__init__.py` — public surface, `ADAPTER_CLASS`, and
                  `STRATIXLiteLLMCallback` backward-compat alias

The legacy flat `providers/litellm_adapter.py` becomes a thin re-export
so existing imports keep working. `providers/__init__.py` gains a
PEP 562 `__getattr__` shim so `LiteLLMAdapter` is importable directly
off the package without forcing the vendor SDK to load eagerly.

Tests at `tests/instrument/adapters/providers/test_litellm.py` mock the
`litellm.completion()` / `litellm.acompletion()` boundary and cover:

- 20 routing cases (`gpt-4` -> openai, `claude-3-5-sonnet` -> anthropic,
  `bedrock/anthropic.claude-3-5-sonnet` -> aws_bedrock, etc.)
- adapter lifecycle (connect/disconnect/degrade-when-missing)
- sync success/failure/streaming callback emission
- async success/failure callback emission via `acompletion`
- legacy flat-module re-export equivalence
- subpackage import does not load `litellm` (lazy-import contract)

Sample at `samples/instrument/providers/litellm/main.py` runs offline
by default (mocked litellm) and exercises six routing scenarios; live
mode opt-in via `LAYERLENS_LITELLM_LIVE=1`.

Pricing: LiteLLM does not add manifest entries — the adapter calls
`litellm.completion_cost` first, then falls through to the canonical
`PRICING` map. Doc at `docs/adapters/providers-litellm.md` updated to
describe routing + pricing inheritance + subpackage layout.

Acceptance:
- pytest tests/instrument/adapters/providers/test_litellm.py: 38/38 pass
- mypy --strict src/.../providers/litellm: clean
- ruff check (changed files): clean
- lazy-import + default-install guards: pass
@mmercuri mmercuri requested a review from m-peko April 26, 2026 08:05
@m-peko m-peko closed this May 21, 2026
@m-peko m-peko deleted the feat/instrument-providers-litellm branch July 2, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants