feat(instrument): Port LiteLLM provider adapter (M3)#107
Closed
mmercuri wants to merge 3 commits into
Closed
Conversation
Bootstraps the LayerLens instrument layer with the abstract base classes,
adapter registry, capture configuration, event sinks, vendored event
schemas, and pydantic v1/v2 compatibility shim that every concrete
adapter (frameworks, protocols, providers) will depend on.
Scope
-----
- src/layerlens/instrument/__init__.py: lean re-export surface
- src/layerlens/instrument/_vendored/: frozen ateam event schemas (no
runtime ateam dependency)
- src/layerlens/instrument/adapters/_base/: BaseAdapter, AdapterRegistry,
AdapterStatus, AdapterHealth, AdapterCapability, ReplayableTrace,
CaptureConfig, EventSink, TraceStoreSink, IngestionPipelineSink,
PydanticCompat
- src/layerlens/_compat/pydantic.py: model_dump/model_validate shim
spanning pydantic v1 + v2
- scripts/{port_adapter,port_protocol,emit_adapter_manifest,
regen_dep_baselines}.py: codegen helpers used to port the rest of M1
- tests/instrument/{test_base_layer,test_lazy_imports,
test_default_install,test_resolved_dep_tree}.py + _baselines/
- .github/workflows/dep-tree-guard.yaml: CI gate that locks the default
install footprint
- docs/adapters/: CONTRIBUTING, STATUS, pydantic-compatibility, testing,
PERSONA_REVIEW
Blast radius
------------
- Pure additions. No public surface changes outside the new
layerlens.instrument namespace.
- Default `pip install layerlens` install set is unchanged (verified by
test_default_install.py against the new baseline).
- Lazy adapter discovery: importing layerlens.instrument MUST NOT pull
in any optional adapter dep (verified by test_lazy_imports.py).
Test plan
---------
- uv run pytest tests/instrument/test_base_layer.py
tests/instrument/test_lazy_imports.py -x -> 45 passed
- The dep-tree-guard workflow exercises test_default_install.py and
test_resolved_dep_tree.py against the new baselines on every PR.
LAY-3400 umbrella: this PR is the prerequisite for the M1.B/M1.C/M1.D
adapter ports, M7 protocol certification, and M8 Cohere/Mistral.
Ports the nine LLM provider adapters from the ateam reference
implementation onto the new layerlens.instrument base layer:
OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Vertex,
Ollama, LiteLLM, Cohere, Mistral
This rolls M1.B (the seven original providers) and M8 (Cohere +
Mistral) into a single PR because they share the same provider _base
helpers (pricing, token counters, provider mixin) and per-PR they would
all be sub-1k LOC.
Scope
-----
- src/layerlens/instrument/adapters/providers/_base/: shared provider
mixin (pricing.py, provider.py, tokens.py)
- src/layerlens/instrument/adapters/providers/{openai,anthropic,
azure_openai,bedrock,google_vertex,ollama,litellm,cohere,mistral}_
adapter.py: per-provider adapter
- tests/instrument/adapters/providers/: unit + live tests for all nine
providers (live tests skip when no API key is set)
- tests/instrument/adapters/test_pydantic_compat.py: shared compat
surface used by every provider
- samples/instrument/{openai,anthropic,cohere,mistral}/: runnable
samples
- docs/adapters/providers-*.md: per-provider integration guide
- pyproject.toml: nine new optional extras
(providers-openai, providers-anthropic, providers-azure-openai,
providers-bedrock, providers-vertex, providers-ollama,
providers-litellm, providers-cohere, providers-mistral) plus the
providers-all umbrella
Blast radius
------------
- Default `pip install layerlens` install set is unchanged. Each
provider's heavy dep (openai, anthropic, boto3, etc.) is gated
behind its own `providers-<name>` extra.
- No changes to existing public API surface.
- Importing layerlens.instrument still does NOT pull in any provider
module (lazy registry lookup).
Test plan
---------
- uv run pytest tests/instrument/adapters/providers/ -x -> 122 passed,
4 skipped (live-only without keys)
- uv run pytest tests/instrument/adapters/test_pydantic_compat.py -x
-> 62 passed
Stacks on
---------
- feat/instrument-base-foundation (M1.A) — required for the BaseAdapter
surface this PR consumes.
LAY-3400 umbrella (M1.B + M8).
Splits the M1.B port of the LiteLLM provider adapter into a subpackage
under `providers/litellm/`:
- `adapter.py` — `LiteLLMAdapter` lifecycle (connect/disconnect/version)
- `callback.py` — `LayerLensLiteLLMCallback` with sync + async hooks
(`log_*_event` and `async_log_*_event`)
- `routing.py` — `detect_provider` mapping LiteLLM model strings to
canonical LayerLens provider names
- `__init__.py` — public surface, `ADAPTER_CLASS`, and
`STRATIXLiteLLMCallback` backward-compat alias
The legacy flat `providers/litellm_adapter.py` becomes a thin re-export
so existing imports keep working. `providers/__init__.py` gains a
PEP 562 `__getattr__` shim so `LiteLLMAdapter` is importable directly
off the package without forcing the vendor SDK to load eagerly.
Tests at `tests/instrument/adapters/providers/test_litellm.py` mock the
`litellm.completion()` / `litellm.acompletion()` boundary and cover:
- 20 routing cases (`gpt-4` -> openai, `claude-3-5-sonnet` -> anthropic,
`bedrock/anthropic.claude-3-5-sonnet` -> aws_bedrock, etc.)
- adapter lifecycle (connect/disconnect/degrade-when-missing)
- sync success/failure/streaming callback emission
- async success/failure callback emission via `acompletion`
- legacy flat-module re-export equivalence
- subpackage import does not load `litellm` (lazy-import contract)
Sample at `samples/instrument/providers/litellm/main.py` runs offline
by default (mocked litellm) and exercises six routing scenarios; live
mode opt-in via `LAYERLENS_LITELLM_LIVE=1`.
Pricing: LiteLLM does not add manifest entries — the adapter calls
`litellm.completion_cost` first, then falls through to the canonical
`PRICING` map. Doc at `docs/adapters/providers-litellm.md` updated to
describe routing + pricing inheritance + subpackage layout.
Acceptance:
- pytest tests/instrument/adapters/providers/test_litellm.py: 38/38 pass
- mypy --strict src/.../providers/litellm: clean
- ruff check (changed files): clean
- lazy-import + default-install guards: pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port of
stratix/sdk/python/adapters/llm_providers/litellm_adapter.py(~355 LOC, ateam) into the M3 fan-out atsrc/layerlens/instrument/adapters/providers/litellm/. Uses the existingproviders/openai_adapter.pyas the structural template (lifecycle /_emit_*helper usage /ADAPTER_CLASSregistry hook), and introduces a subpackage split:LiteLLM is a multi-provider router — a single
litellm.completion(...)call is dispatched to one of ~100 providers (OpenAI, Anthropic, Bedrock, Vertex AI, Cohere, Ollama, Together, Groq, HuggingFace, ...). Rather than monkey-patching every provider client, the adapter installs a single callback intolitellm.callbacksand lets LiteLLM dispatch it. The routing layer normalises the model string (bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0→aws_bedrock,gpt-4o→openai, etc.) into the canonical LayerLens provider name so downstream telemetry lines up across every adapter.Pricing: LiteLLM contributes no new entries to the canonical pricing manifest. The callback calls
litellm.completion_cost(...)first (recorded withcost_source: "litellm"when LiteLLM has its own pricing for the model); when LiteLLM cannot price the call it falls through to the canonicalPRICINGmap atproviders/_base/pricing.py. The routing layer's normalised provider name carries through tocost.recordso Bedrock-routed Anthropic flows throughBEDROCK_PRICING, not directPRICING.STRATIX*aliased toLayerLens*for backward-compat with ateam users;stratix.*paths translated tolayerlens.*. The legacy flatproviders/litellm_adapter.pyis kept as a thin re-export so existing imports keep working.Files
src/layerlens/instrument/adapters/providers/litellm/__init__.py(new, public surface + alias)src/layerlens/instrument/adapters/providers/litellm/adapter.py(new, lifecycle)src/layerlens/instrument/adapters/providers/litellm/callback.py(new, sync + async callbacks)src/layerlens/instrument/adapters/providers/litellm/routing.py(new, detect_provider)src/layerlens/instrument/adapters/providers/litellm_adapter.py(now thin re-export)src/layerlens/instrument/adapters/providers/__init__.py(PEP 562 lazy__getattr__forLiteLLMAdapter)tests/instrument/adapters/providers/test_litellm.py(new, 38 tests; supersedestest_litellm_adapter.py)tests/instrument/adapters/providers/test_litellm_adapter.py(deleted)samples/instrument/providers/litellm/{__init__.py,main.py,README.md}(new, mocked-by-default sample)docs/adapters/providers-litellm.md(rewritten — subpackage layout + pricing inheritance)pyproject.tomlalready carriedproviders-litellm = ["litellm>=1.40,<2"]from the M1.B port — no version bump needed.Test plan
uv run pytest tests/instrument/adapters/providers/test_litellm.py -x— 38/38 passuv run mypy --strict src/layerlens/instrument/adapters/providers/litellm— clean (4 source files)uv run ruff check src/.../providers/litellm src/.../providers/__init__.py src/.../providers/litellm_adapter.py tests/.../test_litellm.py— cleanuv run pytest tests/instrument/adapters/providers/ -x(no live tests) — 137/137 pass (no regression on sibling provider tests)uv run pytest tests/instrument/test_lazy_imports.py::test_layerlens_import_does_not_pull_frameworks tests/instrument/test_lazy_imports.py::test_instrument_import_does_not_pull_frameworks tests/instrument/test_default_install.py— 5/5 passpython -m samples.instrument.providers.litellm.main— runs offline, emits 12 events across 6 routing scenariosRouting coverage
Every prefix listed in the M3 PR description gets a parametrised test:
gpt-4,gpt-4o,o1-mini,o3-miniopenaiopenai/gpt-4o-miniopenaiclaude-3-5-sonnet,anthropic/claude-3-5-sonnetanthropicazure/my-deploymentazure_openaibedrock/anthropic.claude-3-5-sonnet-20241022-v2:0aws_bedrockvertex_ai/gemini-1.5-pro,gemini-2.0-flashgoogle_vertexollama/llama3ollamacohere/command-rcoherehuggingface/...,together_ai/...,groq/...llama-3.1-70bmetamistral-largemistralunknownNotes for review
test_litellm.py(not_adapter.py) per the task spec; old test removed.async_log_*_event) delegate to the sync handlers — LiteLLM hands the async dispatch the same kwargs/response_obj shape, so no separate logic. Theacompletiontest exercises this path with anAsyncMock.connect()degrades cleanly whenlitellmis missing (statusDEGRADED, no crash) — covered bytest_connect_degraded_when_litellm_missing.__getattr__shim onproviders/__init__.pywas the simplest way to exposeproviders.LiteLLMAdapterat the package level without breaking the lazy-import contract; the eagerif TYPE_CHECKING:import is for static analysers only.