+
+[**Run your first eval**](#quick-start) · [**Browse 213 models**](https://stratix.layerlens.ai) · [**Star if useful ⭐**](https://github.com/LayerLens/stratix-python)
+
+
+
+[**Run your first eval**](#quick-start) · [**Browse 213 models**](https://stratix.layerlens.ai) · [**Star if useful ⭐**](https://github.com/LayerLens/stratix-python)
+
+
+ Vendor-neutral evals in 5 lines of Python.
+| -Set your API key as an environment variable: +### Vendor-neutral + +Stratix is not owned by a model provider. The same benchmark runs across 213 public models from 26 providers in one workspace. No labs grading their own homework. No leaderboards optimized for marketing. + + | ++ +### Reproducible by default + +Every score is backed by a verifiable, persisted trace you can re-run, inspect, and cite. Same prompt, same prompt template, same scoring logic, same model version. Every time. + + | ++ +### Production-ready + +Wire evals into CI. Calibrate judges to a quality goal in plain English. Score full agent traces, not just last-token outputs. Ship reliable agents faster. + + | +
| Standard (pip) | +Modern (uv) | +Authenticate | +
|---|---|---|
| + +```bash +pip install layerlens +``` + + | ++ +```bash +uv pip install layerlens +``` + + | ++ +```bash +export LAYERLENS_STRATIX_API_KEY=... +``` + +Or pass `api_key=...` to the client. + + | +
| + +### Model evaluation + +Run any of 213 public models across 59 benchmarks. AIME, GPQA, ARC-AGI-2, HumanEval, Terminal-Bench, MMLU Pro, BIRD-CRITIC, more. Reasoning, coding, math, agentic, multilingual. + +[Docs →](https://layerlens.gitbook.io/stratix-python-sdk) + + | ++ +### Agent trace evaluation + +Upload OpenAI-format trace files and score multi-step agent behavior. Tool use, planning quality, recovery from failures. Not just the final token. + +[Docs →](https://layerlens.gitbook.io/stratix-python-sdk) + + | ++ +### Judge calibration + +Define a quality goal in plain English. Stratix calibrates an LLM-as-judge to that goal, validates against your gold examples, and reuses the judge across runs. + +[Docs →](https://layerlens.gitbook.io/stratix-python-sdk) + + | +
| + +### Custom benchmarks + +Bring your own dataset. Smart benchmark generation for adversarial cases, edge inputs, and domain-specific evals. Reuses public scoring infrastructure. + +[Docs →](https://layerlens.gitbook.io/stratix-python-sdk) + + | ++ +### CI integration + +Fail the build on quality regressions, not just on red unit tests. Use `stratix ci report` in GitHub Actions, GitLab CI, CircleCI, or any Python-capable runner. + +[Sample →](./samples/cicd) + + | ++ +### Reproducible runs + +Every evaluation persists model version, prompt template, judge config, and full traces. Re-run any evaluation by ID. Cite the result with confidence. + +[Docs →](https://layerlens.gitbook.io/stratix-python-sdk) + + | +
| Hand-rolled (typical) | +Stratix | +
|---|---|
| ```python -import os -from layerlens import Stratix +import openai, json, asyncio +from datasets import load_dataset -client = Stratix(api_key=os.environ.get("LAYERLENS_STRATIX_API_KEY")) +ds = load_dataset("aime-2026")["test"] +client = openai.OpenAI() -# Get a model and benchmark by key -model = client.models.get_by_key("openai/gpt-4o") -benchmark = client.benchmarks.get_by_key("arc-agi-2") +results = [] +async def score_one(item): + resp = await client.chat.completions.create( + model="gpt-5.5-20260423", + messages=[{"role":"user","content":item["q"]}], + ) + answer = parse_answer(resp.choices[0].message.content) + return {"q": item["q"], "ans": answer, "expected": item["a"], + "correct": answer == item["a"]} + +# Implement: rate limiting, retries, cost tracking, +# trace storage, judge logic, schema versioning, +# benchmark drift detection, regression alerting. +# Repeat per benchmark. Per model. Per release. +``` + + | ++ +```python +from layerlens import Stratix + +client = Stratix() # reads LAYERLENS_STRATIX_API_KEY -# Create an evaluation (pass the full model and benchmark objects) evaluation = client.evaluations.create( - model=model, - benchmark=benchmark, + model=client.models.get_by_key("openai/gpt-5.5-20260423"), + benchmark=client.benchmarks.get_by_key("aime2026"), ) - -# Wait for results (pass the evaluation object, not just the ID) result = client.evaluations.wait_for_completion(evaluation) -print(f"Accuracy: {result.accuracy}") + +print(result.accuracy) +print(f"https://stratix.layerlens.ai/evaluations/{result.id}") ``` -### Async usage + | +
| + | Stratix | +Braintrust | +LangSmith | +Phoenix | +OpenAI Evals | +
|---|---|---|---|---|---|
| Public-model leaderboard | 213 | none | none | none | limited |
| Independent grading | ✅ | ✅ | ✅ | ✅ | ⚠️ vendor |
| Reproducible scores | ✅ traces persisted | ✅ | ✅ | ✅ | ✅ |
| Agent trace evaluation | ✅ | ✅ | ✅ | ✅ | ⚠️ |
| Judge calibration in SDK | ✅ | ✅ | ⚠️ | ⚠️ | ⚠️ |
| Custom benchmarks | ✅ | ✅ | ✅ | ✅ | ✅ |
| Smart benchmark generation | ✅ | via templates | via templates | manual | manual |
| 59 prebuilt benchmarks out of the box | ✅ | via templates | via templates | via Arize | small core set |
| Context manager (sync) | +Context manager (async) | +
|---|---|
| ```python -from layerlens import Stratix # Primary -from layerlens import AsyncStratix # Async primary -from layerlens import Client # Alias for Stratix -from layerlens import AsyncClient # Alias for AsyncStratix -from layerlens import Atlas # Legacy alias -from layerlens import AsyncAtlas # Legacy alias -from layerlens import PublicClient # Public endpoints -from layerlens import AsyncPublicClient +from layerlens import Stratix + +with Stratix() as client: + eval = client.evaluations.create(...) +# HTTP connection released ``` -## Configuration + | +-| Environment Variable | Description | Default | -| ---------------------------- | ------------------------- | --------------------------------- | -| `LAYERLENS_STRATIX_API_KEY` | Your API key | (required) | -| `LAYERLENS_STRATIX_BASE_URL` | Override the API base URL | `https://api.layerlens.ai/api/v1` | +```python +import asyncio +from layerlens import AsyncStratix -Legacy env vars (`LAYERLENS_ATLAS_API_KEY`, `LAYERLENS_ATLAS_BASE_URL`) are also supported. +async def main(): + async with AsyncStratix() as client: + eval = await client.evaluations.create(...) -## Error handling +asyncio.run(main()) +``` -The SDK raises typed exceptions for API errors: + | +
| Recently shipped | +In progress | +Coming up | +Exploring | +
|---|---|---|---|
| + +- [x] 213 public models +- [x] Agent trace evaluation +- [x] Judge calibration +- [x] Smart benchmark generation +- [x] Async client +- [x] Reproducible runs + + | ++ +- [ ] Deliberation panels +- [ ] Custom-model adapters (open weights) +- [ ] Cost-aware eval routing -Full CLI docs: [docs/cli/](docs/cli/) + | +-| Guide | Description | -| --- | --- | -| [Getting Started](docs/cli/getting-started.md) | Installation, configuration, first commands | -| [Command Reference](docs/cli/commands.md) | All commands and options | -| [Examples](docs/cli/examples.md) | 15 common workflows as copy-paste shell sessions | +- [ ] Per-domain leaderboards +- [ ] Streaming eval results +- [ ] TypeScript SDK -## Requirements + | +-- Python 3.8+ -- Dependencies: `httpx`, `pydantic`, `requests` -- CLI extra: `click>=8.0.0` +- [ ] Cross-model A/B harness +- [ ] Latency-quality Pareto plots +- [ ] OpenTelemetry trace ingest -## Documentation + | +