Bugfix/trace evaluations

README.md

@@ -1,6 +1,6 @@
 # LayerLens Stratix Python SDK
 
-The official Python library for the [LayerLens Stratix](https://layerlens.ai) evaluation API.
+The official Python library for the [LayerLens Stratix](https://app.layerlens.ai) evaluation API.
 
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

docs/README.md

@@ -1,6 +1,6 @@
 # LayerLens Stratix Python SDK
 
-The official Python library for the [LayerLens Stratix](https://layerlens.ai) evaluation API.
+The official Python library for the [LayerLens Stratix](https://app.layerlens.ai) evaluation API.
 
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

docs/getting-started/authentication.md

@@ -0,0 +1,79 @@
+# Authentication & Configuration
+
+## API Key Setup
+
+The SDK authenticates using an API key tied to your LayerLens organization. You can obtain your key from the [LayerLens dashboard](https://app.layerlens.ai).
+
+### Environment Variable (Recommended)
+
+Set the `LAYERLENS_STRATIX_API_KEY` environment variable and the client will pick it up automatically:
+
+```bash
+export LAYERLENS_STRATIX_API_KEY="your-api-key"
+```
+
+```python
+from layerlens import Stratix
+
+# Automatically reads from LAYERLENS_STRATIX_API_KEY
+client = Stratix()
+```
+
+### Explicit API Key
+
+Pass the key directly when constructing the client:
+
+```python
+from layerlens import Stratix
+
+client = Stratix(api_key="your-api-key")
+```
+
+### Using a .env File
+
+```bash
+# .env (add this file to .gitignore)
+LAYERLENS_STRATIX_API_KEY=your-api-key
+```
+
+```python
+from dotenv import load_dotenv
+load_dotenv()
+
+from layerlens import Stratix
+client = Stratix()
+```
+
+## Configuration Options
+
+| Parameter  | Type                             | Default                           | Description     |
+| ---------- | -------------------------------- | --------------------------------- | --------------- |
+| `api_key`  | `str \| None`                    | `LAYERLENS_STRATIX_API_KEY` env   | Your API key    |
+| `base_url` | `str \| httpx.URL \| None`       | `https://api.layerlens.ai/api/v1` | API base URL    |
+| `timeout`  | `float \| httpx.Timeout \| None` | 10 minutes                        | Request timeout |
+
+## Environment Variables
+
+| Variable                     | Description               | Default                           |
+| ---------------------------- | ------------------------- | --------------------------------- |
+| `LAYERLENS_STRATIX_API_KEY`  | Your API key              | (required)                        |
+| `LAYERLENS_STRATIX_BASE_URL` | Override the API base URL | `https://api.layerlens.ai/api/v1` |
+
+Legacy environment variables (`LAYERLENS_ATLAS_API_KEY`, `LAYERLENS_ATLAS_BASE_URL`) are also supported for backward compatibility.
+
+## Organization & Project
+
+On initialization, the client fetches your organization and project IDs automatically using your API key. These are used to scope all API requests.
+
+## Timeout Configuration
+
+```python
+# Global timeout (30 seconds)
+client = Stratix(timeout=30.0)
+
+# Per-request timeout override
+evaluation = client.with_options(timeout=120.0).evaluations.create(
+    model=model,
+    benchmark=benchmark,
+)
+```

docs/getting-started/installation.md

@@ -0,0 +1,47 @@
+# Installation
+
+## Requirements
+
+- Python 3.8 or higher
+- pip package manager
+
+## Install from LayerLens Package Registry
+
+```bash
+pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
+```
+
+## Verify Installation
+
+```python
+import layerlens
+print(layerlens.__version__)
+```
+
+## Dependencies
+
+The SDK installs the following dependencies automatically:
+
+- `httpx` - HTTP client for API requests
+- `pydantic` - Data validation and serialization
+- `requests` - Used for file uploads (presigned S3 URLs)
+
+## Upgrading
+
+To upgrade to the latest version:
+
+```bash
+pip install --upgrade layerlens --extra-index-url https://sdk.layerlens.ai/package
+```
+
+## Virtual Environment (Recommended)
+
+It's recommended to install the SDK in a virtual environment to avoid dependency conflicts:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate  # Linux/macOS
+# .venv\Scripts\activate   # Windows
+
+pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
+```

docs/getting-started/quickstart.md

@@ -0,0 +1,130 @@
+# Quick Start Guide
+
+This guide walks you through the most common SDK workflows.
+
+## Setup
+
+```bash
+pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
+export LAYERLENS_STRATIX_API_KEY="your-api-key"
+```
+
+## Run a Benchmark Evaluation
+
+```python
+from layerlens import Stratix
+
+client = Stratix()
+
+# Get a model and benchmark by key
+model = client.models.get_by_key("openai/gpt-4o")
+benchmark = client.benchmarks.get_by_key("arc-agi-2")
+
+# Create an evaluation
+evaluation = client.evaluations.create(
+    model=model,
+    benchmark=benchmark,
+)
+
+# Wait for results
+result = client.evaluations.wait_for_completion(evaluation)
+print(f"Accuracy: {result.accuracy}")
+```
+
+## Create a Judge and Evaluate Traces
+
+```python
+import time
+from layerlens import Stratix
+
+client = Stratix()
+
+# Create a judge
+judge = client.judges.create(
+    name="Response Quality Judge",
+    evaluation_goal="Rate whether the response is accurate, complete, and well-structured",
+)
+
+# Upload traces from a JSON/JSONL file
+upload = client.traces.upload("./my_traces.json")
+print(f"Uploaded {len(upload.trace_ids)} traces")
+
+# Run a trace evaluation
+trace_eval = client.trace_evaluations.create(
+    trace_id=upload.trace_ids[0],
+    judge_id=judge.id,
+)
+
+# Poll until complete
+while True:
+    evaluation = client.trace_evaluations.get(trace_eval.id)
+    if evaluation.status.value in ("success", "failure"):
+        break
+    time.sleep(2)
+
+# Get results
+result = client.trace_evaluations.get_results(trace_eval.id)
+if result:
+    print(f"Score: {result.score}, Passed: {result.passed}")
+    print(f"Reasoning: {result.reasoning}")
+```
+
+## Async Usage
+
+Every method is available in async form via `AsyncStratix`:
+
+```python
+import asyncio
+from layerlens import AsyncStratix
+
+async def main():
+    client = AsyncStratix()
+
+    model = await client.models.get_by_key("openai/gpt-4o")
+    benchmark = await client.benchmarks.get_by_key("arc-agi-2")
+
+    evaluation = await client.evaluations.create(
+        model=model,
+        benchmark=benchmark,
+    )
+
+    result = await client.evaluations.wait_for_completion(evaluation)
+    print(f"Accuracy: {result.accuracy}")
+
+asyncio.run(main())
+```
+
+## Browse Public Data
+
+```python
+from layerlens import Stratix
+
+client = Stratix()
+
+# List public models
+models = client.public.models.get()
+for model in models.models:
+    print(f"{model.key}: {model.name}")
+
+# List public benchmarks
+benchmarks = client.public.benchmarks.get()
+for bm in benchmarks.benchmarks:
+    print(f"{bm.key}: {bm.name}")
+```
+
+## Error Handling
+
+```python
+from layerlens import Stratix, NotFoundError, AuthenticationError, APIError
+
+client = Stratix()
+
+try:
+    model = client.models.get_by_id("nonexistent-id")
+except NotFoundError as e:
+    print(f"Not found: {e.message}")
+except AuthenticationError as e:
+    print(f"Auth failed: {e.message}")
+except APIError as e:
+    print(f"API error ({e.status_code}): {e.message}")
+```

docs/security/data-privacy.md

@@ -0,0 +1,51 @@
+# Data Privacy
+
+This guide covers data privacy considerations when using the LayerLens Stratix SDK.
+
+## Data Sent to the API
+
+When using the SDK, the following data is transmitted to LayerLens servers:
+
+- **API key** - Used for authentication (sent in request headers)
+- **Trace data** - Contents of trace files you upload via `client.traces.upload()`
+- **Evaluation parameters** - Model and benchmark selections, judge configurations
+- **Judge definitions** - Evaluation goals and settings for custom judges
+
+## Data Storage
+
+- Uploaded traces are stored in your organization's project scope and are not shared across organizations
+- Evaluation results are associated with your organization and project
+- API keys are never logged or stored in SDK-side logs
+
+## Sensitive Data in Traces
+
+If your traces contain sensitive information (PII, credentials, proprietary data), consider:
+
+1. **Sanitize before upload** - Remove or redact sensitive fields from trace files before uploading
+2. **Use test data** - Use synthetic or anonymized data for development and testing
+3. **Review trace contents** - Inspect trace files before upload to ensure no secrets are included
+
+## Logging
+
+The SDK uses Python's standard `logging` module. Sensitive headers (API keys, authorization tokens) are automatically redacted in log output.
+
+```python
+import logging
+
+# Enable SDK logging (API keys will be redacted)
+logging.basicConfig(level=logging.DEBUG)
+```
+
+## Local Data
+
+The SDK does not write any data to disk. All operations are performed in-memory and transmitted directly to the API. Trace files are read from disk for upload but are not modified or cached.
+
+## Environment Variable Security
+
+Store API keys in environment variables rather than in source code:
+
+```bash
+export LAYERLENS_STRATIX_API_KEY="your-api-key"
+```
+
+See [API Key Management](api-key-management.md) and [Environment Variables](environment-variables.md) for detailed guidance.