Skip to content

PublicClient, model & benchmarks management #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
m-peko merged 5 commits into from
Feb 27, 2026
Merged

PublicClient, model & benchmarks management #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
01b0fe8
Add public resources to SDK
m-peko Feb 25, 2026
3766c36
Add model and benchmark managing functionality to the SDK
m-peko Feb 26, 2026
6d9d320
Bump version
m-peko Feb 26, 2026
3a5e1bc
Return evaluation summary
m-peko Feb 26, 2026
bec8217
Fetch public evaluations
m-peko Feb 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

## API Reference
* [Client Configuration](api-reference/client.md)
* [Public Client](api-reference/public-client.md)
* [Evaluations](api-reference/evaluations.md)
* [Results](api-reference/results.md)
* [Models & Benchmarks](api-reference/models-benchmarks.md)
Expand Down
19 changes: 19 additions & 0 deletions docs/api-reference/client.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,25 @@ The client automatically loads configuration from these environment variables:
LAYERLENS_STRATIX_API_KEY="your_api_key_here"
```

## Public Client

For accessing public endpoints (models, benchmarks, comparisons), use `PublicClient` or `AsyncPublicClient`. See the [Public Client](public-client.md) reference for full details.

```python
from layerlens import PublicClient

# Loads API key from the "LAYERLENS_STRATIX_API_KEY" environment variable
public = PublicClient()
models = public.models.get(companies=["OpenAI"])
```

You can also access public endpoints from an authenticated client via the `.public` property:

```python
client = Stratix()
public_models = client.public.models.get(query="claude")
```

## Timeout Configuration

### Simple Timeout
Expand Down
87 changes: 70 additions & 17 deletions docs/api-reference/evaluations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,17 +177,22 @@ async def get_evaluation():
asyncio.run(get_evaluation())
```

### `get_many(page=None, page_size=None, timeout=None)`
### `get_many(page=None, page_size=None, sort_by=None, order=None, model_ids=None, benchmark_ids=None, status=None, timeout=None)`

Retrieves multiple evaluations with optional pagination support.
Retrieves multiple evaluations with optional pagination, sorting, and filtering.

#### Parameters

| Parameter | Type | Required | Description |
| ----------- | -------------------------------- | -------- | ------------------------------------------------------- |
| `page` | `int \| None` | No | Page number for pagination (1-based, defaults to 1) |
| `page_size` | `int \| None` | No | Number of evaluations per page (default: 100, max: 500) |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |
| Parameter | Type | Required | Description |
| --------------- | -------------------------------- | -------- | ------------------------------------------------------- |
| `page` | `int \| None` | No | Page number for pagination (1-based, defaults to 1) |
| `page_size` | `int \| None` | No | Number of evaluations per page (default: 100, max: 500) |
| `sort_by` | `str \| None` | No | Sort by field: `submittedAt`, `accuracy`, or `averageDuration` |
| `order` | `str \| None` | No | Sort order: `asc` or `desc` |
| `model_ids` | `List[str] \| None` | No | Filter by model IDs |
| `benchmark_ids` | `List[str] \| None` | No | Filter by benchmark/dataset IDs |
| `status` | `EvaluationStatus \| None` | No | Filter by evaluation status |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Expand All @@ -198,6 +203,27 @@ Returns an `EvaluationsResponse` object containing:

Returns `None` if the request fails.

#### Example

```python
from layerlens import Stratix
from layerlens.models import EvaluationStatus

client = Stratix()

# Get top evaluations by accuracy
response = client.evaluations.get_many(
sort_by="accuracy",
order="desc",
status=EvaluationStatus.SUCCESS,
page_size=10,
)

if response:
for evaluation in response.evaluations:
print(f"{evaluation.id}: accuracy={evaluation.accuracy:.2f}%")
```

### `get_results(page=None, page_size=None, timeout=None)`

Fetches results for this evaluation with pagination support. This is a synchronous method.
Expand Down Expand Up @@ -378,16 +404,43 @@ The `create`, `get_by_id` and `get_many` method returns an `Evaluation` objects

### Evaluation Object Properties

| Property | Type | Description |
| ------------------ | ------------------ | --------------------------------------------------------- |
| `id` | `str` | Unique evaluation identifier |
| `status` | `EvaluationStatus` | Current evaluation status (enum) |
| `submitted_at` | `int` | Unix timestamp when evaluation was submitted |
| `finished_at` | `int` | Unix timestamp when evaluation finished |
| `model_id` | `str` | ID of the model used in the evaluation |
| `benchmark_id` | `str` | ID of the benchmark used (aliased as "dataset_id" in API) |
| `average_duration` | `int` | Average response time in milliseconds |
| `accuracy` | `float` | Overall accuracy score (0.0 to 1.0) |
| Property | Type | Description |
| -------------------- | ----------------------------- | --------------------------------------------------------- |
| `id` | `str` | Unique evaluation identifier |
| `status` | `EvaluationStatus` | Current evaluation status (enum) |
| `status_description` | `str` | Human-readable status description (default: `""`) |
| `submitted_at` | `int` | Unix timestamp when evaluation was submitted |
| `finished_at` | `int` | Unix timestamp when evaluation finished |
| `model_id` | `str` | ID of the model used in the evaluation |
| `model_name` | `str` | Name of the model (default: `""`) |
| `model_key` | `str` | Key identifier of the model (default: `""`) |
| `model_company` | `str` | Company/provider of the model (default: `""`) |
| `benchmark_id` | `str` | ID of the benchmark used (aliased as "dataset_id" in API) |
| `benchmark_name` | `str` | Name of the benchmark (aliased as "dataset_name" in API, default: `""`) |
| `average_duration` | `int` | Average response time in milliseconds |
| `accuracy` | `float` | Overall accuracy score (0.0 to 1.0) |
| `readability_score` | `float` | Readability score (default: `0.0`) |
| `toxicity_score` | `float` | Toxicity score (default: `0.0`) |
| `ethics_score` | `float` | Ethics score (default: `0.0`) |
| `failed_prompt_count`| `int` | Number of failed prompts (default: `0`) |
| `queue_id` | `int` | Queue identifier (default: `0`) |
| `summary` | `EvaluationSummary \| None` | Rich evaluation summary (see below, default: `None`) |

### EvaluationSummary Object

The `summary` field contains a rich analysis of the evaluation when available.

| Property | Type | Description |
| --------------------- | ------------------------------- | ---------------------------------------- |
| `name` | `str` | Summary title |
| `goal` | `str` | Goal of the evaluation |
| `metrics` | `List[EvaluationMetric]` | Metrics used (each has `name`, `description`) |
| `task_types` | `List[EvaluationTaskType]` | Task types (each has `name`, `description`) |
| `dataset` | `EvaluationDataset \| None` | Dataset info (`total_size`, `training_size`, `test_size`, `characteristics`) |
| `model` | `EvaluationModelInfo \| None` | Model info (`model_name`, `performance`) |
| `performance_details` | `PerformanceDetails \| None` | Strengths and challenges lists |
| `error_analysis` | `ErrorAnalysis \| None` | Common failure modes and example |
| `analysis_summary` | `AnalysisSummary \| None` | Key takeaways list |

#### Evaluation Status

Expand Down
220 changes: 218 additions & 2 deletions docs/api-reference/models-benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,87 @@ Retrieves a specific model by its unique key. Both the `Stratix` and `AsyncStrat

Returns an `Optional[Model]` - a single `Model` object if found, or `None` if the model doesn't exist or there's an error.

### `add(*model_ids, timeout=None)`

Adds public models to the project by their IDs.

#### Parameters

| Parameter | Type | Required | Description |
| ------------ | -------------------------------- | -------- | ------------------------------ |
| `*model_ids` | `str` | Yes | One or more model IDs to add |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` - `True` if the operation succeeded, `False` otherwise.

#### Example

```python
client = Stratix()
success = client.models.add("model-id-1", "model-id-2")
```

### `remove(*model_ids, timeout=None)`

Removes models from the project by their IDs.

#### Parameters

| Parameter | Type | Required | Description |
| ------------ | -------------------------------- | -------- | --------------------------------- |
| `*model_ids` | `str` | Yes | One or more model IDs to remove |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` - `True` if the operation succeeded, `False` otherwise.

### `create_custom(name, key, description, api_url, max_tokens, api_key=None, timeout=None)`

Creates a custom model backed by an OpenAI-compatible API endpoint. This allows you to evaluate any model accessible via a chat completions endpoint.

#### Parameters

| Parameter | Type | Required | Description |
| ------------- | -------------------------------- | -------- | --------------------------------------------------------------------------------- |
| `name` | `str` | Yes | Model name (max 256 characters) |
| `key` | `str` | Yes | Unique model key, lowercase alphanumeric with dots/hyphens/slashes (max 256 chars)|
| `description` | `str` | Yes | Model description (max 500 characters) |
| `api_url` | `str` | Yes | Base URL of the OpenAI-compatible API endpoint |
| `max_tokens` | `int` | Yes | Maximum number of tokens the model supports |
| `api_key` | `str \| None` | No | API key for the model provider |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns an `Optional[CreateModelResponse]` containing:

- `organization_id`: Organization identifier
- `project_id`: Project identifier
- `model_id`: The newly created model's identifier

Returns `None` if the request fails.

#### Example

```python
client = Stratix()

result = client.models.create_custom(
name="My Custom Model",
key="my-org/custom-model-v1",
description="Custom fine-tuned model served via vLLM",
api_url="https://my-model-endpoint.example.com/v1",
api_key="my-provider-api-key",
max_tokens=4096,
)

if result:
print(f"Created model: {result.model_id}")
```

## Benchmarks

### `get(type=None, name=None, timeout=None)`
Expand All @@ -111,8 +192,6 @@ Retrieves a list of available benchmarks with optional filtering parameters. Bot

#### Returns

Returns a `List[Benchmark]` containing available benchmarks that match the filter criteria. Returns `None` if no benchmarks are found or if there's an error.

Returns `Optional[List[Benchmark]]` - a list of `Benchmark` objects that match the filter criteria. Returns an empty list `[]` if no benchmarks match the criteria, or `None` if there's an error.

#### Benchmark Object Properties
Expand Down Expand Up @@ -154,3 +233,140 @@ Retrieves a specific benchmark by its unique key. Both the `Stratix` and `AsyncS
#### Returns

Returns an `Optional[Benchmark]` - a single `Benchmark` object if found, or `None` if the benchmark doesn't exist or there's an error.

### `add(*benchmark_ids, timeout=None)`

Adds benchmarks to the project by their IDs.

#### Parameters

| Parameter | Type | Required | Description |
| ---------------- | -------------------------------- | -------- | ---------------------------------- |
| `*benchmark_ids` | `str` | Yes | One or more benchmark IDs to add |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` - `True` if the operation succeeded, `False` otherwise.

#### Example

```python
client = Stratix()
success = client.benchmarks.add("benchmark-id-1", "benchmark-id-2")
```

### `remove(*benchmark_ids, timeout=None)`

Removes benchmarks from the project by their IDs.

#### Parameters

| Parameter | Type | Required | Description |
| ---------------- | -------------------------------- | -------- | ------------------------------------- |
| `*benchmark_ids` | `str` | Yes | One or more benchmark IDs to remove |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` - `True` if the operation succeeded, `False` otherwise.

### `create_custom(name, description, file_path, additional_metrics=None, custom_scorer_ids=None, input_type=None, timeout=None)`

Creates a custom benchmark by uploading a JSONL file. The file should contain one JSON object per line with `input` and `truth` fields.

#### Parameters

| Parameter | Type | Required | Description |
| -------------------- | -------------------------------- | -------- | -------------------------------------------------------------------- |
| `name` | `str` | Yes | Benchmark name (max 64 characters) |
| `description` | `str` | Yes | Benchmark description (max 280 characters) |
| `file_path` | `str` | Yes | Path to a JSONL file with benchmark prompts |
| `additional_metrics` | `List[str] \| None` | No | Additional metrics: `readability`, `toxicity`, `hallucination` |
| `custom_scorer_ids` | `List[str] \| None` | No | List of custom scorer IDs to use |
| `input_type` | `str \| None` | No | Input type: `messages` or `json_payload` |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### JSONL File Format

Each line should be a JSON object:

```json
{"input": "What is 2+2?", "truth": "4"}
{"input": "Capital of France?", "truth": "Paris"}
```

Optional fields: `subset` (for grouping prompts into categories).

#### Returns

Returns an `Optional[CreateBenchmarkResponse]` containing:

- `organization_id`: Organization identifier
- `project_id`: Project identifier
- `benchmark_id`: The newly created benchmark's identifier

Returns `None` if the request fails.

#### Example

```python
client = Stratix()

result = client.benchmarks.create_custom(
name="QA Benchmark",
description="Tests model factual accuracy",
file_path="benchmark_data.jsonl",
additional_metrics=["hallucination"],
)

if result:
print(f"Created benchmark: {result.benchmark_id}")
```

### `create_smart(name, description, system_prompt, file_paths, metrics=None, timeout=None)`

Creates a smart benchmark from uploaded files. The platform uses AI to automatically generate benchmark prompts from the provided documents. The benchmark is generated asynchronously.

#### Parameters

| Parameter | Type | Required | Description |
| --------------- | -------------------------------- | -------- | ------------------------------------------------------------------- |
| `name` | `str` | Yes | Benchmark name (max 256 characters) |
| `description` | `str` | Yes | Benchmark description (max 500 characters) |
| `system_prompt` | `str` | Yes | System prompt guiding benchmark generation (max 4000 characters) |
| `file_paths` | `List[str]` | Yes | List of file paths to upload (1-20 files, max 50 MB each) |
| `metrics` | `List[str] \| None` | No | Additional metrics: `readability`, `toxicity`, `hallucination` |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Supported File Types

`.txt`, `.pdf`, `.html`, `.docx`, `.csv`, `.json`, `.jsonl`, `.parquet`

#### Returns

Returns an `Optional[CreateBenchmarkResponse]` containing:

- `organization_id`: Organization identifier
- `project_id`: Project identifier
- `benchmark_id`: The newly created benchmark's identifier

Returns `None` if the request fails.

#### Example

```python
client = Stratix()

result = client.benchmarks.create_smart(
name="Product Knowledge Benchmark",
description="Evaluates model knowledge of product docs",
system_prompt="Generate QA pairs testing understanding of product features.",
file_paths=["product_docs.pdf", "faq.txt"],
metrics=["hallucination"],
)

if result:
print(f"Smart benchmark created: {result.benchmark_id}")
print("Check the dashboard for generation progress.")
```
Loading
Loading