Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,17 @@ Stratix powers evaluation workflows at LayerLens and across teams building produ

If your team uses Stratix, [open a PR](https://github.com/LayerLens/stratix-python/pulls) to add your logo here.

## Join the Community

The LayerLens Discord is the best place to:
- Get help with the SDK and trace evaluations
- Share your custom judges and agent workflows
- Access free Stratix Premium Credits for active contributors
- Join weekly Eval Office Hours & model comparison discussions
- Influence the roadmap

[Join the LayerLens Discord!](https://discord.gg/layerlens)

## Documentation

Full documentation is available at [layerlens.gitbook.io/stratix-python-sdk](https://layerlens.gitbook.io/stratix-python-sdk).
Expand Down
56 changes: 54 additions & 2 deletions docs/api-reference/models-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Returns an `Optional[Model]` - a single `Model` object if found, or `None` if th

### `add(*model_ids, timeout=None)`

Adds public models to the project by their IDs.
Adds models (public or custom) to the project by their IDs.

#### Parameters

Expand All @@ -123,7 +123,7 @@ success = client.models.add("model-id-1", "model-id-2")

### `remove(*model_ids, timeout=None)`

Removes models from the project by their IDs.
Removes models (public or custom) from the project's model list. The underlying records are not deleted — use `delete_custom` to fully tear down a custom model.

#### Parameters

Expand Down Expand Up @@ -187,6 +187,58 @@ if result:
print(f"Created model: {result.model_id}")
```

### `update_custom(model_id, *, api_url=None, api_key=None, max_tokens=None, timeout=None)`

Updates a custom model's mutable fields. At least one of `api_url`, `api_key`, or `max_tokens` must be provided. Primary use case: repointing `api_url` for ephemeral vLLM endpoints behind cloudflared tunnels whose URL changes between sessions.

#### Parameters

| Parameter | Type | Required | Description |
| ------------ | -------------------------------- | -------- | -------------------------------------------------------- |
| `model_id` | `str` | Yes | ID of the custom model to update |
| `api_url` | `str \| None` | No | New base URL for the OpenAI-compatible API endpoint |
| `api_key` | `str \| None` | No | New API key for the model provider |
| `max_tokens` | `int \| None` | No | New maximum tokens value |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` — `True` on success, `False` otherwise.

#### Example

```python
client = Stratix()

# Repoint the api_url without re-creating the model
client.models.update_custom(
"model-id-from-create-custom",
api_url="https://my-new-endpoint.example.com/v1",
)
```

### `delete_custom(model_id, *, timeout=None)`

Disables a custom model and removes it from `Project.Models`. The backend tears down the model's S3 yaml artifacts and AWS secret, and marks the record as disabled (preserving any evaluation references). Public models cannot be deleted via the SDK.

#### Parameters

| Parameter | Type | Required | Description |
| ---------- | -------------------------------- | -------- | --------------------------------- |
| `model_id` | `str` | Yes | ID of the custom model to delete |
| `timeout` | `float \| httpx.Timeout \| None` | No | Override request timeout |

#### Returns

Returns `bool` — `True` on success, `False` otherwise.

#### Example

```python
client = Stratix()
client.models.delete_custom("model-id-from-create-custom")
```

## Benchmarks

### `get(type=None, name=None, key=None, categories=None, languages=None, timeout=None)`
Expand Down
63 changes: 63 additions & 0 deletions docs/examples/models-and-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,69 @@ def main():
print(f" - {m.name} (id={m.id}, key={m.key})")


if __name__ == "__main__":
main()
```

## Repointing a Custom Model's `api_url`

Use this when your model's endpoint URL changes — for example, when serving a vLLM instance behind a cloudflared tunnel that rotates its hostname between sessions.

```python
from layerlens import Stratix


def main():
client = Stratix()

result = client.models.create_custom(
name="My Tunnel-backed Model",
key="my-org/tunnel-model-v1",
description="vLLM served behind a cloudflared tunnel",
api_url="https://tunnel-1.example.com/v1",
api_key="my-provider-api-key",
max_tokens=4096,
)
assert result is not None

# Later, when the tunnel URL changes:
client.models.update_custom(
result.model_id,
api_url="https://tunnel-2.example.com/v1",
)

# Run evaluations as usual — the model now points at the new endpoint.


if __name__ == "__main__":
main()
```

## Replacing a Custom Model

`delete_custom` releases the model's name so it can be reused. This is useful for replacing a misconfigured model without picking a new name.

```python
from layerlens import Stratix


def main():
client = Stratix()

# Tear down the old version
client.models.delete_custom("old-model-id")

# Recreate with the same name (now free)
client.models.create_custom(
name="My Custom Model",
key="my-org/custom-model-v2",
description="Replacement after schema migration",
api_url="https://my-endpoint.example.com/v1",
api_key="my-provider-api-key",
max_tokens=4096,
)


if __name__ == "__main__":
main()
```
Expand Down
Loading
Loading