Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 42 additions & 11 deletions eval/harbor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ PYTHONPATH=$PWD/eval/harbor harbor run \
NOTE: hub datasets namespace task names — filters must match the full
name: `-i 'terminal-bench/fix-git'` (or use a glob: `-i '*fix-git*'`).

## Evaluate with claude-opus-4-8 on a Claude subscription
## Evaluate with claude-opus-5 on a Claude subscription

Uses your Claude Pro/Max subscription (OAuth) instead of an API key.
One-time prerequisite on the host: `clawcodex login` (writes
Expand All @@ -70,9 +70,10 @@ covers only the main loop).
PYTHONPATH=$PWD/eval/harbor harbor run \
--dataset terminal-bench/terminal-bench-2-1 \
--agent clawcodex_agent:Clawcodex \
--model anthropic/claude-opus-4-8 \
--model anthropic/claude-opus-5 \
--ak subscription=true \
--ak effort=high \
--ak source=git+https://github.com/agentforce314/clawcodex@main \
--jobs-dir eval/harbor/jobs \
--n-concurrent 2
```
Expand All @@ -82,10 +83,21 @@ Notes:
resources. Keep Harbor's default timeout policy for leaderboard-comparable
runs; use timeout multipliers only for explicitly labeled diagnostics.
- `effort=high` maps to `clawcodex --effort high` →
`output_config.effort` on effort-capable models (Opus 4.6/4.8,
`output_config.effort` on effort-capable models (Opus 5, Opus 4.6/4.8,
Sonnet 4.6, Fable 5). Requires clawcodex > 1.2.1 in the container —
until the next PyPI release, add
`--ak source=git+https://github.com/agentforce314/clawcodex@main`.
until the next PyPI release, keep the
`--ak source=git+https://github.com/agentforce314/clawcodex@main` above.
- **claude-opus-5 needs a clawcodex that registers it** (the model tables
in `src/models/configs.py`, `src/services/pricing.py`, and the three
gates in `src/query/query.py`). On a build without it, opus-5 falls off
the adaptive-thinking allowlist and the request carries
`thinking={"type": "enabled", "budget_tokens": …}` — removed on Opus 5,
so **every request 400s** — while `--effort` is silently dropped and the
context window is assumed to be 200K instead of 1M (early compaction,
the handicap #730 removed for opus-4-8). PyPI 1.2.1 predates this, so
`source=` is required; verify the branch/SHA you point it at actually
carries the registration before starting a full run, and prefer a
commit SHA over `@main` for anything you plan to compare later.
- Subscription rate limits are shared with your interactive Claude
usage — keep `--n-concurrent` low (2-4) and consider
`--max-retries 2 --retry-include ApiRateLimitError`.
Expand All @@ -105,7 +117,7 @@ token), so multi-hour jobs work without a manually exported token:
PYTHONPATH=$PWD/eval/harbor harbor run \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude_code_subscription:ClaudeCodeSubscription \
--model anthropic/claude-opus-4-8 \
--model anthropic/claude-opus-5 \
--ak reasoning_effort=high \
--jobs-dir eval/harbor/jobs \
--n-concurrent 2
Expand All @@ -114,8 +126,16 @@ PYTHONPATH=$PWD/eval/harbor harbor run \
Notes:
- Effort uses the parent agent's kwarg name: `--ak reasoning_effort=`
(low|medium|high|xhigh|max).
- The model is forwarded as the `ANTHROPIC_MODEL` env var with the
provider prefix stripped (`harbor/agents/installed/claude_code.py`
:1372-1386 — so don't expect a `--model` on the exec line when
debugging), and effort as the `--effort` CLI flag. Neither is gated on
a model table anywhere in the adapter, and the bootstrap installs the
latest official CLI in every container, so moving this arm to a new
model is the model string alone — no adapter or clawcodex change.
- Pin the CLI to a leaderboard row's version with `--ak version=2.1.205`
(default: latest).
(default: latest). A pin predating a model's support will fail on that
model, so re-check the pin when changing `--model`.
- `CLAUDE_FORCE_OAUTH` is set by the wrapper, so a host
`ANTHROPIC_API_KEY` can never silently take over and bill the API.
- `--ak subprocess_env_scrub=true` enables the CLI's subprocess-env scrub
Expand Down Expand Up @@ -158,9 +178,19 @@ Notes:
- Subscription auth is env-only (`CLAUDE_CODE_OAUTH_TOKEN` access token,
no refresh token, no credential file in the container); the same
30-min-runway host refresh as the clawcodex adapter applies.
- This openclaude snapshot's effort ladder is low|medium|high|max (no
xhigh), and its model metadata predates claude-opus-4-8, so it assumes
a conservative 128k context for compaction purposes.
- This openclaude snapshot (0.24.0) knows claude-opus-4-8 but **not
claude-opus-5**, and on opus-5 it does not merely degrade — it breaks.
`modelSupportsAdaptiveThinking` (`typescript/src/utils/thinking.ts:159`)
allowlists only opus-4-8/4-7/4-6 and sonnet-4-6, then excludes anything
else matching `opus`, so opus-5 falls to the `budget_tokens` branch at
`typescript/src/services/api/claude.ts:1731` — and `budget_tokens` is
removed on Opus 5, i.e. **HTTP 400 on every request** (the same failure
the clawcodex adapter's `source=` note describes). Model metadata is
missing too, so the context window falls back to the 200K default
(`typescript/src/utils/context.ts:17`). Keep this arm on
`anthropic/claude-opus-4-8` until `typescript/` is refreshed to a
snapshot carrying opus-5.
- Its effort ladder is low|medium|high|xhigh|max as of 0.24.0.

## Evaluate ALL terminal-bench 2.0 tasks

Expand Down Expand Up @@ -193,7 +223,7 @@ aggregate accuracy; each trial dir has the agent's stream-json log under
# Agent kwargs
--ak max_turns=100 # clawcodex --max-turns (default 300)
--ak effort=high # clawcodex --effort (low|medium|high|xhigh|max)
# xhigh is model-dependent (opus-4-8 yes,
# xhigh is model-dependent (opus-5/opus-4-8 yes,
# sonnet-4-6/opus-4-6 no → degraded to high)
--ak version=1.2.1 # pin the clawcodex-cli PyPI version
--ak source=git+https://github.com/agentforce314/clawcodex@main
Expand All @@ -205,6 +235,7 @@ aggregate accuracy; each trial dir has the agent's stream-json log under

# Other models/providers (Harbor convention: provider/model)
--model deepseek/deepseek-v4-pro
--model anthropic/claude-opus-4-8 # the previous tb2.1 baseline
--model anthropic/claude-opus-4-5 # needs ANTHROPIC_API_KEY
```

Expand Down
9 changes: 6 additions & 3 deletions eval/harbor/clawcodex_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,12 @@
own default of 50 is too low for terminal-bench tasks). The
``CLAWCODEX_MAX_TURNS`` host env var works as a fallback.
* ``effort`` — clawcodex ``--effort`` (low|medium|high|xhigh|max) for
models that support ``output_config.effort`` (Opus 4.6/4.8, Sonnet 4.6,
Fable 5). ``xhigh`` is model-dependent (opus-4-8 yes; sonnet-4-6/
opus-4-6 no) — clawcodex degrades it to ``high`` where rejected.
models that support ``output_config.effort`` (Opus 5, Opus 4.6/4.8,
Sonnet 4.6, Fable 5). ``xhigh`` is model-dependent (opus-5/opus-4-8 yes;
sonnet-4-6/opus-4-6 no) — clawcodex degrades it to ``high`` where
rejected. A model missing from clawcodex's effort allowlist drops the
flag SILENTLY, so a new model needs a clawcodex build that registers it
(see ``source`` below) before an effort number means anything.
``CLAWCODEX_EFFORT`` host env var works as a fallback.
* ``version`` — pin a ``clawcodex-cli`` PyPI version (default: latest).
* ``source`` — full pip-installable spec overriding the PyPI package, e.g.
Expand Down
24 changes: 15 additions & 9 deletions src/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,9 @@ def main():
return launch_ink_tui(
provider=args.provider,
model=args.model,
# --effort applies interactively too: forwarded to the agent-server
# child, which seeds the session's /effort level from it.
effort=args.effort,
permission_mode=args._resolved_permission_mode,
is_bypass_available=args._resolved_is_bypass_available,
workspace=(worktree_session.worktree_path if worktree_session else None),
Expand Down Expand Up @@ -474,15 +477,18 @@ def _build_parser() -> argparse.ArgumentParser:
default=None,
help='Override the provider (anthropic, openai, zai, minimax, openrouter, deepseek, meta)',
)
# Mirrors TS ``--effort <level>`` (main.tsx:995). TS registers it
# session-wide; the interactive TUI equivalent here is the persisted
# ``/effort`` setting, so the flag is exposed on the print path where
# no dialog exists. None = auto (settings.effort, else the parameter is
# omitted and the API applies its model default). xhigh acceptance is
# model-dependent — resolve_thinking_effort clamps it to high on models
# that reject it. KNOWN LIMIT: the flag governs the MAIN loop only;
# subagents (Agent tool) resolve from settings.effort — persist
# ``/effort`` (or seed settings) for session-wide coverage.
# Mirrors TS ``--effort <level>`` (main.tsx:995), and like TS it is
# session-wide: the print path reads it via HeadlessOptions, and the
# interactive path forwards it to the agent-server child, which seeds
# the session's ``/effort`` level from it (a later ``/effort`` in the
# TUI overrides; ``/effort auto`` clears back to settings.effort).
# Registered under the noninteractive group for backwards-compatible
# ``--help`` grouping only. None = auto (settings.effort, else the
# parameter is omitted and the API applies its model default). xhigh
# acceptance is model-dependent — resolve_thinking_effort clamps it to
# high on models that reject it. KNOWN LIMIT: the flag governs the MAIN
# loop only; subagents (Agent tool) resolve from settings.effort —
# persist ``/effort`` (or seed settings) for session-wide coverage.
noninteractive.add_argument(
'--effort',
choices=('low', 'medium', 'high', 'xhigh', 'max'),
Expand Down
3 changes: 2 additions & 1 deletion src/command_system/effort_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ def _invalid_msg(raw: str) -> str:

# TS effort.tsx:179 help text. xhigh/max availability is model-dependent;
# resolve_thinking_effort clamps xhigh to high on models that reject it
# (wire-probed 2026-07-18: opus-4-8 accepts xhigh; sonnet-4-6/opus-4-6 400).
# (wire-probed 2026-07-18: opus-4-8 accepts xhigh; sonnet-4-6/opus-4-6 400.
# 2026-07-25: opus-5 accepts xhigh and max. fable-5 unprobed.)
_USAGE = (
"Usage: /effort [low|medium|high|xhigh|max|auto]\n\n"
"Effort levels:\n"
Expand Down
9 changes: 9 additions & 0 deletions src/entrypoints/agent_server_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,14 @@ def run_agent_server_subcommand(argv: list[str]) -> int:
help="Optional bearer token required on POST /sessions.")
parser.add_argument("--provider", default=None, help="Provider name override.")
parser.add_argument("--model", default=None, help="Model override.")
parser.add_argument(
"--effort", default=None,
choices=("low", "medium", "high", "xhigh", "max"),
help="Reasoning effort for the session. Seeds /effort; routed to "
"output_config.effort on Anthropic and reasoning_effort on "
"OpenAI-compatible providers. (_build_runtime re-validates, "
"because --stdio callers bypass this parser.)",
)
parser.add_argument(
"--fallback-model", default=None, dest="fallback_model",
help="Model to switch to after repeated overloaded (529) errors "
Expand Down Expand Up @@ -233,6 +241,7 @@ def run_agent_server_subcommand(argv: list[str]) -> int:
agent_config = AgentServerConfig(
provider_name=args.provider,
model=args.model,
effort=args.effort,
fallback_model=args.fallback_model,
permission_mode=args.permission_mode,
is_bypass_available=is_bypass_available,
Expand Down
12 changes: 12 additions & 0 deletions src/entrypoints/tui_launcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,12 @@ def run_tui_launcher(argv: list[str]) -> int:
)
parser.add_argument("--provider", default=None)
parser.add_argument("--model", default=None)
# choices= so a bad level is a clean argparse error rather than a value
# that silently resolves to settings.effort downstream (the server also
# validates the seed, since it can be driven without a parser).
parser.add_argument("--effort", default=None,
choices=("low", "medium", "high", "xhigh", "max"),
help="Reasoning effort — seeds the session's /effort level")
parser.add_argument("--permission-mode", default="default", dest="permission_mode")
parser.add_argument(
"--dangerously-skip-permissions", action="store_true",
Expand Down Expand Up @@ -151,6 +157,7 @@ def run_tui_launcher(argv: list[str]) -> int:
return launch_ink_tui(
provider=args.provider,
model=args.model,
effort=args.effort,
permission_mode=mode,
is_bypass_available=is_bypass_available,
workspace=args.workspace,
Expand All @@ -163,6 +170,7 @@ def launch_ink_tui(
*,
provider: str | None = None,
model: str | None = None,
effort: str | None = None,
permission_mode: str = "default",
is_bypass_available: bool = False,
workspace: str | None = None,
Expand All @@ -186,6 +194,7 @@ def launch_ink_tui(
args = SimpleNamespace(
provider=provider,
model=model,
effort=effort,
permission_mode=permission_mode,
is_bypass_available=is_bypass_available,
workspace=workspace,
Expand Down Expand Up @@ -256,6 +265,8 @@ def _agent_server_cmd(args) -> list[str]:
cmd += ["--provider", args.provider]
if args.model:
cmd += ["--model", args.model]
if getattr(args, "effort", None):
cmd += ["--effort", args.effort]
return cmd


Expand Down Expand Up @@ -307,6 +318,7 @@ def _print_connect(args) -> int:
if getattr(args, "is_bypass_available", False) else []),
*(["--provider", args.provider] if args.provider else []),
*(["--model", args.model] if args.model else []),
*(["--effort", args.effort] if getattr(args, "effort", None) else []),
"--workspace", workspace,
])

Expand Down
34 changes: 34 additions & 0 deletions src/models/configs.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,40 @@ class ModelConfig:
cost_cache_create_per_mtok=12.50,
cost_cache_read_per_mtok=1.00,
),
# Claude Opus 5 — the current Opus tier, at Opus 4.8's price (5/25 per
# MTok) and shape: 1M context (default AND maximum), 128K true output
# cap, same 32_000 first-attempt wire ``max_tokens`` convention as the
# rows above. Registered LAST among the opus rows because its prefix
# base is the family-wide ``claude-opus`` (``key.rsplit("-", 1)[0]``).
# Every 4.x id still matches the earlier ``claude-opus-4`` base first,
# so their 200K/legacy resolution is unchanged, but two other strings
# now land here: an unregistered future ``claude-opus-<n>``, AND the
# literal ``claude-opus`` — a live MODEL_ALIASES key (aliases.py:15)
# that resolves to claude-opus-4-20250514, so ``display_name()`` on the
# UNRESOLVED alias now reads "Claude Opus 5". Callers that canonicalize
# first (the normal path) are unaffected; test_model_system pins both.
# This inverts the "under-estimate is the safe direction" note above
# for unknown opus ids, and over-estimating is the worse failure —
# auto-compact never fires and the request eventually exceeds the real
# window. Accepted because every Opus since 4.6 ships 1M, and it
# matches ``claude-fable-5``, whose base is likewise family-wide.
# (``src/services/pricing.py`` deliberately went the other way: its
# bare ``claude-opus-4`` prefix was dropped so unknown ids price as
# None rather than wrong. A wrong cost is silent; a wrong window is
# not — but the directions genuinely differ, so don't "harmonize"
# them without re-reading both rationales.)
"claude-opus-5": ModelConfig(
model_id="claude-opus-5",
display_name="Claude Opus 5",
context_window=1_000_000,
max_output_tokens=32_000,
supports_thinking=True,
supports_computer_use=True,
cost_input_per_mtok=5.0,
cost_output_per_mtok=25.0,
cost_cache_create_per_mtok=6.25,
cost_cache_read_per_mtok=0.50,
),

# Claude 3.7 series
"claude-3-7-sonnet-20250219": ModelConfig(
Expand Down
33 changes: 32 additions & 1 deletion src/providers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ class ProviderInfo(TypedDict):
"available_models": [
# Frontier (above Opus tier)
"claude-fable-5",
# Claude 4 series (latest)
# Claude 5 series (Opus tier; sonnet-5 not registered yet —
# it needs the same model-table entry opus-5 got)
"claude-opus-5",
# Claude 4 series
"claude-sonnet-4-6",
"claude-sonnet-4-5",
"claude-sonnet-4-5-20250929",
Expand Down Expand Up @@ -294,6 +297,33 @@ def provider_env_vars(provider_name: str) -> tuple[str, ...]:
return spec.env_vars if spec else ()


def is_anthropic_wire(provider: Any) -> bool:
"""Whether ``provider`` speaks the ANTHROPIC request shape.

The single source of truth for a decision several call sites have to
agree on, because the two families take the same concepts as different
parameters and sending the wrong one is a hard 400:

* reasoning effort — ``output_config.effort`` here vs a top-level
``reasoning_effort`` body field on OpenAI-compatible providers,
* the system prompt — a ``system`` kwarg here vs a prepended
``{"role": "system"}`` message there.

Minimax is included because it speaks the Anthropic shape. A custom
``base_url`` / third-party gateway is a FLAG on AnthropicProvider
(``has_custom_endpoint``), never a subclass, so an isinstance test
cannot be fooled by one. Takes a provider INSTANCE, not a name — the
class is what carries the wire contract.

Imports are local: this module is imported at startup by the fast-path
CLI, and the provider modules pull in their SDKs.
"""
from .anthropic_provider import AnthropicProvider
from .minimax_provider import MinimaxProvider

return isinstance(provider, (AnthropicProvider, MinimaxProvider))


def provider_requires_api_key(provider_name: str) -> bool:
"""Whether a provider needs an API key.

Expand Down Expand Up @@ -377,6 +407,7 @@ def resolve_api_key(
"ChatResponse",
"get_provider_class",
"get_provider_info",
"is_anthropic_wire",
"canonical_provider_name",
"provider_env_vars",
"provider_has_credentials",
Expand Down
5 changes: 4 additions & 1 deletion src/providers/anthropic_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -921,7 +921,10 @@ def get_available_models(self) -> list[str]:
return [
# Frontier (above Opus tier)
"claude-fable-5",
# Claude 4 series (latest)
# Claude 5 series (Opus tier; sonnet-5 not registered yet —
# it needs the same model-table entry opus-5 got)
"claude-opus-5",
# Claude 4 series
"claude-sonnet-4-6",
"claude-sonnet-4-5",
"claude-sonnet-4-5-20250929",
Expand Down
Loading
Loading