Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ All notable changes to SkillOpt are documented here. This project adheres to
## [Unreleased]

### Added
- **Cursor Agent research target harness** (`cursor_exec`) for running
supported benchmark rollouts through an installed, authenticated
`cursor-agent`, with sandboxed workspaces, structured trace capture, and
target-only optimizer separation.
- **Handoff backend** (`--backend handoff`) for SkillOpt-Sleep — runs the
sleep cycle with no model subprocess or API key: the engine writes each
pending model call to `PROMPTS.md`/`pending.json` (exit code 3) and the
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,13 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7

A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
`qwen_chat`, `minimax_chat`, `openai_compatible`, `codex_exec`,
`claude_code_exec`). If a provider implements the OpenAI Chat Completions
`claude_code_exec`, `cursor_exec`). If a provider implements the OpenAI Chat Completions
protocol, try the built-in `openai_compatible` backend before adding code. See
[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
contract; in short you add a `skillopt/model/<name>_backend.py` module,
register it in `skillopt/model/common.py` + `backend_config.py`, and wire
it through the router in `skillopt/model/__init__.py`. `qwen_backend.py`
and `minimax_backend.py` are good templates.
contract. Chat backends add a `skillopt/model/<name>_backend.py` module;
target-only exec backends use the shared harness in `codex_harness.py`.
Both register through `common.py`, `backend_config.py`, and
`skillopt/model/__init__.py`.

### Adding a new benchmark

Expand Down
2 changes: 2 additions & 0 deletions configs/_base_/default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ model:
claude_code_exec_use_sdk: auto
claude_code_exec_effort: medium
claude_code_exec_max_thinking_tokens: 16384
cursor_exec_path: "" # blank uses CURSOR_EXEC_PATH or cursor-agent
cursor_exec_sandbox: "" # blank uses CURSOR_EXEC_SANDBOX or enabled
codex_trace_to_optimizer: true
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
azure_openai_api_version: "2024-12-01-preview"
Expand Down
12 changes: 12 additions & 0 deletions docs/guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ model:
| `minimax_chat` | ✓ | ✓ | MiniMax API |
| `codex_exec` | — | ✓ | Codex CLI execution harness |
| `claude_code_exec` | — | ✓ | Claude Code CLI execution harness |
| `cursor_exec` | — | ✓ | Cursor Agent CLI execution harness |

The current MiniMax adapter has one shared deployment. Set
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
Expand Down Expand Up @@ -183,6 +184,9 @@ Model credentials are loaded from environment variables:
| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence |
| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` |
| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client |
| `CURSOR_EXEC_PATH` | `cursor_exec` | Optional path to `cursor-agent`; defaults to `cursor-agent` |
| `CURSOR_EXEC_SANDBOX` | `cursor_exec` | Cursor sandbox mode: `enabled` (default) or `disabled` |
| `CURSOR_API_KEY` | `cursor_exec` | Optional authentication method understood directly by Cursor Agent |
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint |
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL |
Expand All @@ -197,6 +201,14 @@ and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way
the CLI may authenticate, but SkillOpt does not call the Anthropic API
directly through this backend.

`cursor_exec` is a target-only benchmark harness. Install and authenticate
Cursor Agent first, then select it with `model.target_backend=cursor_exec`.
Read-only rollouts use Ask mode; artifact-producing rollouts add Cursor's
headless `--force` flag inside the benchmark workspace. SkillOpt enables the
Cursor sandbox by default and rejects file-edit rollouts if it is disabled;
read-only Ask-mode rollouts may explicitly disable it. SkillOpt does not approve
MCP servers automatically.

### Three OpenAI-compatible paths

- Research, generic provider: select `openai_compatible` and use
Expand Down
6 changes: 3 additions & 3 deletions docs/guide/new-backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,9 @@ Provider-specific configuration helpers and `count_tokens()` are optional, but
their state must be safe to update while calls may run concurrently. Keep
credentials out of logs and persisted artifacts.

Exec-style targets do not implement this chat contract. They are target-only
and are integrated through `codex_harness.py` plus environment-specific rollout
code.
Exec-style targets such as `claude_code_exec` and `cursor_exec` do not
implement this chat contract. They are target-only and are integrated through
`codex_harness.py` plus environment-specific rollout code.

## Step 2: register and route the backend

Expand Down
1 change: 1 addition & 0 deletions docs/guideline.html
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,7 @@ <h2>Research model backends</h2>
<tr><td><code>minimax_chat</code></td><td>Yes</td><td>Yes</td><td>MiniMax chat endpoint.</td></tr>
<tr><td><code>codex_exec</code></td><td>Yes</td><td>Supported adapters only</td><td>Executes Codex for optimizer calls and as a target agent where supported.</td></tr>
<tr><td><code>claude_code_exec</code></td><td>No</td><td>Supported adapters only</td><td>Executes Claude Code as a target agent.</td></tr>
<tr><td><code>cursor_exec</code></td><td>No</td><td>Supported adapters only</td><td>Executes Cursor Agent as a sandboxed target agent where supported.</td></tr>
</tbody>
</table>
</div>
Expand Down
1 change: 1 addition & 0 deletions docs/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ not via a base class subclass. Supported values (as of this writing):
| `openai_compatible` | ✓ | ✓ |
| `codex_exec` | ✓ | ✓ |
| `claude_code_exec` | — | ✓ |
| `cursor_exec` | — | ✓ |

See `skillopt/model/backend_config.py` for the live whitelist and
[`docs/reference/config.md`](./config.md) for the per-backend
Expand Down
27 changes: 24 additions & 3 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

> **Version note.** This reference tracks `main`. PyPI 0.2.0 does not yet
> include the generic research `openai_compatible` backend, Sleep handoff,
> Sleep support for non-Azure OpenAI-compatible endpoints, or the Sleep
> `--preferences` flag; use a source install from `main` for those features
> until the next release.
> Sleep support for non-Azure OpenAI-compatible endpoints, the Sleep
> `--preferences` flag, or the research `cursor_exec` target harness; use a
> source install from `main` for those features until the next release.

## Training

Expand Down Expand Up @@ -91,6 +91,27 @@ python scripts/train.py \
model.target=deepseek-chat
```

To benchmark an installed, authenticated Cursor Agent through an environment
that supports exec targets:

```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill skills/my_skill.md \
--cfg-options \
model.optimizer_backend=openai_chat \
model.target_backend=cursor_exec \
model.target=composer-2.5
```

`cursor_exec` runs the target only; the optimizer remains separately
configured. Read-only rollouts use Cursor Ask mode. Rollouts that request file
edits use `--force` inside the benchmark workspace, with Cursor sandboxing
enabled. The harness refuses file-edit rollouts when the Cursor sandbox is
disabled. Read-only Ask-mode rollouts may explicitly disable it. Override the
executable or sandbox through `model.cursor_exec_path` and
`model.cursor_exec_sandbox`.

## SkillOpt-Sleep

```bash
Expand Down
3 changes: 3 additions & 0 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ selecting the generic OpenAI-compatible backend.
| `minimax_chat` | ✓ | ✓ |
| `codex_exec` | ✓ | ✓ |
| `claude_code_exec` | — | ✓ |
| `cursor_exec` | — | ✓ |

MiniMax currently has one shared deployment. `model.minimax_model` is applied
when MiniMax is the target; mixed-backend runs cannot independently choose a
Expand Down Expand Up @@ -64,6 +65,8 @@ defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
| `model.minimax_*` | MiniMax `base_url`, `api_key`, shared `minimax_model`, `temperature`, `max_tokens`, and `enable_thinking`; `minimax_model` applies when MiniMax is the target |
| `model.codex_exec_*` | Codex path, sandbox, profile, SDK mode, reasoning, network/search, and approval policy |
| `model.claude_code_exec_*` | Claude path, profile, SDK mode, effort, and thinking-token cap |
| `model.cursor_exec_path` | Cursor Agent executable path; default `cursor-agent` |
| `model.cursor_exec_sandbox` | Cursor sandbox mode: `enabled` (default) or `disabled`; file-edit rollouts require `enabled` |

## Training (`train`)

Expand Down
22 changes: 21 additions & 1 deletion scripts/eval_only.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
configure_azure_openai,
configure_claude_code_exec,
configure_codex_exec,
configure_cursor_exec,
configure_qwen_chat,
configure_minimax_chat,
set_reasoning_effort,
Expand Down Expand Up @@ -139,7 +140,7 @@ def parse_args() -> argparse.Namespace:
# Legacy flat overrides
p.add_argument("--env", type=str)
p.add_argument("--backend", type=str,
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "minimax", "minimax_chat"])
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "cursor", "cursor_exec", "minimax", "minimax_chat"])
p.add_argument("--optimizer_model", type=str)
p.add_argument("--target_model", type=str)
p.add_argument("--optimizer_backend", type=str)
Expand Down Expand Up @@ -181,6 +182,8 @@ def parse_args() -> argparse.Namespace:
p.add_argument("--claude_code_exec_use_sdk", type=str)
p.add_argument("--claude_code_exec_effort", type=str)
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
p.add_argument("--cursor_exec_path", type=str)
p.add_argument("--cursor_exec_sandbox", type=str)
p.add_argument("--minimax_base_url", type=str)
p.add_argument("--minimax_api_key", type=str)
p.add_argument("--minimax_model", type=str)
Expand Down Expand Up @@ -262,6 +265,8 @@ def main() -> None:
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
"claude_code_exec_effort": "model.claude_code_exec_effort",
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
"cursor_exec_path": "model.cursor_exec_path",
"cursor_exec_sandbox": "model.cursor_exec_sandbox",
"minimax_base_url": "model.minimax_base_url",
"minimax_api_key": "model.minimax_api_key",
"minimax_model": "model.minimax_model",
Expand Down Expand Up @@ -327,6 +332,11 @@ def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
elif backend == "claude_code_exec":
cfg.setdefault("optimizer_backend", "openai_chat")
cfg.setdefault("target_backend", "claude_code_exec")
elif backend == "cursor_exec":
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
cfg["optimizer_backend"] = "openai_chat"
if not _has_model_override("model.target_backend", "target_backend"):
cfg["target_backend"] = "cursor_exec"
elif backend in {"minimax", "minimax_chat"}:
cfg.setdefault("optimizer_backend", "openai_chat")
cfg.setdefault("target_backend", "minimax_chat")
Expand Down Expand Up @@ -355,6 +365,12 @@ def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
and not _has_model_override("model.target", "target_model")
):
cfg["target_model"] = default_model_for_backend("claude_chat")
if cfg.get("target_backend") == "cursor_exec":
if (
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.target", "target_model")
):
cfg["target_model"] = default_model_for_backend("cursor_exec")
if cfg.get("target_backend") == "minimax_chat":
if (
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
Expand Down Expand Up @@ -429,6 +445,10 @@ def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
)
configure_cursor_exec(
path=cfg.get("cursor_exec_path") or None,
sandbox=cfg.get("cursor_exec_sandbox") or None,
)
configure_qwen_chat(
base_url=cfg.get("qwen_chat_base_url") or None,
api_key=cfg.get("qwen_chat_api_key") or None,
Expand Down
17 changes: 16 additions & 1 deletion scripts/train.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ def parse_args() -> argparse.Namespace:
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
p.add_argument("--env", type=str)
p.add_argument("--backend", type=str,
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "cursor", "cursor_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
p.add_argument("--optimizer_model", type=str)
p.add_argument("--target_model", type=str)
p.add_argument("--optimizer_backend", type=str)
Expand Down Expand Up @@ -205,6 +205,8 @@ def parse_args() -> argparse.Namespace:
p.add_argument("--claude_code_exec_use_sdk", type=str)
p.add_argument("--claude_code_exec_effort", type=str)
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
p.add_argument("--cursor_exec_path", type=str)
p.add_argument("--cursor_exec_sandbox", type=str)
p.add_argument("--codex_trace_to_optimizer", type=_BOOL)
p.add_argument("--skill_init", type=str)
p.add_argument("--num_epochs", type=int)
Expand Down Expand Up @@ -343,6 +345,8 @@ def parse_args() -> argparse.Namespace:
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
"claude_code_exec_effort": "model.claude_code_exec_effort",
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
"cursor_exec_path": "model.cursor_exec_path",
"cursor_exec_sandbox": "model.cursor_exec_sandbox",
"codex_trace_to_optimizer": "model.codex_trace_to_optimizer",
"num_epochs": "train.num_epochs",
"train_size": "train.train_size",
Expand Down Expand Up @@ -445,6 +449,11 @@ def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
elif backend == "claude_code_exec":
flat.setdefault("optimizer_backend", "openai_chat")
flat.setdefault("target_backend", "claude_code_exec")
elif backend == "cursor_exec":
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
flat["optimizer_backend"] = "openai_chat"
if not _has_model_override("model.target_backend", "target_backend"):
flat["target_backend"] = "cursor_exec"
elif backend in {"qwen", "qwen_chat"}:
flat.setdefault("optimizer_backend", "openai_chat")
flat.setdefault("target_backend", "qwen_chat")
Expand Down Expand Up @@ -482,6 +491,12 @@ def _has_model_override(dotted_key: str, legacy_key: str) -> bool:
and not _has_model_override("model.target", "target_model")
):
flat["target_model"] = default_model_for_backend("claude_chat")
if flat.get("target_backend") == "cursor_exec":
if (
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
and not _has_model_override("model.target", "target_model")
):
flat["target_model"] = default_model_for_backend("cursor_exec")
if flat.get("target_backend") == "qwen_chat":
if (
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
Expand Down
2 changes: 2 additions & 0 deletions skillopt/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@
"model.claude_code_exec_use_sdk": "claude_code_exec_use_sdk",
"model.claude_code_exec_effort": "claude_code_exec_effort",
"model.claude_code_exec_max_thinking_tokens": "claude_code_exec_max_thinking_tokens",
"model.cursor_exec_path": "cursor_exec_path",
"model.cursor_exec_sandbox": "cursor_exec_sandbox",
"model.codex_trace_to_optimizer": "codex_trace_to_optimizer",
"model.azure_endpoint": "azure_endpoint",
"model.azure_api_version": "azure_api_version",
Expand Down
8 changes: 8 additions & 0 deletions skillopt/engine/trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
configure_azure_openai,
configure_claude_code_exec,
configure_codex_exec,
configure_cursor_exec,
configure_minimax_chat,
configure_qwen_chat,
get_token_summary,
Expand Down Expand Up @@ -681,6 +682,9 @@ def _build_eval_env(split: str, env_num: int, seed: int):
elif backend == "claude_code_exec":
optimizer_backend = optimizer_backend or "openai_chat"
target_backend = target_backend or "claude_code_exec"
elif backend in {"cursor", "cursor_exec"}:
optimizer_backend = optimizer_backend or "openai_chat"
target_backend = target_backend or "cursor_exec"
elif backend in {"qwen", "qwen_chat"}:
optimizer_backend = optimizer_backend or "openai_chat"
target_backend = target_backend or "qwen_chat"
Expand Down Expand Up @@ -711,6 +715,10 @@ def _build_eval_env(split: str, env_num: int, seed: int):
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
)
configure_cursor_exec(
path=cfg.get("cursor_exec_path") or None,
sandbox=cfg.get("cursor_exec_sandbox") or None,
)
configure_qwen_chat(
base_url=cfg.get("qwen_chat_base_url") or None,
api_key=cfg.get("qwen_chat_api_key") or None,
Expand Down
2 changes: 1 addition & 1 deletion skillopt/envs/_template/config_template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,5 +51,5 @@ evaluation:
# Override only what differs from the inherited defaults.
model:
optimizer_backend: openai_chat # openai_chat | claude_chat | qwen_chat | minimax_chat | codex_exec
target_backend: openai_chat # chat backends plus codex_exec / claude_code_exec
target_backend: openai_chat # chat backends plus codex_exec / claude_code_exec / cursor_exec
reasoning_effort: medium
8 changes: 8 additions & 0 deletions skillopt/model/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@
from skillopt.model.backend_config import ( # noqa: F401
configure_claude_code_exec,
configure_codex_exec,
configure_cursor_exec,
get_claude_code_exec_config,
get_codex_exec_config,
get_cursor_exec_config,
get_optimizer_backend,
get_target_backend,
is_optimizer_chat_backend,
Expand Down Expand Up @@ -53,6 +55,10 @@ def set_backend(name: str | None) -> str:
set_optimizer_backend("openai_chat")
set_target_backend(normalized)
return normalized
if normalized in {"cursor", "cursor_agent", "cursor_exec"}:
set_optimizer_backend("openai_chat")
set_target_backend("cursor_exec")
return "cursor_exec"
if normalized in {"qwen", "qwen_chat"}:
set_optimizer_backend("openai_chat")
set_target_backend("qwen_chat")
Expand Down Expand Up @@ -84,6 +90,8 @@ def get_backend_name() -> str:
return "qwen_chat"
if optimizer == "openai_chat" and target == "minimax_chat":
return "minimax_chat"
if optimizer == "openai_chat" and target == "cursor_exec":
return "cursor_exec"
if optimizer == "openai_compatible" and target == "openai_compatible":
return "openai_compatible"
return f"{optimizer}+{target}"
Expand Down
Loading