diff --git a/.gitignore b/.gitignore index 222caa50b..c643ab30e 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,8 @@ MANIFEST # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec +.cursor +.qoder # Installer logs pip-log.txt diff --git a/README.md b/README.md index 49aa7deb5..2c02a91a2 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,7 @@ sh INSTALL_MEGATRON.sh | Server startup scripts | transformers/megatron | [Script](cookbook/client/server) | ## Changelog +- 🎉2026-08-04 Sandboxed multi-turn RL is now supported: run model-generated code in isolated [AgentENV](https://github.com/kvcache-ai/AgentENV) Firecracker microVMs, or in an OpenEnv server, with the same `train.py`. See the [cookbook](cookbook/rl/envs) and the [deployment guide](docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md). - 🎉2026-05-20 Support DeepSeek-V4-Flash and DeepSeek-V4-Pro models. - 🎉2026-05-20 Multi-turn rollout and tool calling in RL are now supported. The Cookbook is currently being written. You can use `from twinkle_agentic.rollout import MultiTurnRollout/APIMultiTurnRollout` directly for multi-turn rollout. - 🎉2026-05-20 IM message alerting on training job failure is now supported. Usage: `import twinkle; twinkle.initialize(..., notifier=DingNotifier(...))`. diff --git a/README_ZH.md b/README_ZH.md index dcd95e391..ed86be1ca 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -103,6 +103,7 @@ sh INSTALL_MEGATRON.sh Twinkle✨支持相同的算法接口运行在单GPU、torchrun多机、Ray、Client等各场景下。其算法过程是外露的，非常便于修改和调试。完整的框架介绍请查看[快速开始](https://modelscope.github.io/twinkle-web/zh/docs/usage-guide/quick-start/) ## 更新日志 +- 🎉2026-08-04 支持沙箱环境下的多轮RL训练：模型生成的代码可在隔离的 [AgentENV](https://github.com/kvcache-ai/AgentENV) Firecracker microVM 或 OpenEnv 服务中执行，两个后端共用同一份 `train.py`。参考 [cookbook](cookbook/rl/envs) 和[部署文档](docs/source_zh/使用指引/Agentic%20RL部署与训练.md)。 - 🎉2026-05-20 支持DeepSeek-V4-Flash and DeepSeek-V4-Pro系列模型。 - 🎉2026-05-20 支持多轮rollout和RL中的工具调用，Cookbook正在编写中，可以直接使用`from twinkle_agentic.rollout import MultiTurnRollout/APIMultiTurnRollout`进行多轮rollout。 - 🎉2026-05-20 支持训练任务失败后的IM消息告警, 使用方式: `import twinkle; twinkle.initialize(..., notifier=DingNotifier(...))`。 diff --git a/cookbook/rl/envs/README.md b/cookbook/rl/envs/README.md new file mode 100644 index 000000000..d12d996d2 --- /dev/null +++ b/cookbook/rl/envs/README.md @@ -0,0 +1,113 @@ +# Code RL (MBPP) + +One training script, two environments: openenv and agentenv. + +MBPP dataset, the model writes Python functions: it calls `run_python` to try code in a sandbox, calls `submit_solution`, and the trainer scores it against hidden tests. Qwen3.5-4B + LoRA, GRPO, up to 6 tool-calling turns. + +Concepts, deployment, tuning and troubleshooting live in **[Agentic RL Deployment and Training](../../../docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md)**. This file only lists the commands. + +## Choosing a backend + +| | `openenv` | `agentenv` | +|---|---|---| +| Where code runs | A session on an OpenEnv server | One Firecracker microVM per trajectory | +| Interpreter | AST interpreter (`coding_env`) | Real CPython | +| Memory per env | KBs | ~1GB | +| Environment host | An ordinary CPU machine, can be the training host | Needs `/dev/kvm`, kernel 6.8+ | + +`openenv` is enough for MBPP. Use `agentenv` when you need `unittest` + `@patch`, file writes, or pip installs. + +Either way the training host has 8 GPUs (`--model-gpus 4` + `--sampler-gpus 4`). + +## Run: openenv + +Environment host: + +```bash +sh openenv_server/install.sh # pip install openenv + coding_env from source + +HOST=127.0.0.1 sh openenv_server/serve.sh # training on this same host +# HOST=10.0.1.20 sh openenv_server/serve.sh # across hosts: bind the private NIC +``` + +Training host: + +```bash +pip install openenv +sh run_openenv.sh +# across hosts: OPENENV_BASE_URL=http://10.0.1.20:8000 sh run_openenv.sh +``` + +## Run: agentenv + +On the environment host, install the server and build the template (once; rebuild only when `Dockerfile` changes): + +```bash +sh agentenv_server/install.sh # install + provision host + build template +sh agentenv_server/install.sh --rebuild # delete the old template and rebuild +``` + +On a restricted network neither the base image nor pip resolves; point `BASE_IMAGE` at a reachable registry (it is forwarded to `aenv build --image`, so `Dockerfile` stays untouched): + +```bash +BASE_IMAGE=/library/python:3.11-slim sh agentenv_server/install.sh +``` + +Mirror options and build-failure troubleshooting are in the [appendix](../../../docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md) of the deployment guide. + +Start the server: + +```bash +sh agentenv_server/serve.sh # foreground, binds 127.0.0.1:8000 +NOHUP=1 sh agentenv_server/serve.sh # background +``` + +Training host: + +```bash +pip install e2b +# over HTTP directly: +AENV_API_URL=http://:8000 sh run_agentenv.sh +# or through an SSH tunnel: +ssh -N -L 8000:127.0.0.1:8000 root@ip-of-agentenv +# then, in another terminal +sh run_agentenv.sh +``` + +> Verify a sandbox boots before launching training: +> +> ```bash +> python -c " +> from twinkle_agentic.envs import AgentEnv +> e = AgentEnv(template='twinkle-code', api_url='http://127.0.0.1:8000') +> e.reset(); print('sandbox ok') +> print(e.run_command({'command': 'python -c \"import numpy, sympy; print(numpy.__version__)\"'})) +> " +> ``` + +## Arguments + +Command-line arguments are forwarded to `train.py` and override the `TRAIN_ARGS` defaults in `run_*.sh`: + +```bash +sh run_openenv.sh --max-steps 500 --batch-size 8 +``` + +Smoke test. `batch-size × num-generations` **must be ≥ `--model-gpus`**, otherwise every batch is dropped by the length filter with only a warning: + +```bash +sh run_openenv.sh --batch-size 2 --num-generations 4 --max-steps 2 +``` + +`agentenv` memory = `batch-size × num-generations × 1GB + 8GB`, i.e. ~40GB at the default 32 concurrent sandboxes. + +## Files + +| File | Role | +|---|---| +| `train.py` | Training logic, backend-agnostic | +| `_openenv.py` `_agentenv.py` | env construction, prompt, tools, hidden-test replay (the `_` prefix keeps them from shadowing the same-named pip packages) | +| `openenv_server/` `agentenv_server/` | Per-backend `install.sh` (one-time setup) and `serve.sh` | +| `run_openenv.sh` `run_agentenv.sh` | Launch commands and training hyper-parameters (keep both `TRAIN_ARGS` in sync) | + +To add a backend: write `_xxx.py` (`NAME`, `SYSTEM_PROMPT`, `TOOL_SCHEMA`, `make_env()`, `run_tests()`, `describe()`) and a `run_xxx.sh`, then add the name to `BACKENDS` in `train.py`. diff --git a/cookbook/rl/envs/_agentenv.py b/cookbook/rl/envs/_agentenv.py new file mode 100644 index 000000000..a085c95bc --- /dev/null +++ b/cookbook/rl/envs/_agentenv.py @@ -0,0 +1,146 @@ +import os +import textwrap +from typing import Any, Dict, List, Tuple + +from twinkle_agentic.envs import AgentEnv + +NAME = 'agentenv' + +API_URL = os.environ.get('AENV_API_URL', 'http://127.0.0.1:8000') +TEMPLATE = os.environ.get('AENV_TEMPLATE', 'twinkle-code') +SANDBOX_TIMEOUT = int(os.environ.get('SANDBOX_TIMEOUT', '600')) +COMMAND_TIMEOUT = int(os.environ.get('AENV_COMMAND_TIMEOUT', '60')) + +SYSTEM_PROMPT = """You are an expert Python programmer with access to a Linux sandbox. + +Solve the task by writing a Python function. + +- Use `run_python` to try out your function. Each call runs in a FRESH process, + so every snippet must be self-contained (include the imports and the function + definition, then call it) and must `print(...)` what you want to see. +- The full Python standard library is available, plus `numpy` and `sympy`. +- When you are confident, call `submit_solution` with the complete final source + (imports plus the function definition). + +Submit exactly once, and only after the code runs correctly.""" + +TOOL_SCHEMA: List[Dict[str, Any]] = [ + { + 'type': 'function', + 'function': { + 'name': 'run_python', + 'description': 'Run a self-contained Python snippet in the sandbox and return its stdout/stderr.', + 'parameters': { + 'type': 'object', + 'properties': { + 'code': { + 'type': 'string', + 'description': 'Python source to execute. Must print what you want to inspect.', + }, + }, + 'required': ['code'], + }, + }, + }, + { + 'type': 'function', + 'function': { + 'name': 'submit_solution', + 'description': 'Submit the final solution source and end the coding phase.', + 'parameters': { + 'type': 'object', + 'properties': { + 'code': { + 'type': 'string', + 'description': 'Complete final source: imports plus the function definition.', + }, + }, + 'required': ['code'], + }, + }, + }, +] + + +def _run_python(env: AgentEnv, arguments: Dict[str, Any]) -> str: + """Write the snippet to a file and execute it, avoiding shell quoting issues.""" + code = arguments.get('code') + if not code: + return "Error: 'code' argument is required." + env.sandbox.files.write('/workspace/scratch.py', code) + return env.run_command({'command': 'python /workspace/scratch.py', 'cwd': '/workspace'}) + + +def _submit_solution(env: AgentEnv, arguments: Dict[str, Any]) -> str: + """Record the solution on the env; the training loop scores it later.""" + code = (arguments.get('code') or '').strip() + if not code: + return "Error: 'code' argument is required." + env.submitted_code = code + return 'Solution submitted.' + + +def make_env() -> AgentEnv: + """Boot one sandbox per trajectory, exposing only the two task tools. + + ``include_default_tools=False`` hides AgentEnv's built-ins (raw command + execution, file read/write) so the action space matches the task exactly and + reward attribution stays clean. + """ + env = AgentEnv( + template=TEMPLATE, + api_url=API_URL, + sandbox_timeout=SANDBOX_TIMEOUT, + command_timeout=COMMAND_TIMEOUT, + include_default_tools=False, + ) + env.submitted_code = None + return (env.register_tool(TOOL_SCHEMA[0], _run_python).register_tool(TOOL_SCHEMA[1], _submit_solution)) + + +def _build_test_script(solution: str, test_list: List[str], setup_code: str) -> str: + """Build a script that runs each assertion independently and prints a tally. + + Every test is wrapped in its own ``try`` so that one failing assertion (or + one that raises) does not hide the rest — the reward uses the pass rate. + """ + parts = [solution, ''] + if setup_code: + parts += [setup_code, ''] + parts.append('_passed = 0') + for test in test_list: + body = textwrap.indent(test.strip(), ' ') + parts += ['try:', body, ' _passed += 1', 'except Exception:', ' pass'] + parts.append(f"print('TESTS_PASSED', _passed, {len(test_list)})") + return '\n'.join(parts) + '\n' + + +def run_tests(env: AgentEnv, test_list: List[str], setup_code: str = '') -> Tuple[int, int]: + """Replay the hidden tests against the submitted solution inside the sandbox. + + Real CPython means the tests can run as one ordinary script, unlike the + OpenEnv backend which drives them one expression at a time. + + Returns: + ``(n_passed, n_total)``. ``(0, n)`` when nothing was submitted, or when + the script never reaches its tally line (syntax error, timeout, ...). + """ + total = len(test_list) + solution = getattr(env, 'submitted_code', None) + if not solution: + return 0, total + + script = _build_test_script(solution, test_list, setup_code) + env.sandbox.files.write('/workspace/run_tests.py', script) + output = env.run_command({'command': 'python /workspace/run_tests.py', 'cwd': '/workspace'}) + + for line in reversed(output.splitlines()): + if line.startswith('TESTS_PASSED'): + fields = line.split() + if len(fields) >= 3 and fields[1].isdigit(): + return int(fields[1]), total + return 0, total + + +def describe() -> str: + return f'AgentENV microVM: api_url={API_URL}, template={TEMPLATE}' diff --git a/cookbook/rl/envs/_openenv.py b/cookbook/rl/envs/_openenv.py new file mode 100644 index 000000000..38d6a47b5 --- /dev/null +++ b/cookbook/rl/envs/_openenv.py @@ -0,0 +1,142 @@ +import os +from typing import Any, Dict, List, Tuple + +from twinkle_agentic.envs import OpenEnvClient + +NAME = 'openenv' + +BASE_URL = os.environ.get('OPENENV_BASE_URL', 'http://127.0.0.1:8000') +ENV_NAME = os.environ.get('OPENENV_ENV_NAME', 'coding_env') +# Code execution can be slow; keep the per-message timeout generous. Note the +# executor's own caps still apply (operation count, while-loop iterations, and +# a wall-clock limit in newer smolagents releases). +MESSAGE_TIMEOUT_S = float(os.environ.get('OPENENV_MESSAGE_TIMEOUT_S', '120')) + +SYSTEM_PROMPT = """You are an expert Python programmer with access to a Python interpreter. + +Solve the task by writing a Python function. + +- Use `run_python` to define and test your function. The interpreter keeps its + state between calls, so a function defined in one call stays available. +- Available modules: math, re, collections, itertools, functools, operator, + heapq, bisect, string, statistics, fractions, decimal, datetime, copy, json. + There is no file or network access. +- When you are confident, call `submit_solution` with the complete final source + (imports plus the function definition). + +Submit exactly once, and only after the code runs correctly.""" + +TOOL_SCHEMA: List[Dict[str, Any]] = [ + { + 'type': 'function', + 'function': { + 'name': 'run_python', + 'description': 'Execute a Python snippet in the interpreter and return its stdout/stderr.', + 'parameters': { + 'type': 'object', + 'properties': { + 'code': { + 'type': 'string', + 'description': 'Python source to execute. Use print(...) to inspect values.', + }, + }, + 'required': ['code'], + }, + }, + }, + { + 'type': 'function', + 'function': { + 'name': 'submit_solution', + 'description': 'Submit the final solution source and end the coding phase.', + 'parameters': { + 'type': 'object', + 'properties': { + 'code': { + 'type': 'string', + 'description': 'Complete final source: imports plus the function definition.', + }, + }, + 'required': ['code'], + }, + }, + }, +] + + +def _submit_solution(env: OpenEnvClient, arguments: Dict[str, Any]) -> str: + """Record the solution on the env; the training loop scores it later.""" + code = (arguments.get('code') or '').strip() + if not code: + return "Error: 'code' argument is required." + env.submitted_code = code + return 'Solution submitted.' + + +def make_env() -> OpenEnvClient: + """Open one session per trajectory, exposing only the two task tools. + + ``run_python`` stays server-backed: the default action mapper turns + ``{'code': ...}`` straight into the env's ``CodeAction``. Only + ``submit_solution`` needs a client-side handler. + """ + env = OpenEnvClient( + env_name=ENV_NAME, + base_url=BASE_URL, + tools=[TOOL_SCHEMA[0]], + message_timeout_s=MESSAGE_TIMEOUT_S, + ) + env.submitted_code = None + return env.register_tool(TOOL_SCHEMA[1], _submit_solution) + + +def _ok(result) -> bool: + return not (getattr(result.observation, 'exit_code', 0) or 0) + + +def _last_line(text: str) -> str: + lines = [line for line in (text or '').splitlines() if line.strip()] + return lines[-1].strip() if lines else '' + + +def run_tests(env: OpenEnvClient, test_list: List[str], setup_code: str = '') -> Tuple[int, int]: + """Run the hidden tests against the submitted solution in the same session. + + Each assertion is executed as its own ``print()`` step rather than as + one block of ``assert`` statements. Two reasons: printing the boolean tells + "the assertion evaluated to False" apart from "the code blew up", whereas a + failed ``assert`` surfaces as an exception indistinguishable from a crash + inside the solution; and one step per test isolates a test that raises, so + the remaining ones still run. + + The executor does support ``assert``/``try``, so this is a diagnosability + choice, not a capability workaround. + + Returns: + ``(n_passed, n_total)``. ``(0, n)`` when nothing was submitted or the + solution itself fails to execute. + """ + total = len(test_list) + solution = getattr(env, 'submitted_code', None) + if not solution: + return 0, total + + # Define the submitted solution in the session namespace. + if not _ok(env.execute({'code': solution})): + return 0, total + if setup_code and not _ok(env.execute({'code': setup_code})): + return 0, total + + passed = 0 + for test in test_list: + expr = test.strip() + if expr.startswith('assert '): + expr = expr[len('assert '):] + result = env.execute({'code': f'print({expr})'}) + if _ok(result) and _last_line(getattr(result.observation, 'stdout', '')) == 'True': + passed += 1 + return passed, total + + +def describe() -> str: + return f'OpenEnv server mode: base_url={BASE_URL}, env={ENV_NAME}' diff --git a/cookbook/rl/envs/agentenv_server/Dockerfile b/cookbook/rl/envs/agentenv_server/Dockerfile new file mode 100644 index 000000000..3cc312d5d --- /dev/null +++ b/cookbook/rl/envs/agentenv_server/Dockerfile @@ -0,0 +1,4 @@ +FROM python:3.11-slim +RUN pip install --no-cache-dir numpy sympy +ENV PYTHONUNBUFFERED=1 +WORKDIR /workspace diff --git a/cookbook/rl/envs/agentenv_server/install.sh b/cookbook/rl/envs/agentenv_server/install.sh new file mode 100644 index 000000000..f611f9522 --- /dev/null +++ b/cookbook/rl/envs/agentenv_server/install.sh @@ -0,0 +1,68 @@ +#!/bin/sh +set -eu +TEMPLATE="${TEMPLATE:-twinkle-code}" +CPU_COUNT="${CPU_COUNT:-1}" +MEMORY_MB="${MEMORY_MB:-1024}" +BASE_IMAGE="${BASE_IMAGE:-}" +# Where the runtime config is copied to, readable by the aenv user. serve.sh +# reads the same default. +REPO_ROOT="${REPO_ROOT:-$HOME/AgentENV}" +CONFIG_DIR="${CONFIG_DIR:-/var/lib/aenv/config}" + +SKIP_INSTALL=0 +REBUILD=0 +for arg in "$@"; do + case "$arg" in + --skip-install) SKIP_INSTALL=1 ;; + --rebuild) REBUILD=1 ;; + *) echo "Unknown option: $arg" >&2; exit 2 ;; + esac +done + +cd "$(dirname "$0")" + +if [ "$SKIP_INSTALL" = "0" ]; then + echo "==> Installing AgentENV server + aenv CLI" + curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh \ + | sudo bash + + echo "==> Provisioning the host (kvm group, ublk module, udev, sysctl)" + sudo server --setup-host + sudo install -d -o aenv -g aenv /var/lib/aenv/home + + # A source-built binary defaults to its build-time repo path for the config + # (CARGO_MANIFEST_DIR), which the aenv user cannot read when the repo lives + # under /root. Only default.toml needs copying — deps_manifest.toml is + # include_str!'d into the binary at compile time. + if [ -f "$REPO_ROOT/config/default.toml" ]; then + sudo install -d -o aenv -g aenv "$CONFIG_DIR" + sudo install -o aenv -g aenv -m 0644 \ + "$REPO_ROOT/config/default.toml" "$CONFIG_DIR/config.toml" + echo " config seeded to $CONFIG_DIR/config.toml" + fi +fi + +echo "==> Authenticating the CLI" +if [ -f "$HOME/.config/aenv/credentials" ]; then + echo " already authenticated ($HOME/.config/aenv/credentials)" +else + aenv auth +fi + +if [ "$REBUILD" = "1" ]; then + echo "==> Deleting template '$TEMPLATE'" + aenv template delete "$TEMPLATE" || true +fi + +echo "==> Building template '$TEMPLATE' (cpu=$CPU_COUNT mem=${MEMORY_MB}MiB)" +set -- Dockerfile -t "$TEMPLATE" --cpu-count "$CPU_COUNT" --memory-mb "$MEMORY_MB" +[ -n "$BASE_IMAGE" ] && set -- "$@" --image "$BASE_IMAGE" +aenv build "$@" + +echo +echo "Build runs server-side and takes a few minutes. Follow it with:" +echo " aenv template watch # id printed above" +echo " aenv template list # confirm it reaches ready" +echo +echo "Then start the server:" +echo " sh serve.sh" diff --git a/cookbook/rl/envs/agentenv_server/serve.sh b/cookbook/rl/envs/agentenv_server/serve.sh new file mode 100644 index 000000000..207f2afbd --- /dev/null +++ b/cookbook/rl/envs/agentenv_server/serve.sh @@ -0,0 +1,106 @@ +#!/bin/sh +set -eu +REPO_ROOT="${REPO_ROOT:-$HOME/AgentENV}" +# Read by the server itself, not by this script. +export API_ADDR="${API_ADDR:-127.0.0.1:8000}" +LOG_FILE="${LOG_FILE:-/tmp/aenv-server.log}" +NOHUP="${NOHUP:-0}" + +# The server drops privileges to a non-root user, so it must not inherit root's +# HOME — regctl and docker credential lookups fail with EACCES there, which +# turns into a hard failure once a private registry needs credentials. +AENV_HOME="${AENV_HOME:-/var/lib/aenv/home}" + +# The binary bakes in its build-time repo path as the default config location +# (CARGO_MANIFEST_DIR in src/cfg.rs), so a server built under /root looks for +# /root/AgentENV/config/default.toml — unreadable once it drops to the aenv +# user, since /root is 0700. Point it at a copy the runtime user owns. +AENV_CONFIG_PATH="${AENV_CONFIG_PATH:-/var/lib/aenv/config/config.toml}" + +# run-with-capabilities.sh is primarily a test wrapper: when these are unset it +# defaults them to /tmp/aenv-test-/{home,run}. That sends downloaded +# dependencies (kernel, firecracker, overlaybd — hundreds of MB) to a directory +# that /tmp cleanup wipes, so every restart re-downloads them. Pin the real +# state directory instead; home_path in config.toml points at the same place. +AENV_HOME_PATH="${AENV_HOME_PATH:-/var/lib/aenv}" +AENV_RUNTIME_PATH="${AENV_RUNTIME_PATH:-/run/aenv}" + +if [ ! -r "$AENV_CONFIG_PATH" ]; then + echo "Config not readable: $AENV_CONFIG_PATH" >&2 + echo "Seed it from the repo (install.sh does this for you):" >&2 + echo " sudo install -d -o aenv -g aenv \$(dirname $AENV_CONFIG_PATH)" >&2 + echo " sudo install -o aenv -g aenv -m 0644 \\" >&2 + echo " $REPO_ROOT/config/default.toml $AENV_CONFIG_PATH" >&2 + exit 1 +fi + +# Stop whatever is already running, so this script is a restart rather than a +# "port already in use" failure. Match the binary path, not this script's name: +# run-with-capabilities.sh ends in `exec setpriv ... server`, which replaces the +# process image, so argv[0] of the live process is the server binary. +SERVER_BIN="${SERVER_BIN:-/usr/local/bin/server}" + +stop_running() { + # A systemd-managed instance would be restarted right after a kill, so hand + # it over to systemctl instead. install.sh sets up aenv.service when systemd + # is present. + if [ -d /run/systemd/system ] && systemctl is-active --quiet aenv 2>/dev/null; then + echo "Stopping systemd service aenv" + sudo systemctl stop aenv + return + fi + + pids=$(pgrep -f "^$SERVER_BIN" 2>/dev/null || true) + [ -z "$pids" ] && return + + echo "Stopping running server (pid: $pids)" + # SIGTERM first: the server tears down microVMs, veth pairs and iptables + # rules on shutdown, and SIGKILL would leave those behind. + sudo kill $pids 2>/dev/null || true + i=0 + while [ $i -lt 30 ] && pgrep -f "^$SERVER_BIN" >/dev/null 2>&1; do + sleep 1 + i=$((i + 1)) + done + if pgrep -f "^$SERVER_BIN" >/dev/null 2>&1; then + echo " still alive after 30s, sending SIGKILL" + sudo pkill -KILL -f "^$SERVER_BIN" 2>/dev/null || true + sleep 1 + fi +} + +stop_running + +# STOP_ONLY=1 sh serve.sh — shut down without starting again. +if [ "${STOP_ONLY:-0}" = "1" ]; then + echo "Stopped." + exit 0 +fi + +cd "$REPO_ROOT" + +# run-with-capabilities.sh grants CAP_NET_ADMIN + CAP_SYS_ADMIN via setpriv and +# re-initialises supplementary groups (--init-groups), which is what makes a +# fresh kvm-group membership take effect without re-login. It derives repo_root +# from BASH_SOURCE, so the path above is what matters, not the cwd. +# +# `sudo env VAR=...`, not `sudo VAR=...`: with sudoers env_reset (the default) +# the latter is not guaranteed to pass anything through. +# +# AENV_RUN_USER must be explicit: the script otherwise falls back through +# SUDO_USER -> repo owner -> aenv -> root, and running as root is not supported. +E="AENV_RUN_USER=aenv HOME=$AENV_HOME API_ADDR=$API_ADDR AENV_CONFIG_PATH=$AENV_CONFIG_PATH AENV_HOME_PATH=$AENV_HOME_PATH AENV_RUNTIME_PATH=$AENV_RUNTIME_PATH" + +if [ "$NOHUP" = "1" ]; then + # setsid, not just nohup: the wrapper ends in `exec setpriv`, which replaces + # the process image, and a SIGHUP disposition inherited from nohup is not + # guaranteed to survive that. A new session detaches from the terminal + # regardless. + echo "Starting AgentENV on $API_ADDR (background) -> $LOG_FILE" + sudo env $E setsid nohup ./scripts/run-with-capabilities.sh server \ + >"$LOG_FILE" 2>&1 Installing the openenv client" +pip install openenv + +echo "==> Installing coding_env from source" +# server_app.py imports PythonCodeActEnv and PyExecutor from this package; its +# dependencies pull in smolagents, which is the AST interpreter that actually +# runs the model's code. +if [ -d "$OPENENV_SRC/.git" ]; then + echo " reusing $OPENENV_SRC" + git -C "$OPENENV_SRC" pull --ff-only +else + git clone https://github.com/huggingface/OpenEnv.git "$OPENENV_SRC" +fi +pip install -e "$OPENENV_SRC/envs/coding_env" + +echo +echo "Done. Start the server:" +echo " sh serve.sh # binds 0.0.0.0:8000" +echo " HOST=127.0.0.1 sh serve.sh # same-host training, no network exposure" diff --git a/cookbook/rl/envs/openenv_server/serve.sh b/cookbook/rl/envs/openenv_server/serve.sh new file mode 100644 index 000000000..988cbdd81 --- /dev/null +++ b/cookbook/rl/envs/openenv_server/serve.sh @@ -0,0 +1,22 @@ +#!/bin/sh +set -eu +HOST="${HOST:-0.0.0.0}" +PORT="${PORT:-8000}" +# Uvicorn worker processes. Each worker holds its own app instance, so total +# capacity is WORKERS x MAX_CONCURRENT_ENVS. Size it above the number of +# concurrent trajectories (BATCH_SIZE x NUM_GENERATIONS), otherwise the extra +# WebSocket connections are rejected at capacity. +WORKERS="${WORKERS:-4}" +export MAX_CONCURRENT_ENVS="${MAX_CONCURRENT_ENVS:-64}" + +cd "$(dirname "$0")" + +echo "Serving twinkle_code_env on ${HOST}:${PORT}" +echo " workers=${WORKERS}, max_concurrent_envs=${MAX_CONCURRENT_ENVS} per worker" +echo " capacity=$((WORKERS * MAX_CONCURRENT_ENVS)) concurrent sessions" +if [ "$HOST" = "0.0.0.0" ]; then + echo " WARNING: bound to 0.0.0.0 with no authentication — restrict this port" + echo " to the training host, or use HOST=127.0.0.1 if training is local." +fi + +exec uvicorn server_app:app --host "$HOST" --port "$PORT" --workers "$WORKERS" diff --git a/cookbook/rl/envs/openenv_server/server_app.py b/cookbook/rl/envs/openenv_server/server_app.py new file mode 100644 index 000000000..29ac3c00d --- /dev/null +++ b/cookbook/rl/envs/openenv_server/server_app.py @@ -0,0 +1,75 @@ +import os + +from coding_env.models import CodeAction, CodeObservation +from coding_env.server.python_codeact_env import PythonCodeActEnv +from coding_env.server.python_executor import PyExecutor +from openenv.core.env_server import create_app + +# Modules the sandboxed executor may import. Keep this list tight: it is the +# only thing standing between model-generated code and this process, since +# LocalPythonExecutor is an AST interpreter, not an OS-level sandbox. +ALLOWED_IMPORTS = [ + 'math', + 're', + 'collections', + 'itertools', + 'functools', + 'operator', + 'heapq', + 'bisect', + 'string', + 'statistics', + 'fractions', + 'decimal', + 'datetime', + 'copy', + 'json', +] + +MAX_CONCURRENT_ENVS = int(os.environ.get('MAX_CONCURRENT_ENVS', '64')) + + +class ConcurrentCodeEnv(PythonCodeActEnv): + """Session-isolated Python executor with a task-appropriate config.""" + + SUPPORTS_CONCURRENT_SESSIONS = True + + def __init__(self): + super().__init__() + self._configure() + + def reset(self, **kwargs): + # The parent's reset() rebuilds the executor and the transform with + # upstream defaults, so re-apply our config afterwards. **kwargs + # absorbs the seed / episode_id the server forwards from the client. + observation = super().reset() + self._configure() + return observation + + def _configure(self) -> None: + self._executor = PyExecutor(additional_imports=list(ALLOWED_IMPORTS)) + # Drop the style/safety reward heuristics; reward comes from the tests. + self.transform = None + + +app = create_app( + ConcurrentCodeEnv, + CodeAction, + CodeObservation, + env_name='twinkle_code_env', + max_concurrent_envs=MAX_CONCURRENT_ENVS, +) + + +def main(): + """Entry point for ``python server_app.py`` (single worker).""" + import uvicorn + uvicorn.run( + app, + host=os.environ.get('HOST', '0.0.0.0'), + port=int(os.environ.get('PORT', '8000')), + ) + + +if __name__ == '__main__': + main() diff --git a/cookbook/rl/envs/run_agentenv.sh b/cookbook/rl/envs/run_agentenv.sh new file mode 100644 index 000000000..c743cb5d0 --- /dev/null +++ b/cookbook/rl/envs/run_agentenv.sh @@ -0,0 +1,37 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")" + +export CODE_RL_BACKEND=agentenv +# LOCAL AgentENV instance or: +# ssh -N -L 8000:127.0.0.1:8000 root@xx.xx.xx.xx +# if you can connect the agent env server by ssh +export AENV_API_URL="${AENV_API_URL:-http://127.0.0.1:8000}" +export AENV_TEMPLATE="${AENV_TEMPLATE:-twinkle-code}" +# Sandbox lifetime; must outlast one episode plus the test replay. +export SANDBOX_TIMEOUT="${SANDBOX_TIMEOUT:-600}" +# Per-command timeout inside the sandbox. +export AENV_COMMAND_TIMEOUT="${AENV_COMMAND_TIMEOUT:-60}" +# Max tool-calling turns per episode. +export MAX_TURNS="${MAX_TURNS:-6}" +# Concurrent reset/score calls issued from the driver. +export ENV_CONCURRENCY="${ENV_CONCURRENCY:-16}" + +TRAIN_ARGS=" + --model-id ms://Qwen/Qwen3.5-4B + --model-gpus 4 + --sampler-gpus 4 + --num-generations 8 + --max-tokens 2048 + --batch-size 4 + --mini-batch-size 8 + --micro-batch-size 2 + --max-steps 1000 + --lr 1e-5 + --lora-r 16 + --save-steps 500 + --adapter-name default +" + +echo "Backend: agentenv | api: $AENV_API_URL | template: $AENV_TEMPLATE" +python train.py $TRAIN_ARGS "$@" diff --git a/cookbook/rl/envs/run_openenv.sh b/cookbook/rl/envs/run_openenv.sh new file mode 100644 index 000000000..82b885da8 --- /dev/null +++ b/cookbook/rl/envs/run_openenv.sh @@ -0,0 +1,34 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")" + +export CODE_RL_BACKEND=openenv + +# ---- OpenEnv connection (a load-balancer address works here too) ---- +# For a cross-network setup this is the LOCAL end of an SSH port forward +# (http://127.0.0.1:8000) rather than the remote address — see the deployment +# guide in docs. +export OPENENV_BASE_URL="${OPENENV_BASE_URL:-http://127.0.0.1:8000}" +export OPENENV_ENV_NAME="${OPENENV_ENV_NAME:-coding_env}" +export OPENENV_MESSAGE_TIMEOUT_S="${OPENENV_MESSAGE_TIMEOUT_S:-120}" +export MAX_TURNS="${MAX_TURNS:-6}" +export ENV_CONCURRENCY="${ENV_CONCURRENCY:-16}" + +TRAIN_ARGS=" + --model-id ms://Qwen/Qwen3.5-4B + --model-gpus 4 + --sampler-gpus 4 + --num-generations 8 + --max-tokens 2048 + --batch-size 4 + --mini-batch-size 8 + --micro-batch-size 2 + --max-steps 1000 + --lr 1e-5 + --lora-r 16 + --save-steps 500 + --adapter-name default +" + +echo "Backend: openenv | server: $OPENENV_BASE_URL | env: $OPENENV_ENV_NAME" +python train.py $TRAIN_ARGS "$@" diff --git a/cookbook/rl/envs/train.py b/cookbook/rl/envs/train.py new file mode 100644 index 000000000..85d6f91df --- /dev/null +++ b/cookbook/rl/envs/train.py @@ -0,0 +1,331 @@ +import os +from concurrent.futures import ThreadPoolExecutor +from importlib import import_module +from typing import Any, Dict, List, Tuple + +import torch +from peft import LoraConfig + +import twinkle +from twinkle import DeviceGroup, DeviceMesh, get_device_placement, get_logger +from twinkle.advantage import GRPOAdvantage +from twinkle.checkpoint_engine import CheckpointEngineManager +from twinkle.cli import CLI +from twinkle.data_format import SamplingParams +from twinkle.metric import CompletionRewardMetric +from twinkle.model import TransformersModel +from twinkle.processor import InputProcessor +from twinkle.sampler import vLLMSampler +from twinkle.template import Qwen3_5Template +from twinkle_agentic.envs import EnvTool +from twinkle_agentic.rollout.multi_turn import MultiTurnRollout +from twinkle_agentic.tools.tool_manager import ToolManager + +logger = get_logger() +args = CLI.from_args() + +# ========== Backend selection ========== +BACKENDS = ('openenv', 'agentenv') +backend_name = os.environ.get('CODE_RL_BACKEND', 'openenv') +if backend_name not in BACKENDS: + raise ValueError(f'Unknown CODE_RL_BACKEND {backend_name!r}. Available: {", ".join(BACKENDS)}') +backend = import_module(f'_{backend_name}') + +# ========== Configuration ========== +MODEL_ID = args.model.model_id or 'ms://Qwen/Qwen3.5-4B' + +MODEL_GPUS = args.infra.model_gpus or 4 +SAMPLER_GPUS = args.infra.sampler_gpus or 4 +NUM_GPUS = MODEL_GPUS + SAMPLER_GPUS + +NUM_GENERATIONS = args.rl.num_generations or 8 +MAX_NEW_TOKENS = args.sampling.max_tokens or 2048 +LEARNING_RATE = args.optimizer.learning_rate or 1e-5 +MAX_STEPS = args.training.max_steps or 1000 +BATCH_SIZE = args.training.batch_size or 4 +MINI_BATCH_SIZE = args.training.mini_batch_size or 8 +MICRO_BATCH_SIZE = args.training.micro_batch_size or 2 +GRADIENT_ACCUMULATION_STEPS = args.training.gradient_accumulation_steps or 1 +ADAPTER_NAME = args.lora.adapter_name or 'default' +SAVE_STEPS = args.training.save_steps or 500 +LORA_RANK = args.lora.lora_r or 16 +MAX_TURNS = int(os.environ.get('MAX_TURNS', '6')) + +# Parallelism for the blocking reset/score calls in the driver. +ENV_CONCURRENCY = int(os.environ.get('ENV_CONCURRENCY', '16')) + + +# ========== Dataset (MBPP) ========== +def load_mbpp() -> List[Dict[str, Any]]: + """Load MBPP as [{'prompt', 'test_list', 'test_setup_code'}].""" + from modelscope.msdatasets import MsDataset + raw = MsDataset.load('opencompass/mbpp', subset_name='full', split='train') + samples = [] + for row in raw: + test_list = list(row['test_list'] or []) + if not test_list: + continue + # The first assertion is handed to the model as a signature hint: MBPP + # descriptions alone do not pin the function name, and the hidden tests + # call it by name. + prompt = (f"{row['text']}\n\n" + f'Your function must satisfy this call:\n{test_list[0]}') + samples.append({ + 'prompt': prompt, + 'test_list': test_list, + 'test_setup_code': row.get('test_setup_code') or '', + }) + logger.info(f'MBPP loaded: {len(samples)} samples') + return samples + + +# ========== Environment lifecycle ========== +def prepare_trajectories( + samples: List[Dict[str, Any]], + pool: ThreadPoolExecutor, +) -> Tuple[List[Dict[str, Any]], List[ToolManager], List[Any]]: + """Create one env per trajectory (reset in parallel) and build trajectories.""" + envs = [backend.make_env() for _ in samples] + # reset() blocks on the network (session handshake, or booting a sandbox); + # run them concurrently so a batch of 32 does not serialize. + list(pool.map(lambda env: env.reset(), envs)) + + trajectories = [] + tool_managers = [] + for sample, env in zip(samples, envs): + tool_managers.append(ToolManager(EnvTool.from_env(env))) + trajectories.append({ + 'messages': [ + {'role': 'system', 'content': backend.SYSTEM_PROMPT}, + {'role': 'user', 'content': sample['prompt']}, + ], + 'tools': backend.TOOL_SCHEMA, + }) + return trajectories, tool_managers, envs + + +def close_envs(envs: List[Any], pool: ThreadPoolExecutor) -> None: + """Release envs so their server-side capacity is freed immediately.""" + list(pool.map(lambda env: env.close(), envs)) + + +def extract_rewards( + envs: List[Any], + samples: List[Dict[str, Any]], + pool: ThreadPoolExecutor, +) -> Tuple[List[float], List[float]]: + """Score submitted solutions against the hidden MBPP tests. + + Shaped so that a full pass dominates, while a submission that passes some + tests still beats one that passes none — which beats never submitting. + + Returns: + ``(rewards, pass_rates)``. + """ + + def score(pair) -> Tuple[float, float]: + env, sample = pair + passed, total = backend.run_tests(env, sample['test_list'], sample['test_setup_code']) + rate = passed / total if total else 0.0 + if total and passed == total: + return 1.0, rate + if getattr(env, 'submitted_code', None): + return 0.1 + 0.4 * rate, rate + return 0.0, rate + + scored = list(pool.map(score, zip(envs, samples))) + return [s[0] for s in scored], [s[1] for s in scored] + + + +# ========== Main ========== +def main(): + device_groups = [ + DeviceGroup(name='model', ranks=list(range(MODEL_GPUS)), device_type='GPU'), + DeviceGroup(name='sampler', ranks=list(range(MODEL_GPUS, NUM_GPUS)), device_type='GPU'), + ] + model_mesh = DeviceMesh.from_sizes(world_size=MODEL_GPUS, dp_size=MODEL_GPUS) + sampler_mesh = DeviceMesh.from_sizes(world_size=SAMPLER_GPUS, dp_size=SAMPLER_GPUS) + twinkle.initialize(mode='ray', nproc_per_node=NUM_GPUS, groups=device_groups, lazy_collect=False) + + lora_config = LoraConfig( + target_modules='all-linear', + r=LORA_RANK, + lora_alpha=LORA_RANK * 2, + lora_dropout=0.05, + ) + + model = TransformersModel( + model_id=MODEL_ID, + device_mesh=model_mesh, + remote_group='model', + dtype=torch.float32, + mixed_precision='bf16', + ) + model.add_adapter_to_model(ADAPTER_NAME, lora_config, gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS) + model.set_optimizer('AdamW', lr=LEARNING_RATE) + model.set_lr_scheduler('CosineAnnealingLR', T_max=MAX_STEPS, eta_min=0) + grpo_epsilon = 0.2 + model.set_loss('GRPOLoss', epsilon=grpo_epsilon) + model.add_metric('GRPOMetric', is_training=True, epsilon=grpo_epsilon) + model.set_processor(InputProcessor, padding_free=True) + model.set_template('Qwen3_5Template', model_id=MODEL_ID, enable_thinking=False) + + sampler = vLLMSampler( + model_id=MODEL_ID, + engine_args={ + 'gpu_memory_utilization': 0.8, + 'max_model_len': 8192, + 'max_lora_rank': 32, + 'enable_lora': True, + 'enable_tower_connector_lora': True, + }, + device_mesh=sampler_mesh, + remote_group='sampler', + ) + sampler.set_template('Qwen3_5Template', model_id=MODEL_ID, enable_thinking=False) + + rollout_template = Qwen3_5Template(MODEL_ID, max_length=8192, enable_thinking=False) + rollout_template.truncation_strategy = 'delete' + + ckpt_manager = CheckpointEngineManager(model=model, sampler=sampler) + + sampling_params = SamplingParams( + max_tokens=MAX_NEW_TOKENS, num_samples=1, logprobs=1, + temperature=1.0, top_p=0.95, + ) + rollout = MultiTurnRollout( + sampler=sampler, + template=rollout_template, + sampling_params=sampling_params, + max_turns=MAX_TURNS, + ) + + advantage_fn = GRPOAdvantage() + metrics = CompletionRewardMetric() + dataset = load_mbpp() + env_pool = ThreadPoolExecutor(max_workers=ENV_CONCURRENCY) + + optim_step = 0 + sample_cursor = 0 + logger.info(f'Starting code RL [{backend.NAME}] — {backend.describe()}') + logger.info(f'Concurrent envs needed: {BATCH_SIZE * NUM_GENERATIONS}') + logger.info(get_device_placement()) + + while optim_step < MAX_STEPS: + metrics.reset() + + # BATCH_SIZE problems x NUM_GENERATIONS contiguous copies (GRPO groups). + batch = [dataset[(sample_cursor + i) % len(dataset)] for i in range(BATCH_SIZE)] + sample_cursor += BATCH_SIZE + expanded = [s for s in batch for _ in range(NUM_GENERATIONS)] + n_traj = len(expanded) + + # 1. Create envs and build initial trajectories + logger.info(f'[Step {optim_step}] Preparing {n_traj} envs...') + expand_prompts, tool_managers, envs = prepare_trajectories(expanded, env_pool) + + try: + # 2. Sync model weights to sampler + ckpt_manager.sync_weights(merge_and_sync=False) + sampler.reset_prefix_cache() + + # 3. Multi-turn rollout with per-trajectory ToolManagers + all_trajectories: List[Dict[str, Any]] = rollout( + expand_prompts, + tool_manager=tool_managers, + ) + + # 4. Score solutions by replaying the hidden tests in each env + total_rewards, pass_rates = extract_rewards(envs, expanded, env_pool) + finally: + # Envs occupy server-side capacity; always release them. + close_envs(envs, env_pool) + + all_old_logps: List[List[float]] = [] + all_completion_lengths: List[int] = [] + n_turns_per_rollout: List[int] = [] + for traj in all_trajectories: + logprobs = traj.get('logprobs') or [] + all_old_logps.append([lp[0][1] for lp in logprobs] if logprobs else []) + labels = traj.get('labels') or [] + all_completion_lengths.append(sum(1 for l in labels if l != -100)) + n_turns_per_rollout.append(int(traj.get('turns') or 0)) + + # 5. Group-relative advantages + advantages = advantage_fn( + total_rewards, num_generations=NUM_GENERATIONS, scale='group', + ).tolist() + + # 6. Log metrics + metrics.accumulate( + completion_lengths=all_completion_lengths, + rewards={'total': total_rewards}, + ) + avg_reward = sum(total_rewards) / len(total_rewards) if total_rewards else 0.0 + solve_rate = sum(1 for r in total_rewards if r >= 1.0) / max(len(total_rewards), 1) + avg_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else 0.0 + avg_turns = sum(n_turns_per_rollout) / len(n_turns_per_rollout) if n_turns_per_rollout else 0.0 + max_turns = max(n_turns_per_rollout) if n_turns_per_rollout else 0 + min_turns = min(n_turns_per_rollout) if n_turns_per_rollout else 0 + logger.info(f'[Step {optim_step}] avg_reward={avg_reward:.3f}, solve_rate={solve_rate:.3f}, ' + f'test_pass_rate={avg_pass_rate:.3f}, avg_turns={avg_turns:.1f}') + + # 7. Filter and train + all_input_data: List[Dict[str, Any]] = [] + filtered_old_logps: List[List[float]] = [] + filtered_advantages: List[float] = [] + max_len = rollout_template.max_length or float('inf') + for i, traj in enumerate(all_trajectories): + traj_len = len(traj.get('input_ids') or traj.get('labels') or []) + comp_len = sum(1 for l in (traj.get('labels') or []) if l != -100) + if traj_len > max_len or comp_len == 0: + continue + all_input_data.append(traj) + filtered_old_logps.append(all_old_logps[i]) + filtered_advantages.append(advantages[i]) + + if len(all_input_data) < MODEL_GPUS: + logger.warning(f'[Step {optim_step}] Only {len(all_input_data)} valid trajectories ' + f'after filtering (need >= {MODEL_GPUS}), skipping this batch.') + continue + + total_completions = len(all_input_data) + logger.info(f'[Step {optim_step}] {total_completions}/{n_traj} trajectories ' + f'passed length filter (max_len={max_len})') + + for mb_start in range(0, total_completions, MINI_BATCH_SIZE): + mb_end = min(mb_start + MINI_BATCH_SIZE, total_completions) + model.forward_backward( + inputs=all_input_data[mb_start:mb_end], + old_logps=filtered_old_logps[mb_start:mb_end], + advantages=filtered_advantages[mb_start:mb_end], + micro_batch_size=MICRO_BATCH_SIZE, + ) + model.clip_grad_and_step() + optim_step += 1 + + if optim_step >= MAX_STEPS: + break + if optim_step % SAVE_STEPS == 0: + model.save(f'code-rl-{backend.NAME}-checkpoint-{optim_step}') + + # 8. Step summary + log_dict = metrics.calculate() # train/*_reward, train/completion_length + log_dict.update(model.calculate_metric(is_training=True)) # loss, grad_norm, learning rate, token accuracy... + log_dict['train/code_acc'] = solve_rate + log_dict['train/test_pass_rate'] = avg_pass_rate + log_dict['train/avg_reward'] = avg_reward + log_dict['train/avg_turns'] = avg_turns + log_dict['train/max_turns'] = max_turns + log_dict['train/min_turns'] = min_turns + metrics.reset() + logger.info(f'[Step {optim_step}/{MAX_STEPS}] {log_dict}') + + env_pool.shutdown(wait=False) + logger.info(f'Training completed. optim_steps={optim_step}') + model.save(f'code-rl-{backend.NAME}-final') + + +if __name__ == '__main__': + main() diff --git a/docs/source_en/Components/Agentic/Envs.md b/docs/source_en/Components/Agentic/Envs.md index 3e2e90b3a..988eac6fe 100644 --- a/docs/source_en/Components/Agentic/Envs.md +++ b/docs/source_en/Components/Agentic/Envs.md @@ -78,7 +78,7 @@ manager = ToolManager(env_tools) | `from_env(env)` | Factory: creates one `EnvTool` per tool in `env.tools()`. | | `last_result` | Stores the most recent `StepResult` for inspection. | | `done` | Property: whether the last step terminated the episode. | -| `episode_reward` | Property: cumulative reward from `info['episode_reward']`. | +| `episode_reward` | Property: cumulative reward from `info['episode_reward']`, falling back to the last step's `reward`. | ### Manual Construction @@ -97,41 +97,102 @@ env_tool = EnvTool( ) ``` -## OpenEnv +## OpenEnv: Two Integration Modes -`OpenEnv` adapts an [OpenEnv](https://github.com/OpenEnv) WebSocket-based environment server as a synchronous Twinkle `Env`. +An [OpenEnv](https://github.com/meta-pytorch/OpenEnv) environment package ships both an `Environment` implementation and an `EnvClient`, so Twinkle provides two adapters for two very different deployment shapes: + +| | **Embedded** `OpenEnv` | **Server** `OpenEnvClient` | +|---|---|---| +| Where the env runs | Inside the training process (Environment instantiated directly) | A separate OpenEnv service process / container | +| Transport | None (local function calls) | Persistent WebSocket (one connection = one session) | +| Isolation | None; shares memory and the GPU node with training | Process / container level; can live on a completely separate machine | +| Dependencies | Env package must be installed on the training node | Env package only needed on the environment node | +| Scaling | `EnvPool` shards across Ray workers | The server's own concurrent sessions + replicas | +| Best for | Pure-compute lightweight envs (board games, text games) | Code execution, anything needing isolation or independent scaling | + +> Do **not** wrap `OpenEnvClient` in `EnvPool`: the session's lifetime is owned by the server, so sharding it again through Ray buys nothing and only adds an RPC hop. + +### Mode 1: Embedded `OpenEnv` + +Bypasses OpenEnv's FastAPI server and constructs the Environment in-process — zero network overhead. ```python from twinkle_agentic.envs.openenv import OpenEnv env = OpenEnv( - base_url='http://localhost:8000', - env_cls='coding_env.CodingEnv', # Optional typed client - env_kwargs={'message_timeout_s': 30}, - tool_schema=[...], # Optional tool definitions + env_name='openspiel_env', # Package name; Environment / Action classes auto-discovered + env_kwargs={'game_name': 'blackjack'}, # Passed to the Environment constructor ) +result = env.reset() +result = env.step('play', {'action': 'hit'}) ``` -### Parameters +| Parameter | Type | Description | +|-----------|------|-------------| +| `env_name` | `str` | OpenEnv package name (e.g. `'coding_env'`). The `*Environment` class is discovered from `.server` and the `*Action` class from the package's `__all__`. | +| `env_cls` | `str` or class | Explicit Environment class (`'module:ClassName'`), used instead of `env_name`. | +| `env_kwargs` | `Dict` | Kwargs for the Environment constructor. | +| `action_cls` | `str` or class | Explicit Action class; auto-discovered from `env_name` when omitted. | +| `action_mapper` | `Callable` | `(tool_name, arguments) -> action`. Defaults to passing the tool arguments as Action fields. | + +Note that embedded `OpenEnv` does not implement `tools()`, so it inherits the base class's empty list and `EnvTool.from_env(env)` falls back to a single generic `env_action` tool. Pass explicit schemas to `EnvTool` (or subclass and override `tools()`) when the model needs to see the real action names. + +### Mode 2: Server `OpenEnvClient` + +First, run the OpenEnv environment as an ordinary HTTP/WebSocket service on the environment host (no Docker required): + +```bash +pip install openenv +pip install -e /path/to/OpenEnv/envs/coding_env +uvicorn coding_env.server.app:app --host 0.0.0.0 --port 8000 --workers 4 +``` + +Then create one client per trajectory on the training side. Each instance owns its own WebSocket session, and the server keeps a dedicated Environment instance for it: + +```python +from twinkle_agentic.envs.openenv import OpenEnvClient + +env = OpenEnvClient( + env_name='coding_env', # EnvClient subclass + Action class auto-discovered + base_url='http://10.0.0.5:8000', # Or set OPENENV_BASE_URL + message_timeout_s=120, # Raise it when the env runs long-executing code +) +env.reset() +result = env.step('run_python', {'code': 'print(1 + 1)'}) +print(result.observation) # '2' +env.close() +``` | Parameter | Type | Description | |-----------|------|-------------| -| `base_url` | `str` | URL of the running OpenEnv server. | -| `env_cls` | `str` or class | Dotted import path or class for a typed client. `None` uses `GenericEnvClient`. | -| `env_kwargs` | `Dict` | Extra kwargs for the client constructor. | -| `tool_schema` | `List[ToolInfo]` | Tool definitions exposed via `tools()`. | -| `action_mapper` | `Callable` | Custom function to map `(tool_name, args)` to the action dict sent to the server. | +| `env_name` | `str` | OpenEnv package name; the `EnvClient` subclass and `*Action` class are auto-discovered from it. | +| `env_cls` | `str` or class | Explicit client class (`'module:ClassName'`), used instead of `env_name`. | +| `base_url` | `str` | Server address, `http(s)://` or `ws(s)://` (converted automatically). Falls back to `OPENENV_BASE_URL`. A load-balancer address works here too. | +| `action_cls` | `str` or class | Action class; auto-discovered when omitted. | +| `action_mapper` | `Callable` | `(tool_name, arguments) -> action`, returning an Action instance or a dict of fields. | +| `tools` | `List[ToolInfo]` | Tool schemas exposed to the model. Defaults to a single `run_python(code)`, matching OpenEnv's code environments. | +| `reset_kwargs` | `Dict` | Kwargs forwarded to the server's `reset()` (e.g. `task_prompt` / `expected_answer` for `repl_env`). Also settable per episode via the `env.reset_kwargs` attribute. | +| `connect_timeout_s` | `float` | WebSocket connect timeout, default 10s. | +| `message_timeout_s` | `float` | Per-message timeout, default 120s. | +| `client_kwargs` | `Dict` | Extra kwargs for the OpenEnv client constructor. | + +Additional capabilities: + +- `register_tool(tool_info, handler)`: register a tool handled **locally on the client** instead of being sent to the server. The typical use is a bookkeeping tool such as `submit_solution`, which records the model's answer on the env for the training loop to score later. A tool of the same name shadows the default one. +- `execute(action)`: send an action and return the server's **raw** `StepResult`, so you can read typed fields such as `exit_code`. The training loop uses it to run scoring code inside the same session. +- `episode_reward` / `last_result` / `client`: cumulative reward, last raw result, and the underlying synchronous client. + +**Capacity and concurrency**: `OpenEnvClient` calls the OpenEnv client's `.sync()`, giving every instance a dedicated background event loop, so a thread pool of concurrent resets/steps is safe. But the server must have room for every concurrent session: the environment class has to declare `SUPPORTS_CONCURRENT_SESSIONS = True` and `create_app(..., max_concurrent_envs=N)` must be large enough, otherwise extra connections are rejected. Total capacity is `workers x max_concurrent_envs`, since the limit applies per worker process. OpenEnv's bundled `coding_env` leaves `SUPPORTS_CONCURRENT_SESSIONS` at the conservative default and needs to be subclassed to lift it (`create_app` raises `ConcurrencyConfigurationError` for `max_concurrent_envs > 1` otherwise); `cookbook/rl/envs/openenv_server/server_app.py` shows the full pattern. ### Usage with Rollout +Downstream usage is the same for both modes: + ```python -from twinkle_agentic.envs.openenv import OpenEnv from twinkle_agentic.envs.env_tool import EnvTool from twinkle_agentic.tools.tool_manager import ToolManager from twinkle_agentic.rollout.api_multi_turn import APIMultiTurnRollout -# Set up environment -env = OpenEnv(base_url='http://localhost:8000', tool_schema=[...]) env.reset() # Bridge to ToolManager @@ -143,7 +204,119 @@ rollout = APIMultiTurnRollout(api=api, tool_manager=manager, max_turns=10) results = rollout(trajectories) ``` -### Implementing a Custom Environment +See [Agentic RL Deployment and Training](../../Usage%20Guide/Agentic-RL-Deployment-and-Training.md) for an end-to-end multi-turn GRPO example. + +## AgentEnv: Firecracker microVM Sandboxes + +`AgentEnv` is a client-side `Env` over an [AgentENV](https://github.com/kvcache-ai/AgentENV) deployment, which runs Firecracker microVM sandboxes behind an E2B-compatible HTTP API. One sandbox is created per episode, giving the model **real operating-system semantics**: a genuine CPython interpreter, a writable filesystem, subprocesses, and `pip install`. + +Compared with the OpenEnv adapters: + +| | `OpenEnvClient` | `AgentEnv` | +|---|---|---| +| Isolation | Process / container | microVM (KVM), destroyed on teardown | +| Executor | smolagents AST interpreter | Real CPython | +| Memory per environment | KBs | ~1GB | +| Files / pip / subprocesses | Not supported | Supported | +| Transport | Persistent WebSocket | Stateless HTTP (E2B SDK) | +| Prerequisites | An OpenEnv service | AgentENV server + a built template + `/dev/kvm`, kernel 6.8+ | + +OpenEnv's `coding_env` runs on smolagents' `LocalPythonExecutor`, **an AST interpreter rather than an OS-level sandbox**. It does not handle `decorator_list` at all, so **decorators are silently ignored**: `@patch` has no effect, the test does not error, and the reward comes out as a plausible-looking wrong number. Such silent errors are harder to diagnose than a crash. It is a good fit for enforcing an import allowlist, but not for executing adversarial code. When tests rely on decorators, or the model must write files, install packages, or spawn subprocesses, use `AgentEnv`. + +Three things must be in place before training (all one-time, outside the training loop): the AgentENV server is deployed, a template is built (`aenv pull ubuntu:22.04 --name my-env`), and `pip install e2b` has been run on the training side. + +```python +from twinkle_agentic.envs import AgentEnv + +env = AgentEnv( + template='my-env', # Template built beforehand with the aenv CLI + api_url='http://10.0.0.5:8000', # Server or gateway; falls back to E2B_API_URL + sandbox_timeout=600, # Must outlast one episode plus any test replay +) +env.reset() # Boots a fresh sandbox; the scheduler picks the node +result = env.step('run_command', {'command': 'python -c "print(1 + 1)"'}) +print(result.observation) # '2' +env.close() # Kills the sandbox +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `template` | `str` | AgentENV template name/ID. Required — build it first via `aenv build` / `aenv pull`. | +| `api_url` | `str` | Server or gateway base URL. Falls back to `E2B_API_URL`. | +| `api_key` | `str` | Any non-empty string works, since AgentENV performs no authorization. Falls back to `E2B_API_KEY`, defaulting to `'dummy'`. | +| `sandbox_timeout` | `int` | Sandbox idle timeout in seconds, default 300. Idle sandboxes are **paused**, not killed, and auto-resume on access. | +| `command_timeout` | `int` | Per-command timeout in seconds, default 120. | +| `setup_commands` | `List[str]` | Commands run once after each `reset`; their output becomes the reset observation. | +| `sandbox_envs` | `Dict[str, str]` | Environment variables injected into the sandbox. | +| `metadata` | `Dict[str, str]` | Sandbox metadata, visible in the list APIs — useful for tagging a run name or trajectory id. | +| `refresh_timeout` | `bool` | Extend the timeout after every step, default `True`, so long episodes are not paused mid-flight. | +| `include_default_tools` | `bool` | Expose the built-in `run_command` / `write_file` / `read_file`, default `True`. | + +### Registering task tools + +The AgentENV server defines no tools of its own; it only provides capability primitives (arbitrary command execution, file I/O, port proxying). Tools are therefore a purely client-side concept, registered in one of two ways: + +```python +# 1. Shell command template, formatted with the tool arguments +env.register_command_tool( + {'type': 'function', 'function': { + 'name': 'run_tests', + 'description': 'Run the task test suite.', + 'parameters': {'type': 'object', + 'properties': {'test_file': {'type': 'string'}}, + 'required': ['test_file']}}}, + 'cd /workspace && pytest {test_file} -x -q') + +# 2. Arbitrary Python handler: handler(env, arguments) -> str +def _submit(env, arguments): + env.submitted_code = arguments.get('code', '') + return 'Solution submitted.' + +env.register_tool(submit_schema, _submit) # Both return self, so calls can be chained +``` + +Setting `include_default_tools=False` hides the built-ins so the action space matches the task exactly, which keeps reward attribution clean. A registered tool whose name collides with a built-in overrides it. + +Additional capabilities: + +- `run_command(arguments)`: public, so custom handlers can reuse it. A non-zero exit code is **not** raised; stdout, stderr, and the exit code are formatted into the observation so the model can react to the failure. +- `sandbox` / `sandbox_id`: the underlying E2B handle (giving access to PTY, file watching, and similar) and the current sandbox id. +- Observations are truncated to 32K characters to keep a runaway command from blowing up the context. + +**Error handling and rewards**: `step` never raises. Tool errors come back as `observation='Error: ...'` with `done=False`, letting the rollout loop continue or the model recover, bounded by `max_turns`. The sandbox produces no reward — `evaluate` returns zeros by default — so score trajectories in the training loop, or subclass and override `step` / `evaluate`. + +**Concurrency**: unlike `OpenEnv` / `EnvPool`, `AgentEnv` is deliberately **not** a `@remote_class`. Sandbox placement, load balancing, pause/resume, and node failover are all handled server-side by AgentENV's gateway/scheduler/orchestrator, so the adapter is a stateless HTTP client that can be instantiated directly inside rollout workers. Do **not** wrap it in `EnvPool`. Because `reset()` blocks on a network call while a sandbox boots, create trajectories concurrently from a thread pool. + +The primary capacity constraint is memory: concurrent sandboxes equal `batch_size x num_generations`, and each consumes the template's `--memory-mb` (the cookbook builds with `--memory-mb 1024`). See [Agentic RL Deployment and Training](../../Usage%20Guide/Agentic-RL-Deployment-and-Training.md) for the deployment steps, memory budget, and troubleshooting. + +## EnvPool: Distributed Environment Pool + +`EnvPool` is a `@remote_class` that shards `pool_size` **embedded** `OpenEnv` instances across Ray workers. Each worker manages `ceil(pool_size / world_size)` slots (the last shard is clipped to `pool_size`, so it may hold fewer), and `reset` / `step` are routed to the owning worker automatically via `remote_function`. + +```python +from twinkle_agentic.envs.openenv import EnvPool + +pool = EnvPool( + pool_size=64, + device_mesh=mesh, + env_kwargs={'env_name': 'openspiel_env', 'env_kwargs': {'game_name': 'blackjack'}}, +) + +# Each slot becomes a standard Env, ready for EnvTool / ToolManager +envs = pool.get_adapters(64) +env_tools = EnvTool.from_env(envs[0]) +``` + +| Method | Description | +|--------|-------------| +| `reset(idx)` / `step(idx, tool_name, arguments)` | Operate on a single slot. | +| `reset_batch(indices)` / `step_batch(indices, tool_names, arguments_list)` | Batch operations covering many slots in one RPC, returned in `indices` order. | +| `get_adapters(n)` | Wrap the first `n` slots as `EnvPoolAdapter` (a standard `Env`) on the driver side. Raises if `n > pool_size`. | +| `close()` | Close all environments. | + +`EnvPoolAdapter` implements the standard `Env` interface and proxies `reset` / `step` to the owning worker. On a `step` failure it returns `done=True` with the error in `info['error']`, so one broken environment does not stall the whole rollout batch. Its own `close()` is a no-op — release resources through the pool's `close()`. + +## Implementing a Custom Environment ```python from twinkle_agentic.envs.base import Env, StepResult diff --git a/docs/source_en/Components/Agentic/Multi-Turn-Tool-Usage.md b/docs/source_en/Components/Agentic/Multi-Turn-Tool-Usage.md index 06a962a33..24296f584 100644 --- a/docs/source_en/Components/Agentic/Multi-Turn-Tool-Usage.md +++ b/docs/source_en/Components/Agentic/Multi-Turn-Tool-Usage.md @@ -143,25 +143,26 @@ rollout = APIMultiTurnRollout(api=api, tool_manager=manager, max_turns=10) ## Using OpenEnv Environments -Connect to a remote OpenEnv WebSocket server: +Connect to a running OpenEnv server (each client instance maps to one server-side session): ```python -from twinkle_agentic.envs.openenv import OpenEnv +from twinkle_agentic.envs.openenv import OpenEnvClient from twinkle_agentic.envs.env_tool import EnvTool -env = OpenEnv( +env = OpenEnvClient( + env_name='coding_env', base_url='http://localhost:8000', - env_cls='coding_env.CodingEnv', - tool_schema=[{ + tools=[{ 'type': 'function', 'function': { - 'name': 'submit', - 'description': 'Submit code solution.', + 'name': 'run_python', + 'description': 'Execute Python code in the remote interpreter.', 'parameters': { 'type': 'object', 'properties': { 'code': {'type': 'string'}, }, + 'required': ['code'], }, }, }], @@ -172,6 +173,8 @@ env_tools = EnvTool.from_env(env) manager = ToolManager(env_tools) ``` +To run the environment inside the training process instead (no isolation, zero network overhead), swap `OpenEnvClient` for `OpenEnv(env_name=..., env_kwargs=...)`. See [Environments](./Envs.md) for a full comparison of the two modes. + ## Per-Trajectory Tool Managers For scenarios where each trajectory needs its own tool set (e.g., trajectory-bound state): diff --git a/docs/source_en/Usage Guide/Agentic-RL-Deployment-and-Training.md b/docs/source_en/Usage Guide/Agentic-RL-Deployment-and-Training.md new file mode 100644 index 000000000..d0c2cb218 --- /dev/null +++ b/docs/source_en/Usage Guide/Agentic-RL-Deployment-and-Training.md @@ -0,0 +1,770 @@ +# Agentic RL Deployment and Training + +Agentic RL has two parts: the execution environment where the model performs actions, and the GRPO training loop that consumes the resulting trajectories. This guide covers the deployment of each and how they are wired together. + +The two are orthogonal: switching execution backends requires only a change to the `CODE_RL_BACKEND` environment variable, and moving to a separate host requires only a change to one URL — the training code stays untouched. Validate the full pipeline on a single machine first, then scale out. + +The examples throughout use [`cookbook/rl/envs/`](https://github.com/modelscope/twinkle/tree/main/cookbook/rl/envs): a multi-turn code generation task on the MBPP dataset, where one `train.py` supports both backends. The task, tools, and reward formula are identical; only the location of code execution differs, so differences between experiment curves can be attributed to the execution environment itself. + +## Choosing an Execution Backend + +The three options differ mainly in isolation level and its memory cost. + +| | OpenEnv embedded | OpenEnv server | AgentENV | +|---|---|---|---| +| Where the environment runs | Inside the training process | A standalone HTTP/WebSocket service | A dedicated Firecracker microVM | +| Isolation | None | Process / container | microVM (KVM) | +| Executor | Depends on the environment package | smolagents AST interpreter | Real CPython | +| Memory per environment | KBs | KBs | **~1GB** | +| Files / pip / subprocesses | Depends on the environment package | Not supported | Supported, wiped on teardown | +| Deployment cost | Zero | One `uvicorn` command | Requires a control plane + `/dev/kvm` | + +Selection criteria: + +**Pure-computation environments** (board games, text games, scoring logic that can be safely evaluated inside the training process) — use embedded OpenEnv, with zero deployment. + +**Code execution is required but the standard library suffices** — use OpenEnv server mode. It is three orders of magnitude lighter than a microVM; booting a virtual machine to execute a few lines of `sorted()` is not worth the cost. + +**`unittest` + `@patch` is required, or the model needs to write files, install packages, or spawn subprocesses** — AgentENV is the only option. There is an easily overlooked failure mode here: OpenEnv's `coding_env` is built on smolagents' `LocalPythonExecutor`, **an AST interpreter rather than an OS-level sandbox**. It does not handle `decorator_list` at all, so **decorators are silently ignored** — `@patch` has no effect, the test does not fail, and the reward yields a plausible-looking but incorrect number. Silent errors of this kind are harder to locate than crashes. It is suitable for enforcing an import allowlist, not for executing adversarial code. + +For a full comparison of backend capabilities, see [Execution Environments](../Components/Agentic/Envs.md). + +--- + +# Part One: Deploying the Execution Environment + +## OpenEnv + +### Embedded: no deployment required + +The environment is instantiated directly inside the training process, with no network hop: + +```bash +cd cookbook/rl/multi_turn && python multi_turn_grpo.py +``` + +When there are many environment instances and CPU becomes the bottleneck, use `EnvPool` to move them onto a dedicated CPU `DeviceGroup`, keeping them off the GPU process's memory and GIL: + +```bash +ENV_REMOTE=1 ENV_NUM_WORKERS=8 ENV_POOL_SIZE=64 python multi_turn_grpo.py +``` + +Sharding only actually happens with `ENV_REMOTE=1`; without it, environments run locally in the driver (zero RPC). `ENV_POOL_SIZE=0` means the number of trajectories is used automatically. + +`EnvPool` is only meaningful for embedded OpenEnv. **Do not place `OpenEnvClient` or `AgentEnv` inside an `EnvPool`** — their session / sandbox lifecycle lives on the server side, so having Ray shard them again yields no benefit and merely adds an RPC hop. + +### Server mode + +The environment host needs no GPU, KVM, or Docker: + +```bash +cd cookbook/rl/envs +sh openenv_server/install.sh # pip install openenv + coding_env from source +sh openenv_server/serve.sh # 4 workers x 64 sessions = 256 concurrent +``` + +`coding_env` is a subpackage inside the OpenEnv repository that is not published to PyPI, so it can only be installed from source — `server_app.py` imports `PythonCodeActEnv` and `PyExecutor` from it, and smolagents is pulled in through it as well. + +The training host only needs `pip install openenv`, which provides the client classes; `coding_env` is not required there. + +### Why not use the upstream server + +`serve.sh` starts [the `server_app.py` in this directory](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/envs/openenv_server/server_app.py) rather than the upstream `coding_env.server.app`. Adopting upstream directly runs into three problems: + +```python +class ConcurrentCodeEnv(PythonCodeActEnv): + # Upstream defaults to False; create_app(max_concurrent_envs > 1) then raises + # ConcurrencyConfigurationError and the server is locked to a single session + SUPPORTS_CONCURRENT_SESSIONS = True + + def reset(self, **kwargs): + # The parent reset() rebuilds executor and transform from upstream + # defaults, so they must be reconfigured afterwards + observation = super().reset() + self._configure() + return observation + + def _configure(self) -> None: + # Upstream only authorizes "import json"; math / collections, both common + # in MBPP, would fail outright + self._executor = PyExecutor(additional_imports=list(ALLOWED_IMPORTS)) + # Upstream's create_safe_coding_transform() overwrites observation.reward + # with code-style heuristics (-1.0 on seeing open( / import os, +0.1 for + # short code). This task's reward comes from unit tests; style scores + # sharing the same channel would only be noise + self.transform = None +``` + +Enabling `SUPPORTS_CONCURRENT_SESSIONS` is safe: `create_app` receives the **class** (used as a factory), each WebSocket connection creates a new instance, and both executor and state are instance-private. + +Note also that the HTTP `/step` and `/reset` endpoints are unsuitable for multi-turn scenarios: OpenEnv creates a new env per request on these two endpoints and calls `close()` immediately after returning, so no state is retained. Multi-turn episodes must use WebSocket, which `OpenEnvClient` already implements. + +### Capacity + +``` +concurrent session limit = WORKERS x MAX_CONCURRENT_ENVS # default 4 x 64 = 256 +must be >= BATCH_SIZE x NUM_GENERATIONS # default 4 x 8 = 32 +``` + +Connections beyond capacity are rejected by the server, which shows up as some trajectories in a rollout batch having observations that are entirely `Error:`. When raising `--batch-size` / `--num-generations`, scale `WORKERS` or `MAX_CONCURRENT_ENVS` accordingly. + +## AgentENV + +[AgentENV](https://github.com/kvcache-ai/AgentENV) uses Firecracker microVMs to provide real operating-system semantics, which suits tool-integrated reasoning and SWE-style tasks. + +### Host prerequisite check + +All three of the following must hold: + +```bash +uname -r # >= 6.8 +ls -l /dev/kvm # must exist and be read-writable +modinfo ublk_drv >/dev/null && echo ublk-ok # the ublk kernel module is required +``` + +Bare metal generally satisfies these directly. Cloud VMs and GPU instances require nested virtualization to be enabled by the provider, which is off by default in most cases. Managed environments such as containers, K8s Pods, and notebook services depend on the **host's** kernel and `/dev/kvm`, and additionally need privileged mode with `/dev` mounted — when the host does not qualify, no software workaround inside the container can get around it. + +When the prerequisites are not met, no code changes are needed: deploy AgentENV separately on a machine that does qualify, and point `AENV_API_URL` at that address from the training side. Firecracker **does not support GPU passthrough**, so GPU workloads cannot run inside the sandbox and the environment host does not need a graphics card. + +### Installing the server + +```bash +cd cookbook/rl/envs +sh agentenv_server/install.sh +``` + +The script performs four steps in order: install the server and the `aenv` CLI, provision the host via `server --setup-host` (kvm group, ublk module, udev rules, sysctl), prepare the runtime directories, and build the sandbox template. + +Underneath it calls AgentENV's official `install.sh`, which creates a dedicated non-root `aenv` account (granted only `CAP_NET_ADMIN`/`CAP_SYS_ADMIN` and the kvm group), downloads runtime assets such as Firecracker and the guest kernel, and registers a systemd service. The data directory defaults to `/var/lib/aenv` and holds both image layers and snapshots, so pointing it at a large disk is recommended: + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh \ + | sudo AENV_HOME_PATH=/data/aenv bash +``` + +When only the client is needed (creating templates from the training host, or deploying via Docker): + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | sudo bash +``` + +### Building a sandbox template + +A template is the equivalent of a sandbox's factory image: dependencies are pre-installed and frozen into a snapshot, which is what allows a sandbox to start in roughly 50ms. **Dependencies must be baked into the template** rather than installed with `pip install` at training time — repeating a several-tens-of-seconds install on every trajectory is unacceptable overhead. + +`install.sh` already builds the template. To rebuild it manually: + +```bash +sh agentenv_server/install.sh --rebuild # deletes the old template first +``` + +Or use the CLI directly: + +```bash +aenv auth +# AENV server URL [http://localhost:8000]: http://127.0.0.1:8000 +# API key: dummy # any non-empty string works for a local deployment + +aenv build agentenv_server/Dockerfile -t twinkle-code --cpu-count 1 --memory-mb 1024 +aenv template watch # wait for ready +``` + +`aenv build` supports `FROM / RUN / ENV / WORKDIR / USER` (`ENTRYPOINT` becomes the start command; `EXPOSE / VOLUME / LABEL` are ignored). Both the image pull and the overlaybd conversion happen **on the server**, so the local machine does not need docker. A rebuild is only necessary when the Dockerfile changes. To use an existing image without modification: `aenv pull ubuntu:22.04 --name ubuntu`. + +Aliases cannot be rebound. On `alias 'xxx' already points to ...`, run `aenv template delete xxx` first — `--rebuild` wraps exactly this step. + +Implementations of complex tools are also best baked into the template, leaving the tool handler as a single call: + +```dockerfile +COPY tools/search.py /opt/tools/search.py +``` + +The tool implementation is then version-controlled, distributed with the snapshot, and free at runtime. + +### Starting the service + +```bash +sh agentenv_server/serve.sh # foreground, binds 127.0.0.1:8000 +NOHUP=1 sh agentenv_server/serve.sh # background, logs to /tmp/aenv-server.log +API_ADDR=0.0.0.0:8000 sh agentenv_server/serve.sh # external; read the security note below first +``` + +The script stops any running instance before starting, so restarting requires no manual kill. + +The server's own default listener is `0.0.0.0:8000`; the script narrows this to loopback. **AgentENV provides no authentication whatsoever**, so a reachable port is equivalent to reachable arbitrary code execution — binding `0.0.0.0` requires a security-group allowlist. + +Verify: + +```bash +curl -i http://127.0.0.1:8000/health # expect 204 +``` + +Single-machine setups do not need AgentENV's gateway / scheduler (both target multi-node deployments); connect directly to the server's `:8000`. + +### Memory budget + +Memory is AgentENV's primary capacity constraint: + +``` +concurrent sandboxes = BATCH_SIZE x NUM_GENERATIONS +memory required = concurrent sandboxes x template --memory-mb + 8GB (AgentENV itself + system) +``` + +At `--memory-mb 1024`, the default 32 concurrent sandboxes need 40GB; during pipeline validation, `2 x 4 = 8` concurrent sandboxes need only 16GB, so there is no need to procure a large-memory machine up front. + +When memory is short, adjust in this order: lower the template `--memory-mb` (512MB suffices for most tasks) → reduce `batch_size` → rely on automatic hibernation (idle sandboxes return memory to the host once paused). + +CPU contention also needs consideration: sandboxes compete for CPU cores with the dataloader and tokenizer, so `ENV_CONCURRENCY` should not exceed the number of idle cores. + +## Networking and Security + +twinkle supports two connection methods: **direct HTTP** for the same machine or the same private network / VPC, and **SSH port forwarding** across networks. + +The framework does not introduce VPN or NAT-traversal components — those belong to the network infrastructure layer and are unrelated to the training code. All that is visible from twinkle's side is an `http://host:port`. + +### Direct HTTP + +In enterprise setups, GPU and CPU machines are usually already in the same VPC / IDC / K8s cluster, so no extra network components are needed. `HOST` in `openenv_server/serve.sh` defaults to `0.0.0.0` (in which case the script prints a no-authentication warning), so bind the **private NIC** explicitly: + +```bash +HOST=10.0.1.20 sh openenv_server/serve.sh # OpenEnv +API_ADDR=10.0.1.20:8000 sh agentenv_server/serve.sh # AgentENV +``` + +The training side only needs one environment variable changed: + +```bash +OPENENV_BASE_URL=http://10.0.1.20:8000 sh run_openenv.sh +AENV_API_URL=http://10.0.1.20:8000 sh run_agentenv.sh +``` + +Security-group ingress should admit **only the training host's IP/32 or its security-group ID**, open only port 8000, and never use `0.0.0.0/0`. Confirm the binding took effect with `ss -tlnp | grep 8000` — the output must be `10.0.1.20:8000`, not `0.0.0.0:8000`. + +The environment service needs a keep-alive mechanism: once it goes down, the entire rollout batch is wasted. This belongs to operations infrastructure, so use whatever process manager you already have. A minimal systemd unit: + +```ini +# /etc/systemd/system/openenv-server.service +[Service] +User=openenv +WorkingDirectory=/opt/twinkle/cookbook/rl/envs/openenv_server +Environment=HOST=10.0.1.20 MAX_CONCURRENT_ENVS=64 +ExecStart=/bin/sh serve.sh +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +`Restart=always` is the essential line — it covers both a process crash and an OOM kill. Enable it with `sudo systemctl enable --now openenv-server`. + +### SSH port forwarding + +Across networks, SSH here does more than establish connectivity — it **adds a layer of authentication to a service that has none**, reusing existing SSH keys and login auditing, and it is typically an operations channel already approved within the organization. + +Port forwarding operates at the TCP layer and is fully transparent to WebSocket. Verified in practice: the handshake, session state across messages, and 8 concurrent sessions multiplexed over one connection all work correctly. + +```bash +# Environment side: loopback only, unreachable from the network +HOST=127.0.0.1 sh openenv_server/serve.sh + +# Training side +ssh -N -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes \ + -L 8000:127.0.0.1:8000 user@env-host & +OPENENV_BASE_URL=http://127.0.0.1:8000 sh run_openenv.sh +``` + +`ssh -N` not returning is normal — that blocked state is the tunnel working; add `-f` to move it to the background. + +Two addresses are easily confused: `OPENENV_BASE_URL` takes the **local entry point** `127.0.0.1:8000`, while the `127.0.0.1:8000` in `-L` is the target address as resolved **from the environment host's perspective**. The same applies to AgentENV via `AENV_API_URL`. + +Long-running jobs must have keep-alive configured: if the forward drops, the entire rollout batch is wasted, so it should not run in an interactive shell: + +```bash +autossh -M 0 -N -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \ + -L 8000:127.0.0.1:8000 user@env-host +``` + +All traffic is multiplexed over one TCP connection. Each turn transfers only a few KB of code and output; 8 concurrent sessions showed no pressure in practice. At hundreds of concurrent sessions, multiple forwards mapped to different local ports can spread the load. + +### Sandbox egress + +Restricting what code inside the sandbox can reach prevents model-generated code from scanning the internal network (SSRF) or abusing bandwidth. AgentENV provides a node-level enforced policy in `config/default.toml`, and the defaults are already reasonable: + +```toml +[network.egress] +# A sandbox's own egress policy cannot override these +always_denied_cidrs = [ + "10.0.0.0/8", "100.64.0.0/10", "127.0.0.0/8", + "169.254.0.0/16", # <- critical: blocks the cloud metadata service 169.254.169.254 + "172.16.0.0/12", "192.168.0.0/16", +] +``` + +The `169.254.0.0/16` entry is the most critical one: it blocks cloud provider metadata services, preventing code inside the sandbox from stealing temporary IAM credentials and thereby gaining permissions over the entire cloud account. Standard-library algorithm problems need no outbound access, so the defaults can be used as-is. + +To relax this, pass a policy per sandbox: + +```python +# base_policy: Default | Allow | Deny +{'base_policy': 'Deny', + 'egress': {'allowed_domains': ['pypi.org', 'files.pythonhosted.org'], + 'denied_cidrs': ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '169.254.0.0/16']}} +``` + +One final point deserves emphasis: the API key used by `aenv auth` **provides no authentication**. Any non-empty string passes in a local deployment, and there is no tenant isolation — obtaining any sandbox-id grants read, write, and destroy access to that sandbox, and `GET /sandboxes` lists every sandbox. To expose this as a service, you must build your own authentication / tenancy / quota layer, keep AgentENV confined to a private network, and expose only your own upper-layer API. + +--- + +# Part Two: Wiring Up Training + +This part continues with `cookbook/rl/envs/`. Code generation was chosen as the task for two reasons: its reward can be computed objectively from unit tests, with no judge model required; and multi-turn interaction has intrinsic meaning — write, try, fix, submit. + +## The action space the model sees + +Only two tools are exposed to the model (defined in `_openenv.py` / `_agentenv.py`): `run_python(code)` executes code in the environment, and `submit_solution(code)` submits the final answer. + +The second tool deserves separate mention, as it **is not sent to the server**: + +```python +def _submit_solution(env, arguments: Dict[str, Any]) -> str: + code = (arguments.get('code') or '').strip() + if not code: + return "Error: 'code' argument is required." + env.submitted_code = code + return 'Solution submitted.' +``` + +This tool is handled locally on the client via `register_tool`, recording the source on the env purely for the training loop to score. This is a general pattern: **the model's actions and the bookkeeping training needs are two separate concerns**, and the latter belongs in a local handler rather than polluting the environment protocol. + +The system prompt must describe backend semantics accurately, otherwise the model writes code against the wrong mental model. The two backends are exactly **opposite** on this point: + +- OpenEnv session: `The interpreter keeps its state between calls` — a function defined this turn can be called directly in the next. It also lists the module allowlist and declares that there is no file or network access. +- AgentENV: `Each call runs in a FRESH process, so every snippet must be self-contained`. + +The AgentENV side additionally sets `include_default_tools=False` to disable the built-in `run_command` / `read_file` / `write_file`, aligning the action space precisely with the task to keep reward attribution clean. + +The tool set is a client-side concept. The AgentENV server itself **defines no tools**; it only provides capability primitives: arbitrary command execution, file read/write, and port proxying. Besides `register_tool(schema, handler)`, there is also the command-template form `register_command_tool`: + +```python +env.register_command_tool( + {'type': 'function', 'function': { + 'name': 'run_tests', + 'description': 'Run the task test suite.', + 'parameters': {'type': 'object', + 'properties': {'test_file': {'type': 'string'}}, + 'required': ['test_file']}}}, + 'cd /workspace && pytest {test_file} -x -q') # formatted with the tool arguments +``` + +The handler signature is `handler(env, arguments) -> str`, and the return value becomes the observation. Internally it can use `env.run_command(...)` and `env.sandbox` (the raw E2B handle, giving access to PTY, file watching, and similar). Registering under an existing name overrides the built-in tool. + +## One env per trajectory + +```python +def prepare_trajectories(samples, pool): + envs = [backend.make_env() for _ in samples] + # reset() blocks on the network (a WebSocket handshake, or booting a sandbox), + # so it must be concurrent — otherwise a batch of 32 waits serially + list(pool.map(lambda env: env.reset(), envs)) + + trajectories, tool_managers = [], [] + for sample, env in zip(samples, envs): + tool_managers.append(ToolManager(EnvTool.from_env(env))) + trajectories.append({ + 'messages': [ + {'role': 'system', 'content': backend.SYSTEM_PROMPT}, + {'role': 'user', 'content': sample['prompt']}, + ], + 'tools': backend.TOOL_SCHEMA, + }) + return trajectories, tool_managers, envs +``` + +**One `ToolManager` per trajectory.** It holds a specific env instance, so sharing one would send every trajectory's tool calls to the same session. + +**`envs` must be closed in a `finally` block.** They occupy server-side capacity, and leaking them causes subsequent steps to fail for lack of capacity: + +```python +try: + all_trajectories = rollout(expand_prompts, tool_manager=tool_managers) + total_rewards, pass_rates = extract_rewards(envs, expanded, env_pool) +finally: + close_envs(envs, env_pool) +``` + +One more easily overlooked behavior: `MultiTurnRollout` terminates when **the model stops emitting tool calls** (or when `MAX_TURNS` / the length limit is hit); it **does not read** `EnvTool.done`. The system prompt therefore needs an explicit instruction not to call tools after submitting, otherwise episodes run all the way to `MAX_TURNS`. + +## Reward: unit tests rather than a judge model + +Each MBPP sample ships with a number of `assert` statements. Hidden tests are replayed after the rollout finishes; **the script form is dictated by backend capability, but the reward formula is identical** for both backends. + +The AgentENV side is real CPython, so it generates an ordinary script directly, wrapping each assertion in `try` and printing `TESTS_PASSED n total` at the end. + +The OpenEnv side executes them one at a time **within the same session**, rewriting `assert X` into `print(X)`. The purpose is diagnosability: the exception raised by a failing `assert` is indistinguishable from a crash inside the solution, whereas printing separates "the assertion evaluated to False" from "the code crashed", and one test raising does not stop the rest from running. The executor itself does support `assert` and `try`; this is not a capability workaround. + +The OpenEnv side uses `env.execute()` rather than `env.step()`: `execute()` returns the server's **raw** `StepResult`, exposing structured fields such as `exit_code`, while `step()` returns text rendered for the model and counts toward the episode. Scoring logic should not appear in the model's conversation. + +Reward shaping: + +```python +rate = passed / total if total else 0.0 +if total and passed == total: + return 1.0, rate # all passed +if getattr(env, 'submitted_code', None): + return 0.1 + 0.4 * rate, rate # submitted, partially correct -> 0.1 ~ 0.5 +return 0.0, rate # not submitted +``` + +The shape guarantees "all correct > partially correct > submitted but all wrong > not submitted", and the 1.0 for all-correct is clearly above the 0.5 ceiling for partial credit — otherwise the model learns to submit a fake implementation that just barely passes the first test. + +**Giving the submit action a 0.1 floor** is deliberate: early in training all trajectory rewards are 0, GRPO's within-group advantages are then all 0, and no useful learning signal can be produced. That floor supplies the initial gradient signal. + +The sandbox itself produces no reward (`AgentEnv.evaluate` returns 0 by default); scoring happens uniformly in the training loop. To score inside the sandbox, have the tool emit a structured result and parse it on the driver side. + +## GRPO group layout + +```python +batch = [dataset[(sample_cursor + i) % len(dataset)] for i in range(BATCH_SIZE)] +expanded = [s for s in batch for _ in range(NUM_GENERATIONS)] # N copies of the same problem, contiguous +... +advantages = advantage_fn(total_rewards, num_generations=NUM_GENERATIONS, scale='group').tolist() +``` + +Note how `expanded` is written: `for s in batch for _ in range(N)` places the N rollouts of one problem **contiguously** in the list, and `GRPOAdvantage` splits groups according to that layout. Writing `for _ in range(N) for s in batch` misaligns the grouping completely — training appears to proceed normally while learning nothing useful. + +## Launching training + +Validate the full pipeline at low concurrency first, to save both time and compute cost: + +```bash +cd cookbook/rl/envs + +# OpenEnv +sh run_openenv.sh --batch-size 2 --num-generations 4 --max-steps 2 + +# AgentENV +sh run_agentenv.sh --batch-size 2 --num-generations 4 --max-steps 2 +``` + +Concurrency must not be too low, however: the trajectory count (`batch-size x num-generations`) must be >= `--model-gpus` (4 by default). Otherwise too few remain after length filtering and the whole batch is skipped, with the log merely repeating `skipping this batch` — a symptom that looks like a hang rather than an error, and is harder to diagnose than an outright crash. The `2 x 4 = 8` above leaves a 2x margin. + +Arguments only take effect when placed **after** the script name (the script runs `python train.py $TRAIN_ARGS "$@"`, and `"$@"` must come last to override). + +Scaling up to real training: + +```bash +OPENENV_BASE_URL=http://10.0.0.5:8000 MAX_TURNS=8 ENV_CONCURRENCY=32 \ +sh run_openenv.sh --batch-size 8 --num-generations 16 --max-steps 500 +``` + +Overridable environment variables: + +| Variable | Default | Description | +|---|---|---| +| `MAX_TURNS` | `6` | Maximum tool-calling turns per episode | +| `ENV_CONCURRENCY` | `16` | Driver threads for concurrent create / destroy / scoring | +| `OPENENV_BASE_URL` | `http://127.0.0.1:8000` | OpenEnv service address (a load balancer works too) | +| `OPENENV_ENV_NAME` | `coding_env` | Environment package name; determines the client and Action classes | +| `OPENENV_MESSAGE_TIMEOUT_S` | `120` | Per-message timeout | +| `AENV_API_URL` | `http://127.0.0.1:8000` | AgentENV address | +| `AENV_TEMPLATE` | `twinkle-code` | Template name | +| `SANDBOX_TIMEOUT` | `600` | Sandbox idle timeout (seconds) | +| `AENV_COMMAND_TIMEOUT` | `60` | Per-command timeout inside the sandbox (seconds) | + +Training hyper-parameters are passed as CLI arguments, with defaults defined in the respective `run_*.sh`. The `TRAIN_ARGS` of both backends must stay in sync, otherwise changes in reward cannot be attributed to the execution backend rather than to differing hyper-parameters. + +`AENV_API_URL` cannot be replaced by `E2B_API_URL` — setting only the latter silently falls back to the default. + +## Monitoring metrics + +The log prints one line per step: + +``` +[Step 0] {'train/code_acc': 0.031, 'train/test_pass_rate': 0.208, 'train/avg_reward': 0.145, ...} +``` + +Task metrics: + +| Metric | Meaning and interpretation | +|---|---| +| `train/code_acc` | pass@1, the fraction where all hidden tests pass. This is the metric that actually needs to improve | +| `train/test_pass_rate` | Mean per-test pass rate. Smoother than `code_acc`; watch this one first in the early phase | +| `train/avg_reward` | Mean of the shaped reward. If it rises while `code_acc` does not, the model is farming the 0.1 submit floor | +| `train/avg_turns` | Mean turn count. Sitting near `MAX_TURNS` means the model often exhausts its turns without submitting | +| `train/max_turns` / `train/min_turns` | Turn distribution within the batch. `max_turns` pinned at the limit means some trajectories are truncated; `min_turns` near 1 means trajectories often submit or give up on the first turn | + +Policy health (registered via `model.add_metric('GRPOMetric', ...)`): + +| Metric | Meaning and interpretation | +|---|---| +| `train/approx_kl` | KL between the old and new policy (Schulman K3 estimator). A sudden spike precedes policy collapse | +| `train/clip_ratio` | PPO clipping trigger rate (with `_low` / `_high` broken out by direction). Persistently high means single-step updates are too aggressive | +| `train/token_kl_max` / `train/token_ratio_max` | Per-token KL / probability-ratio extremes, used to localize a collapse | +| `train/policy_confidence` | `exp(mean_logp)`, the policy's mean confidence | + +`GRPOMetric`'s `epsilon` must match `set_loss('GRPOLoss', epsilon=...)`, otherwise the clipping threshold reflected in `clip_ratio` is not the one actually in effect. `train/entropy` only appears with `GRPOLoss(entropy_coef > 0)`, but that option adds an entropy bonus to the loss and **thereby changes the training objective**, so it should not be enabled merely to observe a metric. + +## Porting to your own task + +Reusing this skeleton typically requires changes in four places; the training loop, GRPO configuration, weight synchronization, and metrics all stay as they are: + +1. **Dataset**: replace `load_mbpp()` with your own loader, producing `{'prompt', ...fields needed for scoring}`. +2. **Tools**: `TOOL_SCHEMA` and handlers. Server capabilities go through the default action path; client-side bookkeeping uses `register_tool`. +3. **Reward**: `score()` inside `extract_rewards()`. Prefer signals that can be computed objectively (unit tests, exact match, executable validation), and consider a judge model only when no such signal exists. +4. **System prompt**: must describe backend semantics accurately (whether state persists across turns, which modules and permissions are available, the turn budget). + +To add a backend: write `_xxx.py` (providing `NAME`, `SYSTEM_PROMPT`, `TOOL_SCHEMA`, `make_env()`, `run_tests()`, `describe()`) along with `run_xxx.sh`, and add the name to `BACKENDS` in `train.py`. + +--- + +# Troubleshooting + +## Pre-flight checklist + +- [ ] The service is **not** bound to `0.0.0.0` (confirm with `ss -tlnp | grep 8000`) +- [ ] Direct private-network access: security-group ingress admits only the training host, not `0.0.0.0/0` +- [ ] Across networks: the environment side binds `127.0.0.1` and is reached only through SSH port forwarding +- [ ] Capacity >= `BATCH_SIZE x NUM_GENERATIONS` (for AgentENV, check memory separately) +- [ ] The environment service has keep-alive configured (systemd) and the SSH forward has keep-alive (autossh), rather than running in an interactive shell +- [ ] The full pipeline has been validated with `--batch-size 2 --num-generations 4 --max-steps 2` +- [ ] AgentENV's `always_denied_cidrs` has not been relaxed just to "make the sandbox reachable" + +## Environment side + +| Symptom | Cause and remedy | +|---|---| +| Some observations in a batch are entirely `Error:` | Insufficient server capacity. Verify `WORKERS x MAX_CONCURRENT_ENVS >= BATCH_SIZE x NUM_GENERATIONS` | +| The model reports `import math` is not allowed | The `ALLOWED_IMPORTS` allowlist did not take effect; confirm `_configure()` is called again after `reset()` | +| `ConcurrencyConfigurationError` | The upstream `coding_env.server.app` (single session) was started. Use `openenv_server/server_app.py` instead | +| Anomalous ±0.1 / -1.0 values mixed into the reward | The environment's reward transform was not disabled; check `self.transform = None` | +| Timeout / message-wait errors | Raise `OPENENV_MESSAGE_TIMEOUT_S`. The executor itself has three further limits: 30s wall-clock per execution, 10 million operations, and 1 million while-loop iterations, so the value must stay above 30s to be meaningful | +| `Address already in use` | The port is taken. Find the holding process with `ss -tlnp \| grep 8000`, or switch to `PORT=8001` | + +## AgentENV side + +| Symptom | Cause and remedy | +|---|---| +| `/dev/kvm is not accessible` | The runtime account is not in the kvm group, or the host has no KVM. Run `sudo server --setup-host --runtime-user aenv --runtime-group aenv`, then restart | +| `ublk_drv is not loaded` | Run `sudo modprobe ublk_drv`; kernels older than 6.8 need an upgrade | +| `ImportError: AgentEnv requires the E2B SDK` | `pip install e2b` | +| `Invalid API key format: expected "e2b_"` | Client-side validation in the e2b SDK. `AgentEnv` already sets `E2B_VALIDATE_API_KEY=false` by default, so a persisting error means it was explicitly overridden to `true` | +| `400: template xxx not found` | The template was not created, or the build has not reached ready. Check the state with `aenv template list` | +| `alias 'xxx' already points to ...` | Aliases cannot be rebound; run `aenv template delete xxx` first | +| `pip install` fails inside the sandbox | Egress is blocked by policy, or the current network cannot reach PyPI; pre-install in the template instead | +| A sandbox becomes unavailable mid-trajectory | `SANDBOX_TIMEOUT` is shorter than the trajectory duration and the sandbox was auto-paused. Raise it | +| Batch startup is slow | Raise `ENV_CONCURRENCY`; confirm dependencies are baked into the template rather than installed at runtime | +| Dependencies are re-downloaded on every restart | `AENV_HOME_PATH` was unset and fell back to `/tmp/aenv-test-/` (the test default in `run-with-capabilities.sh`), which `/tmp` cleanup removes. `serve.sh` pins it to `/var/lib/aenv` | +| `load config ... Permission denied` | A source-built binary hard-codes its build-time path as the default config location, which is unreadable under `/root` after dropping privileges to `aenv`. Point `AENV_CONFIG_PATH` at a copy readable by `aenv`; `serve.sh` already handles this | + +## Training side + +| Symptom | Cause and remedy | +|---|---| +| Too few valid trajectories, batch skipped | Most trajectories were filtered for being too long. Lower `MAX_TURNS` / `--max-tokens`, or tighten tool output (observations on the AgentENV side are truncated to 32K characters by default); also confirm the trajectory count is >= `--model-gpus` | +| Reward stays at 0 | First run `run_tests` alone against a known-correct solution to confirm the scoring path itself works, then investigate the model | +| The number of envs does not match `--batch-size` | Arguments were placed before `sh run_*.sh` and never reached `"$@"` | + +--- + +# Appendix: Workarounds for Restricted Networks + +This section is only relevant when **the default path does not work**, and applies to environments that cannot reach public sources such as Docker Hub, GitHub, and PyPI directly (isolated corporate networks, internal clusters, or regions with egress restrictions). If `install.sh` succeeded and the template is built, skip this section. + +The registry addresses and package sources below are examples; substitute the internal mirrors actually reachable from your environment. + +## Mirror sources + +The default Dockerfile depends on external networks in two places — `FROM python:3.11-slim` and `RUN pip install` — which must be handled separately. + +**Base image**: the server only searches the registries listed in `[image.resolver] search_registries`, which defaults to `docker.io` / `ghcr.io`. When neither is reachable, the typical errors are `dial tcp ...: i/o timeout` (timeout) or 401 (authentication required). Writing a fully qualified name bypasses the search list, and **`library/` must not be omitted** (the real path of official images on the Hub is `library/python`): + +```bash +BASE_IMAGE=/library/python:3.11-slim sh agentenv_server/install.sh +``` + +`BASE_IMAGE` is forwarded to `aenv build --image`, overriding `FROM` in the Dockerfile, so the file itself stays untouched. + +Probing candidate mirrors for reachability first avoids a wasted build. Both 200 and 401 count as reachable (401 means a token must be fetched first, which is normal), while a timeout means unavailable: + +```bash +for R in ; do + printf "%-24s " "$R" + timeout 15 curl -s -o /dev/null -w '%{http_code}\n' \ + "https://$R/v2/library/python/manifests/3.11-slim" +done +``` + +A status code alone is not proof of usability: some proxies complete the TLS handshake yet cannot serve a full manifest. To confirm, compare response sizes — a genuine multi-arch index is several KB, whereas an error page is typically under 200 bytes. + +Public proxy sites may become unavailable or rate-limited at any time and are unsuitable for long training runs. For production, mirror the images into your own container registry (Harbor, or a managed registry from your cloud provider) on the same private network as the environment host. + +**pip**: if the sandbox cannot reach pypi.org, set `ENV PIP_INDEX_URL` in the Dockerfile (the default Dockerfile does not include this line, so it must be added) and it must appear **before** `RUN pip install`. + +Use `aenv template watch ` to see why a build failed. Two common causes: + +- `all registry candidates failed during manifest fetch` — the base image cannot be pulled; substitute a mirror as above. +- `overlaybd-commit ... failed to perform commit(), 2: No such file or directory` — the image was pulled but conversion failed. First confirm `FROM` does not point at the wrong image (large images of tens of GB with hundreds of layers tend to stall here); once that is ruled out, check that overlaybd is fully installed: `/var/lib/aenv/deps/overlaybd/bin/` should contain the create / apply / commit / resize binaries, and `/etc/overlaybd/overlaybd.json` should exist. + +The log line `open /root/.regctl/config.json: permission denied` is only a WARN (the server drops privileges to `aenv` and then tries to read root's home directory) and does not affect pulling public images. Configuring private registry credentials, however, requires giving it a writable HOME first: + +```bash +sudo install -d -o aenv -g aenv /var/lib/aenv/home +``` + +## When install.sh cannot download: building from source + +`install.sh` depends on only two external endpoints: `api.github.com` for release metadata, and then `github.com/.../releases/download/` for two assets (`aenv-linux-x86_64` at 9.3MB and `aenv-server-linux-x86_64.tar.gz` at 68MB). The latter redirects (302) to `release-assets.githubusercontent.com`, which in some network environments is prone to being RST mid-transfer (`curl: (56) Recv failure` or `(92)`). + +Try a proxy first: re-run after `export https_proxy=...`, since `install.sh` uses curl throughout and honors the env proxy. Only without a proxy is the approach in this section necessary. + +One point to establish up front: **compiling only solves the Rust binary part.** Besides `server`, the prebuilt tarball also contains a `deps/` directory (firecracker, guest kernel, tools driver, overlaybd, regctl), none of which are build artifacts, so they must be prepared separately. This is where the approach most often runs into trouble. + +### Prerequisites + +If `static.rust-lang.org` / `crates.io` are unreachable, a usable crates mirror must be configured (`` / `` are placeholders below; substitute addresses reachable from your environment). **The variables must be set before installing rustup**; reversing the order requires starting over: + +```bash +# Write to the profile: AgentENV has a rust-toolchain.toml, so the first cargo +# build pulls the toolchain once more +cat >> ~/.bashrc <<'EOF' +export RUSTUP_DIST_SERVER= +export RUSTUP_UPDATE_ROOT=/rustup +EOF +. ~/.bashrc + +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +. "$HOME/.cargo/env" + +# Most mirrors only support the sparse index, requiring cargo >= 1.68. Write by +# overwriting: appending twice produces a duplicate [source.crates-io] and TOML +# parsing fails outright +tee "${CARGO_HOME:-$HOME/.cargo}/config.toml" >/dev/null <<'EOF' +[source.crates-io] +replace-with = 'mirror' + +[source.mirror] +registry = "sparse+/" + +[net] +retry = 5 + +[http] +timeout = 60 +EOF + +sudo apt-get update # upgrade does not refresh the index; skipping this gives Unable to locate package + +# Build dependencies (same as the builder stage of Dockerfile.agentenv). clang / +# libclang-dev are required by uvm-ublk-daemon: it uses bindgen to generate ublk +# kernel bindings, and without them you get Unable to find libclang +sudo apt-get install -y build-essential pkg-config libssl-dev \ + clang libclang-dev libprotobuf-dev protobuf-compiler + +# Runtime dependencies (same as the runtime-base stage). libaio1t64 is the +# Ubuntu 24.04 package name; on 22.04 / Debian 12 it is called libaio1 +sudo apt-get install -y ca-certificates curl dpkg e2fsprogs iproute2 iptables \ + jq libaio1t64 sudo umoci zstd +``` + +`protobuf-compiler` must come from the system package: `make ci-deps-protoc` downloads protoc from GitHub Releases, which is precisely the unreachable path. On Debian 13+, `pkg-config` has been renamed `pkgconf`. + +Whether the mirror took effect is visible on the first line of `cargo build`, which must read ``Updating `mirror` index``. If it still says `Updating crates.io index`, `config.toml` was not read — common causes are `CARGO_HOME` pointing elsewhere, or `~/.cargo` not existing when the file was written. Note that the `an/yh/anyhow` form in error messages is a sparse path, but cargo >= 1.70 uses sparse by default, so **it cannot be taken as evidence that the mirror is in effect**. + +### Building and installing + +```bash +git clone https://github.com/kvcache-ai/AgentENV.git && cd AgentENV +cargo build --release -p agentenv --bin server +cargo build --release -p aenv +cargo build --release -p uvm-ublk -p uvm-ublk-daemon + +sudo install -m 0755 target/release/aenv /usr/local/bin/aenv +sudo install -m 0755 target/release/server /usr/local/bin/server +sudo install -D -m 0755 target/release/uvm-ublk-daemon /var/lib/aenv/ublk/uvm-ublk-daemon +sudo install -D -m 0640 config/default.toml /var/lib/aenv/config/config.toml + +sudo groupadd --system aenv +sudo useradd --system --gid aenv --home-dir /var/lib/aenv \ + --no-create-home --shell /usr/sbin/nologin aenv +``` + +The location of that last `config.toml` is especially important: **a source-built binary hard-codes the build-time repository path as the default config location** (`CARGO_MANIFEST_DIR` in `cfg.rs`). Built under `/root/AgentENV`, it looks for `/root/AgentENV/config/default.toml`, which is inaccessible after dropping privileges to `aenv` because `/root` is mode 0700 — hence `Permission denied`. A copy must therefore be placed somewhere `aenv` can read, with `AENV_CONFIG_PATH` pointing at it on startup. `agentenv_server/serve.sh` already handles this. + +### Preparing deps + +```bash +E="AENV_CONFIG_PATH=/var/lib/aenv/config/config.toml AENV_HOME_PATH=/var/lib/aenv" +sudo env $E /usr/local/bin/server --setup-only +``` + +When downloads fail, first establish where the files come from — these five URLs are everything it fetches (defined in `config/deps_manifest.toml`): + +``` +firecracker + cpu-template-helper https://pub-4ee15c400f554ab7a9eac3f5bc8f53de.r2.dev/firecracker-1.15.1-patch-v1-x86_64.tgz +guest kernel https://pub-4ee15c400f554ab7a9eac3f5bc8f53de.r2.dev/vmlinux-6.1.175 +regctl https://github.com/regclient/regclient/releases/download/v0.11.5/regctl-linux-amd64 +overlaybd .deb https://github.com/containerd/overlaybd/releases/download/v1.0.18/overlaybd-1.0.18-20260710.cee2186.{target}.deb +tools.ext4 ghcr.io/zlzgithub-0801/agentenv-tools:0.1.0 (an OCI image; export with regctl/docker) +``` + +Do not hard-code `{target}`: `overlaybd.rs` reads `/etc/os-release` and substitutes `ubuntu1..` automatically. The only part that must be copied verbatim is the `1.0.18-20260710.cee2186` date-plus-git-hash segment. + +Download the failing assets on a machine with connectivity, then **pre-place the files** — `download_file` skips the download when the target file already exists and is non-empty, and the target path is the `dest=` value from the failure log: + +```bash +# Log: downloading url="https://pub-...r2.dev/firecracker-1.15.1-patch-v1-x86_64.tgz" +# dest=/var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker-1.15.1-patch-v1-x86_64.tgz +# scp to the dest= path (keeping the filename identical), then: +sudo chown -R aenv:aenv /var/lib/aenv/deps +sudo env $E /usr/local/bin/server --setup-only +``` + +This method works for all five dependencies without configuration changes. overlaybd has no local-path switch but responds to the same approach: place the `.deb` under `/var/lib/aenv/deps/overlaybd/downloads/` (filename taken from the last URL segment, with `{target}` already substituted) and `package_url` will never be accessed. + +Alternatively, edit `/var/lib/aenv/config/config.toml` to point at the directory holding the files. Note that the `[firecracker]`, `[kernel]`, and `[tools]` **sections already exist**, so the keys must be added inside them; **do not append duplicate sections** — a repeated table causes TOML parsing to fail outright: + +```toml +[firecracker] # existing section; boot_args etc. stay as-is +binary_path = "/opt/aenv-assets/firecracker" # the extracted binary, not the tgz +[kernel] # existing section (empty) +image_path = "/opt/aenv-assets/vmlinux.bin" +[tools] # existing section; control_plane_port stays as-is +version = "0.1.0" # must be paired with drive_path +drive_path = "/opt/aenv-assets/tools.ext4" +``` + +`/opt/aenv-assets` must be created by hand; it is not a pre-existing directory. + +### Starting the service + +```bash +sudo env $E /usr/local/bin/server --setup-host --runtime-user aenv --runtime-group aenv +sudo chown -R aenv:aenv /var/lib/aenv +sh cookbook/rl/envs/agentenv_server/serve.sh +``` + +Several environment variables in `serve.sh` cannot be omitted: `AENV_RUN_USER=aenv` (without it the three-level fallback `SUDO_USER` → repository owner → `aenv` applies, giving unstable results when run directly as root), `AENV_HOME_PATH=/var/lib/aenv` (without it the path falls back to `/tmp/aenv-test-/`, requiring a few hundred MB to be re-downloaded after cleanup), and `AENV_CONFIG_PATH` (the build-path issue described above). + +One more note: the change from `--setup-host` that adds `aenv` to the kvm group does not apply to existing sessions, which is why `run-with-capabilities.sh` re-initializes them via `--init-groups`. Starting the server directly, bypassing that script, fails for lack of kvm permissions. + +> The build-from-source path has not been validated end to end, and the reachability of the deps downloads varies by machine. + +## Docker deployment + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash +docker run -d --privileged -v /dev:/dev -p 8000:8000 ghcr.io/kvcache-ai/aenv-server:latest +``` + +Docker here is only a **deployment vehicle**; the sandbox is still a Firecracker microVM and still requires KVM on the host. Pulling the image from `ghcr.io` also goes through GitHub distribution, which may be equally unreachable on a restricted network. + +--- + +## Related Documents + +- Component reference: [Execution Environments](../Components/Agentic/Envs.md) (the `Env` abstraction, `EnvTool`, both OpenEnv modes, `EnvPool`) +- Multi-turn tool calling: [Multi-Turn Tool Usage](../Components/Agentic/Multi-Turn-Tool-Usage.md) +- Runnable examples: `cookbook/rl/envs/` (code task, both backends), `cookbook/rl/multi_turn/` (embedded OpenEnv) +- OpenEnv upstream repository: +- AgentENV official documentation: diff --git a/docs/source_en/index.rst b/docs/source_en/index.rst index 9ea9d2e6c..e1c4ba4b6 100644 --- a/docs/source_en/index.rst +++ b/docs/source_en/index.rst @@ -14,6 +14,7 @@ Twinkle DOCUMENTATION Usage Guide/Server and Client/index.rst Usage Guide/NPU-Support.md Usage Guide/Train-as-a-Service.md + Usage Guide/Agentic-RL-Deployment-and-Training.md Usage Guide/Introduction-with-Qwen3.5.md Usage Guide/Embedding-Training.md diff --git a/docs/source_zh/index.rst b/docs/source_zh/index.rst index 2a8c88aeb..57cc87346 100644 --- a/docs/source_zh/index.rst +++ b/docs/source_zh/index.rst @@ -14,6 +14,7 @@ Twinkle DOCUMENTATION 使用指引/服务端和客户端/index.rst 使用指引/NPU的支持.md 使用指引/训练服务.md + 使用指引/Agentic RL部署与训练.md 使用指引/Qwen3.5最佳实践.md 使用指引/Embedding训练.md diff --git "a/docs/source_zh/\344\275\277\347\224\250\346\214\207\345\274\225/Agentic RL\351\203\250\347\275\262\344\270\216\350\256\255\347\273\203.md" "b/docs/source_zh/\344\275\277\347\224\250\346\214\207\345\274\225/Agentic RL\351\203\250\347\275\262\344\270\216\350\256\255\347\273\203.md" new file mode 100644 index 000000000..0841d46cd --- /dev/null +++ "b/docs/source_zh/\344\275\277\347\224\250\346\214\207\345\274\225/Agentic RL\351\203\250\347\275\262\344\270\216\350\256\255\347\273\203.md" @@ -0,0 +1,765 @@ +# Agentic RL 部署与训练 + +Agentic RL 需要两个部分：供模型执行动作的执行环境，以及消费这些轨迹的 GRPO 训练循环。本文依次介绍两者的部署与对接。 + +两者相互正交：更换执行后端仅需修改 `CODE_RL_BACKEND` 环境变量，跨机部署仅需修改一个 URL，训练代码无需变动。因此建议先在单机验证完整链路，再扩展到多机。 + +全文以 [`cookbook/rl/envs/`](https://github.com/modelscope/twinkle/tree/main/cookbook/rl/envs) 为例：MBPP 数据集上的多轮代码生成任务，同一份 `train.py` 支持两个后端。任务、工具、奖励公式完全一致，仅代码执行位置不同，因此实验曲线的差异可归因于执行环境本身。 + +## 执行后端的选择 + +三个选项的差异主要在于隔离级别及其内存代价。 + +| | OpenEnv 嵌入式 | OpenEnv server | AgentENV | +|---|---|---|---| +| 环境跑在哪 | 训练进程内 | 独立 HTTP/WebSocket 服务 | 独立 Firecracker microVM | +| 隔离 | 无 | 进程 / 容器 | microVM（KVM） | +| 执行器 | 看环境包 | smolagents AST 解释器 | 真 CPython | +| 单环境内存 | KB | KB | **~1GB** | +| 文件 / pip / 子进程 | 看环境包 | 不支持 | 支持，销毁即清零 | +| 部署成本 | 零 | 一条 `uvicorn` | 需部署控制面 + `/dev/kvm` | + +选择依据： + +**纯计算环境**（棋类、文本游戏、可在训练进程内安全求值的判分逻辑）——用嵌入式 OpenEnv，零部署。 + +**需要执行代码但标准库已能满足**——用 OpenEnv server 模式。它比 microVM 轻三个数量级，为执行几行 `sorted()` 而启动虚拟机并不划算。 + +**需要 `unittest` + `@patch`，或模型需要写文件、装包、开子进程**——只能用 AgentENV。此处有一个容易被忽略的失败模式：OpenEnv 的 `coding_env` 底层是 smolagents 的 `LocalPythonExecutor`，**一个 AST 解释器，而非操作系统级沙箱**。它对 `decorator_list` 没有任何处理，**装饰器会被静默忽略**——`@patch` 不生效、测试不报错，reward 产出一个形式正常的错误数值。这类隐形错误比崩溃难以定位。它适用于约束「仅允许 import 白名单」，不适用于执行对抗性代码。 + +后端能力的完整对比见[执行环境](../组件/Agentic/Envs.md)。 + +--- + +# 第一部分：执行环境部署 + +## OpenEnv + +### 嵌入式：无需部署 + +环境在训练进程里直接实例化，没有网络跳数： + +```bash +cd cookbook/rl/multi_turn && python multi_turn_grpo.py +``` + +环境实例较多、CPU 成为瓶颈时，用 `EnvPool` 将其迁移到独立的 CPU `DeviceGroup`，不占用 GPU 进程的内存与 GIL： + +```bash +ENV_REMOTE=1 ENV_NUM_WORKERS=8 ENV_POOL_SIZE=64 python multi_turn_grpo.py +``` + +`ENV_REMOTE=1` 才会真正分片，不设则在 driver 本地运行（零 RPC）。`ENV_POOL_SIZE=0` 表示自动取轨迹数。 + +`EnvPool` 仅对嵌入式 OpenEnv 有意义。**不要将 `OpenEnvClient` 或 `AgentEnv` 放入 `EnvPool`**——它们的 session / sandbox 生命周期在服务端，Ray 再分片一次不带来收益，只增加一跳 RPC。 + +### server 模式 + +环境机不需要 GPU、KVM 或 Docker： + +```bash +cd cookbook/rl/envs +sh openenv_server/install.sh # pip install openenv + 从源码装 coding_env +sh openenv_server/serve.sh # 4 workers x 64 sessions = 256 并发 +``` + +`coding_env` 是 OpenEnv 仓库内的子包，未发布到 PyPI，只能从源码安装——`server_app.py` 需从其中 import `PythonCodeActEnv` 与 `PyExecutor`，smolagents 也由它引入。 + +训练机仅需 `pip install openenv`，客户端类由它提供，无需安装 `coding_env`。 + +### 不使用上游 server 的原因 + +`serve.sh` 启动的是[本目录下的 `server_app.py`](https://github.com/modelscope/twinkle/blob/main/cookbook/rl/envs/openenv_server/server_app.py)，而非上游的 `coding_env.server.app`。直接沿用 upstream 会遇到三个问题： + +```python +class ConcurrentCodeEnv(PythonCodeActEnv): + # 上游默认 False，create_app(max_concurrent_envs > 1) 会直接抛 + # ConcurrencyConfigurationError，服务端被锁成单 session + SUPPORTS_CONCURRENT_SESSIONS = True + + def reset(self, **kwargs): + # 父类 reset() 用上游默认值重建 executor 和 transform，之后必须重配 + observation = super().reset() + self._configure() + return observation + + def _configure(self) -> None: + # 上游只授权 import json，MBPP 里常见的 math / collections 会直接失败 + self._executor = PyExecutor(additional_imports=list(ALLOWED_IMPORTS)) + # 上游的 create_safe_coding_transform() 用代码风格启发式覆盖 + # observation.reward（见到 open( / import os 罚 -1.0，短代码奖 +0.1）。 + # 本任务的奖励来自单元测试，风格分挤在同一个通道上只会是噪声 + self.transform = None +``` + +开启 `SUPPORTS_CONCURRENT_SESSIONS` 是安全的：`create_app` 接收的是**类**（作为 factory 使用），每个 WebSocket 连接新建一个实例，executor 与 state 均为实例私有。 + +另需说明，HTTP 的 `/step` `/reset` 端点不适用于多轮场景：OpenEnv 的这两个端点每次请求新建一个 env，返回后立即 `close()`，状态不予保留。多轮 episode 必须使用 WebSocket，`OpenEnvClient` 已实现该通路。 + +### 容量 + +``` +并发 session 上限 = WORKERS x MAX_CONCURRENT_ENVS # 默认 4 x 64 = 256 +必须 ≥ BATCH_SIZE x NUM_GENERATIONS # 默认 4 x 8 = 32 +``` + +超出容量的连接会被服务端拒绝，表现为一批 rollout 中部分轨迹的观测全部为 `Error:`。调大 `--batch-size` / `--num-generations` 时须同步扩大 `WORKERS` 或 `MAX_CONCURRENT_ENVS`。 + +## AgentENV + +[AgentENV](https://github.com/kvcache-ai/AgentENV) 用 Firecracker microVM 提供真实的操作系统语义，适合 tool-integrated reasoning 和 SWE 类任务。 + +### 主机条件核查 + +以下三项须全部满足： + +```bash +uname -r # >= 6.8 +ls -l /dev/kvm # 必须存在且可读写 +modinfo ublk_drv >/dev/null && echo ublk-ok # 需要 ublk 内核模块 +``` + +裸金属通常直接满足。云上 VM 与 GPU 实例需云厂商开启嵌套虚拟化，多数默认关闭。容器 / K8s Pod / DSW 这类托管环境取决于**宿主机**的内核与 `/dev/kvm`，还需要 privileged 与挂载 `/dev`——宿主机不满足时，容器内没有任何软件手段能绕过。 + +条件不满足时无需修改代码：在一台满足条件的机器上单独部署 AgentENV，训练侧将 `AENV_API_URL` 指向该地址即可。Firecracker **不支持 GPU 直通**，沙箱内无法运行 GPU 任务，因此环境机不需要显卡。 + +### 安装服务端 + +```bash +cd cookbook/rl/envs +sh agentenv_server/install.sh +``` + +该脚本依次完成四项工作：安装 server 与 `aenv` CLI、通过 `server --setup-host` 预置主机（kvm 组、ublk 模块、udev 规则、sysctl）、准备运行时目录、构建沙箱模板。 + +其底层调用 AgentENV 官方的 `install.sh`，后者创建专用的非 root `aenv` 账户（仅授予 `CAP_NET_ADMIN`/`CAP_SYS_ADMIN` 与 kvm 组），下载 Firecracker、guest kernel 等运行时资源，并注册 systemd 服务。数据目录默认为 `/var/lib/aenv`，镜像层与快照均存放于此，建议指向大容量磁盘： + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install.sh \ + | sudo AENV_HOME_PATH=/data/aenv bash +``` + +仅需客户端时（在训练机上创建模板，或采用 Docker 部署）： + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/install-cli.sh | sudo bash +``` + +### 构建沙箱模板 + +模板相当于沙箱的出厂镜像：依赖预装完毕并固化为快照，沙箱才能达到约 50ms 的启动速度。**依赖必须预置在模板内**，不应在训练时执行 `pip install`——每条轨迹重复安装数十秒的开销不可接受。 + +`install.sh` 已完成模板构建。手动重建的方式如下： + +```bash +sh agentenv_server/install.sh --rebuild # 先删旧模板 +``` + +也可直接使用 CLI： + +```bash +aenv auth +# AENV server URL [http://localhost:8000]: http://127.0.0.1:8000 +# API key: dummy # 本地部署下任意非空字符串均可 + +aenv build agentenv_server/Dockerfile -t twinkle-code --cpu-count 1 --memory-mb 1024 +aenv template watch # 等 ready +``` + +`aenv build` 支持 `FROM / RUN / ENV / WORKDIR / USER`（`ENTRYPOINT` 转为启动命令，`EXPOSE / VOLUME / LABEL` 忽略）。拉镜像与转 overlaybd 均在**服务端**完成，本机无需 docker。仅当 Dockerfile 变更时才需重建。直接使用现成镜像而不做加工：`aenv pull ubuntu:22.04 --name ubuntu`。 + +别名不可改绑。报 `alias 'xxx' already points to ...` 时需先执行 `aenv template delete xxx`，`--rebuild` 即封装了这一步。 + +复杂工具的实现也建议预置在模板内，工具 handler 仅保留一行调用： + +```dockerfile +COPY tools/search.py /opt/tools/search.py +``` + +如此工具实现具备版本管理、随快照分发、运行时零开销。 + +### 启动服务 + +```bash +sh agentenv_server/serve.sh # 前台，绑 127.0.0.1:8000 +NOHUP=1 sh agentenv_server/serve.sh # 后台，日志到 /tmp/aenv-server.log +API_ADDR=0.0.0.0:8000 sh agentenv_server/serve.sh # 对外，阅读下方安全说明后使用 +``` + +脚本会先停止运行中的实例再启动，因此重启无需手动 kill。 + +server 自身的默认监听为 `0.0.0.0:8000`，脚本已将其收窄至回环。**AgentENV 不提供任何认证**，端口可达即等同于任意代码执行可达，绑定 `0.0.0.0` 时必须配置安全组白名单。 + +验证： + +```bash +curl -i http://127.0.0.1:8000/health # 期望 204 +``` + +单机场景无需 AgentENV 的 gateway / scheduler（二者面向多节点部署），直连 server 的 `:8000` 即可。 + +### 内存预算 + +内存是 AgentENV 最主要的容量约束： + +``` +并发沙箱数 = BATCH_SIZE x NUM_GENERATIONS +需要内存 = 并发沙箱数 x 模板 --memory-mb + 8GB（AgentENV 自身 + 系统） +``` + +按 `--memory-mb 1024` 计算，默认 32 并发需 40GB；验证链路阶段采用 `2 x 4 = 8` 并发仅需 16GB，无需先行采购大内存机器。 + +内存不足时的调整顺序：降低模板 `--memory-mb`（多数任务 512MB 即可满足）→ 减小 `batch_size` → 依靠自动休眠（空闲沙箱 pause 后内存归还宿主机）。 + +CPU 争抢同样需要考虑：沙箱与 dataloader/tokenizer 竞争 CPU 核，`ENV_CONCURRENCY` 不应超过空闲核数。 + +## 网络与安全 + +twinkle 支持两种连接方式：同机或同一内网/VPC 内使用 **HTTP 直连**，跨网络使用 **SSH 端口转发**。 + +框架不引入 VPN / NAT 穿透类组件——这些属于网络基础设施层面，与训练代码无关。twinkle 一侧可见的仅是一个 `http://host:port`。 + +### HTTP 直连 + +企业环境下 GPU 机与 CPU 机通常已处于同一 VPC / IDC / K8s 集群，无需额外网络组件。`openenv_server/serve.sh` 的 `HOST` 默认为 `0.0.0.0`（此时脚本会打印无认证警告），应显式绑定**内网网卡**： + +```bash +HOST=10.0.1.20 sh openenv_server/serve.sh # OpenEnv +API_ADDR=10.0.1.20:8000 sh agentenv_server/serve.sh # AgentENV +``` + +训练侧仅需修改一个环境变量： + +```bash +OPENENV_BASE_URL=http://10.0.1.20:8000 sh run_openenv.sh +AENV_API_URL=http://10.0.1.20:8000 sh run_agentenv.sh +``` + +安全组入方向仅放行**训练机的 IP/32 或其安全组 ID**，端口仅开放 8000，不得使用 `0.0.0.0/0`。用 `ss -tlnp | grep 8000` 确认绑定生效——输出必须为 `10.0.1.20:8000` 而非 `0.0.0.0:8000`。 + +环境服务需配置保活：服务一旦中断，整批 rollout 随之作废。这属于运维基础设施的范畴，使用现有的进程管理器即可。最小可用的 systemd 配置： + +```ini +# /etc/systemd/system/openenv-server.service +[Service] +User=openenv +WorkingDirectory=/opt/twinkle/cookbook/rl/envs/openenv_server +Environment=HOST=10.0.1.20 MAX_CONCURRENT_ENVS=64 +ExecStart=/bin/sh serve.sh +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +`Restart=always` 是关键项——覆盖进程崩溃与 OOM 被杀两种情况。启用：`sudo systemctl enable --now openenv-server`。 + +### SSH 端口转发 + +跨网络时，SSH 在此处的作用不仅是建立连接，更是**为零认证的服务补上一层认证**——复用已有的 SSH 密钥与登录审计，且它通常是企业已批准的运维通道。 + +端口转发工作在 TCP 层，对 WebSocket 完全透明。已实测验证：握手、跨消息的 session 状态、以及 8 个并发 session 复用一条连接，均工作正常。 + +```bash +# 环境侧：只绑回环，网络上彻底不可达 +HOST=127.0.0.1 sh openenv_server/serve.sh + +# 训练侧 +ssh -N -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes \ + -L 8000:127.0.0.1:8000 user@env-host & +OPENENV_BASE_URL=http://127.0.0.1:8000 sh run_openenv.sh +``` + +`ssh -N` 不返回属于正常现象，该阻塞状态即表示隧道正在工作；需转入后台则加 `-f`。 + +两个地址容易混淆：`OPENENV_BASE_URL` 填写的是**本地入口** `127.0.0.1:8000`，`-L` 中的 `127.0.0.1:8000` 则是**在环境机视角**下解析的目标地址。AgentENV 同理，对应修改 `AENV_API_URL`。 + +长时间任务必须配置保活：转发一旦中断，整批 rollout 全部作废，因此不应运行在交互式 shell 中： + +```bash +autossh -M 0 -N -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \ + -L 8000:127.0.0.1:8000 user@env-host +``` + +所有流量复用一条 TCP 连接。每轮交互仅传输几 KB 代码与输出，实测 8 并发无压力；上百并发时可开多条转发并映射到不同本地端口分流。 + +### 沙箱出口 + +限制沙箱内代码的可访问范围，防止模型生成的代码扫描内网（SSRF）或滥用带宽。AgentENV 在 `config/default.toml` 中提供节点级强制策略，默认值已较为合理： + +```toml +[network.egress] +# 沙箱自己的 egress policy 覆盖不了这些 +always_denied_cidrs = [ + "10.0.0.0/8", "100.64.0.0/10", "127.0.0.0/8", + "169.254.0.0/16", # ← 关键：挡住云元数据服务 169.254.169.254 + "172.16.0.0/12", "192.168.0.0/16", +] +``` + +`169.254.0.0/16` 这一条最为关键：封禁云厂商元数据服务，防止沙箱内的代码窃取 IAM 临时凭证进而获得整个云账号的权限。标准库算法题无需出网，直接使用默认值即可。 + +需要放宽时按沙箱粒度传入策略： + +```python +# base_policy: Default | Allow | Deny +{'base_policy': 'Deny', + 'egress': {'allowed_domains': ['pypi.org', 'files.pythonhosted.org'], + 'denied_cidrs': ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '169.254.0.0/16']}} +``` + +最后需强调一点：`aenv auth` 使用的 API key **不具备认证作用**，本地部署下填写任意非空字符串即可通过，也不存在租户隔离——获得任意 sandbox-id 即可读写、销毁对应沙箱，`GET /sandboxes` 会列出全部沙箱。对外提供服务时，必须自建认证/租户/配额层，将 AgentENV 限定在私网内，仅对外暴露自建的上层 API。 + + +--- + +# 第二部分：训练对接 + +本部分继续以 `cookbook/rl/envs/` 为例。选择代码生成作为任务有两个原因：其奖励可由单元测试客观计算，无需裁判模型；且多轮交互具备内在语义——编写、试跑、修正、提交。 + +## 模型可见的动作空间 + +仅对模型暴露两个工具（定义于 `_openenv.py` / `_agentenv.py`）：`run_python(code)` 在环境中执行代码，`submit_solution(code)` 提交最终答案。 + +第二个工具值得单独说明，它**不发送给服务端**： + +```python +def _submit_solution(env, arguments: Dict[str, Any]) -> str: + code = (arguments.get('code') or '').strip() + if not code: + return "Error: 'code' argument is required." + env.submitted_code = code + return 'Solution submitted.' +``` + +该工具用 `register_tool` 在客户端本地处理，仅将源码记录在 env 上供训练循环打分。这是一个通用模式：**模型的动作与训练所需的记账属于两个层面**，后者应置于本地 handler，不应污染环境协议。 + +System prompt 必须准确描述后端语义，否则模型会基于错误的心智模型编写代码。两个后端在这一点上正好**相反**： + +- OpenEnv session：`The interpreter keeps its state between calls`——本轮定义的函数下轮可直接调用。同时列出可用模块白名单，并声明无文件/网络访问。 +- AgentENV：`Each call runs in a FRESH process, so every snippet must be self-contained`。 + +AgentENV 侧另通过 `include_default_tools=False` 关闭了内置的 `run_command` / `read_file` / `write_file`，使动作空间与任务精确对齐，以保证奖励归因的清晰。 + +工具集是客户端概念。AgentENV 服务端本身**不定义工具**，仅提供能力原语：任意命令执行、文件读写、端口代理。除 `register_tool(schema, handler)` 之外，还提供命令模板型的 `register_command_tool`： + +```python +env.register_command_tool( + {'type': 'function', 'function': { + 'name': 'run_tests', + 'description': '运行任务测试集。', + 'parameters': {'type': 'object', + 'properties': {'test_file': {'type': 'string'}}, + 'required': ['test_file']}}}, + 'cd /workspace && pytest {test_file} -x -q') # 用工具参数格式化 +``` + +handler 的签名为 `handler(env, arguments) -> str`，返回值即 observation。内部可使用 `env.run_command(...)` 与 `env.sandbox`（原始 E2B 句柄，可访问 PTY、文件 watch 等能力）。同名注册会覆盖内置工具。 + +## 轨迹与 env 的一一对应 + +```python +def prepare_trajectories(samples, pool): + envs = [backend.make_env() for _ in samples] + # reset() 阻塞在网络上（WebSocket 握手，或启动一个沙箱），必须并发， + # 否则一个 batch 32 条会串行等 + list(pool.map(lambda env: env.reset(), envs)) + + trajectories, tool_managers = [], [] + for sample, env in zip(samples, envs): + tool_managers.append(ToolManager(EnvTool.from_env(env))) + trajectories.append({ + 'messages': [ + {'role': 'system', 'content': backend.SYSTEM_PROMPT}, + {'role': 'user', 'content': sample['prompt']}, + ], + 'tools': backend.TOOL_SCHEMA, + }) + return trajectories, tool_managers, envs +``` + +**每条轨迹对应一个 `ToolManager`。** 它持有具体的 env 实例，共享将导致所有轨迹的工具调用打到同一个 session。 + +**`envs` 必须在 `finally` 中关闭。** 它们占用服务端容量，泄漏会使后续 step 因容量不足而失败： + +```python +try: + all_trajectories = rollout(expand_prompts, tool_manager=tool_managers) + total_rewards, pass_rates = extract_rewards(envs, expanded, env_pool) +finally: + close_envs(envs, env_pool) +``` + +还有一项容易忽略的行为：`MultiTurnRollout` 的终止条件是**模型不再发出工具调用**（或达到 `MAX_TURNS` / 长度上限），它**不读取** `EnvTool.done`。因此 system prompt 中需明确要求「提交后不要再调用工具」，否则回合会一直运行至 `MAX_TURNS`。 + +## 奖励：基于单元测试而非裁判模型 + +MBPP 每条样本自带若干 `assert`。rollout 结束后回放隐藏测试，两个后端的**脚本形态由后端能力决定，奖励公式完全相同**。 + +AgentENV 侧是真 CPython，直接生成一个普通脚本，每条断言用 `try` 包住，最后打印 `TESTS_PASSED n total`。 + +OpenEnv 侧在**同一个 session** 中逐条执行，并将 `assert X` 改写为 `print(X)`。此举的目的是可诊断性：失败的 `assert` 抛出的异常与解法内部崩溃无法区分，改为 print 即可将「断言求值为 False」与「代码崩溃」区分开，且单条测试抛出异常不影响其余测试继续执行。该执行器本身支持 `assert` / `try`，此处并非能力绕行。 + +OpenEnv 侧使用 `env.execute()` 而非 `env.step()`：`execute()` 返回服务端**原始** `StepResult`，可读取 `exit_code` 等结构化字段；`step()` 返回的是渲染给模型的文本，且会计入 episode。打分逻辑不应出现在模型的对话中。 + +奖励整形： + +```python +rate = passed / total if total else 0.0 +if total and passed == total: + return 1.0, rate # 全对 +if getattr(env, 'submitted_code', None): + return 0.1 + 0.4 * rate, rate # 提交了，部分对 → 0.1 ~ 0.5 +return 0.0, rate # 没提交 +``` + +形状上保证「全对 > 部分对 > 提交但全错 > 未提交」，且全对的 1.0 明显高于部分对的上限 0.5，否则模型会学会提交一个「刚好能过第一个测试」的假实现。 + +**提交动作本身给 0.1 底分**是有意设计：训练早期所有轨迹奖励均为 0，GRPO 的组内优势也全为 0，无法产生有效学习信号。该底分提供了最初的梯度信号。 + +沙箱本身不产生 reward（`AgentEnv.evaluate` 默认返回 0），打分统一在训练循环中完成。如需在沙箱内判分，则让工具输出结构化结果，再在 driver 侧解析。 + +## GRPO 的组内布局 + +```python +batch = [dataset[(sample_cursor + i) % len(dataset)] for i in range(BATCH_SIZE)] +expanded = [s for s in batch for _ in range(NUM_GENERATIONS)] # 同题连续 N 份 +... +advantages = advantage_fn(total_rewards, num_generations=NUM_GENERATIONS, scale='group').tolist() +``` + +注意 `expanded` 的写法：`for s in batch for _ in range(N)` 使同一道题的 N 条 rollout 在列表中**连续排列**，`GRPOAdvantage` 按该布局切分组。写成 `for _ in range(N) for s in batch` 会使分组完全错位，训练过程表面上正常，实际上无法学到任何有效信号。 + +## 启动训练 + +建议先以小并发验证全链路，以节约时间与算力成本： + +```bash +cd cookbook/rl/envs + +# OpenEnv +sh run_openenv.sh --batch-size 2 --num-generations 4 --max-steps 2 + +# AgentENV +sh run_agentenv.sh --batch-size 2 --num-generations 4 --max-steps 2 +``` + +但并发不宜过低：轨迹数（`batch-size x num-generations`）必须 ≥ `--model-gpus`（默认 4）。否则长度过滤后剩余不足，整批被跳过，日志仅反复输出 `skipping this batch`——这种表现形似卡住而非报错，比明确的崩溃更难定位。上述 `2 x 4 = 8` 预留了 2 倍余量。 + +参数需置于脚本名**之后**才能生效（脚本内为 `python train.py $TRAIN_ARGS "$@"`，`"$@"` 在后方才能覆盖）。 + +扩展至正式训练： + +```bash +OPENENV_BASE_URL=http://10.0.0.5:8000 MAX_TURNS=8 ENV_CONCURRENCY=32 \ +sh run_openenv.sh --batch-size 8 --num-generations 16 --max-steps 500 +``` + +可覆盖的环境变量： + +| 变量 | 默认 | 说明 | +|---|---|---| +| `MAX_TURNS` | `6` | 单回合最大工具调用轮数 | +| `ENV_CONCURRENCY` | `16` | driver 并发创建/销毁/打分的线程数 | +| `OPENENV_BASE_URL` | `http://127.0.0.1:8000` | OpenEnv 服务地址（也可以是负载均衡器） | +| `OPENENV_ENV_NAME` | `coding_env` | 环境包名，决定客户端 + Action 类 | +| `OPENENV_MESSAGE_TIMEOUT_S` | `120` | 单条消息超时 | +| `AENV_API_URL` | `http://127.0.0.1:8000` | AgentENV 地址 | +| `AENV_TEMPLATE` | `twinkle-code` | 模板名 | +| `SANDBOX_TIMEOUT` | `600` | 沙箱空闲超时（秒） | +| `AENV_COMMAND_TIMEOUT` | `60` | 沙箱内单条命令超时（秒） | + +训练超参通过 CLI 参数传入，默认值定义于各自的 `run_*.sh` 中。两个后端的 `TRAIN_ARGS` 需保持一致，否则奖励的变化无法归因于执行后端而非超参差异。 + +`AENV_API_URL` 不能用 `E2B_API_URL` 替代——只设后者会静默回落到默认值。 + +## 监控指标 + +日志每步输出一行： + +``` +[Step 0] {'train/code_acc': 0.031, 'train/test_pass_rate': 0.208, 'train/avg_reward': 0.145, ...} +``` + +任务指标： + +| 指标 | 含义与解读 | +|---|---| +| `train/code_acc` | pass@1，全部隐藏测试通过的比例。这是真正需要提升的指标 | +| `train/test_pass_rate` | 平均单测通过率。比 `code_acc` 平滑，早期优先观察其变化 | +| `train/avg_reward` | 整形后的奖励均值。它上涨而 `code_acc` 不涨，说明模型在套取 0.1 的提交底分 | +| `train/avg_turns` | 平均轮数。贴着 `MAX_TURNS` 说明模型经常不提交就用完轮数 | +| `train/max_turns` / `train/min_turns` | 批内轮数分布。`max_turns` 顶到上限说明有轨迹被截断；`min_turns` 接近 1 说明常有轨迹第一轮就提交或放弃 | + +策略健康度（由 `model.add_metric('GRPOMetric', ...)` 注册）： + +| 指标 | 含义与解读 | +|---|---| +| `train/approx_kl` | 新旧策略的 KL（Schulman K3 估计）。突然飙升是策略崩塌的前兆 | +| `train/clip_ratio` | PPO 裁剪触发比例（另有 `_low` / `_high` 分向）。长期偏高说明单步更新太激进 | +| `train/token_kl_max` / `train/token_ratio_max` | 单 token 的 KL / 概率比极值，用来定位崩塌 | +| `train/policy_confidence` | `exp(mean_logp)`，策略平均置信度 | + +`GRPOMetric` 的 `epsilon` 需与 `set_loss('GRPOLoss', epsilon=...)` 保持一致，否则 `clip_ratio` 统计的裁剪阈值与实际生效的不是同一个。`train/entropy` 需 `GRPOLoss(entropy_coef > 0)` 才会出现，但该选项会向 loss 中加入 entropy bonus，**从而改变训练目标**，不应仅为观察指标而开启。 + +## 迁移到自定义任务 + +复用该骨架时通常仅需修改四处，训练循环、GRPO 配置、权重同步、指标均无需变动： + +1. **数据集**：将 `load_mbpp()` 替换为自定义加载函数，产出 `{'prompt', ...打分所需字段}`。 +2. **工具**：`TOOL_SCHEMA` 与 handler。服务端能力走默认 action 通路，客户端记账用 `register_tool`。 +3. **奖励**：`extract_rewards()` 中的 `score()`。优先选择可客观计算的信号（单测、精确匹配、可执行校验），无可用信号时再考虑裁判模型。 +4. **System prompt**：必须准确描述后端语义（状态是否跨轮保留、有哪些模块和权限、轮数预算）。 + +新增一个后端：编写 `_xxx.py`（提供 `NAME`、`SYSTEM_PROMPT`、`TOOL_SCHEMA`、`make_env()`、`run_tests()`、`describe()`）与 `run_xxx.sh`，并在 `train.py` 的 `BACKENDS` 中加入名称。 + +--- + +# 故障排查 + +## 上线前检查清单 + +- [ ] 服务**未**绑在 `0.0.0.0`（`ss -tlnp | grep 8000` 确认） +- [ ] 内网直连：安全组入方向仅放行训练机，而非 `0.0.0.0/0` +- [ ] 跨网络：环境侧绑 `127.0.0.1`，仅经 SSH 端口转发访问 +- [ ] 容量 ≥ `BATCH_SIZE x NUM_GENERATIONS`（AgentENV 需另核内存） +- [ ] 环境服务已配保活（systemd），SSH 转发已配保活（autossh），而非运行在交互式 shell 上 +- [ ] 已用 `--batch-size 2 --num-generations 4 --max-steps 2` 验证过全链路 +- [ ] 未为了「让沙箱联通」而放宽 AgentENV 的 `always_denied_cidrs` + +## 环境侧 + +| 现象 | 原因与处理 | +|---|---| +| 一批轨迹里部分观测全是 `Error:` | 服务端容量不足。核对 `WORKERS x MAX_CONCURRENT_ENVS ≥ BATCH_SIZE x NUM_GENERATIONS` | +| 模型报 `import math` 不被允许 | `ALLOWED_IMPORTS` 白名单未生效；确认 `reset()` 之后重新调用了 `_configure()` | +| `ConcurrencyConfigurationError` | 启动的是上游 `coding_env.server.app`（单 session）。改用 `openenv_server/server_app.py` | +| 奖励里混进 ±0.1 / -1.0 的异常值 | 环境的 reward transform 未关闭，检查 `self.transform = None` | +| 超时 / 消息等待报错 | 调大 `OPENENV_MESSAGE_TIMEOUT_S`。执行器自身另有三重上限：单次执行 30s wall-clock、1000 万次操作、100 万次 while 迭代，因此该值需保留在 30s 以上才有意义 | +| `Address already in use` | 端口被占。`ss -tlnp \| grep 8000` 查占用进程，或换用 `PORT=8001` | + +## AgentENV 侧 + +| 现象 | 原因与处理 | +|---|---| +| `/dev/kvm is not accessible` | 运行账户不在 kvm 组，或宿主机无 KVM。执行 `sudo server --setup-host --runtime-user aenv --runtime-group aenv` 后重启 | +| `ublk_drv is not loaded` | 执行 `sudo modprobe ublk_drv`；内核 < 6.8 需升级 | +| `ImportError: AgentEnv requires the E2B SDK` | `pip install e2b` | +| `Invalid API key format: expected "e2b_"` | e2b SDK 的客户端本地校验。`AgentEnv` 已默认设置 `E2B_VALIDATE_API_KEY=false`，仍报错说明被显式覆盖为 `true` | +| `400: template xxx not found` | 模板未创建，或 build 未达到 ready。用 `aenv template list` 查看状态 | +| `alias 'xxx' already points to ...` | 别名不可改绑，需先执行 `aenv template delete xxx` | +| 沙箱内 `pip install` 失败 | 出口网络被策略拦截，或当前网络不可达 PyPI；改为在模板里预装 | +| 轨迹中途报沙箱不可用 | `SANDBOX_TIMEOUT` 小于轨迹耗时，被自动 pause。需调大 | +| batch 启动很慢 | 调大 `ENV_CONCURRENCY`；确认依赖已预置在模板内而非运行时安装 | +| 每次重启都重新下载依赖 | `AENV_HOME_PATH` 未设置，落到了 `/tmp/aenv-test-/`（`run-with-capabilities.sh` 的测试默认值），会被 `/tmp` 清理。`serve.sh` 已固定为 `/var/lib/aenv` | +| `load config ... Permission denied` | 源码编译的二进制将编译时路径固化为默认配置位置，降权为 `aenv` 后读不到 `/root`。用 `AENV_CONFIG_PATH` 指向 `aenv` 能读的副本，`serve.sh` 已处理 | + +## 训练侧 + +| 现象 | 原因与处理 | +|---|---| +| 有效轨迹数不足被跳过 | 多数轨迹超长被过滤。调小 `MAX_TURNS` / `--max-tokens`，或收敛工具输出（AgentENV 侧观测默认截断到 32K 字符）；也确认轨迹数 ≥ `--model-gpus` | +| 奖励长期为 0 | 先单独运行 `run_tests` 给一个已知正确解打分，确认打分链路本身可用，再排查模型 | +| env 数量与 `--batch-size` 不符 | 参数加在了 `sh run_*.sh` 之前，未进入 `"$@"` | + +--- + +# 附录：受限网络下的部署变通 + +本节仅在**默认路径不可用**时需要参考，适用于无法直达 Docker Hub、GitHub、PyPI 等公共源的环境（企业隔离网、内网集群、或存在出网限制的区域）。若 `install.sh` 已执行成功、模板已构建完毕，可跳过本节。 + +下文的镜像地址与包源均为示例，需替换为当前环境实际可达的内部镜像。 + +## 镜像源 + +默认 Dockerfile 有两处依赖外部网络：`FROM python:3.11-slim` 与 `RUN pip install`，两者需分别处理。 + +**基础镜像**：服务端仅按 `[image.resolver] search_registries` 搜索，默认为 `docker.io` / `ghcr.io`。两者不可达时典型报错为 `dial tcp ...: i/o timeout`（超时）或 401（需认证）。写全限定名可绕开搜索列表，**`library/` 不可省略**（官方镜像在 Hub 上的真实路径为 `library/python`）： + +```bash +BASE_IMAGE=/library/python:3.11-slim sh agentenv_server/install.sh +``` + +`BASE_IMAGE` 会传给 `aenv build --image`，覆盖 Dockerfile 中的 `FROM`，无需修改文件。 + +建议先探测候选镜像源的可达性，避免无效的 build。200 与 401 均计为可达（401 表示需先获取 token，属于正常），超时则说明不可用： + +```bash +for R in ; do + printf "%-24s " "$R" + timeout 15 curl -s -o /dev/null -w '%{http_code}\n' \ + "https://$R/v2/library/python/manifests/3.11-slim" +done +``` + +仅返回状态码不足以证明可用：部分代理能完成 TLS 握手却取不到完整 manifest。如需确认，可对比响应体大小——真实的 multi-arch index 为数 KB，而错误页面通常不足 200 字节。 + +公共代理站点存在随时不可用或限速的可能，不宜用于长期训练。生产环境建议将镜像转存至自建的容器镜像仓库（Harbor、或云厂商提供的托管仓库），置于与环境机同一内网。 + +**pip**：若沙箱内无法访问 pypi.org，需在 Dockerfile 中设置 `ENV PIP_INDEX_URL`（默认 Dockerfile 未包含该行，需自行添加），且必须写在 `RUN pip install` **之前**。 + +构建失败时用 `aenv template watch ` 查看原因。两个常见原因： + +- `all registry candidates failed during manifest fetch`——基础镜像无法拉取，按上述方式更换。 +- `overlaybd-commit ... failed to perform commit(), 2: No such file or directory`——镜像已拉取但转换失败。先确认 `FROM` 是否指错（数十 GB、上百层的大镜像容易卡在此处），确认无误后再检查 overlaybd 是否安装完整：`/var/lib/aenv/deps/overlaybd/bin/` 应包含 create / apply / commit / resize 四个二进制，`/etc/overlaybd/overlaybd.json` 应存在。 + +日志中的 `open /root/.regctl/config.json: permission denied` 仅为 WARN（server 降权为 `aenv` 后去读取 root 家目录），拉取公开镜像不受影响。但配置私有 registry 凭据时，需先为其提供一个可写 HOME： + +```bash +sudo install -d -o aenv -g aenv /var/lib/aenv/home +``` + +## install.sh 下载失败时：源码编译 + +`install.sh` 仅依赖两个外网端点：`api.github.com` 取 release 元数据，然后从 `github.com/.../releases/download/` 下载两个资产（`aenv-linux-x86_64` 9.3MB、`aenv-server-linux-x86_64.tar.gz` 68MB）。后者会 302 至 `release-assets.githubusercontent.com`，部分网络环境下易在传输中途被 RST（`curl: (56) Recv failure` 或 `(92)`）。 + +先尝试代理：`export https_proxy=...` 后重跑，`install.sh` 全程使用 curl，识别 env 代理。无代理时才需采用本节方案。 + +需先明确一点：**编译只能解决 Rust 二进制部分。** 预编译 tarball 中除 `server` 外还包含一个 `deps/`（firecracker、guest kernel、tools 驱动、overlaybd、regctl），这些不在编译产物中，需单独准备。这是本方案最易遭遇问题的环节。 + +### 前置 + +若无法直达 `static.rust-lang.org` / `crates.io`，需配置可用的 crates 镜像（下文以 `` / `` 占位，替换为当前环境可达的地址）。**必须先设置变量再安装 rustup**，顺序颠倒则需重新执行： + +```bash +# 写进 profile：AgentENV 有 rust-toolchain.toml，首次 cargo build 会再拉一次工具链 +cat >> ~/.bashrc <<'EOF' +export RUSTUP_DIST_SERVER= +export RUSTUP_UPDATE_ROOT=/rustup +EOF +. ~/.bashrc + +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +. "$HOME/.cargo/env" + +# 多数镜像仅支持 sparse 索引，需 cargo >= 1.68。用覆盖写：>> 跟两次会出现重复的 +# [source.crates-io]，TOML 直接解析失败 +tee "${CARGO_HOME:-$HOME/.cargo}/config.toml" >/dev/null <<'EOF' +[source.crates-io] +replace-with = 'mirror' + +[source.mirror] +registry = "sparse+/" + +[net] +retry = 5 + +[http] +timeout = 60 +EOF + +sudo apt-get update # upgrade 不刷索引；漏了会报 Unable to locate package + +# 编译依赖（同 Dockerfile.agentenv 的 builder 阶段）。clang / libclang-dev 是 +# uvm-ublk-daemon 必需：它用 bindgen 生成 ublk 内核绑定，缺了报 Unable to find libclang +sudo apt-get install -y build-essential pkg-config libssl-dev \ + clang libclang-dev libprotobuf-dev protobuf-compiler + +# 运行依赖（同 runtime-base 阶段）。libaio1t64 是 Ubuntu 24.04 的包名， +# 22.04 / Debian 12 上叫 libaio1 +sudo apt-get install -y ca-certificates curl dpkg e2fsprogs iproute2 iptables \ + jq libaio1t64 sudo umoci zstd +``` + +`protobuf-compiler` 必须使用系统包：`make ci-deps-protoc` 会从 GitHub Releases 下载 protoc，而那正是不可达的路径。Debian 13+ 上 `pkg-config` 已改名为 `pkgconf`。 + +镜像是否生效看 `cargo build` 的第一行，必须为 ``Updating `mirror` index``。仍为 `Updating crates.io index` 则说明 `config.toml` 未被读取——常见原因是 `CARGO_HOME` 指向其他位置，或写入文件时 `~/.cargo` 尚不存在。注意报错中的 `an/yh/anyhow` 是 sparse 路径，但 cargo >= 1.70 默认即使用 sparse，**不能以此作为镜像生效的依据**。 + +### 编译与安装 + +```bash +git clone https://github.com/kvcache-ai/AgentENV.git && cd AgentENV +cargo build --release -p agentenv --bin server +cargo build --release -p aenv +cargo build --release -p uvm-ublk -p uvm-ublk-daemon + +sudo install -m 0755 target/release/aenv /usr/local/bin/aenv +sudo install -m 0755 target/release/server /usr/local/bin/server +sudo install -D -m 0755 target/release/uvm-ublk-daemon /var/lib/aenv/ublk/uvm-ublk-daemon +sudo install -D -m 0640 config/default.toml /var/lib/aenv/config/config.toml + +sudo groupadd --system aenv +sudo useradd --system --gid aenv --home-dir /var/lib/aenv \ + --no-create-home --shell /usr/sbin/nologin aenv +``` + +最后那个 `config.toml` 的位置尤为关键：**源码编译的二进制将编译时的仓库路径固化为默认配置位置**（`cfg.rs` 中的 `CARGO_MANIFEST_DIR`）。若在 `/root/AgentENV` 编译，它会查找 `/root/AgentENV/config/default.toml`，而降权为 `aenv` 后无权访问 `/root`（0700），因此报 `Permission denied`。所以需将配置副本放至 `aenv` 可读的位置，启动时用 `AENV_CONFIG_PATH` 指向该副本。`agentenv_server/serve.sh` 已处理该问题。 + +### 准备 deps + +```bash +E="AENV_CONFIG_PATH=/var/lib/aenv/config/config.toml AENV_HOME_PATH=/var/lib/aenv" +sudo env $E /usr/local/bin/server --setup-only +``` + +下载失败时，需先确认文件的来源——以下五个 URL 即其需下载的全部内容（定义于 `config/deps_manifest.toml`）： + +``` +firecracker + cpu-template-helper https://pub-4ee15c400f554ab7a9eac3f5bc8f53de.r2.dev/firecracker-1.15.1-patch-v1-x86_64.tgz +guest kernel https://pub-4ee15c400f554ab7a9eac3f5bc8f53de.r2.dev/vmlinux-6.1.175 +regctl https://github.com/regclient/regclient/releases/download/v0.11.5/regctl-linux-amd64 +overlaybd .deb https://github.com/containerd/overlaybd/releases/download/v1.0.18/overlaybd-1.0.18-20260710.cee2186.{target}.deb +tools.ext4 ghcr.io/zlzgithub-0801/agentenv-tools:0.1.0 （OCI 镜像，需 regctl/docker 导出） +``` + +`{target}` 不要写死，`overlaybd.rs` 会读取 `/etc/os-release` 自动替换为 `ubuntu1..`。真正需要照抄的仅有 `1.0.18-20260710.cee2186` 这一段日期加 git hash。 + +在可联网的机器上将失败的资源下载完毕，然后**预放文件**——`download_file` 发现目标文件已存在且非空则跳过下载，目标路径即失败日志中的 `dest=`： + +```bash +# 日志：downloading url="https://pub-...r2.dev/firecracker-1.15.1-patch-v1-x86_64.tgz" +# dest=/var/lib/aenv/deps/firecracker/1.15.1-patch-v1/firecracker-1.15.1-patch-v1-x86_64.tgz +# scp 到 dest= 路径（文件名一模一样），然后： +sudo chown -R aenv:aenv /var/lib/aenv/deps +sudo env $E /usr/local/bin/server --setup-only +``` + +该方法对五项依赖均适用，无需修改配置。overlaybd 没有本地路径开关，但同样适用此方法：将 `.deb` 放入 `/var/lib/aenv/deps/overlaybd/downloads/`（文件名按 URL 末段，`{target}` 已替换），`package_url` 就不会被访问。 + +也可修改 `/var/lib/aenv/config/config.toml` 指向存放文件的目录。注意 `[firecracker]`、`[kernel]`、`[tools]` **这三段原本已存在**，需将键加入段内，**不要追加同名段**——TOML 中重复的表会直接解析失败： + +```toml +[firecracker] # 已有段，boot_args 等原样保留 +binary_path = "/opt/aenv-assets/firecracker" # 解开 tgz 后的二进制，不是 tgz +[kernel] # 已有段（空） +image_path = "/opt/aenv-assets/vmlinux.bin" +[tools] # 已有段，control_plane_port 原样保留 +version = "0.1.0" # 与 drive_path 必须成对 +drive_path = "/opt/aenv-assets/tools.ext4" +``` + +`/opt/aenv-assets` 需自行创建，并非现成目录。 + +### 启动服务 + +```bash +sudo env $E /usr/local/bin/server --setup-host --runtime-user aenv --runtime-group aenv +sudo chown -R aenv:aenv /var/lib/aenv +sh cookbook/rl/envs/agentenv_server/serve.sh +``` + +`serve.sh` 中有几个不可省略的环境变量：`AENV_RUN_USER=aenv`（不设则走 `SUDO_USER` → 仓库 owner → `aenv` 三层 fallback，root 直接运行时结果不稳定）、`AENV_HOME_PATH=/var/lib/aenv`（不设则落到 `/tmp/aenv-test-/`，被清理后需重新下载数百 MB）、`AENV_CONFIG_PATH`（上述的编译路径问题）。 + +另有一点：`--setup-host` 将 `aenv` 加入 kvm 组的变更对已有会话不生效，需依靠 `run-with-capabilities.sh` 的 `--init-groups` 重新初始化。绕过它直接启动 server 会因无法获得 kvm 权限而失败。 + +> 源码编译这条路未做端到端实测，deps 的下载可达性因机器而异。 + +## Docker 部署 + +```bash +curl -fsSL https://raw.githubusercontent.com/kvcache-ai/AgentENV/main/scripts/docker-setup.sh | sudo bash +docker run -d --privileged -v /dev:/dev -p 8000:8000 ghcr.io/kvcache-ai/aenv-server:latest +``` + +这里的 Docker 仅作为**部署载体**，沙箱仍为 Firecracker microVM，同样需要宿主机的 KVM。`ghcr.io` 的镜像拉取同样经由 GitHub 分发，在受限网络下可能一样不可达。 + +--- + +## 相关文档 + +- 组件参考：[执行环境](../组件/Agentic/Envs.md)（`Env` 抽象、`EnvTool`、OpenEnv 两种模式、`EnvPool`） +- 多轮工具调用：[多轮工具调用](../组件/Agentic/Multi-Turn-Tool-Usage.md) +- 可运行示例：`cookbook/rl/envs/`（代码任务，两个后端）、`cookbook/rl/multi_turn/`（嵌入式 OpenEnv） +- OpenEnv 上游仓库： +- AgentENV 官方文档： diff --git "a/docs/source_zh/\347\273\204\344\273\266/Agentic/Envs.md" "b/docs/source_zh/\347\273\204\344\273\266/Agentic/Envs.md" index 675f05baf..76c729eb7 100644 --- "a/docs/source_zh/\347\273\204\344\273\266/Agentic/Envs.md" +++ "b/docs/source_zh/\347\273\204\344\273\266/Agentic/Envs.md" @@ -78,7 +78,7 @@ manager = ToolManager(env_tools) | `from_env(env)` | 工厂方法：为 `env.tools()` 中的每个工具创建一个 `EnvTool`。 | | `last_result` | 存储最近一次 `StepResult` 供调用方检查。 | | `done` | 属性：最后一步是否终止了回合。 | -| `episode_reward` | 属性：来自 `info['episode_reward']` 的累计奖励。 | +| `episode_reward` | 属性：来自 `info['episode_reward']` 的累计奖励，缺失时回退到最后一步的 `reward`。 | ### 手动构造 @@ -97,41 +97,102 @@ env_tool = EnvTool( ) ``` -## OpenEnv +## OpenEnv：两种接入模式 -`OpenEnv` 将基于 WebSocket 的 [OpenEnv](https://github.com/OpenEnv) 环境服务器适配为同步的 Twinkle `Env`。 +[OpenEnv](https://github.com/meta-pytorch/OpenEnv) 的环境包同时提供「Environment 实现」和「EnvClient 客户端」，因此 Twinkle 提供两个适配器，对应两种截然不同的部署形态： + +| | **嵌入式** `OpenEnv` | **服务端** `OpenEnvClient` | +|---|---|---| +| 环境运行位置 | 训练进程内（直接实例化 Environment） | 独立的 OpenEnv 服务进程/容器 | +| 通信 | 无（本地函数调用） | WebSocket 长连接（一个连接 = 一个 session） | +| 隔离性 | 无，与训练进程共享内存和 GPU 节点 | 进程级/容器级，可部署在完全独立的机器 | +| 依赖 | 环境包需装在训练节点 | 环境包只需装在环境节点 | +| 扩展方式 | `EnvPool` 按 Ray worker 分片 | 服务端自身的并发 session + 多副本 | +| 适用场景 | 纯计算型轻量环境（棋类、文本游戏） | 代码执行、需要隔离或需独立扩缩容的环境 | + +> **不要**把 `OpenEnvClient` 放进 `EnvPool`：session 的生命周期在服务端，用 Ray 再分片一次不会带来任何收益，只会多一层 RPC。 + +### 模式一：嵌入式 `OpenEnv` + +绕过 OpenEnv 的 FastAPI 服务，在训练进程内直接构造 Environment，零网络开销。 ```python from twinkle_agentic.envs.openenv import OpenEnv env = OpenEnv( - base_url='http://localhost:8000', - env_cls='coding_env.CodingEnv', # 可选的类型化客户端 - env_kwargs={'message_timeout_s': 30}, - tool_schema=[...], # 可选的工具定义 + env_name='openspiel_env', # 环境包名，自动发现 Environment / Action 类 + env_kwargs={'game_name': 'blackjack'}, # 传给 Environment 构造函数 ) +result = env.reset() +result = env.step('play', {'action': 'hit'}) ``` -### 参数 +| 参数 | 类型 | 说明 | +|------|------|------| +| `env_name` | `str` | OpenEnv 环境包名（如 `'coding_env'`）。自动从 `.server` 中发现 `*Environment` 类，并从包的 `__all__` 中发现 `*Action` 类。 | +| `env_cls` | `str` 或 class | 显式指定 Environment 类（`'module:ClassName'`），与 `env_name` 二选一。 | +| `env_kwargs` | `Dict` | 传给 Environment 构造函数的参数。 | +| `action_cls` | `str` 或 class | 显式指定 Action 类；省略时从 `env_name` 自动发现。 | +| `action_mapper` | `Callable` | `(tool_name, arguments) -> action`。默认把工具参数直接作为 Action 的字段。 | + +需注意嵌入式 `OpenEnv` 并未实现 `tools()`，因此继承基类的空列表，`EnvTool.from_env(env)` 会走回退分支，只生成一个通用的 `env_action` 工具。当模型需要看到真实的动作名时，应给 `EnvTool` 传入显式 schema，或子类化重载 `tools()`。 + +### 模式二：服务端 `OpenEnvClient` + +先在环境节点上把 OpenEnv 环境跑成一个普通 HTTP/WebSocket 服务（不需要 Docker）： + +```bash +pip install openenv +pip install -e /path/to/OpenEnv/envs/coding_env +uvicorn coding_env.server.app:app --host 0.0.0.0 --port 8000 --workers 4 +``` + +然后在训练侧为每条轨迹创建一个客户端。每个实例都持有自己的 WebSocket session，服务端为它维护一个独立的 Environment 实例： + +```python +from twinkle_agentic.envs.openenv import OpenEnvClient + +env = OpenEnvClient( + env_name='coding_env', # 自动发现 EnvClient 子类 + Action 类 + base_url='http://10.0.0.5:8000', # 也可用 OPENENV_BASE_URL 环境变量 + message_timeout_s=120, # 环境要跑长耗时代码时调大 +) +env.reset() +result = env.step('run_python', {'code': 'print(1 + 1)'}) +print(result.observation) # '2' +env.close() +``` | 参数 | 类型 | 说明 | |------|------|------| -| `base_url` | `str` | 运行中的 OpenEnv 服务器 URL。 | -| `env_cls` | `str` 或 class | 类型化客户端的点分导入路径或类。`None` 使用 `GenericEnvClient`。 | -| `env_kwargs` | `Dict` | 传递给客户端构造函数的额外参数。 | -| `tool_schema` | `List[ToolInfo]` | 通过 `tools()` 暴露的工具定义。 | -| `action_mapper` | `Callable` | 自定义函数，将 `(tool_name, args)` 映射为发送给服务器的动作字典。 | +| `env_name` | `str` | OpenEnv 环境包名，从中自动发现 `EnvClient` 子类与 `*Action` 类。 | +| `env_cls` | `str` 或 class | 显式指定客户端类（`'module:ClassName'`），与 `env_name` 二选一。 | +| `base_url` | `str` | 服务地址，`http(s)://` 或 `ws(s)://` 均可（自动转换）。缺省读取 `OPENENV_BASE_URL`。也可以填负载均衡器地址。 | +| `action_cls` | `str` 或 class | Action 类；省略时自动发现。 | +| `action_mapper` | `Callable` | `(tool_name, arguments) -> action`，返回 Action 实例或字段字典。 | +| `tools` | `List[ToolInfo]` | 暴露给模型的工具 schema。默认是单个 `run_python(code)`，与 OpenEnv 代码类环境对齐。 | +| `reset_kwargs` | `Dict` | 转发给服务端 `reset()` 的参数（如 `repl_env` 的 `task_prompt` / `expected_answer`）。也可以按 episode 修改 `env.reset_kwargs` 属性。 | +| `connect_timeout_s` | `float` | WebSocket 连接超时，默认 10s。 | +| `message_timeout_s` | `float` | 单条消息超时，默认 120s。 | +| `client_kwargs` | `Dict` | 传给 OpenEnv 客户端构造函数的额外参数。 | + +补充能力： + +- `register_tool(tool_info, handler)`：注册一个**在客户端本地执行**的工具，不发往服务端。典型用途是 `submit_solution` 这类记账工具——把模型的答案存到 env 上，供训练循环后续打分。同名工具会覆盖默认工具。 +- `execute(action)`：直接发送动作并返回服务端**原始** `StepResult`，可以读取 `exit_code` 等结构化字段。训练循环用它在同一个 session 里追加执行单元测试。 +- `episode_reward` / `last_result` / `client`：累计奖励、上一次原始结果、底层同步客户端。 + +**容量与并发**：`OpenEnvClient` 内部调用 OpenEnv 客户端的 `.sync()`，每个实例拥有独立的后台事件循环，因此可以放在线程池里并发 reset/step。但服务端必须能容纳全部并发 session：环境类需声明 `SUPPORTS_CONCURRENT_SESSIONS = True`，且 `create_app(..., max_concurrent_envs=N)` 要够大，否则多出的连接会被拒绝。由于该上限按 worker 进程生效，总容量为 `workers x max_concurrent_envs`。OpenEnv 自带的 `coding_env` 将 `SUPPORTS_CONCURRENT_SESSIONS` 留在保守的默认值，需要子类化后打开（否则 `create_app` 会在 `max_concurrent_envs > 1` 时抛出 `ConcurrencyConfigurationError`）；`cookbook/rl/envs/openenv_server/server_app.py` 给出了完整写法。 ### 与 Rollout 集成使用 +两种模式的下游用法一致： + ```python -from twinkle_agentic.envs.openenv import OpenEnv from twinkle_agentic.envs.env_tool import EnvTool from twinkle_agentic.tools.tool_manager import ToolManager from twinkle_agentic.rollout.api_multi_turn import APIMultiTurnRollout -# 设置环境 -env = OpenEnv(base_url='http://localhost:8000', tool_schema=[...]) env.reset() # 桥接到 ToolManager @@ -143,7 +204,119 @@ rollout = APIMultiTurnRollout(api=api, tool_manager=manager, max_turns=10) results = rollout(trajectories) ``` -### 实现自定义环境 +端到端的多轮 GRPO 训练示例见[Agentic RL 部署与训练](../../使用指引/Agentic%20RL部署与训练.md)。 + +## AgentEnv：Firecracker microVM 沙箱 + +`AgentEnv` 是面向 [AgentENV](https://github.com/kvcache-ai/AgentENV) 部署的客户端 `Env`，后者在 E2B 兼容的 HTTP API 之后运行 Firecracker microVM 沙箱。每个 episode 创建一个沙箱，为模型提供**真实的操作系统语义**：真 CPython 解释器、可写文件系统、子进程、`pip install`。 + +与 OpenEnv 适配器的差异： + +| | `OpenEnvClient` | `AgentEnv` | +|---|---|---| +| 隔离 | 进程 / 容器 | microVM（KVM），销毁即清零 | +| 执行器 | smolagents AST 解释器 | 真 CPython | +| 单环境内存 | KB | ~1GB | +| 文件 / pip / 子进程 | 不支持 | 支持 | +| 传输 | 长连接 WebSocket | 无状态 HTTP（E2B SDK）| +| 前置条件 | 一个 OpenEnv 服务 | AgentENV 服务端 + 已构建的模板 + `/dev/kvm`、内核 6.8+ | + +OpenEnv 的 `coding_env` 底层是 smolagents 的 `LocalPythonExecutor`，**一个 AST 解释器，而非操作系统级沙箱**。它对 `decorator_list` 没有任何处理，**装饰器会被静默忽略**——`@patch` 不生效、测试不报错，reward 产出一个形式正常的错误数值。这类隐形错误比崩溃难以定位。它适用于约束「仅允许 import 白名单」，不适用于执行对抗性代码。当测试依赖装饰器，或模型需要写文件、装包、开子进程时，使用 `AgentEnv`。 + +训练前需具备三个条件（均为一次性工作，在训练循环之外完成）：AgentENV 服务端已部署、模板已构建（`aenv pull ubuntu:22.04 --name my-env`）、训练侧已执行 `pip install e2b`。 + +```python +from twinkle_agentic.envs import AgentEnv + +env = AgentEnv( + template='my-env', # 预先用 aenv CLI 构建的模板 + api_url='http://10.0.0.5:8000', # server 或 gateway；缺省读 E2B_API_URL + sandbox_timeout=600, # 需大于单个 episode 加测试回放的耗时 +) +env.reset() # 启动一个新沙箱，由 scheduler 选节点 +result = env.step('run_command', {'command': 'python -c "print(1 + 1)"'}) +print(result.observation) # '2' +env.close() # 销毁沙箱 +``` + +| 参数 | 类型 | 说明 | +|-----------|------|-------------| +| `template` | `str` | AgentENV 模板名/ID。必填——需先通过 `aenv build` / `aenv pull` 构建。 | +| `api_url` | `str` | server 或 gateway 的基础 URL。缺省读 `E2B_API_URL`。 | +| `api_key` | `str` | AgentENV 不做任何鉴权，任意非空字符串均可。缺省读 `E2B_API_KEY`，默认 `'dummy'`。 | +| `sandbox_timeout` | `int` | 沙箱空闲超时（秒），默认 300。空闲沙箱会被**pause 而非销毁**，访问时自动恢复。 | +| `command_timeout` | `int` | 单条命令超时（秒），默认 120。 | +| `setup_commands` | `List[str]` | 每次 `reset` 后执行一次的命令，输出作为 reset 的 observation。 | +| `sandbox_envs` | `Dict[str, str]` | 注入沙箱的环境变量。 | +| `metadata` | `Dict[str, str]` | 沙箱元数据，在 list 接口中可见——适合标记运行名或轨迹 id。 | +| `refresh_timeout` | `bool` | 每次 step 后延长超时，默认 `True`，避免长 episode 中途被 pause。 | +| `include_default_tools` | `bool` | 是否暴露内置的 `run_command` / `write_file` / `read_file`，默认 `True`。 | + +### 注册任务工具 + +AgentENV 服务端本身不定义工具，仅提供能力原语（任意命令执行、文件读写、端口代理）。因此工具完全是客户端概念，有两种注册方式： + +```python +# 1. shell 命令模板，用工具参数格式化 +env.register_command_tool( + {'type': 'function', 'function': { + 'name': 'run_tests', + 'description': '运行任务测试集。', + 'parameters': {'type': 'object', + 'properties': {'test_file': {'type': 'string'}}, + 'required': ['test_file']}}}, + 'cd /workspace && pytest {test_file} -x -q') + +# 2. 任意 Python handler：handler(env, arguments) -> str +def _submit(env, arguments): + env.submitted_code = arguments.get('code', '') + return 'Solution submitted.' + +env.register_tool(submit_schema, _submit) # 两者均返回 self，可链式调用 +``` + +设置 `include_default_tools=False` 可隐藏内置工具，使动作空间与任务精确对齐，从而保证奖励归因的清晰。注册的工具名与内置工具冲突时会覆盖后者。 + +其他能力： + +- `run_command(arguments)`：公开方法，供自定义 handler 复用。非零退出码**不会抛异常**，stdout、stderr 与退出码会被格式化进 observation，让模型能针对失败做出反应。 +- `sandbox` / `sandbox_id`：底层 E2B 句柄（可访问 PTY、文件 watch 等能力）与当前沙箱 id。 +- observation 默认截断到 32K 字符，避免失控的命令输出冲爆上下文。 + +**错误处理与奖励**：`step` 不抛异常。工具错误以 `observation='Error: ...'` 且 `done=False` 的形式返回，使 rollout 循环可以继续、模型有机会自行恢复，重试次数由 `max_turns` 约束。沙箱本身不产生奖励（`evaluate` 默认返回 0），打分应在训练循环中完成，或子类化重载 `step` / `evaluate`。 + +**并发**：与 `OpenEnv` / `EnvPool` 不同，`AgentEnv` 有意**不**使用 `@remote_class`。沙箱的调度落位、负载均衡、pause/resume 与节点故障转移均由 AgentENV 的 gateway/scheduler/orchestrator 在服务端完成，因此该适配器是无状态 HTTP 客户端，可直接在 rollout worker 内实例化。**不要**将其放入 `EnvPool`。由于 `reset()` 会阻塞在启动沙箱的网络调用上，应用线程池并发创建轨迹。 + +主要的容量约束是内存：并发沙箱数等于 `batch_size x num_generations`，每个占用模板的 `--memory-mb`（cookbook 构建时使用 `--memory-mb 1024`）。部署步骤、内存预算与故障排查见[Agentic RL 部署与训练](../../使用指引/Agentic%20RL部署与训练.md)。 + +## EnvPool：分布式环境池 + +`EnvPool` 是一个 `@remote_class`，把 `pool_size` 个**嵌入式** `OpenEnv` 实例按 Ray worker 分片。每个 worker 管理 `ceil(pool_size / world_size)` 个槽位（最后一个分片会被裁到 `pool_size`，因此可能更少），`reset`/`step` 通过 `remote_function` 自动路由到持有该槽位的 worker。 + +```python +from twinkle_agentic.envs.openenv import EnvPool + +pool = EnvPool( + pool_size=64, + device_mesh=mesh, + env_kwargs={'env_name': 'openspiel_env', 'env_kwargs': {'game_name': 'blackjack'}}, +) + +# 每个槽位包装成一个标准 Env，可直接交给 EnvTool / ToolManager +envs = pool.get_adapters(64) +env_tools = EnvTool.from_env(envs[0]) +``` + +| 方法 | 说明 | +|------|------| +| `reset(idx)` / `step(idx, tool_name, arguments)` | 操作单个槽位。 | +| `reset_batch(indices)` / `step_batch(indices, tool_names, arguments_list)` | 批量操作，一次 RPC 覆盖多个槽位，按 `indices` 顺序返回。 | +| `get_adapters(n)` | 在 driver 侧把前 `n` 个槽位包装为 `EnvPoolAdapter`（标准 `Env`）。`n > pool_size` 时抛异常。 | +| `close()` | 关闭全部环境。 | + +`EnvPoolAdapter` 实现标准 `Env` 接口，把 `reset`/`step` 代理到对应 worker；`step` 出错时返回 `done=True` 并把错误写入 `info['error']`，避免单个环境异常拖垮整批 rollout。它自身的 `close()` 是空操作——资源需通过池的 `close()` 释放。 + +## 实现自定义环境 ```python from twinkle_agentic.envs.base import Env, StepResult diff --git "a/docs/source_zh/\347\273\204\344\273\266/Agentic/Multi-Turn-Tool-Usage.md" "b/docs/source_zh/\347\273\204\344\273\266/Agentic/Multi-Turn-Tool-Usage.md" index 1c13bfbe0..8b94b2ed4 100644 --- "a/docs/source_zh/\347\273\204\344\273\266/Agentic/Multi-Turn-Tool-Usage.md" +++ "b/docs/source_zh/\347\273\204\344\273\266/Agentic/Multi-Turn-Tool-Usage.md" @@ -143,25 +143,26 @@ rollout = APIMultiTurnRollout(api=api, tool_manager=manager, max_turns=10) ## 使用 OpenEnv 环境 -连接远程 OpenEnv WebSocket 服务器： +连接一个运行中的 OpenEnv 服务（每个客户端实例对应服务端一个独立 session）： ```python -from twinkle_agentic.envs.openenv import OpenEnv +from twinkle_agentic.envs.openenv import OpenEnvClient from twinkle_agentic.envs.env_tool import EnvTool -env = OpenEnv( +env = OpenEnvClient( + env_name='coding_env', base_url='http://localhost:8000', - env_cls='coding_env.CodingEnv', - tool_schema=[{ + tools=[{ 'type': 'function', 'function': { - 'name': 'submit', - 'description': '提交代码解决方案。', + 'name': 'run_python', + 'description': '在远程解释器中执行 Python 代码。', 'parameters': { 'type': 'object', 'properties': { 'code': {'type': 'string'}, }, + 'required': ['code'], }, }, }], @@ -172,6 +173,8 @@ env_tools = EnvTool.from_env(env) manager = ToolManager(env_tools) ``` +如果希望环境跑在训练进程内（无隔离、零网络开销），把 `OpenEnvClient` 换成 `OpenEnv(env_name=..., env_kwargs=...)`。两种模式的完整对比见[执行环境](./Envs.md)。 + ## 每轨迹独立 ToolManager 当每个轨迹需要独立工具集时（例如，轨迹绑定的状态）： diff --git a/src/twinkle_agentic/envs/__init__.py b/src/twinkle_agentic/envs/__init__.py index a3cf38814..4633039c8 100644 --- a/src/twinkle_agentic/envs/__init__.py +++ b/src/twinkle_agentic/envs/__init__.py @@ -1,4 +1,5 @@ # Copyright (c) ModelScope Contributors. All rights reserved. +from .agentenv import AgentEnv from .base import Env, StepResult from .env_tool import EnvTool -from .openenv import EnvPool, EnvPoolAdapter, OpenEnv +from .openenv import EnvPool, EnvPoolAdapter, OpenEnv, OpenEnvClient diff --git a/src/twinkle_agentic/envs/agentenv.py b/src/twinkle_agentic/envs/agentenv.py new file mode 100644 index 000000000..eed3a9e81 --- /dev/null +++ b/src/twinkle_agentic/envs/agentenv.py @@ -0,0 +1,373 @@ +# Copyright (c) ModelScope Contributors. All rights reserved. +"""AgentENV adapter: thin client-side Env over an AgentENV (AENV) deployment. + +AgentENV (https://github.com/kvcache-ai/AgentENV) runs Firecracker microVM +sandboxes behind an E2B-compatible HTTP API. Unlike ``OpenEnv``/``EnvPool``, +this adapter deliberately does NOT use ``@remote_class``: sandbox placement, +load balancing, pause/resume and node failover are all handled server-side by +AgentENV's gateway/scheduler/orchestrator. The adapter is a stateless HTTP +client and can be instantiated directly inside rollout workers. + +Prerequisites (done once, outside training): + 1. Deploy the AgentENV server (single node) or gateway+scheduler cluster. + 2. Build a template, e.g. ``aenv pull ubuntu:22.04 --name my-env``. + 3. ``pip install e2b`` on the training side. + +Usage:: + + env = AgentEnv(template='my-env', api_url='http://gateway:8080') + result = env.reset() + result = env.step('run_command', {'command': 'echo hello'}) + env.close() +""" +import os +from typing import Any, Callable, Dict, List, Optional + +from twinkle.data_format import Trajectory +from twinkle.data_format.message import Tool as ToolInfo +from twinkle.utils import get_logger +from .base import Env, StepResult + +logger = get_logger() + +_MAX_OBSERVATION_CHARS = 32 * 1024 + +_DEFAULT_TOOLS: List[ToolInfo] = [ + { + 'type': 'function', + 'function': { + 'name': 'run_command', + 'description': 'Run a shell command inside the sandbox and return its output.', + 'parameters': { + 'type': 'object', + 'properties': { + 'command': { + 'type': 'string', + 'description': 'The shell command to execute.' + }, + 'cwd': { + 'type': 'string', + 'description': 'Working directory (optional).' + }, + }, + 'required': ['command'], + }, + }, + }, + { + 'type': 'function', + 'function': { + 'name': 'write_file', + 'description': 'Write text content to a file inside the sandbox.', + 'parameters': { + 'type': 'object', + 'properties': { + 'path': { + 'type': 'string', + 'description': 'Absolute file path in the sandbox.' + }, + 'content': { + 'type': 'string', + 'description': 'Text content to write.' + }, + }, + 'required': ['path', 'content'], + }, + }, + }, + { + 'type': 'function', + 'function': { + 'name': 'read_file', + 'description': 'Read a text file from the sandbox.', + 'parameters': { + 'type': 'object', + 'properties': { + 'path': { + 'type': 'string', + 'description': 'Absolute file path in the sandbox.' + }, + }, + 'required': ['path'], + }, + }, + }, +] + + +def _require_e2b(): + """Import the e2b SDK lazily with an actionable error message.""" + try: + from e2b import Sandbox + except ImportError as e: + raise ImportError('AgentEnv requires the E2B SDK to talk to an AgentENV server:\n' + ' pip install e2b\n' + 'Then point it at your deployment via api_url/api_key or the ' + 'E2B_API_URL / E2B_SANDBOX_URL / E2B_API_KEY environment variables.') from e + return Sandbox + + +def _truncate(text: str, limit: int = _MAX_OBSERVATION_CHARS) -> str: + if len(text) <= limit: + return text + return text[:limit] + f'\n... [truncated, {len(text) - limit} chars omitted]' + + +def _format_command_output(stdout: str, stderr: str, exit_code: int) -> str: + parts = [] + if stdout: + parts.append(stdout) + if stderr: + parts.append(f'[stderr]\n{stderr}') + if exit_code != 0: + parts.append(f'[exit code: {exit_code}]') + return _truncate('\n'.join(parts)) if parts else '(no output)' + + +class AgentEnv(Env): + """Env backed by one AgentENV sandbox per episode. + + Lifecycle mapping: + * ``reset`` -> kill the previous sandbox (if any) and create a fresh + one from ``template``; AgentENV's scheduler picks the node. + * ``step`` -> execute a tool inside the sandbox (sticky-routed to + the owning node via the sandbox id header, handled by the SDK). + * ``close`` -> kill the sandbox. + + Built-in tools (can be disabled via ``include_default_tools=False``): + ``run_command``, ``write_file``, ``read_file``. Task-specific tools can + be added with :meth:`register_tool` (arbitrary python handler) or + :meth:`register_command_tool` (shell command template), or by + subclassing. Tool errors never raise; they come back as observations so + the rollout loop can continue or let the model recover. + + Note: rewards are not produced by the sandbox. Keep the default + ``evaluate`` (zeros) and score trajectories with a separate reward + function, or subclass and override ``step``/``evaluate``. + """ + + def __init__(self, + template: str, + api_url: Optional[str] = None, + api_key: Optional[str] = None, + sandbox_timeout: int = 300, + command_timeout: int = 120, + setup_commands: Optional[List[str]] = None, + sandbox_envs: Optional[Dict[str, str]] = None, + metadata: Optional[Dict[str, str]] = None, + refresh_timeout: bool = True, + include_default_tools: bool = True, + **kwargs): + """ + Args: + template: AgentENV template name/ID (``aenv pull ... --name