Support agentenv

.gitignore

@@ -39,6 +39,8 @@ MANIFEST
 #  before PyInstaller builds the exe, so as to inject date/other infos into it.
 *.manifest
 *.spec
+.cursor
+.qoder
 
 # Installer logs
 pip-log.txt

README.md

@@ -107,6 +107,7 @@ sh INSTALL_MEGATRON.sh
 | Server startup scripts               | transformers/megatron | [Script](cookbook/client/server)                 |
 
 ## Changelog
+- 🎉2026-08-04 Sandboxed multi-turn RL is now supported: run model-generated code in isolated [AgentENV](https://github.com/kvcache-ai/AgentENV) Firecracker microVMs, or in an OpenEnv server, with the same `train.py`. See the [cookbook](cookbook/rl/envs) and the [deployment guide](docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md).
 - 🎉2026-05-20 Support DeepSeek-V4-Flash and DeepSeek-V4-Pro models.
 - 🎉2026-05-20 Multi-turn rollout and tool calling in RL are now supported. The Cookbook is currently being written. You can use `from twinkle_agentic.rollout import MultiTurnRollout/APIMultiTurnRollout` directly for multi-turn rollout.
 - 🎉2026-05-20 IM message alerting on training job failure is now supported. Usage: `import twinkle; twinkle.initialize(..., notifier=DingNotifier(...))`.

README_ZH.md

@@ -103,6 +103,7 @@ sh INSTALL_MEGATRON.sh
 Twinkle✨支持相同的算法接口运行在单GPU、torchrun多机、Ray、Client等各场景下。其算法过程是外露的，非常便于修改和调试。完整的框架介绍请查看[快速开始](https://modelscope.github.io/twinkle-web/zh/docs/usage-guide/quick-start/)
 
 ## 更新日志
+- 🎉2026-08-04 支持沙箱环境下的多轮RL训练：模型生成的代码可在隔离的 [AgentENV](https://github.com/kvcache-ai/AgentENV) Firecracker microVM 或 OpenEnv 服务中执行，两个后端共用同一份 `train.py`。参考 [cookbook](cookbook/rl/envs) 和[部署文档](docs/source_zh/使用指引/Agentic%20RL部署与训练.md)。
 - 🎉2026-05-20 支持DeepSeek-V4-Flash and DeepSeek-V4-Pro系列模型。
 - 🎉2026-05-20 支持多轮rollout和RL中的工具调用，Cookbook正在编写中，可以直接使用`from twinkle_agentic.rollout import MultiTurnRollout/APIMultiTurnRollout`进行多轮rollout。
 - 🎉2026-05-20 支持训练任务失败后的IM消息告警, 使用方式: `import twinkle; twinkle.initialize(..., notifier=DingNotifier(...))`。

cookbook/rl/envs/README.md

@@ -0,0 +1,113 @@
+# Code RL (MBPP)
+
+One training script, two environments: openenv and agentenv.
+
+MBPP dataset, the model writes Python functions: it calls `run_python` to try code in a sandbox, calls `submit_solution`, and the trainer scores it against hidden tests. Qwen3.5-4B + LoRA, GRPO, up to 6 tool-calling turns.
+
+Concepts, deployment, tuning and troubleshooting live in **[Agentic RL Deployment and Training](../../../docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md)**. This file only lists the commands.
+
+## Choosing a backend
+
+| | `openenv` | `agentenv` |
+|---|---|---|
+| Where code runs | A session on an OpenEnv server | One Firecracker microVM per trajectory |
+| Interpreter | AST interpreter (`coding_env`) | Real CPython |
+| Memory per env | KBs | ~1GB |
+| Environment host | An ordinary CPU machine, can be the training host | Needs `/dev/kvm`, kernel 6.8+ |
+
+`openenv` is enough for MBPP. Use `agentenv` when you need `unittest` + `@patch`, file writes, or pip installs.
+
+Either way the training host has 8 GPUs (`--model-gpus 4` + `--sampler-gpus 4`).
+
+## Run: openenv
+
+Environment host:
+
+```bash
+sh openenv_server/install.sh        # pip install openenv + coding_env from source
+
+HOST=127.0.0.1 sh openenv_server/serve.sh          # training on this same host
+# HOST=10.0.1.20 sh openenv_server/serve.sh        # across hosts: bind the private NIC
+```
+
+Training host:
+
+```bash
+pip install openenv
+sh run_openenv.sh
+# across hosts: OPENENV_BASE_URL=http://10.0.1.20:8000 sh run_openenv.sh
+```
+
+## Run: agentenv
+
+On the environment host, install the server and build the template (once; rebuild only when `Dockerfile` changes):
+
+```bash
+sh agentenv_server/install.sh                    # install + provision host + build template
+sh agentenv_server/install.sh --rebuild          # delete the old template and rebuild
+```
+
+On a restricted network neither the base image nor pip resolves; point `BASE_IMAGE` at a reachable registry (it is forwarded to `aenv build --image`, so `Dockerfile` stays untouched):
+
+```bash
+BASE_IMAGE=<your-registry>/library/python:3.11-slim sh agentenv_server/install.sh
+```
+
+Mirror options and build-failure troubleshooting are in the [appendix](../../../docs/source_en/Usage%20Guide/Agentic-RL-Deployment-and-Training.md) of the deployment guide.
+
+Start the server:
+
+```bash
+sh agentenv_server/serve.sh                      # foreground, binds 127.0.0.1:8000
+NOHUP=1 sh agentenv_server/serve.sh             # background
+```
+
+Training host:
+
+```bash
+pip install e2b
+# over HTTP directly:
+AENV_API_URL=http://<env-host-ip>:8000 sh run_agentenv.sh
+# or through an SSH tunnel:
+ssh -N -L 8000:127.0.0.1:8000 root@ip-of-agentenv
+# then, in another terminal
+sh run_agentenv.sh
+```
+
+> Verify a sandbox boots before launching training:
+>
+> ```bash
+> python -c "
+> from twinkle_agentic.envs import AgentEnv
+> e = AgentEnv(template='twinkle-code', api_url='http://127.0.0.1:8000')
+> e.reset(); print('sandbox ok')
+> print(e.run_command({'command': 'python -c \"import numpy, sympy; print(numpy.__version__)\"'}))
+> "
+> ```
+
+## Arguments
+
+Command-line arguments are forwarded to `train.py` and override the `TRAIN_ARGS` defaults in `run_*.sh`:
+
+```bash
+sh run_openenv.sh --max-steps 500 --batch-size 8
+```
+
+Smoke test. `batch-size × num-generations` **must be ≥ `--model-gpus`**, otherwise every batch is dropped by the length filter with only a warning:
+
+```bash
+sh run_openenv.sh --batch-size 2 --num-generations 4 --max-steps 2
+```
+
+`agentenv` memory = `batch-size × num-generations × 1GB + 8GB`, i.e. ~40GB at the default 32 concurrent sandboxes.
+
+## Files
+
+| File | Role |
+|---|---|
+| `train.py` | Training logic, backend-agnostic |
+| `_openenv.py` `_agentenv.py` | env construction, prompt, tools, hidden-test replay (the `_` prefix keeps them from shadowing the same-named pip packages) |
+| `openenv_server/` `agentenv_server/` | Per-backend `install.sh` (one-time setup) and `serve.sh` |
+| `run_openenv.sh` `run_agentenv.sh` | Launch commands and training hyper-parameters (keep both `TRAIN_ARGS` in sync) |
+
+To add a backend: write `_xxx.py` (`NAME`, `SYSTEM_PROMPT`, `TOOL_SCHEMA`, `make_env()`, `run_tests()`, `describe()`) and a `run_xxx.sh`, then add the name to `BACKENDS` in `train.py`.

cookbook/rl/envs/_agentenv.py

@@ -0,0 +1,146 @@
+import os
+import textwrap
+from typing import Any, Dict, List, Tuple
+
+from twinkle_agentic.envs import AgentEnv
+
+NAME = 'agentenv'
+
+API_URL = os.environ.get('AENV_API_URL', 'http://127.0.0.1:8000')
+TEMPLATE = os.environ.get('AENV_TEMPLATE', 'twinkle-code')
+SANDBOX_TIMEOUT = int(os.environ.get('SANDBOX_TIMEOUT', '600'))
+COMMAND_TIMEOUT = int(os.environ.get('AENV_COMMAND_TIMEOUT', '60'))
+
+SYSTEM_PROMPT = """You are an expert Python programmer with access to a Linux sandbox.
+
+Solve the task by writing a Python function.
+
+- Use `run_python` to try out your function. Each call runs in a FRESH process,
+  so every snippet must be self-contained (include the imports and the function
+  definition, then call it) and must `print(...)` what you want to see.
+- The full Python standard library is available, plus `numpy` and `sympy`.
+- When you are confident, call `submit_solution` with the complete final source
+  (imports plus the function definition).
+
+Submit exactly once, and only after the code runs correctly."""
+
+TOOL_SCHEMA: List[Dict[str, Any]] = [
+    {
+        'type': 'function',
+        'function': {
+            'name': 'run_python',
+            'description': 'Run a self-contained Python snippet in the sandbox and return its stdout/stderr.',
+            'parameters': {
+                'type': 'object',
+                'properties': {
+                    'code': {
+                        'type': 'string',
+                        'description': 'Python source to execute. Must print what you want to inspect.',
+                    },
+                },
+                'required': ['code'],
+            },
+        },
+    },
+    {
+        'type': 'function',
+        'function': {
+            'name': 'submit_solution',
+            'description': 'Submit the final solution source and end the coding phase.',
+            'parameters': {
+                'type': 'object',
+                'properties': {
+                    'code': {
+                        'type': 'string',
+                        'description': 'Complete final source: imports plus the function definition.',
+                    },
+                },
+                'required': ['code'],
+            },
+        },
+    },
+]
+
+
+def _run_python(env: AgentEnv, arguments: Dict[str, Any]) -> str:
+    """Write the snippet to a file and execute it, avoiding shell quoting issues."""
+    code = arguments.get('code')
+    if not code:
+        return "Error: 'code' argument is required."
+    env.sandbox.files.write('/workspace/scratch.py', code)
+    return env.run_command({'command': 'python /workspace/scratch.py', 'cwd': '/workspace'})
+
+
+def _submit_solution(env: AgentEnv, arguments: Dict[str, Any]) -> str:
+    """Record the solution on the env; the training loop scores it later."""
+    code = (arguments.get('code') or '').strip()
+    if not code:
+        return "Error: 'code' argument is required."
+    env.submitted_code = code
+    return 'Solution submitted.'
+
+
+def make_env() -> AgentEnv:
+    """Boot one sandbox per trajectory, exposing only the two task tools.
+
+    ``include_default_tools=False`` hides AgentEnv's built-ins (raw command
+    execution, file read/write) so the action space matches the task exactly and
+    reward attribution stays clean.
+    """
+    env = AgentEnv(
+        template=TEMPLATE,
+        api_url=API_URL,
+        sandbox_timeout=SANDBOX_TIMEOUT,
+        command_timeout=COMMAND_TIMEOUT,
+        include_default_tools=False,
+    )
+    env.submitted_code = None
+    return (env.register_tool(TOOL_SCHEMA[0], _run_python).register_tool(TOOL_SCHEMA[1], _submit_solution))
+
+
+def _build_test_script(solution: str, test_list: List[str], setup_code: str) -> str:
+    """Build a script that runs each assertion independently and prints a tally.
+
+    Every test is wrapped in its own ``try`` so that one failing assertion (or
+    one that raises) does not hide the rest — the reward uses the pass rate.
+    """
+    parts = [solution, '']
+    if setup_code:
+        parts += [setup_code, '']
+    parts.append('_passed = 0')
+    for test in test_list:
+        body = textwrap.indent(test.strip(), '    ')
+        parts += ['try:', body, '    _passed += 1', 'except Exception:', '    pass']
+    parts.append(f"print('TESTS_PASSED', _passed, {len(test_list)})")
+    return '\n'.join(parts) + '\n'
+
+
+def run_tests(env: AgentEnv, test_list: List[str], setup_code: str = '') -> Tuple[int, int]:
+    """Replay the hidden tests against the submitted solution inside the sandbox.
+
+    Real CPython means the tests can run as one ordinary script, unlike the
+    OpenEnv backend which drives them one expression at a time.
+
+    Returns:
+        ``(n_passed, n_total)``. ``(0, n)`` when nothing was submitted, or when
+        the script never reaches its tally line (syntax error, timeout, ...).
+    """
+    total = len(test_list)
+    solution = getattr(env, 'submitted_code', None)
+    if not solution:
+        return 0, total
+
+    script = _build_test_script(solution, test_list, setup_code)
+    env.sandbox.files.write('/workspace/run_tests.py', script)
+    output = env.run_command({'command': 'python /workspace/run_tests.py', 'cwd': '/workspace'})
+
+    for line in reversed(output.splitlines()):
+        if line.startswith('TESTS_PASSED'):
+            fields = line.split()
+            if len(fields) >= 3 and fields[1].isdigit():
+                return int(fields[1]), total
+    return 0, total
+
+
+def describe() -> str:
+    return f'AgentENV microVM: api_url={API_URL}, template={TEMPLATE}'