Preserve OpenHands context budget

[Yiminnn]

What changed

• generate a complete OpenHands Agent settings file instead of only the minimal LLM route
• pass explicit input/output limits when BENCHFLOW_OPENHANDS_CONTEXT_LIMIT and BENCHFLOW_OPENHANDS_OUTPUT_LIMIT are set
• enable LLMSummarizingCondenser with BENCHFLOW_OPENHANDS_CONTEXT_RESERVE (default 4096)
• install the settings writer during the root-owned agent install phase, then invoke it through a host-owned launch override
• keep install_cmd and launch_cmd byte-identical to the external agent manifest by modeling the compatibility hooks as shim-only fields
• add regression coverage for generated settings, root-owned installation, and the real rollout launch seam

Root cause

BenchFlow wrote only llm.model, llm.api_key, and optional provider fields to agent_settings.json. That bypassed the OpenHands CLI's default agent construction and left condenser=None plus no explicit input/output limits. A long rollout reached 262,216 input tokens against a 262,144-token endpoint and ended in a provider context-window rejection instead of condensing first.

Behavior

With context 262144, output 32768, and the default 4096 reserve, condensation starts at 225280 tokens. Existing provider URL and optional LiteLLM boolean flags are preserved. When limits are omitted, the writer still restores the normal summarizing condenser without inventing a token threshold.

Validation

• Ruff passes
• 259 targeted tests pass
• 12/12 tests pass against the live benchflow-ai/agents@main manifests
• GitHub test, manifest-parity, and real OpenHands rollout-smoke checks pass
• live Daytona read-back verified Agent, input 262144, output 32768, LLMSummarizingCondenser, trigger 225280
• 24 publishable post-patch rollouts / 540 assistant turns: zero context rejections, malformed tool calls, raw tool XML leaks, or <think> leaks

The production experiment patch was deployed only to newly launched jobs; existing active jobs were not restarted.

Latest automation follow-up (2026-07-23)

• Pushed 3f14b1f55 to clear the current user-simulation blockers from https://github.com/benchflow-ai/benchflow/pull/927#issuecomment-5058266206:
  • moved the OpenHands settings-writer source out of the already-large registry into src/benchflow/agents/openhands_settings_writer.py;
  • made the launch override preserve LLM_TIMEOUT, LLM_REASONING_EFFORT (max, typed efforts, and untyped values), and BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS=1;
  • added PR Preserve OpenHands context budget #927 regression docstrings and focused tests for the override path.
• Pushed 8d1106c84 to clear the new exact-head pip-audit failure by locking mcp==1.28.1 for CVE-2026-59950.
• Local validation passed: OpenHands/agent registry/manifest suite (226 passed, 2 skipped), runtime/rollout install-adjacent slice (28 passed, 3 skipped), ruff check ., ruff format --check src tests, ty check src, uv lock --check, and the exact CI pip-audit command.
• Current-head CI is green on 8d1106c844928cedd5b68248d4e55269047d20b6: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke pass.
• Current-head integration-light-jobs artifact from run 30010209498 passed benchflow-experiment-review: 1/1 rollout healthy, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 284,890 tokens, 14 tool calls, and 16/16 LLM responses with usage.
• Remaining gate: PR is still draft, so it should not merge until marked ready and reviewed.

[bingran-you]

Daily scan hygiene (2026-07-13): added rollout/eval tracking labels while this remains a draft PR.

Current state:

• Head ff29b1637b97f30f025ad544df79e51d0aac3a01 has green core checks (test, pip-audit, manifest-parity, integration-light / rollout-smoke).
• Not merge-safe today because the PR is still draft and has no non-author human approval.

Labels: enhancement, P1, status:in-progress, area:rollout, area:eval.

[bingran-you]

Users Simulation automation review (2026-07-13): behavior-ready, but still draft/in-progress on head ff29b1637b97f30f025ad544df79e51d0aac3a01.

Scope checked: OpenHands settings generation, context/output budget propagation, summarizing condenser setup, manifest-neutral install/launch behavior, CLI/SDK regression risk, and artifact health.

Evidence:

• Shared base smoke passed: uv sync --extra dev --extra sandbox-daytona --locked, ruff check ., ty check src, CLI help/list/check surfaces, and SDK Scene.single(agent="oracle") + RolloutConfig.
• PR worktree checks passed: uv sync --extra dev --locked; pytest tests/test_openhands_context_budget.py tests/test_registry_invariants.py tests/agents/test_manifest_parity.py tests/test_runtime_config_wired.py tests/test_source_adapters.py -q -> 213 passed, 2 skipped; focused ruff; ty check src; CLI + SDK smoke.
• Direct writer smoke with BENCHFLOW_OPENHANDS_CONTEXT_LIMIT=262144, BENCHFLOW_OPENHANDS_OUTPUT_LIMIT=32768, and reserve 4096 produced Agent settings with LLMSummarizingCondenser, agent/condenser usage IDs, max input/output limits, and condenser max_tokens=225280.
• GitHub current-head checks are green: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke passed.

Artifact health:

• Downloaded run 29217686724 and validated it with benchflow-experiment-review: 1/1 healthy, ACP + LLM trajectories present, results.jsonl training-ready, reward 1.0, 18/18 LLM exchanges with provider usage, 309,996 tokens, 16 tool calls, timing total 296.1s.
• Caveat: GLM user-endpoint cost accounting is still unpriced (agent_result.cost_usd=null, price_source=null) even though token/timing telemetry is complete.

Thermo-nuclear review: no blocker from this diff. The PR edits an already-large registry file, but the new settings writer is cohesive and avoids scattering context-budget branches across rollout code. I left labels as draft status:in-progress; do not merge until it is marked ready and gets non-author human review.

[bingran-you]

Users Simulation automation review (2026-07-16): blocked by draft/dirty state and one new user-facing CLI drift on head ff29b1637b97f30f025ad544df79e51d0aac3a01.

Behavioral OpenHands context-budget validation is healthy, and the existing integration-light-jobs artifact validates with ACP + LLM trajectories and a training-ready results.jsonl row. However this is not ready to merge:

• Live PR state is still draft and mergeStateStatus=DIRTY; read-only merge simulation conflicts in src/benchflow/agents/registry.py.
• bench agent show openhands still prints the stale cfg.launch_cmd, while runtime/rollout use launch_override_cmd. That makes the user-facing diagnostic disagree with the command BenchFlow actually launches.
• The artifact is healthy but still has the shared GLM user-endpoint caveat: result.json.agent_result.cost_usd / price_source are null even though token/timing telemetry is complete.

Passing evidence from isolated review: OpenHands/context focused tests 213 passed, 2 skipped; nearby rollout/runtime/agent tests 80 passed; ruff and ty check src passed; CLI/SDK smoke passed; artifact run 29217686724 passed benchflow-experiment-review with 419462 tokens, 23 tool calls, and 310.7s timing.

Moving labels to status:blocked + review:changes-requested until the branch is undrafted, rebased, and the bench agent show openhands display path matches the effective launch command.

[bingran-you]

Re: #927 (comment)

Pushed 64edbc1f304e60d93941c5556b53633075d76c72 to clear the current automation blockers:

• merged current origin/main and resolved the src/benchflow/agents/registry.py conflict;
• preserved main's OpenHands CLI/SDK/tools pins (2df8a283, 1.28.1) while keeping the context-budget settings writer;
• changed bench agent show openhands to display the effective launch_override_cmd, matching the runtime launch path;
• added the PR Preserve OpenHands context budget #927 CLI regression coverage.

Validation:

• uv run python -m pytest tests/test_agent_cli.py tests/test_openhands_context_budget.py tests/test_agent_registry.py tests/test_registry_invariants.py tests/agents/test_manifest_parity.py -q -> 222 passed, 2 skipped
• uv run python -m pytest tests/test_runtime.py tests/test_rollout_planes_contract.py tests/test_trial_install_agent_timeout.py -q -> 28 passed, 3 skipped
• focused ruff check, ruff format --check, and ty check -> passed
• uv run bench agent show openhands now shows /opt/benchflow/bin/openhands-settings-writer
• GitHub current-head checks are green: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke
• Current-head integration-light-jobs artifact passed benchflow-experiment-review: 1/1 healthy, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 347,955 tokens, 17 tool calls

Updated labels from status:blocked / review:changes-requested to status:in-progress / review:pending: the code/config blockers are fixed, but the PR is still draft and should not merge until it is marked ready and reviewed.

[bingran-you]

Users Simulation automation review (2026-07-17): behavior-ready, still draft/in-progress on head 64edbc1f304e60d93941c5556b53633075d76c72.

The current head clears the previous automation blockers:

• PR merge state is now CLEAN.
• bench agent show openhands now displays the effective settings-writer launch command with /opt/benchflow/bin/openhands-settings-writer.
• Focused OpenHands/context-budget checks passed: tests/test_openhands_context_budget.py + tests/test_agent_cli.py (6 passed), plus focused ruff, ty, and CLI smoke.
• Current-head integration-light-jobs artifact from run 29501047638 passed benchflow-experiment-review: healthy 1/1 with ACP + LLM trajectories, training-ready results.jsonl, reward 1.0, 347,955 tokens, 17 tool calls, 29 ACP events, and 288.6s timing.

Remaining before merge-ready: the PR is still draft, and the new regression tests in tests/test_openhands_context_budget.py should add AGENTS-compliant docstrings naming PR #927. Thermo-nuclear note: no blocker, but this continues adding harness wiring to an already-large src/benchflow/agents/registry.py file (1496 -> 1590 lines), so future harness-script growth should be decomposed.

Label state synced to status:in-progress + review:pending. GLM user-endpoint accounting caveat persists: token/timing telemetry is complete, but result.json.agent_result.cost_usd and price_source are null.

[bingran-you]

Users Simulation automation review (2026-07-23): blocked on current head 64edbc1f304e60d93941c5556b53633075d76c72.

What still passes:

• PR-scoped sidecar ran the full local suite (4904 passed, 16 skipped, 7 deselected), ruff, format, ty, CLI/SDK smoke, and current exact-head artifact validation.
• The exact-head integration-light artifact validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, one training-ready results.jsonl row, reward 1.0, 347,955 tokens, 17 tool calls, and 18/18 LLM responses with usage.
• GitHub checks are green for test/audit/parity/rollout-smoke, and the PR is clean/mergeable, but still draft.

Blocking behavior regression:

• Runtime now selects launch_override_cmd for OpenHands, but the new /opt/benchflow/bin/openhands-settings-writer only writes model/base_url/tool/context fields. It does not preserve the existing launch behavior for LLM_TIMEOUT, LLM_REASONING_EFFORT, or BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS.
• Repro with LLM_TIMEOUT=115200, LLM_REASONING_EFFORT=max, and BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS=1: the generated agent_settings.json contains model, api_key, base_url, max_input_tokens, and max_output_tokens, but has no timeout, no reasoning_effort, and no litellm_extra_body. The legacy launch_cmd still contains those behaviors, but it is bypassed by launch_override_cmd.

Repo-convention blocker:

• The three new tests in tests/test_openhands_context_budget.py lack docstrings naming PR Preserve OpenHands context budget #927 or the guarding commit, contrary to AGENTS.md.

Recommended fix: migrate the timeout/reasoning/subagent-disable behavior into the settings-writer/override path, add regression coverage for those env vars, and add PR #927 guard docstrings. Keeping this blocked until then.

[bingran-you]

Re: #927 (comment)

Pushed two commits to clear the current user-simulation blockers on PR #927:

• 3f14b1f55 moves the OpenHands settings writer out of the already-large registry and makes the launch_override_cmd path preserve the legacy launch behavior for LLM_TIMEOUT, LLM_REASONING_EFFORT (max, typed efforts, and untyped values), and BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS=1.
• 8d1106c84 updates the locked mcp package to 1.28.1, clearing the new exact-head pip-audit failure for CVE-2026-59950.

Regression coverage now includes PR #927 guard docstrings in tests/test_openhands_context_budget.py, plus direct checks that the generated agent_settings.json contains timeout/reasoning fields and that subagent delegation is patched when disabled.

Validation:

• uv run python -m pytest tests/test_openhands_context_budget.py tests/test_agent_cli.py tests/test_agent_registry.py tests/test_registry_invariants.py tests/agents/test_manifest_parity.py -q -> 226 passed, 2 skipped
• uv run python -m pytest tests/test_runtime.py tests/test_rollout_planes_contract.py tests/test_trial_install_agent_timeout.py -q -> 28 passed, 3 skipped
• uv run ruff check ., uv run ruff format --check src tests, uv run ty check src, uv lock --check -> passed
• exact workflow pip-audit command -> passed
• GitHub current-head checks on 8d1106c844928cedd5b68248d4e55269047d20b6 are green: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke
• downloaded run 30010209498 artifact and validated with benchflow-experiment-review: 1/1 rollout healthy, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 284,890 tokens, 14 tool calls, and 16/16 LLM responses with usage

Updated labels from status:blocked / review:changes-requested to status:in-progress / review:pending. Remaining gate is non-code: the PR is still draft and needs normal review before merge.

[bingran-you]

Users Simulation automation review (2026-07-25): behavior-ready by simulation, still draft/in-progress on current head 8d1106c844928cedd5b68248d4e55269047d20b6.

What now passes:

• PR worktree gate: uv sync --extra dev --locked; focused OpenHands/agent/parity tests -> 195 passed, 2 skipped; focused ruff; focused ty; bench agent show openhands; git diff --check.
• The prior override-path blockers are fixed: settings writer preserves LLM_TIMEOUT, LLM_REASONING_EFFORT, and BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS; the new PR Preserve OpenHands context budget #927 regression tests have guard docstrings.
• Current GitHub checks are green/skipped by scope: test, pip-audit, manifest-parity / parity, integration-light / rollout-smoke, and detect-scope jobs.

Artifact gate:

• Exact-head integration-light-jobs run 30010209498 structurally passes benchflow-experiment-review: healthy 1/1, ACP + LLM trajectories, training-ready results.jsonl, reward 1.0, 284,890 tokens, and 296.5s timing.
• Manual trajectory audit still quarantines this artifact as capability evidence: the shared jax-computing-basics task exposes /app/reference/*.npy, and the no-skill trajectory reads reference outputs. GLM agent_result.cost_usd / price_source are also null.

Recommendation: keep status:in-progress while the PR is draft; do not promote to ready until the draft/review gate and clean artifact rerun are resolved.

AI-generated automation review posted on behalf of Bingran You.