Skip to content

feat(sandbox): Enforce network_mode allowlist on docker#785

Open
Yiminnn wants to merge 34 commits into
mainfrom
feat/network-mode-enforcement
Open

feat(sandbox): Enforce network_mode allowlist on docker#785
Yiminnn wants to merge 34 commits into
mainfrom
feat/network-mode-enforcement

Conversation

@Yiminnn

@Yiminnn Yiminnn commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Problem

network_mode is the documented outbound-network control, but the launch path
historically keyed off the deprecated allow_internet boolean. That made
allowlist validated but unenforceable: a task could declare allowed_hosts,
yet the sandbox would either run open or reject too late.

Current fix

This PR makes network_mode authoritative and fail-closed across the runtime
path.

  • Docker allowlist enforcement: the main container is moved onto an
    internal no-egress network after agent install, and HTTP(S) traffic goes
    through a stdlib egress proxy sidecar that forwards only to allowed_hosts
    plus the model lane when allowed.
  • Daytona allowlist enforcement: exact-host allowlists are mapped to
    Daytona's IPv4 CIDR control when faithfully expressible; hosts are pinned in
    /etc/hosts; wildcard, unresolvable, over-limit, and DinD cases fail closed.
  • Role-level network policy enforcement: agent.network_mode and
    verifier.network_mode now collapse into the effective shared sandbox policy
    before launch instead of being accepted by validation but ignored by lockdown.
  • Hermetic mode: allow_model_endpoint: false prevents the provider/model
    host from being appended to Docker/Daytona allowlists and now fails closed
    before model-backed agent launch.
  • Daytona no-network model lane: no-network tasks that keep the model lane
    enabled install first and then relock to a provider-only CIDR allowlist; fully
    hermetic or no-model Daytona tasks use platform block-all.
  • Snapshot restore guards: Docker snapshot restore fails closed under
    restrictive network policies, and Daytona direct restore now reapplies the
    active allowlist/model lane or verifies platform block-all after restoring
    from a snapshot.
  • Provider host correctness under restrictive policies: lockdown now
    allowlists explicit BENCHFLOW_PROVIDER_BASE_URL endpoints, including
    user-supplied vllm/... endpoints, instead of falling back to the registry
    default host.
  • Required usage tracking fail-closed behavior: usage_tracking=required
    no longer skips the LiteLLM proxy under restrictive Docker/Daytona policies
    and fails only after launch; it now fails before agent launch with an explicit
    message.
  • Heterogeneous scene roles: connect_as() now extends/relocks the model
    lane when a later role switches provider, preserving the original provider
    host and adding the role provider host before opening the ACP session.
  • Egress readiness/canary hardening: Docker relock waits for the bf-egress
    sidecar before returning proxy env; Daytona canary selection avoids
    allowlisted IPs and fails closed if it cannot prove a non-allowlisted host is
    blocked.
  • Audit hygiene: lockfile mcp is refreshed to 1.28.1 to clear
    CVE-2026-59950 in the current PR merge checkout.
  • Compatibility guard: sandbox setup tolerates partial task-config fakes and
    older test helpers that expose only environment.
  • Maintainability cleanup: backend orchestration lives in
    docker_network_lockdown.py and daytona_network_lockdown.py; docker.py
    and daytona.py remain below the 1k-line review threshold. The July 23
    regression tests are isolated in tests/test_network_pr785_regressions.py so
    tests/test_network_runtime.py stays below the 1k-line threshold.

Latest validation

Current head: f4a35bcf07e34a0f7ee8c7fa74f5cec4d1a58a1b.

  • Pushed f4a35bcf0 to address the July 24 code/config blocker: Daytona restrictive relock now preserves every accumulated model/provider lane host across connect_as() role switches, not only the first host. The Daytona CIDR plan receives the task allowlist plus all model-lane hosts, de-duplicated.
  • Added/updated PR feat(sandbox): Enforce network_mode allowlist on docker #785 guard coverage:
    tests/test_network_pr785_regressions.py::test_daytona_relock_threads_every_model_lane_host and
    tests/test_network_runtime.py::test_daytona_relock_no_network_model_lane_allowlists_provider.
  • Exact regression validation passed:
    uv run --extra dev --extra sandbox-daytona python -m pytest tests/test_network_pr785_regressions.py tests/test_network_runtime.py::test_daytona_relock_no_network_model_lane_allowlists_provider tests/test_trial_litellm_runtime.py::test_trial_connect_as_extends_restrictive_model_lane_for_role -q -> 8 passed.
  • Focused network/runtime validation passed:
    uv run --extra dev --extra sandbox-daytona python -m pytest tests/test_network_pr785_regressions.py tests/test_network_runtime.py tests/test_network_policy.py tests/test_network_modes.py tests/test_runtime_capabilities.py tests/test_trial_litellm_runtime.py -q -> 106 passed.
  • Broader runtime/config validation passed:
    uv run --extra dev --extra sandbox-daytona python -m pytest tests/test_env_setup.py tests/test_internet_policy.py tests/test_network_modes.py tests/test_network_runtime.py tests/test_network_pr785_regressions.py tests/test_runtime_capabilities.py tests/test_connect_as_env.py tests/test_task_config.py tests/test_task_document.py tests/test_trial_litellm_runtime.py tests/test_providers.py tests/test_litellm_config.py -q -> 320 passed, 1 skipped, 2 expected deprecation warnings.
  • Static/merge checks passed: uv run ruff check .;
    uv run ruff format --check src tests tools; uv run ty check src/;
    uv lock --check; git diff --check; git merge-tree --write-tree origin/main HEAD.
  • Current-head GitHub checks are green/skipped by scope on f4a35bcf0:
    test, pip-audit, manifest-parity / parity,
    integration-light / detect-scope, integration-scope / detect-scope, and
    integration-light / rollout-smoke passed; integration-scope plan,
    run-matrix, and review-pack skipped by scope.
  • Successful artifact audit: downloaded run 30095697606, successful
    rollout-smoke job 89489136877, and validated integration-light-jobs with
    .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py /tmp/pr785-artifacts.3sjdna --json.
    Result: healthy 1/1; ACP + LLM trajectories present; rollout-level
    results.jsonl training-ready; reward 1.0; 228,682 tokens; 12 tool calls;
    21 ACP events; 16/16 LLM responses with usage; timing total 261.4s.

Remaining gate

The direct July 24 code/config blocker is fixed on-branch and current-head CI is green. The smoke artifact is healthy but still Docker/public (network_mode: null), so it does not by itself prove restrictive Daytona multi-provider enforcement. Labels intentionally remain status:blocked and review:changes-requested because this is a security-sensitive sandbox/network-control PR with an active human CHANGES_REQUESTED review gate. Human security re-review should decide whether the deterministic exact-path regression is sufficient or whether a live restrictive Daytona/FrontierPhysics canary should be recorded before clearing the gate.

network_mode is now the authoritative network control across the launch
path; the deprecated allow_internet boolean is a derived shim. Previously
every backend keyed off allow_internet, so allowed_hosts never reached the
sandbox and network_mode: allowlist was validated but unenforceable -
rejected at preflight on every backend.

- allowlist on the docker sandbox is now ENFORCED: the agent container
  joins an internal (no-egress) network and its HTTP(S) traffic routes
  through a stdlib egress-proxy sidecar that forwards only to allowed_hosts.
  Other hosts, raw-IP connections, and proxy-ignoring tools have no route
  off-box (default deny). New: sandbox/_egress.py, _egress_proxy.py.
- sandbox/network_policy.py resolves the effective policy (open/block-all/
  allowlist) and gates allowlist to backends that can enforce it; daytona/
  modal fail closed (never silently open).
- runtime_capabilities: allowlist supported on docker, rejected at preflight
  elsewhere with a clear message.
- setup.py: preserve_agent_network lifts ANY restrictive policy (no-network
  OR allowlist) to public for web-disabled agent runs (they need the model
  API; no-web is enforced at the agent layer).
- docs/task-authoring-task-md.md network-policy section + CHANGELOG.

Verified on real docker (egress e2e): allowed host -> 200, non-allowed ->
blocked, direct socket with proxy stripped -> no route. Full suite green.
@Yiminnn
Yiminnn temporarily deployed to pypi-internal-preview June 15, 2026 17:42 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Review (automated thorough pass) — ✅ no blocking defects

I ran an adversarial correctness/security review of the allowlist enforcement plus the targeted test + lint suite on a fresh checkout of the PR head.

The enforcement is genuinely sound. Key claims verified:

  • Default-deny is real. The main service joins only bf_egress_internal (internal: true, no off-box route); the compose sequence-replace on services.main.networks correctly drops default. Tools that ignore HTTP(S)_PROXY and raw sockets simply have no route off-box — the proxy is the sole bridge.
  • Host matching is dot-anchored (any(host == a or host.endswith("." + a))): evil-example.com does not match example.com, example.com.evil.com is denied, case/trailing-dot normalized. Covered by tests.
  • Raw-IP / IPv6 / IP-literal CONNECT are denied (config rejects non-hostname allowed_hosts; raw IPs never match). DNS-rebind / SNI-fronting are inherent hostname-allowlist limitations, not bypasses.
  • Fail-closed off-docker: allowlist on daytona/modal is rejected at preflight, and resolve_network_decision fails closed as backstop.
  • allow_internet back-compat shim preserves legacy no-network behavior.
  • preserve_agent_network lift does NOT weaken allowlist — it only fires when web tools are disabled (which for an allowlist task requires self_gen_no_internet=True), mutates a deep copy, and runs after preflight.
  • No injection (allowed_hosts validated to [a-z0-9-.], JSON env, no shell); cleanup tears down sidecar + both networks via compose down with _force_kill_project backstop.

Tests/lint: tests/test_network_policy.py + tests/test_runtime_capabilities.py41 passed; ruff check clean; ty check src/benchflow/sandbox/ clean.

Non-blocking nits (follow-ups, not merge blockers)

  1. _egress.py docstring overstates coverage — says "docker and daytona-dind," but ALLOWLIST_CAPABLE_SANDBOXES = {"docker"} and daytona_dind.py still keys off legacy allow_internet; it doesn't call resolve_network_decision/build_egress_override. Recommend wording it docker-only (matches the PR's own "Scope/follow-ups").
  2. In-backend fail-closed is docker-only. modal_impl.py, daytona.py, daytona_dind.py still read allow_internet directly; the real gate is preflight (which runs first, so it's safe today). Worth routing non-docker backends through resolve_network_decision as defense-in-depth.
  3. Minor: _network_policy_compose_paths rebuilds the override on every compose invocation (incl. down) — consider memoizing. Proxy sidecar binds 0.0.0.0:8080 on both nets — could restrict to the internal interface.

Verdict: MERGE-READY (with the above as follow-ups). Leaving the actual merge to @Yiminnn since the branch was updated very recently and may still be iterating.

@mintlify

mintlify Bot commented Jun 15, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
benchflow-bff148e7 🟢 Ready View Preview Jun 15, 2026, 6:23 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Wildcard allowed_hosts: a single leading `*.` label (e.g. *.example.com)
matches subdomains at any depth (Harbor/nginx semantics) but not the apex;
mid/trailing wildcards are rejected at parse time. Validator in task/config.py,
matcher in sandbox/_egress_proxy.py.

Model lane: a restrictive network_mode (no-network or allowlist) on docker now
keeps one always-allow lane to the host-side benchflow model proxy open, so an
agent run reaches the model without opening the sandbox to the public internet
(no-network becomes model-only egress). The egress sidecar permits the docker
host (_docker_host_address()) and routes to it via the bridge gateway /
host-gateway. Replaces the blanket lift-to-public for web-disabled docker runs
(extracted to _lift_agent_network_to_public, now docker-excluded); other
sandboxes keep the lift. allow_model_endpoint:false (default true) closes the
lane for a hermetic, no-model run.

Live-verified through the real egress proxy: lane host + allowed host reachable,
non-allowed host blocked (403); wildcard subdomain reachable, apex blocked.

ENG-219
@greptile-apps

greptile-apps Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes network_mode the authoritative network control across the docker sandbox launch path, retiring allow_internet to a back-compat shim, and actually enforces the allowlist policy on docker by wiring an internal-network egress-proxy sidecar (bf-egress) that allows only declared allowed_hosts while blocking everything else by default-deny.

  • New egress enforcement (_egress.py, _egress_proxy.py): an allowlist task's main container is placed on an internal: true Docker network and all HTTP(S) traffic is routed through a stdlib proxy sidecar; relock_network() performs post-install network surgery with docker network connect/disconnect and verifies the swap with lockdown_complete().
  • Policy resolution (network_policy.py): introduces resolve_network_decision, a lockdown_complete guard, and Daytona-specific IPv4-CIDR mapping; allowlist on modal/unsupported backends now fails closed at preflight rather than running open.
  • restore() bypass (docker.py): the snapshot-restore path hardcodes --network {project}_default, bypassing the egress proxy — this pre-existing gap is noted in earlier review threads and is not addressed in this PR.

Confidence Score: 4/5

The core egress enforcement logic is sound and the lockdown verification loop prevents silent policy failures, but restore() in docker.py still reconnects restored containers to the unrestricted public bridge — an outstanding gap flagged in earlier review threads that this PR does not address.

The new egress proxy, policy resolution, and lockdown_complete guard are all well-designed and tested. The two new inline comments are minor (IPv6 port parsing, custom-network error clarity) and do not affect the main enforcement path. The unresolved restore() bypass remains present and is the primary reason confidence falls short of the maximum.

src/benchflow/sandbox/docker.py — specifically the restore() method which bypasses egress network enforcement after a snapshot restore.

Important Files Changed

Filename Overview
src/benchflow/sandbox/_egress.py New file: generates the compose override that confines main to an internal network and routes traffic through the bf-egress proxy sidecar; depends_on uses list form (no health-gate, flagged in earlier thread).
src/benchflow/sandbox/_egress_proxy.py New stdlib egress proxy: handles CONNECT tunnelling and plain HTTP with absolute-URI rewrite (_to_origin_form); upstream socket now cleaned up in finally block (fixed from earlier review); IPv6 authority with explicit port parses to wrong port 80.
src/benchflow/sandbox/docker.py Key changes: relock_network() with lockdown_complete() verification; restore() still hardcodes --network {project}_default, bypassing egress enforcement after a snapshot restore (pre-existing unfixed gap from earlier review).
src/benchflow/sandbox/network_policy.py New module: resolve_network_decision, lockdown_complete, plan_daytona_allowlist — clean policy resolution with fail-closed semantics for unsupported sandboxes.
src/benchflow/sandbox/setup.py _lift_agent_network_to_public now correctly excludes docker/daytona sandboxes from the blanket lift; no new issues.
src/benchflow/task/runtime_capabilities.py _append_network_issue now correctly gates allowlist at preflight for unsupported sandboxes; wildcard preflight rejection for daytona's IPv4-CIDR mode added.
tests/test_network_policy.py Good coverage: resolution logic, override shape, proxy host-matching, capability gate; lockdown_complete not directly unit-tested for the task-custom-network edge case.
src/benchflow/sandbox/daytona.py Uses new network_policy helpers; IPv4-CIDR allowlist integration; plan_daytona_allowlist resolve-at-lockdown path is sound but requires DNS resolution at lockdown time.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant rollout as Rollout
    participant docker as DockerSandbox
    participant compose as docker compose
    participant main as main container
    participant sidecar as bf-egress sidecar
    participant internet as Internet

    rollout->>docker: "start() [_network_locked=False]"
    docker->>compose: up --wait (no egress override)
    compose->>main: start (on project_default, full internet)
    rollout->>docker: relock_network(extra_allowed_hosts)
    docker->>docker: "_network_locked = True"
    docker->>docker: build_egress_override() write JSON
    docker->>compose: up --no-deps bf-egress (egress override now in paths)
    compose->>sidecar: start (on bf_egress_internal + bf_egress_external)
    docker->>main: docker network connect bf_egress_internal
    docker->>main: docker network disconnect project_default
    docker->>main: docker inspect verify attached networks
    main->>docker: lockdown_complete() check
    alt lockdown verified
        docker->>rollout: return HTTP_PROXY env
    else lockdown failed
        docker->>rollout: raise SandboxStartupError
    end
    rollout->>main: exec agent with proxy env
    main->>sidecar: HTTP(S) via proxy bf-egress:8080
    sidecar->>sidecar: _host_allowed(hostname)?
    alt allowed host
        sidecar->>internet: forward origin-form rewrite for HTTP
    else denied
        sidecar->>main: 403 Forbidden
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant rollout as Rollout
    participant docker as DockerSandbox
    participant compose as docker compose
    participant main as main container
    participant sidecar as bf-egress sidecar
    participant internet as Internet

    rollout->>docker: "start() [_network_locked=False]"
    docker->>compose: up --wait (no egress override)
    compose->>main: start (on project_default, full internet)
    rollout->>docker: relock_network(extra_allowed_hosts)
    docker->>docker: "_network_locked = True"
    docker->>docker: build_egress_override() write JSON
    docker->>compose: up --no-deps bf-egress (egress override now in paths)
    compose->>sidecar: start (on bf_egress_internal + bf_egress_external)
    docker->>main: docker network connect bf_egress_internal
    docker->>main: docker network disconnect project_default
    docker->>main: docker inspect verify attached networks
    main->>docker: lockdown_complete() check
    alt lockdown verified
        docker->>rollout: return HTTP_PROXY env
    else lockdown failed
        docker->>rollout: raise SandboxStartupError
    end
    rollout->>main: exec agent with proxy env
    main->>sidecar: HTTP(S) via proxy bf-egress:8080
    sidecar->>sidecar: _host_allowed(hostname)?
    alt allowed host
        sidecar->>internet: forward origin-form rewrite for HTTP
    else denied
        sidecar->>main: 403 Forbidden
    end
Loading

Reviews (14): Last reviewed commit: "fix(network): address multi-agent audit ..." | Re-trigger Greptile

Comment thread src/benchflow/sandbox/_egress_proxy.py Outdated
Comment on lines +119 to +139
# Plain HTTP: target is an absolute URI (http://host/path) in proxy form.
host = ""
if "://" in target:
host = target.split("://", 1)[1].split("/", 1)[0]
if not host:
for hl in header.split(b"\r\n"):
if hl.lower().startswith(b"host:"):
host = hl.split(b":", 1)[1].decode("latin-1").strip()
break
hostname = host.rsplit(":", 1)[0].strip("[]")
port = int(host.rsplit(":", 1)[1]) if ":" in host and "]" not in host else 80
if not _host_allowed(hostname):
_deny(client, hostname)
return
try:
upstream = socket.create_connection((hostname, port), timeout=30)
except OSError:
client.sendall(b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\n\r\n")
return
upstream.sendall(header)
_pipe(client, upstream)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Plain-HTTP request forwarded in absolute-URI form

For plain-HTTP requests the proxy forwards the raw header bytes — including the proxy request-line (GET http://host/path HTTP/1.1) — directly to the upstream server. RFC 7230 §5.3.2 says servers MUST accept the absolute-form, but a non-negligible number of back-ends (nginx with default config, some Python WSGI servers, and many embedded HTTP stacks) return 400 Bad Request for requests that arrive in absolute-URI form, which would cause tasks to see inexplicable failures for allowlisted plain-HTTP hosts.

The fix is to rewrite the request-line before forwarding: extract the path component from target (already split at line 122), reconstruct METHOD /path HTTP/1.1\r\n, and replace the first line of header before calling upstream.sendall.

Comment thread src/benchflow/sandbox/_egress.py Outdated
"NO_PROXY": "localhost,127.0.0.1",
"no_proxy": "localhost,127.0.0.1",
},
"depends_on": [_EGRESS_SERVICE],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 depends_on list form doesn't wait for proxy readiness

Using a plain list for depends_on tells Compose to start main as soon as bf-egress has been created, not once it is actually listening on port 8080. In practice the proxy binds almost instantly, but on a slow or loaded host the main container can fire its first proxied request (e.g. during agent framework import) before srv.accept() is running, resulting in Connection refused with no retry path. Switching to the long-form with condition: service_started is equivalent but explicitly documented; condition: service_healthy with a TCP healthcheck would eliminate the race entirely.

Suggested change
"depends_on": [_EGRESS_SERVICE],
"depends_on": {
_EGRESS_SERVICE: {"condition": "service_started"},
},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0c78eadd35647bc47fb7bb4536d64d8555696589.

The egress override now gives bf-egress a TCP healthcheck and makes main depend on service_healthy. relock_docker_network() also fails closed unless the sidecar container reports healthy before returning HTTP(S)_PROXY to the agent, so the proxy env is no longer exposed after mere service creation.

Validation on the pushed head:

  • uv run pytest tests/test_network_policy.py tests/test_network_modes.py tests/test_network_runtime.py -q -> 62 passed
  • uv run ruff check src/benchflow/sandbox/_egress.py src/benchflow/sandbox/docker_network_lockdown.py tests/test_network_policy.py tests/test_network_runtime.py
  • uv run ruff format --check src/benchflow/sandbox/_egress.py src/benchflow/sandbox/docker_network_lockdown.py tests/test_network_policy.py tests/test_network_runtime.py
  • uv run ty check src/benchflow/sandbox/_egress.py src/benchflow/sandbox/docker_network_lockdown.py
  • GitHub CI is green on 0c78eadd3, including test, pip-audit, manifest-parity, and integration-light / rollout-smoke.
  • The integration-light-jobs artifact validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, results.jsonl, reward present, 334350 tokens, 18 tool calls, and training_ready=1.

Moving the PR back to status:ready; keeping review:changes-requested because the security-sensitive PR still needs human re-review to clear the existing review decision.

Comment thread src/benchflow/sandbox/_egress_proxy.py Outdated
Comment on lines +133 to +140
try:
upstream = socket.create_connection((hostname, port), timeout=30)
except OSError:
client.sendall(b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\n\r\n")
return
upstream.sendall(header)
_pipe(client, upstream)
upstream.close()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 upstream socket not closed on exception in the plain-HTTP path

upstream is created at line 134, but if upstream.sendall(header) (line 138) raises OSError the outer except Exception: pass silently swallows the error and upstream.close() (line 140) is never called. In CPython this is reclaimed by reference-counting when _handle returns, but in PyPy/non-reference-counted runtimes the descriptor leaks until GC. Wrapping the post-create_connection block in a try/finally (or using upstream as a context manager) would make cleanup explicit and deterministic.

New docs/network-mode-prior-art.md records the network-mode design model
(no-network/public/allowlist + wildcard + model lane), a mode/mechanism
taxonomy, and credits to the prior-art platforms the design draws on
(Harbor, Modal, AISI Inspect, WAREX, SWE-bench, SWE-bench-Live, WebArena,
Cybench, tau2-bench, browser-use) with primary-source links. Registered in
the Mintlify nav and cross-linked from the task-authoring guide and
sandbox-hardening. Citations verified against primary sources; corrects the
Harbor PR#1854 framing (wildcard-depth clarification, not the feature origin)
and notes the "N0/N1/N2" shorthand is ours, not AISI nomenclature.

ENG-219
@Yiminnn

Yiminnn commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Network-mode verification — citation-check across network strategies × {docker, daytona}

Built six citation-check task variants that differ only in their environment network block, and ran them through benchflow's real launch path on both sandboxes. citation-check is a good probe because its skills call external APIs (CrossRef, NCBI/PubMed, DOI, arXiv), so enforcement is observable end-to-end.

1 · Deterministic policy + preflight matrix (authoritative, no model spend)

Each variant resolved through the real validate_task_runtime_support + resolve_network_decision:

variant   sandbox  preflight          policy     lane   allowed_hosts
public    docker   ok                 OPEN       False  -
public    daytona  ok                 OPEN       False  -
allow     docker   ok                 ALLOWLIST  True   api.crossref.org,eutils.ncbi.nlm.nih.gov,doi.org,export.arxiv.org
allow     daytona  REJECT(network)    BLOCK_ALL  True   …  <=allowlist (fail-closed)
wildcard  docker   ok                 ALLOWLIST  True   *.crossref.org,*.ncbi.nlm.nih.gov,doi.org,*.arxiv.org
wildcard  daytona  REJECT(network)    BLOCK_ALL  True   …  <=allowlist (fail-closed)
deny      docker   ok                 ALLOWLIST  True   example.com
deny      daytona  REJECT(network)    BLOCK_ALL  True   example.com  <=allowlist (fail-closed)
nonet     docker   ok                 BLOCK_ALL  True   -
nonet     daytona  ok                 BLOCK_ALL  True   -
hermetic  docker   ok                 BLOCK_ALL  False  -      ← allow_model_endpoint:false → lane OFF
hermetic  daytona  ok                 BLOCK_ALL  False  -

✅ docker resolves allowlist/no-network correctly with the model lane; hermetic (allow_model_endpoint:false) correctly turns the lane off. ✅ daytona fails closed on every allowlist variant (rejected at preflight and backstopped to BLOCK_ALL).

2 · Live parallel matrix — real openhands agent runs, --concurrency 5 on each sandbox

strategy (environment.network_mode) docker daytona
public ✅ reward 1.0 ✅ reward 1.0
allowlist — full (*.ncbi.nlm.nih.gov … + agent/model hosts) ⚠️ ERR -32603 (model 404 — see note 3, not a proxy block) 🚫 preflight REJECT (fail-closed)
allowlist — deny (example.com only) 🚫 ERR — proxy 403'd agent install 🚫 preflight REJECT
no-network 🚫 ERR — no route, agent can't install ✅ reward 1.0 (see note 2)
hermetic (no-network + allow_model_endpoint:false) 🚫 ERR — no route ✅ reward 1.0 (see note 2)

docker batch — Score: 1/5, errors=4:

[ERR] citation-check-nonet     (Agent openhands install failed (rc=127))
[ERR] citation-check-deny      (Agent openhands install failed (rc=127))
[ERR] citation-check-hermetic  (Agent openhands install failed (rc=127))
[ERR] citation-check-wildcard  (ACP error -32603: Internal error)
Job complete: 1/5 (20.0%), errors=4, time=7.2min          # the 1 pass = public (reward 1.0)

daytona batch — Score: 3/5, errors=2:

benchflow.task.runtime_capabilities.UnsupportedTaskFeatureError:
- environment.network_mode: network_mode='allowlist' is enforced only on the 'docker' sandbox
  (egress proxy); not available on this sandbox — use 'docker', 'no-network', or 'public' (ENG-219) (sandbox=daytona)
[ERR] citation-check-deny      (Task uses parsed runtime features that BenchFlow…)
[ERR] citation-check-wildcard  (Task uses parsed runtime features that BenchFlow…)
Job complete: 3/5 (60.0%), errors=2, time=6.2min          # passes = public, nonet, hermetic

3 · The proxy is enforcing in a live agent run (the load-bearing evidence)

For the deny variant the egress override is wired straight from the task.md, and the sidecar actively 403s every non-allowlisted host the agent reaches:

# applied egress override (from task.md allowed_hosts: [example.com])
{'ALLOWED_HOSTS': 'example.com', 'PORT': '8080', 'BENCHFLOW_EGRESS_LANE_HOST': '172.17.0.1'}

# openhands install log — proxy (172.21.0.2:8080) denies the Ubuntu mirrors (not allowlisted):
E: Failed to fetch http://archive.ubuntu.com/.../InRelease   403  Forbidden [IP: 172.21.0.2 8080]
E: Failed to fetch http://security.ubuntu.com/.../InRelease  403  Forbidden [IP: 172.21.0.2 8080]

When those install hosts are allowlisted, the proxy forwards them and the agent installs cleanly (apt over plain-HTTP through the proxy worked — no absolute-URI 400). So the allowlist permits exactly the listed hosts and denies the rest, in a real run.

Notes / findings

  1. Model lane covers host.docker.internal, but this deployment calls the model provider directly. The agent env sets LLM_BASE_URL=https://api.deepseek.com (not a host-side LiteLLM proxy at host.docker.internal:<port>), so the lane (172.17.0.1) is inert here and the real provider host must be allowlisted for a restrictive run. The lane mechanism itself is sound (verified separately: a host listener at 172.17.0.1 is reachable through the sidecar), but its automatic benefit only fires in the host-proxy topology. → worth folding into ENG-262 (the lane should also admit the resolved provider host under a restrictive policy, since that host is known at launch).

  2. docker vs daytona no-network strictness differs. docker enforces no-network from the start — the agent can't even install (rc=127). daytona's no-network let the agent install, reach the model, and solve (nonet/hermetic → reward 1.0). daytona backends still read allow_internet directly rather than routing through resolve_network_decision (the ENG-262 C1 defense-in-depth item), and appear to install the agent before applying the policy. Flagging as a real cross-backend inconsistency.

  3. The -32603 on docker allowlist is a model-routing quirk, not a network block. deepseek-v4-flash 404s on the configured api.deepseek.com (the model is actually served by a separate internal endpoint); even the public run hit 37× 404 before succeeding via retries to the working endpoint. The proxy permitted every listed host (install succeeded through it); the agent just couldn't complete the model call cleanly under the tighter time budget. Orthogonal to this PR.

Bottom line: the network_mode enforcement is solid — policy resolution is correct on both sandboxes, docker enforces allowlist/no-network (live 403s on unlisted hosts, agent install blocked under restrictive policy), daytona fails closed on allowlist at preflight, and allow_model_endpoint:false correctly closes the lane. The two items above (model-lane direct-provider coverage, daytona no-network strictness) are follow-ups, both already in ENG-262's scope.

Yiminnn added 4 commits June 16, 2026 03:34
Derive daytona block-all from resolve_network_decision (not the deprecated
allow_internet bool) and verify it after start with a raw-TCP egress canary;
abort with SandboxStartupError if a block-all policy can still reach the
network (platform did not honor network_block_all).
…ctive policy (ENG-263)

Under no-network/allowlist the agent cannot silently fall back to the direct
provider (the egress allowlist blocks it) — so a skipped/unavailable LiteLLM
usage proxy must abort the run instead of leaving the agent pointed at a
blocked, untracked provider endpoint. Adds network_is_restrictive +
proxy_unavailable_is_fatal and threads it through ensure_litellm_runtime.
…263)

The container now comes up open so the agent can install; relock_network()
then drops it off the public bridge (and, for allowlist/model-lane, starts the
bf-egress sidecar + moves it onto the internal-only net, returning HTTP_PROXY
for the agent). The main container is never recreated, so the install survives.
Called from install_agent() after lockdown_paths. Fixes rc=127 agent-install
failures under no-network/tight-allowlist on docker.
…s probe) (ENG-263)

The daytona no-network leak was the _lift_agent_network_to_public blanket lift
to public (the same P0 the model lane replaced for docker), still firing on
non-docker. Gate the lift for daytona/daytona-dind too (no lane there -> the
honest behavior is to keep the policy and fail closed). Also fix the egress
canary probe to a single-line python -c (the heredoc form did not execute
through daytona exec) so block-all enforcement is actually verified at start.

Verified live: daytona no-network now resolves block_all=True and the canary
(1.1.1.1:443) is unreachable -- genuinely offline, no longer silently public.
@bingran-you bingran-you added enhancement New feature or request P1 Important debt — must fix soon, but does not block the current release. status:blocked Waiting on external dependency. Add a comment explaining why. review:changes-requested Author needs to push more commits before this can merge. area:sandbox Issue / PR lives primarily in the "sandbox" subsystem. labels Jun 16, 2026
@bingran-you

Copy link
Copy Markdown
Collaborator

Automation triage (2026-06-16): still belongs in benchflow-ai/benchflow and the live verification evidence is strong, but GitHub now reports the PR as merge-conflicting (mergeStateStatus=DIRTY) with no checks on the latest head. I labeled it status:blocked / review:changes-requested / area:sandbox.\n\nNext step: rebase/merge origin/main, resolve conflicts, rerun CI + the targeted docker/daytona network-mode verification, then re-review before merge.

Resolves conflicts from #787 (Mintlify docs retirement):
- docs/docs.json: honored main's deletion (#787 retired the Mintlify
  config); the network-mode-prior-art nav entry is moot.
- docs/sandbox-hardening.md: kept main's reworded task-authoring.md line
  and Yimin's network-mode-prior-art cross-link.

Code merged cleanly. Verified: test_network_modes + test_network_policy
+ test_internet_policy + test_runtime_capabilities = 92 passed; ruff +
format + ty clean on the sandbox surface.
@bingran-you
bingran-you temporarily deployed to pypi-internal-preview June 16, 2026 06:07 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Automation (2026-06-16): merged origin/main into the branch and resolved the conflicts (pushed 70303ce5).

The only conflicts were docs from #787 (Mintlify config retirement):

Code merged cleanly (config.py / test files auto-merged, no logic conflicts). Re-verified locally: test_network_modes + test_network_policy + test_internet_policy + test_runtime_capabilities = 92 passed; ruff + format + ty clean on the sandbox surface.

PR should now be mergeable and CI can run on the fresh head. Still needs a human review (security-sensitive egress enforcement) + the docker/daytona network-mode E2E re-run before merge — removing status:blocked once CI is green.

@bingran-you bingran-you added review:pending PR is ready-for-review, no reviewer engagement yet. and removed status:blocked Waiting on external dependency. Add a comment explaining why. review:changes-requested Author needs to push more commits before this can merge. labels Jun 16, 2026
@Yiminnn

Yiminnn commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Runtime-behavior follow-up (ENG-263) — fixes + live verification on both sandboxes

Verifying this PR's enforcement with a citation-check network-strategy matrix on docker and daytona surfaced three runtime divergences from the designed behavior (policy resolution was correct; the runtime wasn't). All three are now fixed on feat/eng-263-network-runtime, TDD'd (tests/test_network_runtime.py), 678 tests green, ruff/ty clean. Live results below (terminal captures).

Fix #1 — docker install-before-lockdown (was: agent install rc=127)

The agent installs after the policy goes live at compose up, so under no-network/tight-allowlist the install (apt/uv/pypi/github) was 403'd → openhands exit 127. Now the container comes up open for the install phase, then relock_network() drops it off the bridge non-destructively (main is never recreated — docker network disconnect/connect + up --no-deps bf-egress), returning HTTP_PROXY for the agent.

citation-check-allowcitation-only allowlist, no install/model hosts (died at rc=127 before):

Starting environment: citation-check-allow
relock_network: ALLOWLIST applied (sidecar=True)
Session: 53f168d2-...    Prompt 1/1: You are helping a research team verify the integrity of their bibliography...

→ agent installs and reaches the prompt; no rc=127.

Fix #3 — force the host usage proxy under a restrictive policy

A restrictive policy can no longer silently fall back to the (blocked, untracked) direct provider. Same run + citation-check-nonet:

relock_network: BLOCK_ALL applied (sidecar=True)
LiteLLM proxy listening on http://172.17.0.1:39921
applied egress  →  ALLOWED_HOSTS:''  +  BENCHFLOW_EGRESS_LANE_HOST:172.17.0.1   (model-only egress)

→ host LiteLLM proxy is forced + listening, the lane covers it, and no-network resolves to empty-allowlist + model-lane exactly as designed.

Fix #2 — daytona no-network now actually enforced

The daytona "leak" wasn't a leak — it was _lift_agent_network_to_public lifting no-networkpublic (the same P0 the model lane replaced for docker), still firing on non-docker. Gated for daytona/daytona-dind + a reliable egress canary at start:

verify_network_enforcement: block_all=True
egress canary 1.1.1.1:443 reachable=False        ← genuinely offline now
[ERR] citation-check-nonet (offline → agent install can't run)

Before: solved with reward 1.0 by reaching CrossRef + the model. Now: correctly fails-closed / cannot run offline. The probe also aborts the run if a future platform ever ignores the block-all flag.

Honest caveats (none are network-mode bugs)

  • The -32603 on the docker runs is an orthogonal model-routing quirk — the model deepseek-v4-flash 404s on the configured api.deepseek.com (it's served by a separate internal endpoint). The network path is correct (proxy forced, lane covers it); the proxy just relays the 404. Fix belongs in the provider registry.
  • daytona no-network + a runtime-installed agent fails at install — correct: daytona's network is fixed at sandbox creation, so it can't reorder; such tasks need pre-baked agents. It now fails cleanly instead of silently running public.
  • modal still lifts to public (not gated) — follow-up.

Full design + decisions in ENG-263. Screenshots of each run available on request.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation follow-up (2026-07-13): functionally ready by simulation, still blocked by active human review on current head 9f7599f638e9737cae3e51325fc21e321df645c8.

What changed:

  • Merged current origin/main into feat/network-mode-enforcement, resolving the only conflict in CHANGELOG.md while preserving both network-mode and mainline TRL/LLM provenance notes.
  • Refreshed click to 8.4.2 to clear PYSEC-2026-2132.
  • Updated the removed-short-option regression to tolerate Click 8.4's quoted error wording while preserving the same behavior check.

Validation:

  • Local sandbox/network gate: uv sync --extra dev --extra sandbox-daytona --extra sandbox-modal --locked; targeted network/runtime tests -> 120 passed; exact pip-audit export passed with the existing cryptography ignore; ruff; format check; ty check src/; bench --help; git diff --check.
  • GitHub current-head checks are green: test, pip-audit, manifest-parity / parity, integration-light / rollout-smoke, and detect-scope.
  • Current-head integration-light-jobs artifact passed benchflow-experiment-review: 1/1 rollout healthy, reward 1.0, ACP + LLM trajectories and training-ready results.jsonl present, 321,587 tokens, 15 tool calls, total timing 369.6s.

No direct code/config blocker remains from this automation pass. Labels intentionally stay status:blocked + review:changes-requested because GitHub still reports reviewDecision=CHANGES_REQUESTED; next action is security/human re-review or dismissal of that review gate.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation review (2026-07-14): functionally ready by simulation, still blocked by active human review on current head 9f7599f638e9737cae3e51325fc21e321df645c8.

Scope checked: restrictive network-mode enforcement, allow_model_endpoint=false behavior, Docker/Daytona relock helpers, provider-lane handling, CLI/SDK regression risk, and current rollout artifact health.

Evidence:

  • Shared base gates passed in the main checkout: uv sync --extra dev --extra sandbox-daytona --locked, repo ruff check ., repo ty check src, CLI help/list/check surfaces, and SDK smoke.
  • Current-head isolated worktree gates passed: uv sync --extra dev --extra sandbox-daytona --locked; focused network/runtime suite -> 120 passed; focused ruff; ty check src; CLI + SDK smoke.
  • GitHub current-head checks are green: test, pip-audit, manifest-parity / parity, and integration-light / rollout-smoke passed.
  • Current-head integration-light artifact run 29253274941 validated healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, one training-ready results.jsonl row, reward present, 21/21 LLM responses with usage, 321,587 tokens, 15 tool calls, timing total 369.6s.

Blocking state:

  • GitHub still reports reviewDecision=CHANGES_REQUESTED and mergeStateStatus=BLOCKED. I did not find a fresh code/config blocker in today’s simulation, but this security-sensitive sandbox PR should stay status:blocked / review:changes-requested until the human review gate is cleared.

Thermo-nuclear review:

  • No new structural blocker found. The network policy is split into focused helper modules; docker.py is 931 lines and daytona.py is 970 lines, both below the 1k threshold; no CLI/SDK regression surfaced.

Cost caveat:

  • The GLM user-endpoint artifact has complete token/timing telemetry, but result.json.agent_result.cost_usd and price_source are still null.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation review (2026-07-19): behavior/artifact ready by simulation, still blocked by human review gate on current head 9f7599f638e9737cae3e51325fc21e321df645c8.

Fresh validation passed in an isolated checkout:

  • uv sync --extra dev --extra sandbox-daytona --locked.
  • Focused network/runtime suite: 97 passed across tests/test_network_runtime.py, tests/test_network_policy.py, tests/test_network_modes.py, tests/test_connect_as_env.py, and tests/test_runtime_capabilities.py.
  • ruff check ., ruff format --check src tests, ty check src, and git diff --check origin/main...HEAD.
  • CLI/SDK smoke: bench --help, bench agent list, bench eval run --help, bench tasks check --help, Scene.single(...), and RolloutConfig(...).
  • Current-head integration-light-jobs artifact from run 29253274941 validates healthy with benchflow-experiment-review: 1/1 healthy, ACP trajectory, LLM trajectory, training-ready results.jsonl, reward present, 321,587 tokens, 15 tool calls, 21 LLM exchanges with usage, and timing metadata.

Security/runtime checks remain covered: allow_model_endpoint=false fail-closed behavior, Docker bf-egress readiness before proxy env, restrictive snapshot restore rejection, and Daytona relock fail-closed cases all have focused regression coverage. Thermo-nuclear review found no new structural blocker on this head; large changed files stay below the 1k threshold for the sandbox backends (docker.py 931 lines, daytona.py 970 lines), with enforcement split into focused helper modules.

Caveat: the GLM user-endpoint artifact still has result.json.agent_result.cost_usd=null and price_source=null; token usage and timing are complete, and the validator treats the rollout as healthy. Keeping status:blocked + review:changes-requested because GitHub still reports the live human CHANGES_REQUESTED review gate.

@bingran-you

Copy link
Copy Markdown
Collaborator

FrontierPhysics blocker impact — July 22, 2026

This network-mode enforcement is now the shared terminal blocker for the FrontierPhysics open-PR QA campaign. Multiple exact-head Daytona canaries declare agent / environment / verifier network_mode: no-network, yet the released runtime still permits shell egress to GitHub/arXiv after agent install. Environment-only declarations also proved insufficient.

We have moved affected task PRs back to reward-hack-fixing / task-review-changes-fixing and are withholding every terminal reward-hack-green / task-review-approved label until a live denial canary passes. PR #785 current head is green by checks and focused runtime simulation; the remaining blocker is the active security/human review. Please prioritize re-review rather than bypassing it.

Representative FrontierPhysics follow-ups: #263/#264 and current task heads #148/#152/#258/#260/#262.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation review (2026-07-22): blocked by current code/config findings, not just the human review gate, on head 9f7599f638e9737cae3e51325fc21e321df645c8.

Fresh PR-scoped validation still passes the expected deterministic surface:

  • uv sync --extra dev --extra sandbox-daytona --extra sandbox-modal --locked
  • network-focused tests: 126 passed
  • broader CLI/SDK/runtime regression slice: 393 passed, 1 skipped
  • ruff check ., ruff format --check src tests tools, ty check src/, uv lock --check, git diff --check, and git merge-tree passed
  • CLI smoke and task check/digest paths passed
  • local egress proxy probe allowed the allowlisted localhost path and denied the non-allowlisted path
  • GitHub test, pip-audit, manifest-parity, and integration-light / rollout-smoke are green
  • current integration-light-jobs artifact from run 29253274941 validates healthy 1/1 with ACP + LLM trajectories, training-ready results.jsonl, reward 1.0, 321,587 tokens, 15 tool calls, and timing metadata

Blocking findings reproduced in the sidecar:

  • agent.network_mode and verifier.network_mode are accepted by capability validation but ignored at launch; only environment.network_mode reaches _lock_down_network(). The sidecar reproduced runtime_issues=[] while the effective launch policy stayed OPEN.
  • Daytona no-network resolves to BLOCK_ALL with model_lane=True, but startup passes platform network_block_all=True, relock_daytona_network() no-ops for block-all, and the restrictive LiteLLM setup can fall back to direct provider access. That does not provision the documented model lane under restrictive Daytona policy.
  • pick_daytona_canary() falls back to 1.1.1.1 when every candidate is allowlisted, which can produce a false enforcement failure.
  • Validation gap: local Docker is unavailable here, and the green CI artifact uses network_mode: null, so it does not prove allowlist/no-network enforcement.

This matches the FrontierPhysics blocker impact noted earlier today: terminal no-network canaries cannot be treated as secure until these policy paths are fixed and re-reviewed. Keeping status:blocked + review:changes-requested.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 22, 2026 13:09 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation fix (2026-07-22): pushed eee107b380d1ffe02c0978a7b1d56cf296eda041 to address the current code/config blockers on this PR.

What changed:

  • agent.network_mode and verifier.network_mode now collapse into the effective shared sandbox policy before environment creation, so validation can no longer accept role-level no-network/allowlist settings that launch as OPEN.
  • Rollout lockdown reads the sandbox's effective task_env_config, not the raw task environment block.
  • Daytona no-network with the model lane enabled now installs first and then relocks to a provider-only CIDR allowlist; allow_model_endpoint=false remains fully hermetic/platform block-all.
  • Daytona canary selection now uses a pool larger than the 10-CIDR platform limit and fails closed instead of falling back to an already-allowlisted IP.

Validation on the pushed head:

  • uv run --extra dev --extra sandbox-daytona --extra sandbox-modal python -m pytest tests/test_network_modes.py tests/test_network_runtime.py tests/test_runtime_capabilities.py -q -> 84 passed
  • broader runtime/config slice -> 182 passed, 2 expected deprecation warnings
  • uv run ruff check .; uv run ruff format --check src tests tools; uv run ty check src/; uv lock --check; git diff --check origin/main...HEAD; git merge-tree $(git merge-base origin/main HEAD) origin/main HEAD; CLI smoke (bench --help, bench agent list, bench eval run --help, bench tasks check --help) -> passed

Labels intentionally remain status:blocked + review:changes-requested: this is still a security-sensitive network-control PR and needs current-head CI plus human security re-review before merge.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 22, 2026 13:13 — with GitHub Actions Inactive
@bingran-you

bingran-you commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Follow-up on the same automation fix: pushed 3476e05479f3966becb79a73e63255ee628b9a4d to clear the new current-head pip-audit failure.

The failed job reported mcp 1.27.2 / CVE-2026-59950; the branch now has a surgical uv.lock refresh to mcp 1.28.1. Local validation:

  • exact CI audit export + pip-audit --ignore-vuln GHSA-537c-gmf6-5ccf -> no known vulnerabilities found, 1 existing ignore
  • focused network/runtime/capability tests -> 84 passed
  • ruff check ., ruff format --check src tests tools, ty check src/, uv lock --check, and git diff --check origin/main...HEAD -> passed

Labels remain status:blocked + review:changes-requested while current-head CI reruns and the security review gate remains active.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 22, 2026 13:17 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Second follow-up on the automation fix: pushed a5c564cb3f4a1af30870e5de0c2f7ea32a6921fa to fix the new test failure after the network-policy patch.

Root cause: the sandbox setup change correctly folded role-level network policies for full TaskConfig, but a few env setup/internet policy tests use partial task-config fakes with only environment. The setup path now uses getattr(task.config, "agent"/"verifier", None), preserving those compatibility surfaces while still enforcing real agent/verifier network declarations.

Validation after the fix:

  • failed-CI reproduction slice: tests/test_env_setup.py tests/test_internet_policy.py tests/test_network_modes.py tests/test_network_runtime.py tests/test_runtime_capabilities.py -> 141 passed
  • ruff check src tests tools; ruff format --check src tests tools; ty check src/; uv lock --check; git diff --check origin/main...HEAD -> passed

Labels remain status:blocked + review:changes-requested while current-head CI reruns and the human security review gate remains active.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation follow-up (2026-07-22): current-head CI and artifact validation are now complete on a5c564cb3f4a1af30870e5de0c2f7ea32a6921fa.

Current-head GitHub checks are green: test, pip-audit, manifest-parity / parity, integration-light / detect-scope, integration-scope / detect-scope, and integration-light / rollout-smoke passed; integration-scope matrix/final-review jobs skipped by scope.

Artifact audit: downloaded run 29923258051 integration-light-jobs and validated with .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py /tmp/pr785-artifacts.msoxXb/integration-light-jobs --json: healthy 1/1, ACP + LLM trajectories present, rollout-level results.jsonl training-ready, reward present, 313,299 tokens, 14 tool calls, 25 ACP events, and 16/16 LLM responses with usage.

The direct code/config blockers found earlier today have been addressed on-branch and current-head CI is green. Labels intentionally remain status:blocked + review:changes-requested because this is still a security-sensitive sandbox/network-control PR with an active human CHANGES_REQUESTED review gate.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation review (2026-07-23): still blocked on current head a5c564cb3f4a1af30870e5de0c2f7ea32a6921fa.

What passes:

  • PR-scoped sidecar reran focused network/runtime checks (101 passed) and an expanded runtime/provider slice (303 passed). ruff, format, ty, uv lock --check, and git diff --check passed.
  • Exact-head CI is green (test, pip-audit, manifest-parity, rollout-smoke).
  • Existing exact-head artifact from run 29923258051 validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, training-ready results.jsonl, reward 1.0, 313,299 tokens, and complete timing.

Why this is still not ready:

  • The validated artifact has network_mode: null, so it does not prove the restrictive network enforcement this PR changes.
  • Daytona restrictive snapshot restore can reopen networking: DaytonaSandbox.restore() recreates with network_block_all but does not reapply network_allow_list or call relock_network after restore.
  • Custom provider endpoints are not handled for restrictive allowlisting: provider_host_for_model() ignores explicit BENCHFLOW_PROVIDER_BASE_URL, so a custom OpenAI endpoint can allowlist the wrong host and vllm/... can resolve to None.
  • Under restrictive policies, the LiteLLM runtime skip path can still return a direct-provider path with runtime=None even when usage_tracking=required; missing usage is then caught only after the provider call rather than failing before launch.
  • Heterogeneous scene roles can switch provider in connect_as() without extending/relocking the model lane; initial lockdown only covers the primary model host.
  • Daytona oracle + no-network fails by default when allow_model_endpoint=true because there is no provider host and the empty model-lane plan is rejected.

Keeping this blocked until the restrictive Daytona restore, custom-provider allowlisting, required-usage fail-closed behavior, heterogeneous-role relock behavior, and oracle/no-network default are fixed and revalidated.

@bingran-you bingran-you added the bug Something isn't working label Jul 23, 2026

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@bingran-you
bingran-you had a problem deploying to pypi-internal-preview July 23, 2026 13:17 — with GitHub Actions Failure
@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 23, 2026 13:31 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation fix (2026-07-23): pushed b2b3652940339d6cbdebcb72ac6dafc864a67b09 to address the current code/config blockers called out on this head.

What changed:

  • Daytona direct snapshot restore now reapplies the active restrictive allowlist/model lane, or verifies platform block-all after restore.
  • Daytona no-network with no model host (for example oracle) now relocks to platform block-all instead of trying to build an empty model-lane allowlist.
  • Restrictive lockdown now honors explicit BENCHFLOW_PROVIDER_BASE_URL endpoints, including user-supplied vllm/... endpoints.
  • usage_tracking=required fails before launch when restrictive Docker/Daytona policy would skip the LiteLLM proxy and run direct-provider.
  • connect_as() now extends/relocks the model lane when a later scene role switches provider.
  • Added PR feat(sandbox): Enforce network_mode allowlist on docker #785 guard coverage in tests/test_network_pr785_regressions.py plus the role relock test in tests/test_trial_litellm_runtime.py, keeping tests/test_network_runtime.py below the 1k-line review threshold.

Validation:

  • Focused network/provider/role suite: 191 passed.
  • Broader runtime/config slice: 320 passed, 2 expected deprecation warnings.
  • Static/merge gates passed: ruff, format check, ty check src/, uv lock --check, git diff --check, and git merge-tree --write-tree origin/main HEAD.
  • Current-head GitHub checks are green/skipped by scope: test, pip-audit, manifest-parity / parity, both detect-scope jobs, and integration-light / rollout-smoke passed.
  • The first smoke attempt produced a complete reward-1.0 artifact but failed the CI gate by hitting the 600s wall-clock budget at 600.7s. I reran the failed job without code changes; the rerun passed.
  • Successful artifact audit from run 30010552324, job 89220843073: benchflow-experiment-review reports healthy 1/1, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 319,434 tokens, 16 tool calls, 27 ACP events, and 18/18 LLM responses with usage.

Labels intentionally remain status:blocked + review:changes-requested: the direct code/config blockers are fixed and CI is green, but this is still a security-sensitive sandbox/network PR with an active human CHANGES_REQUESTED review gate. Human security re-review should decide whether the deterministic coverage is enough or whether to record an additional live restrictive Daytona/FrontierPhysics canary before clearing the gate.

@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation review (2026-07-24): still blocked on current head b2b3652940339d6cbdebcb72ac6dafc864a67b09.

What passes:

  • The July 23 fixes cover the prior blockers around role-level policy folding, custom BENCHFLOW_PROVIDER_BASE_URL, usage_tracking=required fail-closed behavior, Daytona snapshot restore relock, and oracle + no-network block-all behavior.
  • PR-scoped sidecar validation passed uv sync --extra dev --extra sandbox-daytona --locked; focused network/provider/runtime tests (275 passed, 2 skipped); ruff, format, ty check src/, uv lock --check, git diff --check, merge-tree, CLI smoke, and public SDK smoke.
  • Exact-head integration-light-jobs run 30010552324 structurally passes benchflow-experiment-review: 1/1 healthy, ACP + LLM trajectories, one training-ready results.jsonl, reward 1.0, 319,434 tokens, 18/18 LLM responses with usage, 16 tool calls, and 360.5s timing.

Current blocker:

  • Heterogeneous Daytona connect_as() is still not fully enforced. src/benchflow/sandbox/daytona_network_lockdown.py stores all extra_allowed_hosts, but then plans the Daytona allowlist with only extra_allowed_hosts[0] as model_host. A sidecar repro with multiple providers retained only the first host (api.openai.com); a later role/provider host is omitted from the relock plan and can be blocked even though connect_as() accepted it.
  • The current artifact is Docker/public (network_mode: null), so it does not prove restrictive Daytona multi-provider behavior.

Recommended fix: thread every extra model/provider host into the Daytona allowlist plan, add a PR #785 regression for multiple provider hosts across connect_as(), and rerun a restrictive Daytona canary or equivalent exact-path test before moving this out of status:blocked.

Labels are already correct: keeping status:blocked + review:changes-requested.

AI-generated automation review posted on behalf of Bingran You.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 24, 2026 13:08 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator

Users Simulation automation fix (2026-07-24): pushed f4a35bcf07e34a0f7ee8c7fa74f5cec4d1a58a1b to address the current code/config blocker on PR #785.

What changed:

  • Daytona restrictive relock now carries every accumulated model/provider lane host into the platform CIDR allowlist plan, not only extra_allowed_hosts[0].
  • The planner input is the de-duplicated task allowlist plus all model-lane hosts, so heterogeneous connect_as() role switches keep both the original provider host and later role provider hosts enforceable.
  • Added a PR feat(sandbox): Enforce network_mode allowlist on docker #785 regression for multiple model-lane hosts and updated the existing Daytona model-lane relock expectation.

Validation:

  • uv sync --extra dev --extra sandbox-daytona --locked -> passed.
  • Exact regression slice -> 8 passed.
  • Focused network/runtime slice -> 106 passed.
  • Broader runtime/config slice -> 320 passed, 1 skipped, 2 expected deprecation warnings.
  • Static/merge gates passed: ruff, format check, ty check src/, uv lock --check, git diff --check, and git merge-tree --write-tree origin/main HEAD.
  • Current-head GitHub checks are green/skipped by scope: test, pip-audit, manifest-parity / parity, both detect-scope jobs, and integration-light / rollout-smoke passed; integration-scope matrix/review jobs skipped by scope.
  • Artifact audit from run 30095697606, rollout-smoke job 89489136877: benchflow-experiment-review validator reports healthy 1/1, ACP + LLM trajectories present, rollout-level results.jsonl training-ready, reward 1.0, 228,682 tokens, 12 tool calls, 21 ACP events, 16/16 LLM responses with usage, and 261.4s total timing.

Labels intentionally remain status:blocked + review:changes-requested: the direct July 24 code/config blocker is fixed and current-head CI is green, but this remains a security-sensitive sandbox/network PR with an active human CHANGES_REQUESTED review gate. The current smoke artifact is Docker/public (network_mode: null), so human security re-review should decide whether the deterministic exact-path regression is enough or whether to record a live restrictive Daytona/FrontierPhysics canary before clearing the gate.

AI-generated automation fix posted on behalf of Bingran You.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Issue / PR lives primarily in the "sandbox" subsystem. bug Something isn't working enhancement New feature or request P1 Important debt — must fix soon, but does not block the current release. review:changes-requested Author needs to push more commits before this can merge. status:blocked Waiting on external dependency. Add a comment explaining why.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants