Skip to content

fix(sandbox): keep programmatic tool calling items in memory rollouts - #4260

Open
hsusul wants to merge 1 commit into
openai:mainfrom
hsusul:fix/rollout-program-item-types
Open

fix(sandbox): keep programmatic tool calling items in memory rollouts#4260
hsusul wants to merge 1 commit into
openai:mainfrom
hsusul:fix/rollout-program-item-types

Conversation

@hsusul

@hsusul hsusul commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Sandbox memory rollouts silently drop Programmatic Tool Calling items.

_should_include_memory_item() in src/agents/sandbox/memory/rollouts.py keeps an item only when its type is listed in _INCLUDED_MEMORY_ITEM_TYPES (an explicit _EXCLUDED_MEMORY_ITEM_TYPES set names the types that are deliberately dropped; everything else falls through to dropped). That set lists both halves of every call/output pair in _TOOL_CALL_TO_OUTPUT_TYPE (run_internal/items.py) — function_call, custom_tool_call, shell_call, apply_patch_call, computer_call, local_shell_call, tool_search_call and their outputs — plus web_search_call, mcp_call, mcp_approval_request, mcp_approval_response.

The one pair missing is program / program_output.

The set was written in #2889 (2026-04-15) and has not been edited since. Programmatic Tool Calling added the program and program_output item types later, in #3833 (2026-07-17), and this call site was not updated — so a run that uses ProgrammaticToolCallingTool writes a rollout with the program code and the program result removed.

The rollout is the raw evidence that phase-one/phase-two memory generation reads, so the loss is not cosmetic. Worse, the filter is inconsistent with itself: the function_call items a program issues are retained, and each one carries caller: {"type": "program", "caller_id": ...} pointing at a program item that was just removed. The stored rollout therefore contains dangling caller references.

Before, for a single programmatic turn (programfunction_callfunction_call_outputprogram_output):

generated_items: ["function_call", "function_call_output"]
  function_call.caller = {"type": "program", "caller_id": "call_prog_1"}   # target not in the rollout

After:

generated_items: ["program", "function_call", "function_call_output", "program_output"]

Fix: add "program" and "program_output" to the existing frozenset. Two strings — no new branch, no new helper, no public API or stored-format change.

Deliberately not widened. file_search_call, code_interpreter_call, mcp_list_tools and hosted_tool_call are also absent from the set, but they are not part of the contract this change asserts: the claim here is only that every call/output pair the SDK knows how to pair should survive into the rollout, and program is the sole pair that does not. image_generation_call stays excluded via _EXCLUDED_MEMORY_ITEM_TYPES. There is a boundary test covering exactly this.

Test plan

Three tests added to tests/sandbox/test_memory.py:

  • test_build_rollout_payload_keeps_programmatic_tool_calling_items — a full programmatic turn as RunItems (Program, a function_call carrying CallerProgram, its output, ProgramOutput). Asserts all four survive in order, that the program's code and the program output's result are preserved, and that the retained function_call.caller resolves to a program item that is present.
  • test_build_rollout_payload_keeps_program_items_from_input — the same types arriving on the input side, which goes through the same filter.
  • test_build_rollout_payload_still_drops_hosted_items_outside_the_included_set — boundary: file_search_call and image_generation_call are still dropped while program is kept, so the narrow scope is pinned.

All three fail on upstream/main @ f3b6c617 and pass with the change; verified by reverting only the source hunk and re-running (3 failed, 3 passed6 passed). The three pre-existing build_rollout_payload tests pass unchanged in both states.

Commands run:

  • uv run pytest tests/sandbox/test_memory.py -q73 passed
  • uv run pytest tests/sandbox tests/test_programmatic_tool_calling.py -q1130 passed, 2 skipped
  • make format862 files left unchanged
  • make lintAll checks passed!
  • make typecheck → mypy and pyright clean (exit 0)
  • make tests6744 passed, 29 skipped (parallel) + 77 passed, 5 skipped (serial)
  • git diff --check → clean

No API key, no network, no live service.

Not run: make coverage, make build-docs, the Python 3.10 matrix, and the integration-test profiles.

One unrelated flake seen while validating: tests/extensions/memory/test_advanced_sqlite_session.py::test_branch_allocation_is_serialized_across_processes fails intermittently (~1 in 5) under CPU contention at assert first_ready.wait(timeout=10), a fixed wall-clock wait on a multiprocessing.Event. It reproduces with this change reverted and is not in this change's import graph; make tests is green on this branch when the machine is idle.

Issue number

None.

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

_INCLUDED_MEMORY_ITEM_TYPES lists both halves of every call/output pair in
_TOOL_CALL_TO_OUTPUT_TYPE, plus web_search_call, mcp_call and the MCP approval
items. The program/program_output pair was the only one missing: the set was
written in openai#2889 and Programmatic Tool Calling introduced those two types later
in openai#3833 without updating this call site.

A run that uses ProgrammaticToolCallingTool therefore wrote a rollout with the
program code and the program result stripped, while the function_call items the
program issued were retained -- each still carrying a caller reference to a
program item that was no longer in the rollout.

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you resolve the typecheck error?

@seratch seratch added this to the 0.20.x milestone Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants