fix(sandbox): keep programmatic tool calling items in memory rollouts - #4260
Open
hsusul wants to merge 1 commit into
Open
fix(sandbox): keep programmatic tool calling items in memory rollouts#4260hsusul wants to merge 1 commit into
hsusul wants to merge 1 commit into
Conversation
_INCLUDED_MEMORY_ITEM_TYPES lists both halves of every call/output pair in _TOOL_CALL_TO_OUTPUT_TYPE, plus web_search_call, mcp_call and the MCP approval items. The program/program_output pair was the only one missing: the set was written in openai#2889 and Programmatic Tool Calling introduced those two types later in openai#3833 without updating this call site. A run that uses ProgrammaticToolCallingTool therefore wrote a rollout with the program code and the program result stripped, while the function_call items the program issued were retained -- each still carrying a caller reference to a program item that was no longer in the rollout.
seratch
requested changes
Aug 6, 2026
seratch
left a comment
Member
There was a problem hiding this comment.
Can you resolve the typecheck error?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sandbox memory rollouts silently drop Programmatic Tool Calling items.
_should_include_memory_item()insrc/agents/sandbox/memory/rollouts.pykeeps an item only when itstypeis listed in_INCLUDED_MEMORY_ITEM_TYPES(an explicit_EXCLUDED_MEMORY_ITEM_TYPESset names the types that are deliberately dropped; everything else falls through to dropped). That set lists both halves of every call/output pair in_TOOL_CALL_TO_OUTPUT_TYPE(run_internal/items.py) —function_call,custom_tool_call,shell_call,apply_patch_call,computer_call,local_shell_call,tool_search_calland their outputs — plusweb_search_call,mcp_call,mcp_approval_request,mcp_approval_response.The one pair missing is
program/program_output.The set was written in #2889 (2026-04-15) and has not been edited since. Programmatic Tool Calling added the
programandprogram_outputitem types later, in #3833 (2026-07-17), and this call site was not updated — so a run that usesProgrammaticToolCallingToolwrites a rollout with the program code and the program result removed.The rollout is the raw evidence that phase-one/phase-two memory generation reads, so the loss is not cosmetic. Worse, the filter is inconsistent with itself: the
function_callitems a program issues are retained, and each one carriescaller: {"type": "program", "caller_id": ...}pointing at aprogramitem that was just removed. The stored rollout therefore contains dangling caller references.Before, for a single programmatic turn (
program→function_call→function_call_output→program_output):After:
Fix: add
"program"and"program_output"to the existing frozenset. Two strings — no new branch, no new helper, no public API or stored-format change.Deliberately not widened.
file_search_call,code_interpreter_call,mcp_list_toolsandhosted_tool_callare also absent from the set, but they are not part of the contract this change asserts: the claim here is only that every call/output pair the SDK knows how to pair should survive into the rollout, andprogramis the sole pair that does not.image_generation_callstays excluded via_EXCLUDED_MEMORY_ITEM_TYPES. There is a boundary test covering exactly this.Test plan
Three tests added to
tests/sandbox/test_memory.py:test_build_rollout_payload_keeps_programmatic_tool_calling_items— a full programmatic turn asRunItems (Program, afunction_callcarryingCallerProgram, its output,ProgramOutput). Asserts all four survive in order, that the program'scodeand the program output'sresultare preserved, and that the retainedfunction_call.callerresolves to aprogramitem that is present.test_build_rollout_payload_keeps_program_items_from_input— the same types arriving on theinputside, which goes through the same filter.test_build_rollout_payload_still_drops_hosted_items_outside_the_included_set— boundary:file_search_callandimage_generation_callare still dropped whileprogramis kept, so the narrow scope is pinned.All three fail on
upstream/main@f3b6c617and pass with the change; verified by reverting only the source hunk and re-running (3 failed, 3 passed→6 passed). The three pre-existingbuild_rollout_payloadtests pass unchanged in both states.Commands run:
uv run pytest tests/sandbox/test_memory.py -q→73 passeduv run pytest tests/sandbox tests/test_programmatic_tool_calling.py -q→1130 passed, 2 skippedmake format→862 files left unchangedmake lint→All checks passed!make typecheck→ mypy and pyright clean (exit 0)make tests→6744 passed, 29 skipped(parallel) +77 passed, 5 skipped(serial)git diff --check→ cleanNo API key, no network, no live service.
Not run:
make coverage,make build-docs, the Python 3.10 matrix, and the integration-test profiles.One unrelated flake seen while validating:
tests/extensions/memory/test_advanced_sqlite_session.py::test_branch_allocation_is_serialized_across_processesfails intermittently (~1 in 5) under CPU contention atassert first_ready.wait(timeout=10), a fixed wall-clock wait on amultiprocessing.Event. It reproduces with this change reverted and is not in this change's import graph;make testsis green on this branch when the machine is idle.Issue number
None.
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PR