Configure CUA image context and response threading

[rgarcia]

Summary

• limit tool-result images in each model request to the newest four by default while preserving text, metadata, ordering, agent state, and persisted session history
• expose toolResultImageReplayLimit and responseThreading on CuaAgent, CuaAgentHarness, and the CLI harness builder
• apply the harness image limit after pi selects the final context hook result, so user context handlers compose with the built-in limit instead of replacing it
• apply context controls consistently across Models.stream, complete, streamSimple, and completeSimple
• replace the process-wide response-threading environment variable with code-level configuration; threading remains enabled by default for OpenAI and Tzafon
• document how the image projection fits with pi's transformContext, harness context hooks, session entry transforms, and persistent compact() summaries

Testing

• npm run typecheck
• env -u GOOGLE_API_KEY -u GEMINI_API_KEY npm test --workspace @onkernel/cua-ai (142 passed)
• npm test --workspace @onkernel/cua-agent -- --exclude '**/*.live.test.ts' (121 passed)
• npm test --workspace @onkernel/cua-cli (82 passed, 5 skipped)
• git diff --check

---

Note

Medium Risk
Changes what every provider request sends (images and OpenAI/Tzafon threading), which can affect model behavior and token usage; defaults aim to preserve prior threading-on behavior while capping images.

Overview
Adds request-time context controls so long browser runs do not ship unbounded screenshot history to providers, while agent state and sessions keep full images.

toolResultImageReplayLimit (default 4) trims older tool-result image blocks per provider call, inserts a single [stale tool-result images omitted] marker where needed, and leaves text/metadata/order intact. CuaAgent applies it after any caller transformContext; CuaAgentHarness applies it at the Models boundary after harness context hooks so custom handlers compose with the built-in cap. Set false to disable projection.

responseThreading (default true) toggles OpenAI/Tzafon previous_response_id chaining vs full-context requests via disableResponseThreading. The CUA_DISABLE_RESPONSE_THREADING env toggle is removed in favor of this constructor/CLI option.

Options are exposed on CuaAgent, CuaAgentHarness, buildCuaHarness, and documented in the agent README, with broad test coverage for limits, threading, batch/multi-image cases, and validation.

^{Reviewed by Cursor Bugbot for commit 51eb722. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Exported replay limit type lacks TSDoc
  • Added TSDoc to ToolResultImageReplayLimit clarifying its non-negative integer cap semantics and false bypass behavior.

Or push these changes by commenting:

@cursor push ae5e864071

Preview (ae5e864071)

diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -40,6 +40,11 @@
 const DEFAULT_TOOL_RESULT_IMAGE_REPLAY_LIMIT = 4;
 const OMITTED_TOOL_RESULT_IMAGES = "[stale tool-result images omitted]";
 
+/**
+ * Maximum number of tool-result image blocks replayed from local history.
+ * Use a non-negative integer cap (`0` omits all images) or `false` to skip
+ * projection and keep the original message history unchanged.
+ */
 export type ToolResultImageReplayLimit = number | false;
 
 function resolveToolResultImageReplayLimit(limit: ToolResultImageReplayLimit | undefined): ToolResultImageReplayLimit {

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Threading disable skips stream
  • Updated withoutResponseThreading to inject disableResponseThreading: true for stream and complete in addition to streamSimple/completeSimple, so all harness model entry points stay full-context when threading is disabled.

Or push these changes by commenting:

@cursor push 103d4b0533

Preview (103d4b0533)

diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -351,8 +351,12 @@
 }
 
 function withoutResponseThreading(models: Models): Models {
+	const withResponseThreadingDisabled = (options: CuaSimpleStreamOptions | undefined): CuaSimpleStreamOptions => ({
+		...options,
+		disableResponseThreading: true,
+	});
 	const streamSimple: Models["streamSimple"] = (model, context, options) =>
-		models.streamSimple(model, context, { ...options, disableResponseThreading: true } as CuaSimpleStreamOptions);
+		models.streamSimple(model, context, withResponseThreadingDisabled(options as CuaSimpleStreamOptions | undefined));
 	return {
 		getProviders: () => models.getProviders(),
 		getProvider: (id) => models.getProvider(id),
@@ -360,8 +364,9 @@
 		getModel: (provider, id) => models.getModel(provider, id),
 		refresh: (provider) => models.refresh(provider),
 		getAuth: (model) => models.getAuth(model),
-		stream: (model, context, options) => models.stream(model, context, options),
-		complete: (model, context, options) => models.complete(model, context, options),
+		stream: (model, context, options) => models.stream(model, context, withResponseThreadingDisabled(options as CuaSimpleStreamOptions | undefined)),
+		complete: (model, context, options) =>
+			models.complete(model, context, withResponseThreadingDisabled(options as CuaSimpleStreamOptions | undefined)),
 		streamSimple,
 		completeSimple: (model, context, options) => streamSimple(model, context, options).result(),
 	};

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Harness completeSimple skips image cap
  • Updated withContextManagement so completeSimple delegates to the capped streamSimple path, and adjusted the harness test expectation to verify the image cap is applied.

Or push these changes by commenting:

@cursor push 9c7c960e3b

Preview (9c7c960e3b)

diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -380,7 +380,7 @@
 			withResponseThreading(options, responseThreading),
 		);
 	const completeSimple: Models["completeSimple"] = (model, context, options) =>
-		models.completeSimple(model, context, withResponseThreading(options, responseThreading));
+		streamSimple(model, context, options).result();
 	return {
 		getProviders: () => models.getProviders(),
 		getProvider: (id) => models.getProvider(id),

diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts
--- a/packages/agent/test/agent.test.ts
+++ b/packages/agent/test/agent.test.ts
@@ -786,7 +786,7 @@
 
 		await harness.models.completeSimple(model, context);
 		expect(streamOptions?.disableResponseThreading).toBe(expected);
-		expect(providerImageCount).toBe(5);
+		expect(providerImageCount).toBe(4);
 	});
 
 	it("applies image projection after user context hooks", async () => {

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit a2bf729. Configure here.}