Add verified browser action batches by rgarcia · Pull Request #60 · kernel/cua

rgarcia · 2026-07-13T15:54:51Z

summary

separate complete, generation-fenced browser observations from filtered text presentation, including recursive iframe stitching and persisted ref ownership
add browser_act for dependent browser actions with semantic expectations, honest outcomes, stop boundaries, and stable successor diffs
keep mixed batches from continuing past browser stop boundaries and use compact reusable schemas for standalone and batched tools

testing

npm run typecheck
env -u GOOGLE_API_KEY -u GEMINI_API_KEY -u ANTHROPIC_API_KEY -u OPENAI_API_KEY npm test --workspace @onkernel/cua-ai (145 passed)
npm test --workspace @onkernel/cua-agent (191 passed, 12 live tests skipped)
npm run build --workspace @onkernel/cua-cli
git diff --check origin/main

Note

High Risk
Large change to CDP browser execution, ref lifecycle, and iframe handling; incorrect stop/expectation logic could mislead agents or leave stale refs. Extensive new tests mitigate but the surface area is core agent infrastructure.

Overview
Adds browser_act: a dependent list of ref/focus browser steps with optional per-step and final semantic expectations, honest worked / didnt / unknown outcomes, explicit stop reasons (navigation, dialog, stale ref, etc.), and a single successor snapshot plus accessibility-tree diff back to the baseline.

Refactors browser observation into generation-fenced BrowserObservation / presentation layers (browser-observation.ts, runBrowserAct in browser-act.ts), with recursive iframe stitching, navigation epochs, frame-parent tracking, and tighter ref invalidation. Snapshots and find now share this observe path with retries on mid-collection page changes.

Tooling: browser_act is wired through BrowserExecutor, batch execution stops after a browser_act stop boundary, model-facing text is formatted via formatBrowserActResult (capped diff lines), and @onkernel/cua-ai exposes schemas with $defs for nested expectations in unions/batches. Hybrid/browser modes include the new action as act / browser_act.

^{Reviewed by Cursor Bugbot for commit 0e9edb8. Bugbot is set up for automated code reviews on this repo. Configure here.}

rgarcia · 2026-07-13T15:55:07Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using high effort and found 5 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for all 5 issues found in the latest run.

✅ Fixed: Iframe churn false navigation stops
- Navigation change detection now compares only generations for frames present in both observations, so iframe insertions/removals alone no longer trigger false navigation stops.
✅ Fixed: Main nav leaks frame ancestry
- Main-frame invalidation now prunes descendant frame ancestry and generation entries while invalidating refs, preventing unbounded frameParents/generations growth across top-level navigations.
✅ Fixed: Missing exported type docs
- Added TSDoc comments to the exported BrowserAct and related result types to document outcome semantics, stop boundaries, and successor payload contracts.
✅ Fixed: Final expect stop reason lost
- Final expectation handling now overrides to expectation_failed when needed and only rewrites stopped_at to the final-expectation boundary when that phase actually introduced the stop.
✅ Fixed: Stale iframe session fallback
- Legacy refs without sessionTargetId now rebind to a frame session only when one exists and otherwise correctly fall back to the page session instead of treating frameId as a required owner target.

Or push these changes by commenting:

@cursor push d8684e9c72

Preview (d8684e9c72)

diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -636,6 +636,7 @@
 			steps.length === action.steps.length &&
 			steps.at(-1)?.evidence.includes("input delivered") === true;
 		if (action.expect && (!stopReason || terminalNavigation)) {
+			const stopReasonBeforeFinalExpectation = stopReason;
 			const before = this.evaluateExpectation(action.expect, baseline, baseline);
 			const deadline = Date.now() + EXPECTATION_TIMEOUT_MS;
 			while (true) {
@@ -673,9 +674,16 @@
 					break;
 				}
 			}
-			if (!stopReason && finalExpectation?.status === "failed") stopReason = "expectation_failed";
-			else if (!stopReason && finalExpectation?.status === "unverifiable") stopReason = "control_flow";
-			if (stopReason) stoppedAt = action.steps.length;
+			if (finalExpectation?.status === "failed") stopReason = "expectation_failed";
+			else if (finalExpectation?.status === "unverifiable") stopReason ??= "control_flow";
+			if (
+				stopReason &&
+				(stopReason !== stopReasonBeforeFinalExpectation ||
+					finalExpectation?.status === "failed" ||
+					finalExpectation?.status === "unverifiable")
+			) {
+				stoppedAt = action.steps.length;
+			}
 		}
 
 		let successor: BrowserActResult["successor"] | undefined;
@@ -1294,13 +1302,18 @@
 	 */
 	private async refSession(entry: RefEntry): Promise<string> {
 		if (!entry.sessionId) {
-			await this.attach(entry.targetId);
-			const sessionTargetId = entry.sessionTargetId ?? entry.frameId;
-			const frameSession = this.frameSessions.get(sessionTargetId);
-			if (sessionTargetId !== entry.targetId && !frameSession) {
+			const pageSession = await this.attach(entry.targetId);
+			if (!entry.sessionTargetId) {
+				const frameSession = this.frameSessions.get(entry.frameId);
+				entry.sessionTargetId = frameSession ? entry.frameId : entry.targetId;
+				entry.sessionId = frameSession ?? pageSession;
+				return entry.sessionId;
+			}
+			const frameSession = this.frameSessions.get(entry.sessionTargetId);
+			if (entry.sessionTargetId !== entry.targetId && !frameSession) {
 				throw new Error("owning frame session is unavailable; the element ref is stale");
 			}
-			entry.sessionId = frameSession ?? (await this.attach(entry.targetId));
+			entry.sessionId = frameSession ?? pageSession;
 		}
 		return entry.sessionId;
 	}
@@ -1320,9 +1333,24 @@
 	}
 
 	private invalidateRefs(targetId: string): void {
+		const descendants = new Set<string>();
+		let found = true;
+		while (found) {
+			found = false;
+			for (const [child, parent] of this.frameParents) {
+				if ((parent === targetId || descendants.has(parent)) && !descendants.has(child)) {
+					descendants.add(child);
+					found = true;
+				}
+			}
+		}
 		this.generations.set(targetId, this.generation(targetId) + 1);
+		for (const frameId of descendants) {
+			this.frameParents.delete(frameId);
+			this.generations.delete(frameId);
+		}
 		for (const [ref, entry] of this.refs) {
-			if (entry.targetId === targetId) this.refs.delete(ref);
+			if (entry.targetId === targetId || descendants.has(entry.frameId)) this.refs.delete(ref);
 		}
 	}
 
@@ -1457,12 +1485,12 @@
 	after: BrowserObservation,
 	live: ReadonlyMap<string, number>,
 ): boolean {
-	const keys = new Set([...before.generations.keys(), ...after.generations.keys()]);
-	return [...keys].some(
-		(key) =>
-			before.generations.get(key) !== after.generations.get(key) ||
-			(after.generations.has(key) && after.generations.get(key) !== (live.get(key) ?? 0)),
-	);
+	for (const [key, generation] of before.generations) {
+		const afterGeneration = after.generations.get(key);
+		if (afterGeneration === undefined) continue;
+		if (generation !== afterGeneration || afterGeneration !== (live.get(key) ?? 0)) return true;
+	}
+	return false;
 }
 
 function browserControlChange(

diff --git a/packages/agent/src/translator/types.ts b/packages/agent/src/translator/types.ts
--- a/packages/agent/src/translator/types.ts
+++ b/packages/agent/src/translator/types.ts
@@ -1,6 +1,9 @@
+/** High-level result of a browser action plan. */
 export type BrowserActOutcome = "worked" | "didnt" | "unknown";
+/** Per-expectation verification status used in step and final expectation evidence. */
 export type BrowserExpectationStatus = "newly_verified" | "preexisting" | "failed" | "unverifiable";
 
+/** Before/after expectation evaluation details captured for diagnostics and tool output. */
 export interface BrowserExpectationEvidence {
 	status: BrowserExpectationStatus;
 	before?: boolean;
@@ -8,6 +11,7 @@
 	details: string[];
 }
 
+/** Execution outcome for one `browser_act` step in the submitted order. */
 export interface BrowserActStepResult {
 	index: number;
 	type: string;
@@ -16,6 +20,7 @@
 	expectation?: BrowserExpectationEvidence;
 }
 
+/** Structured diff between baseline and successor observations. */
 export interface BrowserObservationDiff {
 	changed: boolean;
 	added: string[];
@@ -24,6 +29,7 @@
 	title?: { before: string; after: string };
 }
 
+/** Successor payload when a post-act observation was collected successfully. */
 export interface BrowserActObservedSuccessor {
 	status: "observed";
 	text: string;
@@ -32,13 +38,23 @@
 	diff: BrowserObservationDiff;
 }
 
+/** Successor payload when the post-act observation cannot be collected. */
 export interface BrowserActUnavailableSuccessor {
 	status: "unavailable";
 	error: string;
 }
 
+/** Post-act observation payload describing either an observed or unavailable successor. */
 export type BrowserActSuccessor = BrowserActObservedSuccessor | BrowserActUnavailableSuccessor;
 
+/**
+ * Full `browser_act` result contract returned to callers.
+ *
+ * `stopped_at` is a zero-based step index when execution stops during dispatch,
+ * or `steps.length` when it stops while evaluating the final `expect`.
+ * `stop_reason` names the boundary that prevented continued verification or
+ * dispatch, while `successor` always describes the best-effort post-action state.
+ */
 export interface BrowserActResult {
 	outcome: BrowserActOutcome;
 	steps: BrowserActStepResult[];
@@ -48,6 +64,7 @@
 	successor: BrowserActSuccessor;
 }
 
+/** Normalized read payloads emitted by translator batch execution. */
 export type BatchReadResult =
 	| { type: "screenshot"; data: Buffer; mimeType: string }
 	| { type: "url"; url: string }
@@ -55,6 +72,7 @@
 	| { type: "browser_text"; label: string; text: string }
 	| { type: "browser_act"; result: BrowserActResult };
 
+/** Aggregate read results returned for a batch. */
 export interface BatchExecutionResult {
 	readResults: BatchReadResult[];
 }

_{You can send follow-ups to the cloud agent here.}

rgarcia · 2026-07-13T16:09:50Z

bugbot run

rgarcia · 2026-07-13T16:16:48Z

bugbot run

rgarcia · 2026-07-13T16:26:11Z

bugbot run

rgarcia · 2026-07-13T16:32:49Z

bugbot run

rgarcia · 2026-07-13T16:42:33Z

bugbot run

rgarcia · 2026-07-13T16:51:53Z

bugbot run

rgarcia · 2026-07-13T17:03:26Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: OOPIF root invalidated twice
- invalidateSessionTarget now excludes refs whose frameId equals the targetId so the OOPIF root is invalidated only once per navigation.

Or push these changes by commenting:

@cursor push 58b34f29dd

Preview (58b34f29dd)

diff --git a/packages/agent/src/translator/browser.ts b/packages/agent/src/translator/browser.ts
--- a/packages/agent/src/translator/browser.ts
+++ b/packages/agent/src/translator/browser.ts
@@ -1517,7 +1517,7 @@
 	private invalidateSessionTarget(targetId: string): void {
 		const ownedFrames = new Set<string>();
 		for (const entry of this.refs.values()) {
-			if (entry.sessionTargetId === targetId) ownedFrames.add(entry.frameId);
+			if (entry.sessionTargetId === targetId && entry.frameId !== targetId) ownedFrames.add(entry.frameId);
 		}
 		this.invalidateFrame(targetId);
 		for (const frameKey of ownedFrames) this.invalidateFrame(frameKey);

_{You can send follow-ups to the cloud agent here.}

rgarcia · 2026-07-13T17:10:32Z

bugbot run

pulumi · 2026-07-13T17:19:28Z

⚠️ Pulumi could not deploy preview(s) for this pull request because GitHub reports it is not mergeable (mergeable state: dirty). This usually means the branch has merge conflicts with its base branch. Resolve the conflicts and push a new commit to retry.

rgarcia · 2026-07-13T17:19:35Z

bugbot run

rgarcia · 2026-07-13T17:29:02Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b7656ef. Configure here.}

rgarcia · 2026-07-13T17:35:40Z

bugbot run

rgarcia · 2026-07-13T17:44:15Z

bugbot run

rgarcia · 2026-07-13T17:51:56Z

bugbot run

rgarcia · 2026-07-13T18:00:23Z

bugbot run

rgarcia · 2026-07-13T18:09:37Z

bugbot run

rgarcia · 2026-07-13T18:17:35Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 469ffbf. Configure here.}

rgarcia · 2026-07-13T19:34:25Z

bugbot run

rgarcia · 2026-07-13T19:42:17Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0e9edb8. Configure here.}

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/agent/src/translator/types.ts

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/ai/src/actions/browser.ts

Comment thread packages/ai/src/providers/common.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/ai/src/providers/common.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/tools.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/agent/src/tools.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts Outdated

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser.ts Outdated

cursor Bot reviewed Jul 13, 2026

View reviewed changes

rgarcia added 7 commits July 13, 2026 17:34

Add verified browser action batches

3393220

Address browser action review feedback

2c3eb97

Address remaining review feedback

79e0221

Tighten frame state and schema lifecycles

366b9a6

Prune incomplete frame observations

66d24cb

Preserve OOPIF refs across nested navigation

baaeb88

Bound incomplete frame metadata

24a5ffa

rgarcia added 4 commits July 13, 2026 17:34

Handle frame edge cases in action results

fc5a36a

Avoid duplicate OOPIF invalidation

dd55706

Align semantic checks with rendered snapshots

fb3c6a4

Scope navigation epochs by target

2a507d6

rgarcia force-pushed the hypeship/browser-action-outcomes branch from b7656ef to 2a507d6 Compare July 13, 2026 17:35

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Stop action batches on iframe detach

806cf28

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

Keep attached frame sessions during pruning

1e61218

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Stop action batches on SPA navigation

14ab4cb

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

Comment thread packages/agent/src/translator/browser.ts

Preserve nested frame context in fallback snapshots

56f64bb

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts

Guard fallback frame traversal cycles

469ffbf

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Split browser action and observation logic

75bd0d2

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread packages/agent/src/translator/browser.ts Outdated

Comment thread packages/agent/src/translator/browser-observation.ts

Comment thread packages/agent/src/translator/browser-act.ts

Document modular browser internals

0e9edb8

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Uh oh!

Conversation

rgarcia commented Jul 13, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

summary

testing

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

pulumi Bot commented Jul 13, 2026

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

rgarcia commented Jul 13, 2026 •

edited by cursor Bot

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading