AppControlBench bsky-24 (unfollow from a profile — the #1105 profile-wedge screen family) fails identically in 3/3 analyzed cells, consuming 809.6s + 900.1s + 900.1s (two hard timeouts):
Error (XCTEST_RECORDED_FAILURE): XCTest recorded a failure while executing tap; the action may not have been performed.
Hint: The iOS runner session will be restarted.
Error (COMMAND_FAILED): Daemon request timed out
Two compounding defects:
- False negatives: taps reported as failed actually landed. Evidence: a "failed" tap opened an image viewer (the agent's next command
press 'label="Close image"' presupposes it), and in the haiku_high run the judge found the CORRECT profile open in final.png despite the transcript recording continuous tap failures. With no reliable landed/didn't-land signal, agents re-tap and re-open in a spiral.
- The advertised restart doesn't restore reliability: errors persist across
open --relaunch, daemon pkill, and full reopen cycles.
Proposed direction: before reporting a tap as failed on XCTEST_RECORDED_FAILURE, corroborate against evidence the action landed (post-action capture diff — cheap now that penalized captures are ~0.5s after #1587) and report success-with-warning when the UI changed; separately root-cause why the restart path doesn't clear the wedge on profile-class screens.
Transcripts: AppControlBench {gpt_low,haiku_high,haiku_low}__agent-device/bsky-24/.
AppControlBench bsky-24 (unfollow from a profile — the #1105 profile-wedge screen family) fails identically in 3/3 analyzed cells, consuming 809.6s + 900.1s + 900.1s (two hard timeouts):
Two compounding defects:
press 'label="Close image"'presupposes it), and in the haiku_high run the judge found the CORRECT profile open in final.png despite the transcript recording continuous tap failures. With no reliable landed/didn't-land signal, agents re-tap and re-open in a spiral.open --relaunch, daemon pkill, and full reopen cycles.Proposed direction: before reporting a tap as failed on XCTEST_RECORDED_FAILURE, corroborate against evidence the action landed (post-action capture diff — cheap now that penalized captures are ~0.5s after #1587) and report success-with-warning when the UI changed; separately root-cause why the restart path doesn't clear the wedge on profile-class screens.
Transcripts: AppControlBench
{gpt_low,haiku_high,haiku_low}__agent-device/bsky-24/.