Skip to content

feat: add ios_screenshot MCP tool#30

Merged
docer1990 merged 4 commits into
mainfrom
feat/add-iOS-screenshot
Apr 17, 2026
Merged

feat: add ios_screenshot MCP tool#30
docer1990 merged 4 commits into
mainfrom
feat/add-iOS-screenshot

Conversation

@docer1990

Copy link
Copy Markdown
Owner

Summary

  • Adds ios_screenshot MCP tool that captures the current iOS simulator
    display as a PNG via XCUITest and saves it to the host filesystem.
  • Default output path is ./screenshots/ios_screenshot_<yyyyMMdd_HHmmss>.png
    resolved against the MCP server's CWD (the user's project when launched
    by a coding agent), overridable via an optional outputPath parameter.
  • Keeps transport consistent with existing tools: new ui.screenshot
    JSON-RPC method returns a base64-encoded PNG; the Kotlin tool decodes
    and writes to disk.

Changes

iOS automation server (Swift)

  • ScreenshotResult model with toDictionary() in AutomationModels.swift
  • XCUITestBridge.screenshot() using XCUIScreen.main.screenshot().pngRepresentation
  • ui.screenshot case added to JsonRpcServer.executeMethod

MCP server (Kotlin)

  • IOSAutomationClient.screenshot() suspend function
  • ios_screenshot tool registered in IOSAutomationToolRegistrar with
    timeoutMs = 30000, CWD-relative default path, auto-created parent
    directories, and graceful handling of malformed responses, missing
    pngBase64 (outdated-bundle hint), and server-not-running.

Documentation

  • New row in the iOS tool table in CLAUDE.md
  • OpenSpec change archived at openspec/changes/archive/2026-04-16-add-ios-screenshot/
  • New capability spec at openspec/specs/ios-screenshot/spec.md

Design notes

  • Base64-over-JSON-RPC was chosen over a separate binary endpoint to
    preserve the single-method dispatch pattern. For typical iPhone-sized
    simulators the payload is well under 3 MB base64.
  • File writing happens on the Kotlin host side (not Swift) so path
    semantics are unambiguous — the simulator sandbox would resolve paths
    differently from the host.
  • Default path is CWD-relative, not VISIONTEST_DIR, because a
    coding agent taking a screenshot is almost always working on a user
    project and screenshots should land there.
  • Heads-up for users on the pre-built bundle: the ui.screenshot
    method ships in a new bundle release. Users on an older tarball will
    get a clear "outdated bundle" error from the tool until they update.

Test plan

  • ./gradlew :app:test — all Kotlin tests pass including 10 new
    cases in IOSScreenshotToolTest (path resolution, base64 decoding,
    parent-dir creation, server-not-running, success:false, missing
    pngBase64, missing result, malformed JSON).
  • xcodebuild test ... -only-testing:IOSAutomationServerTests
    all 69 tests pass including 2 new ScreenshotResult.toDictionary()
    cases (tested on iPhone 17 simulator).
  • Manual end-to-end: started the iOS automation server on a booted
    simulator, invoked ios_screenshot, confirmed the PNG opens and
    renders the simulator display at native resolution.

Copilot AI review requested due to automatic review settings April 16, 2026 20:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new iOS screenshot capability to VisionTest by extending the iOS automation server with a ui.screenshot JSON-RPC method and exposing an ios_screenshot MCP tool in the Kotlin server that decodes and writes the PNG to disk.

Changes:

  • iOS (Swift) JSON-RPC: add ui.screenshot dispatch plus ScreenshotResult model and bridge implementation.
  • MCP server (Kotlin): add IOSAutomationClient.screenshot() and register ios_screenshot tool (path resolution + base64 decode + file write).
  • Specs/docs/tests: add OpenSpec capability docs, CLAUDE tool table row, and unit tests in both Swift and Kotlin.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
openspec/specs/ios-screenshot/spec.md New spec describing ios_screenshot tool + ui.screenshot JSON-RPC method requirements.
openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md Archived implementation checklist for the change.
openspec/changes/archive/2026-04-16-add-ios-screenshot/specs/ios-screenshot/spec.md Archived copy of the new spec.
openspec/changes/archive/2026-04-16-add-ios-screenshot/proposal.md Archived proposal describing motivation/scope for the new tool.
openspec/changes/archive/2026-04-16-add-ios-screenshot/design.md Archived design notes covering transport/path decisions and risks.
openspec/changes/archive/2026-04-16-add-ios-screenshot/.openspec.yaml OpenSpec metadata for the archived change.
ios-automation-server/IOSAutomationServerUITests/Server/JsonRpcServer.swift Adds ui.screenshot method dispatch to the Swift JSON-RPC server.
ios-automation-server/IOSAutomationServerUITests/Models/AutomationModels.swift Introduces ScreenshotResult and toDictionary() serialization.
ios-automation-server/IOSAutomationServerUITests/Bridge/XCUITestBridge.swift Implements screenshot capture via XCUIScreen.main.screenshot().pngRepresentation.
ios-automation-server/IOSAutomationServerTests/AutomationModelsTests.swift Adds unit tests for ScreenshotResult.toDictionary().
app/src/test/kotlin/com/example/visiontest/tools/IOSScreenshotToolTest.kt Adds Kotlin unit tests for path resolution, decoding, error cases, and file writing.
app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Registers ios_screenshot tool and implements capture + write logic.
app/src/main/kotlin/com/example/visiontest/ios/IOSAutomationClient.kt Adds screenshot() client method delegating to sendRequest("ui.screenshot").
CLAUDE.md Documents the new ios_screenshot tool in the iOS tools table.

Comment thread openspec/specs/ios-screenshot/spec.md
Comment thread openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md
Comment thread app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Outdated
Comment thread app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Outdated
Comment thread openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md
Comment thread openspec/changes/archive/2026-04-16-add-ios-screenshot/proposal.md

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comment thread app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Outdated
Comment thread openspec/specs/ios-screenshot/spec.md

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Comment thread app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Outdated
@docer1990 docer1990 merged commit 388b606 into main Apr 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants