feat: add android MCP tool#31
Merged
Merged
Conversation
added 5 commits
April 16, 2026 22:04
Contributor
There was a problem hiding this comment.
Pull request overview
Adds cross-platform screenshot support to VisionTest by extending both automation servers with a ui.screenshot JSON-RPC method and exposing corresponding MCP tools that save PNGs to disk (defaulting to ./screenshots/ under the MCP server’s CWD), plus tests/spec docs to lock in behavior.
Changes:
- Add
android_screenshotMCP tool and Android JSON-RPCui.screenshotimplementation (base64-over-JSON-RPC, host-side atomic write). - Add/extend
ios_screenshotsupport and shared hardening around screenshot JSON parsing and file writes. - Add unit tests + openspec documentation/specs for screenshot behavior and error handling.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| openspec/specs/ios-screenshot/spec.md | Adds iOS screenshot requirements/spec. |
| openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md | Archived iOS implementation task checklist. |
| openspec/changes/archive/2026-04-16-add-ios-screenshot/specs/ios-screenshot/spec.md | Archived iOS spec snapshot. |
| openspec/changes/archive/2026-04-16-add-ios-screenshot/proposal.md | Archived iOS proposal text. |
| openspec/changes/archive/2026-04-16-add-ios-screenshot/design.md | Archived iOS design doc. |
| openspec/changes/archive/2026-04-16-add-ios-screenshot/.openspec.yaml | Openspec metadata for archived iOS change. |
| openspec/changes/add-android-screenshot/tasks.md | Android screenshot implementation task checklist. |
| openspec/changes/add-android-screenshot/specs/android-screenshot/spec.md | Android screenshot requirements/spec. |
| openspec/changes/add-android-screenshot/proposal.md | Android screenshot proposal text. |
| openspec/changes/add-android-screenshot/design.md | Android screenshot design doc. |
| openspec/changes/add-android-screenshot/.openspec.yaml | Openspec metadata for Android change. |
| ios-automation-server/IOSAutomationServerUITests/Server/JsonRpcServer.swift | Adds ui.screenshot dispatch case on iOS server. |
| ios-automation-server/IOSAutomationServerUITests/Models/AutomationModels.swift | Adds ScreenshotResult model (iOS). |
| ios-automation-server/IOSAutomationServerUITests/Bridge/XCUITestBridge.swift | Implements iOS screenshot capture + base64 encoding. |
| ios-automation-server/IOSAutomationServerTests/AutomationModelsTests.swift | Adds iOS ScreenshotResult unit tests. |
| automation-server/src/test/java/com/example/automationserver/uiautomator/UiAutomatorModelsTest.kt | Adds Android ScreenshotResult model tests. |
| automation-server/src/main/java/com/example/automationserver/uiautomator/UiAutomatorModels.kt | Adds Android ScreenshotResult data class. |
| automation-server/src/main/java/com/example/automationserver/uiautomator/BaseUiAutomatorBridge.kt | Implements Android screenshot capture/compress/base64 on-device. |
| automation-server/src/androidTest/java/com/example/automationserver/JsonRpcServerInstrumented.kt | Adds ui.screenshot dispatch case in Android JSON-RPC server. |
| app/src/test/kotlin/com/example/visiontest/tools/IOSScreenshotToolTest.kt | Adds MCP-side iOS screenshot tool tests (MockWebServer). |
| app/src/test/kotlin/com/example/visiontest/tools/AndroidScreenshotToolTest.kt | Adds MCP-side Android screenshot tool tests (MockWebServer). |
| app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt | Registers ios_screenshot, adds parsing + atomic file write. |
| app/src/main/kotlin/com/example/visiontest/tools/AndroidAutomationToolRegistrar.kt | Registers android_screenshot, adds parsing + atomic file write. |
| app/src/main/kotlin/com/example/visiontest/ios/IOSAutomationClient.kt | Adds screenshot() client method (iOS). |
| app/src/main/kotlin/com/example/visiontest/android/AutomationClient.kt | Adds screenshot() client method (Android). |
| CLAUDE.md | Documents new android_screenshot and ios_screenshot tools. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
the host, mirroring the existing ios_screenshot tool
UiAutomation.takeScreenshot() with in-memory PNG compression and base64 encoding
error, and pngBase64 are now checked for correct primitive types before access, preventing
Gson crashes on malformed server responses
server's CWD (the user's project)
Details
overwrites existing files
guidance
Test plan
server-not-running, JSON-RPC errors, malformed responses, invalid base64, atomic write)
lint — clean
correctly
./screenshots/