Skip to content

feat: add android MCP tool#31

Merged
docer1990 merged 8 commits into
mainfrom
feat/add-android-screenshot
Apr 17, 2026
Merged

feat: add android MCP tool#31
docer1990 merged 8 commits into
mainfrom
feat/add-android-screenshot

Conversation

@docer1990

Copy link
Copy Markdown
Owner

Summary

  • Add android_screenshot MCP tool that captures the device display and saves it as a PNG on
    the host, mirroring the existing ios_screenshot tool
  • Add ui.screenshot JSON-RPC method to the Android automation server using
    UiAutomation.takeScreenshot() with in-memory PNG compression and base64 encoding
  • Add screenshot() suspend function to AutomationClient
  • Harden JSON field validation in both iOS and Android screenshot tool handlers — success,
    error, and pngBase64 are now checked for correct primitive types before access, preventing
    Gson crashes on malformed server responses
  • Default output path: ./screenshots/android_screenshot_.png relative to the MCP
    server's CWD (the user's project)

Details

  • Optional outputPath parameter (absolute or CWD-relative), auto-creates parent directories,
    overwrites existing files
  • Atomic write via temp file + Files.move(ATOMIC_MOVE) — no partial PNGs on failure
  • Handles outdated APK gracefully: JSON-RPC methodNotFound → clear "rebuild or update"
    guidance
  • Handles FLAG_SECURE / unavailable display: null bitmap → descriptive error
  • 30s timeout, consistent with ios_screenshot and get_ui_hierarchy

Test plan

  • ScreenshotResult model tests in UiAutomatorModelsTest (3 cases)
  • AndroidScreenshotToolTest via MockWebServer (12 cases: success, path resolution,
    server-not-running, JSON-RPC errors, malformed responses, invalid base64, atomic write)
  • ./gradlew :automation-server:test — all pass
  • ./gradlew :app:test — all pass
  • ./gradlew :automation-server:assembleDebug :automation-server:assembleDebugAndroidTest +
    lint — clean
  • Manual E2E: install APKs, start server, invoke android_screenshot, verify PNG renders
    correctly
  • Manual cross-platform: confirm ios_screenshot and android_screenshot coexist in
    ./screenshots/

Copilot AI review requested due to automatic review settings April 17, 2026 06:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds cross-platform screenshot support to VisionTest by extending both automation servers with a ui.screenshot JSON-RPC method and exposing corresponding MCP tools that save PNGs to disk (defaulting to ./screenshots/ under the MCP server’s CWD), plus tests/spec docs to lock in behavior.

Changes:

  • Add android_screenshot MCP tool and Android JSON-RPC ui.screenshot implementation (base64-over-JSON-RPC, host-side atomic write).
  • Add/extend ios_screenshot support and shared hardening around screenshot JSON parsing and file writes.
  • Add unit tests + openspec documentation/specs for screenshot behavior and error handling.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
openspec/specs/ios-screenshot/spec.md Adds iOS screenshot requirements/spec.
openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md Archived iOS implementation task checklist.
openspec/changes/archive/2026-04-16-add-ios-screenshot/specs/ios-screenshot/spec.md Archived iOS spec snapshot.
openspec/changes/archive/2026-04-16-add-ios-screenshot/proposal.md Archived iOS proposal text.
openspec/changes/archive/2026-04-16-add-ios-screenshot/design.md Archived iOS design doc.
openspec/changes/archive/2026-04-16-add-ios-screenshot/.openspec.yaml Openspec metadata for archived iOS change.
openspec/changes/add-android-screenshot/tasks.md Android screenshot implementation task checklist.
openspec/changes/add-android-screenshot/specs/android-screenshot/spec.md Android screenshot requirements/spec.
openspec/changes/add-android-screenshot/proposal.md Android screenshot proposal text.
openspec/changes/add-android-screenshot/design.md Android screenshot design doc.
openspec/changes/add-android-screenshot/.openspec.yaml Openspec metadata for Android change.
ios-automation-server/IOSAutomationServerUITests/Server/JsonRpcServer.swift Adds ui.screenshot dispatch case on iOS server.
ios-automation-server/IOSAutomationServerUITests/Models/AutomationModels.swift Adds ScreenshotResult model (iOS).
ios-automation-server/IOSAutomationServerUITests/Bridge/XCUITestBridge.swift Implements iOS screenshot capture + base64 encoding.
ios-automation-server/IOSAutomationServerTests/AutomationModelsTests.swift Adds iOS ScreenshotResult unit tests.
automation-server/src/test/java/com/example/automationserver/uiautomator/UiAutomatorModelsTest.kt Adds Android ScreenshotResult model tests.
automation-server/src/main/java/com/example/automationserver/uiautomator/UiAutomatorModels.kt Adds Android ScreenshotResult data class.
automation-server/src/main/java/com/example/automationserver/uiautomator/BaseUiAutomatorBridge.kt Implements Android screenshot capture/compress/base64 on-device.
automation-server/src/androidTest/java/com/example/automationserver/JsonRpcServerInstrumented.kt Adds ui.screenshot dispatch case in Android JSON-RPC server.
app/src/test/kotlin/com/example/visiontest/tools/IOSScreenshotToolTest.kt Adds MCP-side iOS screenshot tool tests (MockWebServer).
app/src/test/kotlin/com/example/visiontest/tools/AndroidScreenshotToolTest.kt Adds MCP-side Android screenshot tool tests (MockWebServer).
app/src/main/kotlin/com/example/visiontest/tools/IOSAutomationToolRegistrar.kt Registers ios_screenshot, adds parsing + atomic file write.
app/src/main/kotlin/com/example/visiontest/tools/AndroidAutomationToolRegistrar.kt Registers android_screenshot, adds parsing + atomic file write.
app/src/main/kotlin/com/example/visiontest/ios/IOSAutomationClient.kt Adds screenshot() client method (iOS).
app/src/main/kotlin/com/example/visiontest/android/AutomationClient.kt Adds screenshot() client method (Android).
CLAUDE.md Documents new android_screenshot and ios_screenshot tools.

Comment thread openspec/changes/archive/2026-04-16-add-ios-screenshot/tasks.md
Comment thread openspec/changes/archive/2026-04-16-add-ios-screenshot/proposal.md

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Comment thread openspec/specs/ios-screenshot/spec.md
Comment thread openspec/specs/ios-screenshot/spec.md
@docer1990 docer1990 merged commit 3300986 into main Apr 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants