Skip to content

feat(vega): Amazon Vega (Fire TV) platform support#366

Open
finloop wants to merge 56 commits into
software-mansion:mainfrom
finloop:krawipio/vega-virtual-device
Open

feat(vega): Amazon Vega (Fire TV) platform support#366
finloop wants to merge 56 commits into
software-mansion:mainfrom
finloop:krawipio/vega-virtual-device

Conversation

@finloop

@finloop finloop commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds Vega (Amazon Fire TV) as a third Argent platform alongside iOS and Android. Vega apps are React Native 0.72 + Hermes running on a QEMU Virtual Device (VVD), driven by a D-pad tv-remote rather than touch. This PR extends Argent's platform abstraction so the existing tool surface gates correctly on vega, and adds the Vega-specific tools needed to actually drive a Fire TV app end to end.

Platform abstraction

  • Platform gains vega; DeviceKind gains vvd (the QEMU VVD); ToolCapability gains a vega { vvd, device } block; ToolDependency gains vega.
  • assertSupported() selects the vega capability matrix for Vega devices; dispatchByPlatform() gains an optional vega branch, and a Vega dispatch with no branch throws NotImplementedOnPlatformError (501).
  • New vega-cli util (modeled on adb.ts) resolves vega/kepler from PATH or ~/vega/bin with the same hardened exec/quoting; check-deps resolves and hints the vega dependency.
  • The simulator-server blueprint is never wired for vega (Vega uses adb + the vega/kepler CLI, not the proprietary binary).

How Vega is wired

  • Input / screenshot / describe go over adb (input via inputd-cli, screenshots host-side via the Android emulator console, element tree via the on-device automation toolkit).
  • App lifecycle goes over the vega/kepler CLI.
  • v1 is Virtual-Device-only: every Vega capability is declared vvd (physical Fire TV is unverified — declaring device would advertise support that silently returns empty results, so the capability gate rejects hardware up front).
  • Vega runs React Native 0.72, which predates the new React Native DevTools. The CDP/Hermes debugger surface (debugger-*, React/native profilers) depends on integration points the new DevTools provides, so debugging and profiling are out of scope for this PR — Vega profiling/crash analysis is handled by the amazon-devices-buildertools-mcp server instead.

Also includes

  • argent-vega skill documenting the full Vega workflow (target discovery, the describe → compute-path → tv-remote navigation loop, text injection).
  • Vega VVD end-to-end CI workflow.

Feature support by platform

Legend: ✅ supported · ❌ not supported · ⚠️ degraded / conditional · — not applicable

Feature (tool) iOS Android Chromium Vega (VVD) Vega notes
Device discovery (list-devices) Tagged platform:"vega", kind:"vvd"; pass serial as udid
Boot / start device
Launch app (launch-app) bundleId = interactive-component app id from manifest.toml
Restart app (restart-app) Terminate + relaunch
Reinstall app (reinstall-app) Uninstall + install a .vpkg
Screenshot (screenshot) Host-side via emulator console (needs adb); rotation ignored (fixed landscape)
Describe / element tree (describe) On-device automation toolkit; surfaces [focused]/[selected] + normalized frames
Touch gestures (gesture-tap/swipe/scroll/drag/custom) Vega is tv-remote-driven; gestures error — use tv-remote
Pinch / rotate gesture (gesture-pinch/gesture-rotate) ⚠️ N/A on a TV
Remote / D-pad (tv-remote) Vega-only tool; single key, repeat, or a whole path in one call
Hardware buttons (button) Remote keys live in remote instead
Text input (keyboard) Whole-string injection via inputd-cli send_text
Rotate orientation (rotate) TV framebuffer is fixed landscape
Open URL / deep link (open-url) Not wired on Vega
JS debugger (debugger-*) RN 0.72 predates the new RN DevTools — out of scope for this PR
Native devtools / AX (native-describe-screen, etc.) iOS-only
Native profiler (native-profiler-*) Use amazon-devices-buildertools-mcp (Perfetto) instead
React profiler (react-profiler-*) Out of scope; RN 0.72 / new RN DevTools not supported
Screenshot diff (screenshot-diff) Host-side image comparison; platform-agnostic
Flows record/replay (flow-*) Records any tool-call sequence

Test plan

  • list-devices surfaces connected Vega devices (platform:"vega", kind:"vvd")
  • launch-app, restart-app, reinstall-app, list-installed-apps work on Vega
  • tv-remote drives focus (single key, repeat, and multi-key path) and keyboard injects whole strings
  • screenshot returns a frame via the host-side VVD path
  • describe returns the on-screen element tree with [focused] state and normalized coordinates
  • Vega VVD e2e CI workflow passes
  • Test multiple platforms being driven at the same time
    • Android + Vega
    • iOS + Vega
    • chrome + Vega

finloop and others added 29 commits June 18, 2026 10:07
Extend the platform abstraction with a third platform, Vega (Amazon Fire TV):
- Platform gains vega; DeviceKind gains virtual (the QEMU VVD); ToolCapability
  gains a vega { virtual, device } block; ToolDependency gains vega.
- assertSupported() selects the vega capability matrix for vega devices.
- dispatchByPlatform() gains an optional vega branch; a vega dispatch with no
  branch throws NotImplementedOnPlatformError (501).
- New vega-cli util (modeled on adb.ts): resolves vega/kepler (PATH or ~/vega/bin),
  runVega/vegaDevice/vegaShell with the same hardened exec/quoting as adb.
- check-deps resolves and hints the vega dependency.
- simulator-server blueprint rejects vega (Vega uses vega-control, not the binary);
  button tool's per-platform map stays total with an empty vega set.

Verified: tsc --build clean, 31 util tests pass, vega-cli reaches the live VVD.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New vega-devices util: listVegaDevices() parses `vega device list`, enriches
  the single connected device via `vega device info` (product/profile/build/
  simulated), and resolves virtual vs physical kind. Registers results in the
  device-classification inventory.
- device-info gains a Vega inventory that resolveDevice/classifyDevice consult
  first, since a Vega serial (amazon-<hex>) is shape-indistinguishable from an
  adb serial. startVegaWatcher() keeps it warm in the running server (wired into
  index.ts start/shutdown); no-op when the Vega SDK is absent.
- list-devices merges Vega devices (platform: "vega") and floats running ones up.
- vega-cli: rely on the single-connected-device default rather than `-d <serial>`,
  which does not match the VVD by its reported serial (documented inline).

Verified on the live VVD: list-devices returns
{ platform: vega, kind: virtual, state: running, product: vvrp-tv-arm64 };
resolveDevice maps the serial to vega; 16 unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add vega platform branches to the app-lifecycle tools, dispatched via the new
dispatchByPlatform vega slot (requires: ["vega"]):
- launch-app  -> `vega device launch-app -a <appId>`
- restart-app -> terminate-app then launch-app (terminate is non-fatal)
- reinstall-app -> uninstall-app (non-fatal) then `install-app -p <vpkg>`,
  asserting the CLI's "…success" line.

bundleId carries the Vega interactive component app id (…​.main); appPath is a
.vpkg. Commands target the single connected VVD (no -d, per the Phase 1 finding).
describe and open-url intentionally get no vega branch yet (no AX tree / taps on
Vega; deep-link mechanism unconfirmed) — they return a clean unsupported error.

Verified on the live VVD with the multi-tv sample (com.giolaq.multitv.vega.main,
vega_aarch64.vpkg): reinstall -> launch -> restart all succeed and the app reports
running; launching a nonexistent app id fails with the CLI error surfaced through
the tool. 16 tool tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dict)

New vega-qmp util: a minimal QMP (QEMU Machine Protocol) client for the VVD's
unix socket (auto-discovered at /tmp|tmpdir /qmp-socket-*.sock). Provides the
two primitives Vega interaction needs:
- send-key / qcode helpers (REMOTE_QCODES TV-remote map) for Phase 4 input
- screendump -> PPM -> PNG (pngjs), nearest-neighbour scaled

screenshot tool is now platform-aware: Vega captures host-side via QMP (no
simulator-server), iOS/Android unchanged.

SPIKE VERDICT (verified on the live macOS VVD): VVD screen capture is NOT
viable. QMP screendump returns an all-black frame because the VVD renders
through host GPU acceleration QEMU's console can't read (and macOS cannot
disable GL accel), while the on-device gwsi-tool-screenshooter segfaults on the
virtual device (matching Amazon's 'physical-only' note). The implementation is
correct (works where the framebuffer is in the QEMU console, e.g. a Linux
--no-gl-accel VVD) and now DETECTS the blank frame and throws an actionable
error pointing at a physical Fire TV, instead of returning a misleading black
image. The QMP client itself is display-independent and drives Phase 4 input.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New `remote` tool (vega-only, alwaysLoad): up/down/left/right/select/back/home/
  menu/playPause/rewind/fastForward with optional repeat, injected via QMP
  send-key (pressRemoteButton, one connection per call). This replaces touch
  gestures for Vega's directional-remote model.
- keyboard gains a Vega branch: named keys + free text over QMP send-key, with a
  char→qcode map (uppercase/symbols via shift chords). Vega path needs no
  simulator-server.
- Registered remote in setup-registry.

Design note: no separate vega-control blueprint — the per-call QMP connection
(shared by remote/keyboard/screenshot) is simpler and reconnects to the current
socket every time, which a persistent service would only complicate.

Verified on the live VVD against the running sample app via device-log capture:
`remote down x3`/`select` and `keyboard text:"Ab9"`/`key:"enter"` all reach the
app as EV_KEY / GWSI_LOG:KB key events. 20 tool tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New `read-device-logs` tool (vega): runs `vega device start-log-stream` for a
bounded window (durationMs, default 5s/max 30s), optional case-insensitive
substring filter, tail-trimmed to maxLines, then stops the stream. Returns the
text plus lines/truncated/capturedMs and a text-file artifact. Backed by a new
vega-logs util (substring filter, not regex, to avoid ReDoS on agent input).

Verified on the live VVD: filter "KB key" captured injected remote key events
during the window and wrote the artifact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…unblocks Vega CDP)

Metro discovery hard-required the X-React-Native-Project-Root header and threw
when absent. Amazon's forked Metro for Vega (RN 0.72, Metro 0.76.5) does not set
it, which blocked the entire CDP debugger suite on Vega even though the Hermes
inspector targets are present and connectable. The header only feeds source-map
path resolution, so fall back to process.cwd() instead of failing the session.

SPIKE VERDICT (verified live: Vega VVD, RN 0.72 debug build, Metro running,
8081 reverse-forwarded):
- /json/list exposes a 'Hermes React Native' CDP target (vm: Hermes, legacy /
  isNewDebugger:false) — argent's discovery + target-selection handle it.
- debugger-connect -> connected:true; debugger-evaluate '1+1' -> 2; globalThis
  exposes __fbBatchedBridge, __REACT_DEVTOOLS_GLOBAL_HOOK__, HermesInternal.
  => connect + evaluate are fully usable on Vega.
- debugger-component-tree / inspect-element / logbox-disable do NOT work as-is:
  Vega's Hermes rejects the bare `global` identifier (vs globalThis) and some
  iterator patterns the injected scripts use. Adapting those shared scripts is a
  scoped follow-up (risks regressing iOS/Android), deliberately out of scope here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New argent-vega SKILL.md teaches the verified Vega flow: list-devices ->
lifecycle (launch/restart/reinstall by app id + .vpkg) -> remote D-pad / keyboard
-> read-device-logs -> debugger connect+evaluate (use globalThis). Documents the
honest limitations: VVD screenshots unavailable (enable disable-auto-screenshot
to skip the dead post-action delay); gestures/describe/open-url unsupported;
component-tree/inspect not yet working; profiling/crashes via
amazon-devices-buildertools-mcp.

Cross-cutting decisions (no code):
- gesture-* tools keep no vega capability -> clean 'unsupported on vega' at the
  HTTP capability gate (verified).
- Deliberately did NOT add `remote` to the MCP auto-screenshot set: VVD capture
  always fails, so it would only add a ~1.4s dead delay per D-pad press.

Full suite green: 977 tests pass, tsc --build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sole

The VVD is an Android-emulator-derived QEMU (exposes console 5554 / adb 5555).
Capture host-side through the emulator console — `adb emu screenrecord
screenshot` against the auto-discovered `emulator-<consolePort>` — which grabs
the composited GL display that QMP screendump can't read on macOS. `adb emu`
manages the emulator console auth token automatically, so no direct-console
token / sim-server patch is needed (the console port is derived from the QMP
socket name).

- New vega-screen util: emulator-console capture (primary) with the QMP
  screendump path renamed to captureVegaScreenshotViaQmp as a Linux/no-gl-accel
  fallback. Nearest-neighbour scaling.
- screenshot tool routes Vega capture here and requires adb.
- Harden parseVegaDeviceList: only accept the `<DeviceType> : … - <serial>`
  shape, skipping adb-transport rows (emulator-NNNN / host:port) that appear if
  something runs `adb connect` against the VVD — which otherwise switches
  `vega device list` to adb form and breaks discovery. Do NOT `adb connect` the
  VVD; the emulator console is auto-discovered.
- Skill updated: screenshots now supported; added test.

Verified live: argent's screenshot tool returns real VVD content (960x540, full
colour range) via udid amazon-…; 980 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New `list-installed-apps` tool (vega): wraps `vega device installed-apps` and
returns { apps, count } — the installed component app ids, filtered to
reverse-DNS shape. Registered in setup-registry.

Verified live on the VVD: returns 113 app ids including the developer apps
(com.giolaq.multitv.vega.main, com.amazondeveloper.keplervideoapp.main, …).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a 'Live development (Fast Refresh)' section to the argent-vega skill:
debug-build requirement, npm start, reverse port-forward
(start-port-forwarding --port 8081 --forward false), launch, and the
/json/list Hermes-target check. Renumber subsequent sections; the debugger
section now points at the shared prerequisites.

Verified live: editing a .tsx title hot-reloaded onto the running VVD app
without restart (observed via argent screenshots before/after).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a note to the Vega Fast Refresh section: the steps are manual developer
actions — argent only connects to a running Metro (same as iOS/Android), it does
not start Metro or set up port-forwarding on any platform.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tighten prose to conserve tokens while preserving every command, param, and
gotcha: vega/remote model, lifecycle, input, screenshot (adb / no adb-connect),
logs, manual Metro/Fast-Refresh setup, debugger connect+evaluate (globalThis),
unsupported tools, and profiling deferral. Adds list-installed-apps; folds the
Fast Refresh + debugger sections (shared Metro prereqs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… clone 404)

argent init derived the skills source as ARGENT_SKILLS_REPO#v<version>, which
points at the upstream repo. For a -vega fork build that tag only exists on the
fork, so `npx skills add` failed: 'Remote branch v0.10.0-vega not found'. The
skills are bundled in the package anyway — fall back to the local SKILLS_DIR for
-vega builds. Upstream -next.N prereleases keep cloning their published tags.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e focus)

The remote/keyboard tools injected input via QMP send-key, which hits the QEMU
virtual keyboard (qwerty2). Vega delivers that to the app as a CHARACTER key
event that the Cartesian focus engine ignores — keys reached the app (visible in
logs) but D-pad navigation never moved.

Fix: inject on-device through inputd-cli (the input daemon), which routes via the
inputmgr-key-injection device and produces real remote/navigation events:
- remote  -> `inputd-cli button_press KEY_*` (KEY_UP/DOWN/LEFT/RIGHT, KEY_SELECT,
  KEY_BACK, KEY_HOMEPAGE, KEY_MENU, KEY_PLAYPAUSE/REWIND/FASTFORWARD). The build's
  inputd-cli rejects the documented [repeat] positional, so repeats are `;`-joined
  button_press calls in one `vega device run-cmd` round trip.
- keyboard -> `inputd-cli send_text` for text, `button_press` for named keys.
New vega-input util; the dead QMP send-key/qcode input code is removed from
vega-qmp (it keeps only the QMP client + screendump screenshot fallback).

Verified live on the VVD via argent's tools: `remote right x2` moves the focus
ring (Home -> Favourites, by screenshot); `keyboard "hello"` lands 5 CHARACTER
events in the app. 20 tool tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… set

Per a known-good Vega navigation script:
- select: KEY_SELECT -> KEY_ENTER (KEY_SELECT is a no-op; KEY_ENTER activates)
- home:   KEY_HOMEPAGE -> KEY_HOME
- add buttons: next (KEY_NEXTSONG), previous (KEY_PREVIOUSSONG),
  volumeUp (KEY_VOLUMEUP), volumeDown (KEY_VOLUMEDOWN), mute (KEY_MUTE)
- keyboard function keys: KEY_F<n> -> KEY_FN_F<n> (F1–F11)

Verified live: `remote select` now activates the focused item (screen changes);
all new key names accepted by inputd-cli. 20 tool tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oolkit

argent navigated Vega (Fire TV) blind — no `describe`, so D-pad presses
were guesses. The Vega automation toolkit (the server Amazon's Appium Vega
driver uses) exposes the full accessibility tree and is reachable without
Appium: device TCP 8383 over `adb forward` + a JSON-RPC `getPageSource`.

This wires that in as the `describe` Vega branch, emitting the same
DescribeNode tree contract as iOS/Android so the formatter, tap-point
header, and agent UX are unchanged. Each element surfaces `[focused]` /
`[selected]`, so the agent can see where the D-pad cursor is and drive the
existing `remote` tool deliberately instead of blind.

- vega-automation.ts: ensureAutomationToolkitEnabled (idempotently writes
  the `/tmp/automation-toolkit.enable` flag the toolkit reads at app
  launch), vegaJsonRpc (adb-forward + fetch), fetchVegaPageSource. A closed
  socket / empty root maps to a typed "toolkit-unavailable" signal.
- describe/platforms/vega + vega/source-parser.ts: a minimal XML parser
  (modeled on the uiautomator one) that skips `<traits>` metadata subtrees,
  captures `<text>` labels, flattens structural containers, and normalizes
  pixel frames to [0,1]. Unreachable toolkit → empty tree + relaunch hint.
- contract: add `vega-automation` source + optional focused/selected;
  format-tree renders them as flags and treats the source as nested.
- launch-app / restart-app Vega branches set the enable flag before launch
  (best-effort) so argent-launched apps are introspectable.
- Skill doc: document `describe` as the Vega discovery tool.

Verified live on the VVD: describe returns the Jellyfin tree; pressing
`remote right` x2 moves the `[selected]` marker Home -> Favourites. Element
tap is intentionally deferred — interaction stays on `remote`. 1016 tests
pass (18 new).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds packages/native-devtools-vega: a static aarch64 Rust agent that runs on
the Vega (Fire TV) virtual device and exposes an HTTP API for input injection
and UI inspection, reached from the host over a single adb forward.

Replaces the ~1.6-1.8s per-call `vega device run-cmd`/vda handshake. The agent
holds an `inputd-cli start` REPL open and pipes button_press/send_text per line
with a short holdDuration, dropping D-pad presses from ~1800ms to ~3-6ms
(~300x) — verified on a live VVD (focus moves; text + getPageSource proxy work).

Phase 0 (this commit): the Rust agent + README documenting context/decisions.
Phase 1 (next): VegaDevice blueprint + resolveVegaTransport (agent -> adb-shell
-> vega-cli fallbacks) + tool rewiring in packages/tool-server.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The post-tool auto-screenshot ignored a caller's explicit
`includeImageInContext: false`, attaching an image the caller asked to
suppress. Gate the auto-screenshot on that flag so opting out is honored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hase 1)

Land the agent-only transport for Vega input + inspection. `remote`,
`keyboard`, and `describe` now go through the on-device `argent-vega-agent`
over a keep-alive HTTP socket instead of per-call `vega device run-cmd` /
`adb forward` + fresh `fetch`.

- vega-agent-manager: lifecycle singleton (deploy-if-missing, start, forward,
  health, restart-on-death); teardown wired into stop-all-simulator-servers.
- vega-agent-client: keep-alive HTTP/1.1 client; transport faults raise a typed
  VegaAgentTransportError, logical {ok:false} envelopes raise a plain Error.
- vega-transport: agent-only transport; restarts the agent once only on a
  transport fault (a bad command rethrows, no needless redeploy). No
  adb/vega-cli fallback by design.
- vega-agent-assets / -install: resolve + deploy the binary (base64 over
  adb shell stdin; rename-over to dodge ETXTBSY), version-gated.
- describe: getPageSource now comes over the agent; removed the dead host-side
  JSON-RPC path (vegaJsonRpc/fetchVegaPageSource) and the redundant
  per-describe toolkit-flag touch (lifecycle tools own the flag).
- remote: `button` accepts a path (array) run in one round-trip.
- Tests: add vega-transport.test.ts (getPageSource + agent restart policy);
  trim vega-automation.test.ts to the surviving flag helper.

Verified end-to-end on a live VVD: describe ~2 ms steady-state (≈27× vs the
old adb-forward path), select reliable at holdMs 1, transparent restart on a
killed agent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace argent's in-tree TS lifecycle/transport for the Vega on-device agent
with shell-outs to `vega-fast-cli` — a standalone host binary (its own repo,
software-mansion-labs/vega-fast-cli) that discovers the VVD, deploys + starts
the embedded on-device server, and runs the command. Single source of truth;
others can use the CLI directly.

- Delete vega-agent-{manager,transport,client,install,assets}.ts and the
  migrated agent/ Rust source + on-device manifest. The lifecycle now lives in
  the Rust CLI.
- New utils/vega-fast-cli.ts: per-platform binary resolver (mirrors
  simulatorServerBinaryPath: bin/<platform>/vega-fast-cli, env overrides) +
  runVegaFastCli() spawn helper.
- Rewire tools to execFile the CLI: remote → `press <button>...`, keyboard →
  `type`/`key`, describe → `inspect` → parseVegaPageSource, stop-all → `stop`.
  vega-input trimmed to the button-name vocabulary (the CLI owns KEY_ mapping).
- Bundle the per-platform host binary like simulator-server
  (bin/<platform>/vega-fast-cli); download-vega-fast-cli.sh fetches the
  per-platform release assets with sha256 verify; CI asserts per-platform
  presence, gated by ARGENT_REQUIRE_VEGA_AGENT until the first release lands.

Supersedes the in-process keep-alive transport (commit 3ccb9b5). Typechecks;
996 tests pass. End-to-end device verification pending a vega-fast-cli release
(or a local build via ARGENT_VEGA_FAST_CLI_BIN).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Direct the agent to type into a focused Vega field with a single
`keyboard {text}` call (injects via inputd-cli send_text in one shot)
instead of D-padding to each letter on the soft keyboard. Falls back to
on-screen key-picking only if describe confirms the field rejected the
injected text.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vega-fast-cli's automation channel is a no-op on the CI VVD — `remote` and
`describe` silently did nothing there (only screenshots worked), and bundling a
per-host binary reintroduced an arch/glibc build for no capability gain.
Everything it did is reachable via adb + stock on-device facilities, so remove it:

- remote/keyboard -> adb shell inputd-cli button_press/send_text
  (utils/vega-input.ts), gated on an `inputd-cli get_screen_size` liveness probe
  so a dead channel throws instead of silently no-op'ing. KEY_ map restored from
  before the vega-fast-cli refactor.
- describe -> adb forward + JSON-RPC getPageSource to the on-device automation
  toolkit on :8383 (utils/vega-inspect.ts); vega-fast-cli was only a proxy to it.
- screenshot unchanged (adb emu screenrecord).
- Delete utils/vega-fast-cli.ts, scripts/download-vega-fast-cli.sh, and the
  bundle-tools / publish / build-package vega-fast-cli steps + ARGENT_REQUIRE_VEGA_AGENT.

Add a CI e2e (.github/workflows/vega-vvd-e2e.yml + scripts/ci/vega-vvd-test.sh):
builds the tool-server from source, boots the VVD via finloop/vega-virtual-device-action,
installs the kepler fixture, and drives remote + screenshot over the tool-server
HTTP API. typecheck clean; 996 tool-server tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-existing un-formatted files on the branch (not touched by the vega-fast-cli
removal) that the Format check flagged: prettier --write only, no logic changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e.hostPath)

The HTTP screenshot result is an image artifact ({data:{image:{hostPath,…}}}),
not {data:{path}}. The first run proved remote works and screenshots are real
2 MB PNGs; the script just read the wrong field and reported them missing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e guard, cleanup

Follow-up to the Vega (Fire TV) platform review on #1.

- Capability honesty: tighten every Vega tool to `vega: { virtual: true }`
  (drop `device: true`) until physical Fire TV is validated — describe's device
  path returned a misleading empty tree on hardware rather than a clean
  "unsupported" error.
- Multi-device safety: discoverQmpSocket now throws when more than one VVD
  socket is present instead of silently picking matches.sort()[0]. Every
  host-side Vega tool funnels through this chokepoint, so this fixes the silent
  wrong-targeting in one place. Dedupe by socket filename (= console port = one
  device) so the same socket seen under both probed dirs counts once. Comments
  on the threaded-but-unused serial/udid are now honest: the single-device
  contract is enforced, not silently assumed.
- Dead code: rename vega-qmp.ts -> vega-vvd.ts, dropping the QMP screen-capture
  client (blank on macOS, never the path that succeeds) and keeping only the
  load-bearing socket discovery; vega-screen calls the emulator console
  directly. Remove dead vegaShell/vegaShellQuote and the leftover
  packages/native-devtools-vega/ (+ its .gitignore block).
- Stale docs/comments: drop the fictional "vega-control service (CLI + QMP)"
  and the "~1.6s handshake" perf model (a remote call is now a single
  adb inputd-cli round-trip); fix keyboard schema drift (Vega serials, f1–f11,
  delayMs ignored on Vega); make the describe coordinate header remote-oriented
  on Vega (no tap formula); reinstall-app mentions .vpkg; document the
  auto-screenshot remote/keyboard rationale.

Tests: tool-server suite 1001 passed (+5 new vega-vvd guard tests); tsc --build,
test typecheck, and prettier all clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ts, surface multi-VVD on describe

Round-1 review of the #1 review-fix commit:

- SHOULD-FIX: the capability comment on the CLI-backed lifecycle/logs/list tools
  (reinstall-app, list-installed-apps, read-device-logs) wrongly claimed they
  target the VVD "via its emulator/QMP socket" — those reach the device through
  the `vega`/`kepler` CLI, which never touches the socket. Reworded to the
  accurate "target the single running VVD" across all five non-describe tools
  (launch/restart also touch the socket only for the toolkit flag), so the
  comments no longer reintroduce the stale-comment defect the review flagged.
- NIT → fix: `discoverQmpSocket` now throws a typed `MultipleVegaDevicesError`,
  and `describeVega` rethrows it instead of swallowing every fetch failure into
  the "relaunch the app" hint. So with >1 VVD running, `describe` surfaces the
  guard's "stop all but one VVD" message rather than misleading relaunch advice;
  all other toolkit failures keep the best-effort empty-tree+hint behavior.

Tests: new vega-describe.test.ts covers the rethrow vs hint paths; vega-vvd
guard test asserts the typed error. Full tool-server suite green; tsc --build
and prettier clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From the live VVD verification (PR #3 review comments):

1. vega-CLI tools crashed when the long-lived tool-server's cwd was a
   since-deleted dir (e.g. a torn-down worktree): the inherited dead cwd
   made the `vega` Python CLI fail in os.getcwd(). `runVega` now spawns
   with a guaranteed-live cwd via `resolveSpawnCwd` (validates process.cwd(),
   falls back to tmpdir) so no tool-server restart is needed. Unit-tested
   with a simulated missing cwd.

2. debugger-component-tree / debugger-inspect-element now turn the Vega
   Hermes failure signatures (binding timeout / `_internalInstanceHandle`
   TypeError) into a clean unsupported message pointing at debugger-evaluate,
   instead of a raw timeout / TypeError.

3. react-profiler-start / -renders give a Vega-aware message (build is debug;
   the Vega RN 0.72 runtime doesn't inject the React DevTools backend) and
   point to amazon-devices-buildertools-mcp, instead of the misleading
   "production build" hint.

4. view-network-logs description + empty-state now state the fetch()-only
   scope explicitly (XMLHttpRequest/axios traffic is not captured).

tool-server: 1011 tests pass; tsc --build + prettier clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/build-package-artifact.yml Outdated
Comment thread packages/argent-mcp/src/auto-screenshot.ts Outdated
Comment thread packages/argent-mcp/src/mcp-server.ts
Comment thread packages/argent-mcp/test/auto-screenshot.test.ts
finloop and others added 2 commits June 18, 2026 16:43
Add Vega/Fire TV to the always-on rule headline and use-cases, a
dedicated routing entry, and a VVD lifecycle (start/stop via vega CLI)
section to the skill, so Vega tasks discover argent-vega without manual
hunting.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review, the Vega PR shouldn't touch the general MCP auto-screenshot
machinery. Revert auto-screenshot.ts, mcp-server.ts, and auto-screenshot.test.ts
to upstream/main (re-adds `describe` to AUTO_SCREENSHOT_TOOLS, drops the TODO and
the test tweaks). The argent-vega skill already navigates "on the tree alone"
without claiming describe is screenshot-free, so it stays accurate. Any change to
describe's auto-screenshot behaviour can land in a separate, non-Vega PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid PR, we are really thankful as a team for you contribution :)

I only really dislike that this PR changes some unrelated things that are out of scope.

It also seems to me that remote could potentially map its gestures clearly onto tools we already expose, but i could be wrong.

Comment thread packages/registry/src/types.ts Outdated
Comment thread packages/skills/skills/argent-vega/SKILL.md
Comment thread packages/tool-server/src/tools/button/index.ts
Comment thread packages/tool-server/src/tools/describe/platforms/vega/source-parser.ts Outdated
Comment thread packages/tool-server/src/tools/devices/boot-device.ts Outdated
Comment thread packages/tool-server/src/tools/keyboard/index.ts Outdated
Comment thread packages/tool-server/src/tools/list-installed-apps/index.ts Outdated
Comment thread packages/tool-server/src/tools/read-device-logs/index.ts Outdated
Comment thread packages/tool-server/src/tools/remote/index.ts Outdated
finloop and others added 15 commits June 18, 2026 16:53
The Vega source-parser tokenized getPageSource XML with hand-rolled
regexes plus a skipDepth counter to drop <traits> subtrees — flaky on
edge cases. Replace it with fast-xml-parser (preserveOrder) and a small
adapter into the existing VegaXmlNode shape, so the <traits> skip becomes
structural and entity decoding is handled by the parser. Downstream
convert/findScreenSize logic is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- argent-vega SKILL.md: add a "Use when" trigger to the description so the
  Skill Description Quality grader scores it 10.0; reformat with prettier.
- source-parser: narrow the parsed text node to a string instead of
  String()-ing an unknown, clearing a no-base-to-string eslint error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Low-hanging review fixes:
- types.ts: collapse the ToolCapability.vega doc to a one-liner (matches the
  other platform blocks).
- describe/platforms/vega.ts + source-parser.ts: trim the long header docs.
- argent-vega skill: fix "Konwledgebase" heading typo.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The apple/android/chromium capability blocks carry no comment; the vega block
doesn't need one either (the `vvd`/`device` field names are self-describing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the Vega read-device-logs tool and everything that supported it:
- delete the tool (tools/read-device-logs) and its only consumer util
  (utils/vega-logs)
- unregister it from setup-registry
- remove it from the argent-vega SKILL + the vega routing rule
- drop its e2e check from scripts/ci/vega-vvd-test.sh and renumber the
  remaining tests

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add Vega (Amazon Fire TV) Virtual Device support: boot-device accepts a
`vvdImage` to start the SDK-managed VVD, and list-devices/device-info discover
and classify running and stopped VVDs.

VVD liveness is decided from `vega device list` rather than `vega
virtual-device status` (which misreports running:false when the VVD's ports are
bumped off 5554/5555 by a co-running Android emulator); discoverQmpSocket
filters orphaned QMP sockets by cross-checking adb; and startVvd boots the
single SDK-default instance (no `-p`) so installed-app state persists across
runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the now-unused `spawn` import from vega-cli.ts (eslint
no-unused-vars) and reflows a long line in vega-vvd.ts so prettier
--check passes. Unblocks the lint/format CI gates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Vega VVD E2E install step ran `vega device install-app` once,
immediately after `adb wait-for-device`. adb seeing the device precedes
the Vega device agent attaching, and the boot action's non-black-frame
readiness gate can fire before it — so install-app intermittently failed
with "No connected Vega devices found" (seen after the host image
updated). Wait for `vega virtual-device status` running:true (the same
signal argent's isVvdRunning uses), then retry install — matching the
retry pattern every other step in this script already uses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
REMOTE_KEYCODES mapped `home` to KEY_HOME, which is inert on the VVD —
inputd-cli injects the key but nothing happens. The VVD remote skin
keymap (vmtools/agent/skins/tv-remote/layout) uses KEY_HOMEPAGE; verified
on-device that KEY_HOMEPAGE backgrounds the foreground app while KEY_HOME
does nothing. Fix the code and the test that pinned the wrong value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…removal

The E2E smoke test still called the old `remote` tool (renamed to
`tv-remote` in 0aa65ff) and `list-installed-apps` (removed in e340b3d),
both of which returned empty responses and failed the job. Point the
remote test at `tv-remote` and drop the list-installed-apps check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@finloop finloop marked this pull request as ready for review June 19, 2026 11:33
@finloop finloop requested a review from latekvo June 19, 2026 11:42

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of the Vega (Fire TV) platform support — no devices booted. Findings inline.

Comment thread packages/tool-server/src/tools/devices/list-devices.ts
Comment thread packages/tool-server/src/utils/vega-input.ts Outdated
Comment thread packages/tool-server/src/tools/tv-remote/index.ts Outdated
Comment thread packages/tool-server/src/tools/describe/platforms/vega.ts Outdated
Comment thread packages/tool-server/src/tools/keyboard/index.ts
Comment thread packages/tool-server/src/utils/vega-vvd.ts Outdated
Comment thread packages/tool-server/src/tools/describe/format-tree.ts Outdated
Comment thread packages/tool-server/src/tools/devices/boot-device.ts Outdated
Comment thread packages/tool-server/src/utils/vega-devices.ts Outdated
Comment thread packages/tool-server/src/tools/devices/boot-device.ts Outdated
Resolve the unresolved review threads on the Vega PR:

- Preflight Vega input/describe on `adb` (not `vega`): tv-remote and
  describe now `requires: ["adb"]`, and the keyboard vega branch calls
  `ensureDep("adb")` — a running VVD with an unsourced `~/vega/env` is no
  longer blocked, and a missing adb yields a clean 424 instead of ENOENT.
- Scale the `inputd-cli` adb timeout with path length so long `tv-remote`
  paths aren't SIGKILLed mid-sequence.
- list-devices: retry the getprop probe so a transiently-failing real
  emulator isn't misclassified as a VVD shadow and dropped.
- discoverQmpSocket: retry the adb correlation post-boot so the first
  call doesn't throw the misleading "stale socket / crashed" error.
- boot-device: include `force` in the Vega boot coalescing key so a force
  boot can't join a non-force in-flight boot and skip the restart; fix the
  recovery hint to point at the real `vvdImage` field.
- vega-devices: resolve the running VVD's image against installed images
  (fallback to the sole image) so a missing/mismatched `info.profile` no
  longer emits a phantom `stopped` duplicate.
- format-tree: scope the Vega `button`/`text`/`image` content roles to the
  vega source so they no longer alter Chromium describe output.
- vega-screen: reuse `getScreenshotScale()` + lanczos3 `resizeDecodedPng()`
  instead of a divergent hand-rolled nearest-neighbour downscaler.

Add regression tests for the getprop-retry and phantom-stopped dedup cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@finloop finloop requested a review from latekvo June 19, 2026 13:25

@finloop finloop left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the Vega (Fire TV) platform support. Findings below are inline, ordered roughly by severity — a few correctness issues around boot image selection, VVD/adb device correlation, describe parsing and dedup, plus some CI-guard and consistency notes.

await stopVvd();
}

await startVvd({ timeoutSeconds: Math.ceil(params.bootTimeoutMs / 1_000) });

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bootVegaImpl resolves and validates image from vvdImage, but image.path is then never used and startVvd() is called without it — it runs vega virtual-device start -t <n> with no -p <path>. Combined with isVvdRunning() matching any virtual device, the vvdImage selector isn't actually honored:

  • With tv already running, boot-device {vvdImage:"tablet"} returns early with tv's serial but labeled vvdImage:"tablet", so every later tool drives tv while the caller believes it's on tablet.
  • With nothing running, startVvd boots the SDK default rather than the requested image.

Invisible on a single-image SDK, wrong for multi-image setups. vega-sdk.ts resolves VvdImage.path specifically to feed start -p, so either thread image.path into startVvd (and make the already-running check image-aware), or drop the per-image selector and document boot as 'the single SDK VVD'.

const liveSerials = new Set(connected.filter((d) => d.state === "device").map((d) => d.serial));
return names.filter((name) => {
const port = consolePortFromSocketName(name);
return port !== null && liveSerials.has(`emulator-${port}`);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filterLiveSockets keeps a QMP socket whenever emulator-<port> is a live adb device, but never verifies that device is the VVD rather than a real Android emulator. If a prior VVD crashed leaving /tmp/qmp-socket-5554.sock while an Android emulator runs as emulator-5554, the stale socket is treated as live, discoverVegaConsolePort() returns 5554, and screenshot/tv-remote/keyboard/describe all drive the Android emulator instead of the VVD.

Separately, if (!connected) return names (line 44) fails open — a transient adb error returns every socket unfiltered, which can spuriously trip MultipleVegaDevicesError. Consider confirming the port maps to a VVD (cross-check vega device list / the running-VVD serial) before accepting the socket.

if (xml.length < PAGE_SOURCE_EMPTY_LENGTH) {
return { tree: EMPTY_TREE, source: "vega-automation", hint: UNAVAILABLE_HINT };
}
return { tree: parseVegaPageSource(xml), source: "vega-automation" };

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseVegaPageSource(xml) runs outside the try/catch above, so a parse failure escapes as an unhandled error instead of the graceful empty-tree + relaunch hint. This is reachable: fetchVegaPageSource returns JSON.stringify(result ?? "") when the toolkit result isn't a string (vega-inspect.ts:54), and postJson never checks res.statusCode. So a toolkit HTTP error / structured body becomes non-XML JSON — if it's >50 chars, parseVegaXml returns null and throw new Error('Failed to parse Vega page source') crashes describe; if it's "", the screen is silently reported as empty. Suggest wrapping the parse in the same empty-tree fallback and checking the HTTP status in postJson.

});
// `install-app` prints "Installing/Updating '…' ...success" on success.
const output = `${stdout}\n${stderr}`;
if (!/success/i.test(output)) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/success/i.test(output) matches any substring success, including unsuccessful / was not successful. A failed install-app whose output contains those phrases is reported as { reinstalled: true }. Anchor the match to the actual success line (e.g. /\.\.\.\s*success\b/i) so a failure can't read as success.

rows = []; // CLI listing failed; still surface installed images below
}

const info = rows.length === 1 ? await readVegaInfo() : null;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const info = rows.length === 1 ? await readVegaInfo() : null plus the resolveVvdImageName fallback means that when vega device info fails or omits profile and ≥2 images are installed, the running VVD gets vvdImage:null. connectedImages is then empty, so the stopped-image loop re-emits that same running image as a state:"stopped" row — the device shows up twice (once running with vvdImage:null, once stopped with the image name). The existing test only covers the single-installed-image case, where the installedImages.length === 1 fallback happens to mask this.

if (isVega) {
header.push(
"Vega is remote-driven, not touch — there is no tap. Use the frames as spatial hints to plan " +
"D-pad moves with the `tv-remote` tool: compare the target's frame to the `[focused]` element's " +

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This header tells the agent to count rows/columns from 'the [focused] element', but per the contract.ts note the toolkit usually reports the highlighted item via selected while focused stays false (the fixture has two [selected] and zero [focused]). As written the agent is told to anchor on a flag that's frequently absent, breaking the navigation instruction. Mention [selected] as the cursor fallback here so the guidance matches the data.

if (subcommands.length === 0) return;
const { serial } = await emulatorSerial();
const presses = subcommands
.map((s) => `inputd-cli ${s} >/dev/null 2>&1 || true`)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each press is ... >/dev/null 2>&1 || true, and the only liveness gate is get_screen_size — which (per the TODO above) still succeeds when developer mode is off, even though button_press/send_text then silently fail. So tv-remote/keyboard return { pressed/typed, count } as if the input landed while nothing actually happened. Flagging since the false success is user-visible; gating on a dev-mode-only probe (or checking vsm developer-mode) would make it fail loudly as the TODO already notes.

Comment thread .github/workflows/vega-vvd-e2e.yml Outdated
branches:
- main
paths:
- "packages/tool-server/src/tools/remote/**"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two paths: entries don't match the files they're meant to guard:

  • tools/remote/** — the tool lives at tools/tv-remote/, so there is no tools/remote/ directory; edits to tv-remote/index.ts never trigger this workflow.
  • describe/platforms/vega.ts (line 30) matches only that file, not the parser at describe/platforms/vega/source-parser.ts.

So the regression guard won't run on changes to its two headline subjects — the tv-remote tool and the describe XML parser.

Suggested change
- "packages/tool-server/src/tools/remote/**"
- "packages/tool-server/src/tools/tv-remote/**"

(and broaden the describe glob to describe/platforms/vega/**).

resp="$(post_tool describe "$(printf '{"udid":"%s"}' "$SERIAL")")" || resp=""
src="$(jget "$resp" source)"
desc="$(jget "$resp" description)"
if [ "$src" = "vega-automation" ] && [ -n "$desc" ]; then desc_ok=1; break; fi

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gate can't detect a broken describe. describeVega returns source:"vega-automation" even in the empty/unreachable case, and format-tree always emits a non-empty header (Source:/Mode:/ROOT ...), so both src == "vega-automation" and -n "$desc" hold even when the toolkit never attached and the tree is empty. Assert on real tree content instead — e.g. a known label like Search/Home, or that the body has element lines beyond the header.

if (device.platform === "chromium") {
return { chromium: chromiumCdpRef(device) };
}
if (device.platform === "vega") {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keyboard and screenshot resolve Vega by hand — services() returns {} and ensureDep("adb") is called imperatively inside execute — while launch/restart/reinstall use the declarative PlatformImpl.requires path through dispatchByPlatform. Two dispatch styles for one platform invites drift, and the dep matrix is already inconsistent: tv-remote requires adb; launch/restart/reinstall require only vega (though they also shell to adb via ensureAutomationToolkitEnabled); keyboard/screenshot require only adb. Routing Vega through the same dispatchByPlatform/requires mechanism (or a small Vega blueprint) would keep gating in one place.

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the Vega (Fire TV) platform addition. Specific findings inline.

Comment on lines +61 to +62
: platform === "vega"
? "vvd"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolveDevice forces kind: "vvd" for every amazon- serial, so a Vega device's kind is derived purely from serial shape. A physical Fire TV is surfaced by listVegaDevices/classifyKind in vega-devices.ts as kind: "device", but the capability gate goes through resolveDevice and therefore only ever sees "vvd". assertSupported checks matrix[device.kind], so the vega: { vvd: true } declarations (which deliberately omit device) never reject hardware — a physical Fire TV is accepted and driven by the VVD-only adb paths. The v1 "Virtual-Device-only, reject hardware up front" guarantee from the PR description is not actually enforced at the gate.

await stopVvd();
}

await startVvd({ timeoutSeconds: Math.ceil(params.bootTimeoutMs / 1_000) });

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resolved image (line 1063) is only used for the existence check above — it is never passed to startVvd, and startVvd (vega-vvd.ts) runs vega virtual-device start -t <secs> with no image argument. The returned payload echoes vvdImage: params.vvdImage (lines 1078 and 1092) regardless of which image is actually running, and isVvdRunning() is image-agnostic. So requesting image B while image A is running returns A's serial labelled B, and force re-starts the SDK default rather than B. On a multi-image host the requested image is silently ignored.

if (xml.length < PAGE_SOURCE_EMPTY_LENGTH) {
return { tree: EMPTY_TREE, source: "vega-automation", hint: UNAVAILABLE_HINT };
}
return { tree: parseVegaPageSource(xml), source: "vega-automation" };

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseVegaPageSource(xml) is outside the try/catch that wraps fetchVegaPageSource above, and it throws on unparseable input. A non-empty but malformed/truncated page source (length ≥ 50, so it passes the empty-length check) makes the whole describe call fail with a raw parse error, instead of degrading to the EMPTY_TREE + hint path used for every other failure in this function.

}
const base = Array.isArray(params.button) ? params.button : [params.button];
const repeat = Math.max(1, Math.floor(params.repeat ?? 1));
const buttons = repeat === 1 ? base : Array.from({ length: repeat }, () => base).flat();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

button accepts up to 64 entries and repeat up to 50, so one schema-valid call flattens to 3200 presses here. injectViaInputd (vega-input.ts) settles 0.3s between each press, so this runs ~16 min on-device with an adb timeout budgeted to ~27 min, holding the connection open the whole time. The tool isn't marked long-running, so the request can also collide with the idle-shutdown timer.

Comment thread .github/workflows/vega-vvd-e2e.yml Outdated
branches:
- main
paths:
- "packages/tool-server/src/tools/remote/**"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path glob points at tools/remote/**, but the tool lives in tools/tv-remote/, so it matches nothing. With the rest of this list, changes to tv-remote/index.ts, describe/platforms/vega/source-parser.ts, boot-device.ts, list-devices.ts, and device-info.ts do not trigger this workflow — so the "remote + screenshot" regression guard never runs when the remote tool or the source-parser change.

return null;
}

function isTrue(attrs: Record<string, string>, key: string): boolean {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isTrue only accepts the exact literal "true", so focused/selected/focusable/clickable arriving as "True" or "1" are silently treated as false. The fixture only uses lowercase, so any casing variance in real toolkit output would silently drop these state flags with no error.

// A single regex alternation scans left-to-right and consumes each match
// once, so a decoded `&` produced by one step never feeds the next step.
function decodeXmlEntities(s: string): string {
export function decodeXmlEntities(s: string): string {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This export is the only change to this file, and decodeXmlEntities has no importers outside this module (the new Vega parser uses fast-xml-parser). The widening to export adds an unused public symbol — looks like a leftover from an earlier approach.

Comment on lines +18 to +21
* `amazon-<id>` (e.g. `amazon-4a27df03c9777152`); no Android adb serial
* (`emulator-<port>`, a hardware serial, or `ip:port`) starts with it, so the
* prefix classifies Vega by shape — the same approach as Chromium above. v1
* supports the Virtual Device only, so a Vega serial resolves to kind `vvd`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment states that no Android adb serial starts with amazon-, but ro.serialno is vendor-defined and not constrained by adb. No known shipping device collides, so this is informational, but the claim is more absolute than is guaranteed — an Android device with such a serial would be misrouted to the Vega paths.


### `describe`

Nested element tree from the on-device automation toolkit — each line is a `button`/`text`/`image` with its label, `id` (test_id), `[clickable]`, and **`[focused]`/`[selected]`** + a normalized [0,1] frame. `[focused]` is the live D-pad cursor (track this); `[selected]` is just an active-tab/highlight state. Navigate on the tree alone. If the tree comes back empty → `restart-app` and retry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tells the agent to navigate by the [focused] cursor and to treat [selected] as "just an active-tab/highlight state", but the only captured page source has zero focused="true" nodes and uses selected="true" for the active item. Following this verbatim, an agent would find no [focused] line and stall. The guidance and the parser's focus handling don't agree.


- Metro connects only on port **8081** — fixed, cannot be changed.
- Profiling / crashes → use the `amazon-devices-buildertools-mcp` server (`analyze_perfetto_traces`, `get_app_hot_functions`, `symbolicate_acr`).
- Unsupported tools, with the Vega equivalent: `gesture-*` → use `tv-remote`; `open-url` → not wired; `debugger-*` → JS debugger not supported on Vega. These return "unsupported on vega".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says unsupported tools return "unsupported on vega", but the actual error text is Tool '<id>' is not supported on vega vvd. (UnsupportedOperationError) or ... is not yet implemented on vega. An agent matching on the documented literal would never match.

…robustness

Resolve the remaining unresolved review threads on the Vega PR:

- boot-device: thread the resolved image's package path into `startVvd`
  (`vega virtual-device start -p <path>`) so the `vvdImage` selector is
  actually honored instead of always booting the SDK default. When a VVD is
  already running, resolve the *actually running* image via `listVegaDevices`
  and reject a non-force boot of a different image (v1 is single-VVD) rather
  than returning the running device mislabeled.
- describe/source-parser: scope the tree to the foreground app(s), dropping the
  persistent `com.amazon.keplerlauncherapp` overlay so its controls/duplicate
  test_ids no longer leak in. Read boolean state flags case-insensitively
  (`true`/`1`).
- describe/vega: wrap `parseVegaPageSource` in the empty-tree+hint fallback so a
  malformed/truncated page source degrades gracefully instead of throwing.
- reinstall-app/vega: anchor the success check on the `...success` line so
  "unsuccessful"/"was not successful" can't read as success. Validate bundleId
  with BUNDLE_ID_PATTERN to match launch/restart.
- vega-cli: give the binary memo a 60s TTL (mirrors android-binary) so a
  negative result no longer sticks for the process lifetime.
- format-tree + SKILL.md: document `[selected]` as the D-pad cursor fallback
  when no element reports `[focused]`, and correct the unsupported-tool error
  text.
- uiautomator-parser: drop the now-unused `decodeXmlEntities` export.
- vega-vvd-e2e.yml: fix the path globs for the tv-remote rename and the
  describe parser/devices files so the regression guard actually triggers.

Add tests for launcher scoping, case-insensitive flags, image -p threading,
and the different-image rejection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants