Skip to content

feat(mcp): send input actions to a scene with no target - #9549

Draft
eordano wants to merge 1 commit into
decentraland:devfrom
eordano:feat/mcp-input-action
Draft

feat(mcp): send input actions to a scene with no target#9549
eordano wants to merge 1 commit into
decentraland:devfrom
eordano:feat/mcp-input-action

Conversation

@eordano

@eordano eordano commented Jul 31, 2026

Copy link
Copy Markdown
Member

The MCP server can drive everything about the client except the one thing an SDK7 game is usually built on: input that no entity owns.

click_entity, the only tool that produces an InputAction, delivers it through the reticle pipeline — a camera-origin ray has to hit a collider, that collider has to resolve to a scene entity, and the entity has to carry a PointerEvents component qualified for the hit distance. Miss any of those and it reports why it missed instead of delivering an input edge. That is the right contract for clicking a door and the wrong one for a game whose controls are inputSystem.isTriggered(IA_PRIMARY, ...) with no entity argument at all, so the scenes worth driving from a bot were the ones no tool could reach.

The client already writes exactly that shape for a real key press: PrepareGlobalInputEventsSystem turns every bound Unity input action into a GlobalInputEvents entry, and each running current scene's WritePointerEventResultsSystem appends it as a PBPointerEventsResult on SCENE_ROOT_ENTITY with a null hit — what the SDK's inputSystem reads for its no-entity overloads. So this needs no new wire format and no new CRDT path, only a way into that buffer.

press_input_action is a new tool rather than a mode on click_entity: with no target there is nothing for the occlusion, maxDistance and hover diagnostics that make up most of click_entity to say, and every one of its failure messages names an entity. McpInputActionSystem publishes the edge, ordered after PrepareGlobalInputEventsSystem so it lands once that system has cleared the buffer and before the scene worlds read it later in the same frame.

Two details are not cosmetic:

A press owns its release. The hold lives in the ECS request, not in the awaiting tool call, so an agent that disconnects mid-hold cannot leave the scene believing a button is still down. This follows walk / McpMovementOverride, which already holds a duration that way, rather than click_entity, which composes its legs from the tool side — a click's two legs are one frame apart, a hold can be thirty seconds.

The release waits for the scene to leave the press's tick. The SDK keeps pointer results in a value set keyed by timestamp, which the writer fills with the scene tick number, and a down and an up sharing one key collapse into an ambiguous button state. The press tick is read on the frame after the press was published, because the scene may stamp it with either of the two ticks that frame spans; reading it late keeps the gate from ever being too permissive.

click_entity's button gains action_3 through action_6 in the same pass. A PointerEvents entry may name any InputAction as its button and the delivery path already carries whichever one it is given, so the previous {pointer, primary, secondary} restriction was arbitrary; all four are added rather than only action_5 because there is no per-action machinery and a gap at 3/4/6 would need a justification none of them has. Movement actions and IA_ANY stay out — they are not buttons a cursor clicks with, and press_input_action sends them with no target. Both tools now narrow one shared McpInputAction enum through the allowed parameter McpJsonSchema.Enum already had; the existing wire values are unchanged.

Three limits the tool states rather than hides: delivered means the edge reached a running current scene, not that the scene reacted; the scene writer drops the whole global buffer for a frame in which an entity-targeted result was written, so a call racing a click_entity can be swallowed; and one action is in flight at a time, so a second call preempts a held press and reports releaseMissed rather than silently leaving a stuck button.

The MCP server can drive everything about the client except the one thing
an SDK7 game is usually built on: input that no entity owns.

click_entity, the only tool that produces an InputAction, delivers it
through the reticle pipeline — a camera-origin ray has to hit a collider,
that collider has to resolve to a scene entity, and the entity has to
carry a PointerEvents component qualified for the hit distance. Miss any
of those and it reports why it missed instead of delivering an input
edge. That is the right contract for clicking a door and the wrong one
for a game whose controls are inputSystem.isTriggered(IA_PRIMARY, ...)
with no entity argument at all, so the scenes worth driving from a bot
were the ones no tool could reach.

The client already writes exactly that shape for a real key press:
PrepareGlobalInputEventsSystem turns every bound Unity input action into
a GlobalInputEvents entry, and each running current scene's
WritePointerEventResultsSystem appends it as a PBPointerEventsResult on
SCENE_ROOT_ENTITY with a null hit — what the SDK's inputSystem reads for
its no-entity overloads. So this needs no new wire format and no new CRDT
path, only a way into that buffer.

press_input_action is a new tool rather than a mode on click_entity:
with no target there is nothing for the occlusion, maxDistance and hover
diagnostics that make up most of click_entity to say, and every one of
its failure messages names an entity. McpInputActionSystem publishes the
edge, ordered after PrepareGlobalInputEventsSystem so it lands once that
system has cleared the buffer and before the scene worlds read it later
in the same frame.

Two details are not cosmetic:

A press owns its release. The hold lives in the ECS request, not in the
awaiting tool call, so an agent that disconnects mid-hold cannot leave
the scene believing a button is still down. This follows walk /
McpMovementOverride, which already holds a duration that way, rather than
click_entity, which composes its legs from the tool side — a click's two
legs are one frame apart, a hold can be thirty seconds.

The release waits for the scene to leave the press's tick. The SDK keeps
pointer results in a value set keyed by timestamp, which the writer fills
with the scene tick number, and a down and an up sharing one key collapse
into an ambiguous button state. The press tick is read on the frame after
the press was published, because the scene may stamp it with either of
the two ticks that frame spans; reading it late keeps the gate from ever
being too permissive, and the frame that reads it is paid for by the gate
it then fails.

PrepareGlobalInputEventsSystem gains a summary recording that it refills
the buffer per frame for the scene worlds to drain later in the same one,
so a second writer must order after it — the invariant this feature now
depends on, written where someone moving the Clear() will see it.

click_entity's button gains action_3 through action_6 in the same pass. A
PointerEvents entry may name any InputAction as its button and the
delivery path already carries whichever one it is given, so the previous
{pointer, primary, secondary} restriction was arbitrary; all four are
added rather than only action_5 because there is no per-action machinery
and a gap at 3/4/6 would need a justification none of them has. Movement
actions and IA_ANY stay out — they are not buttons a cursor clicks with,
and press_input_action sends them with no target. Both tools now narrow
one shared McpInputAction enum through the allowed parameter
McpJsonSchema.Enum already had; the existing wire values are unchanged.

Three limits the tool states rather than hides: delivered means the edge
reached a running current scene, not that the scene reacted; the scene
writer drops the whole global buffer for a frame in which an
entity-targeted result was written, so a call racing a click_entity can
be swallowed; and one action is in flight at a time, so a second call
preempts a held press and reports releaseMissed rather than silently
leaving a stuck button.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@eordano
eordano force-pushed the feat/mcp-input-action branch from 98fba4b to 3e8f051 Compare July 31, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant