feat(mcp): send input actions to a scene with no target - #9549
Draft
eordano wants to merge 1 commit into
Draft
Conversation
The MCP server can drive everything about the client except the one thing
an SDK7 game is usually built on: input that no entity owns.
click_entity, the only tool that produces an InputAction, delivers it
through the reticle pipeline — a camera-origin ray has to hit a collider,
that collider has to resolve to a scene entity, and the entity has to
carry a PointerEvents component qualified for the hit distance. Miss any
of those and it reports why it missed instead of delivering an input
edge. That is the right contract for clicking a door and the wrong one
for a game whose controls are inputSystem.isTriggered(IA_PRIMARY, ...)
with no entity argument at all, so the scenes worth driving from a bot
were the ones no tool could reach.
The client already writes exactly that shape for a real key press:
PrepareGlobalInputEventsSystem turns every bound Unity input action into
a GlobalInputEvents entry, and each running current scene's
WritePointerEventResultsSystem appends it as a PBPointerEventsResult on
SCENE_ROOT_ENTITY with a null hit — what the SDK's inputSystem reads for
its no-entity overloads. So this needs no new wire format and no new CRDT
path, only a way into that buffer.
press_input_action is a new tool rather than a mode on click_entity:
with no target there is nothing for the occlusion, maxDistance and hover
diagnostics that make up most of click_entity to say, and every one of
its failure messages names an entity. McpInputActionSystem publishes the
edge, ordered after PrepareGlobalInputEventsSystem so it lands once that
system has cleared the buffer and before the scene worlds read it later
in the same frame.
Two details are not cosmetic:
A press owns its release. The hold lives in the ECS request, not in the
awaiting tool call, so an agent that disconnects mid-hold cannot leave
the scene believing a button is still down. This follows walk /
McpMovementOverride, which already holds a duration that way, rather than
click_entity, which composes its legs from the tool side — a click's two
legs are one frame apart, a hold can be thirty seconds.
The release waits for the scene to leave the press's tick. The SDK keeps
pointer results in a value set keyed by timestamp, which the writer fills
with the scene tick number, and a down and an up sharing one key collapse
into an ambiguous button state. The press tick is read on the frame after
the press was published, because the scene may stamp it with either of
the two ticks that frame spans; reading it late keeps the gate from ever
being too permissive, and the frame that reads it is paid for by the gate
it then fails.
PrepareGlobalInputEventsSystem gains a summary recording that it refills
the buffer per frame for the scene worlds to drain later in the same one,
so a second writer must order after it — the invariant this feature now
depends on, written where someone moving the Clear() will see it.
click_entity's button gains action_3 through action_6 in the same pass. A
PointerEvents entry may name any InputAction as its button and the
delivery path already carries whichever one it is given, so the previous
{pointer, primary, secondary} restriction was arbitrary; all four are
added rather than only action_5 because there is no per-action machinery
and a gap at 3/4/6 would need a justification none of them has. Movement
actions and IA_ANY stay out — they are not buttons a cursor clicks with,
and press_input_action sends them with no target. Both tools now narrow
one shared McpInputAction enum through the allowed parameter
McpJsonSchema.Enum already had; the existing wire values are unchanged.
Three limits the tool states rather than hides: delivered means the edge
reached a running current scene, not that the scene reacted; the scene
writer drops the whole global buffer for a frame in which an
entity-targeted result was written, so a call racing a click_entity can
be swallowed; and one action is in flight at a time, so a second call
preempts a held press and reports releaseMissed rather than silently
leaving a stuck button.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eordano
force-pushed
the
feat/mcp-input-action
branch
from
July 31, 2026 20:00
98fba4b to
3e8f051
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The MCP server can drive everything about the client except the one thing an SDK7 game is usually built on: input that no entity owns.
click_entity, the only tool that produces an InputAction, delivers it through the reticle pipeline — a camera-origin ray has to hit a collider, that collider has to resolve to a scene entity, and the entity has to carry a PointerEvents component qualified for the hit distance. Miss any of those and it reports why it missed instead of delivering an input edge. That is the right contract for clicking a door and the wrong one for a game whose controls are inputSystem.isTriggered(IA_PRIMARY, ...) with no entity argument at all, so the scenes worth driving from a bot were the ones no tool could reach.
The client already writes exactly that shape for a real key press: PrepareGlobalInputEventsSystem turns every bound Unity input action into a GlobalInputEvents entry, and each running current scene's WritePointerEventResultsSystem appends it as a PBPointerEventsResult on SCENE_ROOT_ENTITY with a null hit — what the SDK's inputSystem reads for its no-entity overloads. So this needs no new wire format and no new CRDT path, only a way into that buffer.
press_input_action is a new tool rather than a mode on click_entity: with no target there is nothing for the occlusion, maxDistance and hover diagnostics that make up most of click_entity to say, and every one of its failure messages names an entity. McpInputActionSystem publishes the edge, ordered after PrepareGlobalInputEventsSystem so it lands once that system has cleared the buffer and before the scene worlds read it later in the same frame.
Two details are not cosmetic:
A press owns its release. The hold lives in the ECS request, not in the awaiting tool call, so an agent that disconnects mid-hold cannot leave the scene believing a button is still down. This follows walk / McpMovementOverride, which already holds a duration that way, rather than click_entity, which composes its legs from the tool side — a click's two legs are one frame apart, a hold can be thirty seconds.
The release waits for the scene to leave the press's tick. The SDK keeps pointer results in a value set keyed by timestamp, which the writer fills with the scene tick number, and a down and an up sharing one key collapse into an ambiguous button state. The press tick is read on the frame after the press was published, because the scene may stamp it with either of the two ticks that frame spans; reading it late keeps the gate from ever being too permissive.
click_entity's button gains action_3 through action_6 in the same pass. A PointerEvents entry may name any InputAction as its button and the delivery path already carries whichever one it is given, so the previous {pointer, primary, secondary} restriction was arbitrary; all four are added rather than only action_5 because there is no per-action machinery and a gap at 3/4/6 would need a justification none of them has. Movement actions and IA_ANY stay out — they are not buttons a cursor clicks with, and press_input_action sends them with no target. Both tools now narrow one shared McpInputAction enum through the allowed parameter McpJsonSchema.Enum already had; the existing wire values are unchanged.
Three limits the tool states rather than hides: delivered means the edge reached a running current scene, not that the scene reacted; the scene writer drops the whole global buffer for a frame in which an entity-targeted result was written, so a call racing a click_entity can be swallowed; and one action is in flight at a time, so a second call preempts a held press and reports releaseMissed rather than silently leaving a stuck button.