From 3e8f051f3e59a2e3b7cfeaedcc268b23157f422f Mon Sep 17 00:00:00 2001 From: Esteban Ordano Date: Fri, 31 Jul 2026 16:40:06 +0200 Subject: [PATCH] feat(mcp): send input actions to a scene with no target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The MCP server can drive everything about the client except the one thing an SDK7 game is usually built on: input that no entity owns. click_entity, the only tool that produces an InputAction, delivers it through the reticle pipeline — a camera-origin ray has to hit a collider, that collider has to resolve to a scene entity, and the entity has to carry a PointerEvents component qualified for the hit distance. Miss any of those and it reports why it missed instead of delivering an input edge. That is the right contract for clicking a door and the wrong one for a game whose controls are inputSystem.isTriggered(IA_PRIMARY, ...) with no entity argument at all, so the scenes worth driving from a bot were the ones no tool could reach. The client already writes exactly that shape for a real key press: PrepareGlobalInputEventsSystem turns every bound Unity input action into a GlobalInputEvents entry, and each running current scene's WritePointerEventResultsSystem appends it as a PBPointerEventsResult on SCENE_ROOT_ENTITY with a null hit — what the SDK's inputSystem reads for its no-entity overloads. So this needs no new wire format and no new CRDT path, only a way into that buffer. press_input_action is a new tool rather than a mode on click_entity: with no target there is nothing for the occlusion, maxDistance and hover diagnostics that make up most of click_entity to say, and every one of its failure messages names an entity. McpInputActionSystem publishes the edge, ordered after PrepareGlobalInputEventsSystem so it lands once that system has cleared the buffer and before the scene worlds read it later in the same frame. Two details are not cosmetic: A press owns its release. The hold lives in the ECS request, not in the awaiting tool call, so an agent that disconnects mid-hold cannot leave the scene believing a button is still down. This follows walk / McpMovementOverride, which already holds a duration that way, rather than click_entity, which composes its legs from the tool side — a click's two legs are one frame apart, a hold can be thirty seconds. The release waits for the scene to leave the press's tick. The SDK keeps pointer results in a value set keyed by timestamp, which the writer fills with the scene tick number, and a down and an up sharing one key collapse into an ambiguous button state. The press tick is read on the frame after the press was published, because the scene may stamp it with either of the two ticks that frame spans; reading it late keeps the gate from ever being too permissive, and the frame that reads it is paid for by the gate it then fails. PrepareGlobalInputEventsSystem gains a summary recording that it refills the buffer per frame for the scene worlds to drain later in the same one, so a second writer must order after it — the invariant this feature now depends on, written where someone moving the Clear() will see it. click_entity's button gains action_3 through action_6 in the same pass. A PointerEvents entry may name any InputAction as its button and the delivery path already carries whichever one it is given, so the previous {pointer, primary, secondary} restriction was arbitrary; all four are added rather than only action_5 because there is no per-action machinery and a gap at 3/4/6 would need a justification none of them has. Movement actions and IA_ANY stay out — they are not buttons a cursor clicks with, and press_input_action sends them with no target. Both tools now narrow one shared McpInputAction enum through the allowed parameter McpJsonSchema.Enum already had; the existing wire values are unchanged. Three limits the tool states rather than hides: delivered means the edge reached a running current scene, not that the scene reacted; the scene writer drops the whole global buffer for a frame in which an entity-targeted result was written, so a call racing a click_entity can be swallowed; and one action is in flight at a time, so a second call preempts a held press and reports releaseMissed rather than silently leaving a stuck button. Co-Authored-By: Claude Opus 5 --- .claude/skills/mcp-scene-iteration/SKILL.md | 1 + .../Global/Dynamic/DynamicWorldContainer.cs | 1 + .../Systems/PrepareGlobalInputEventsSystem.cs | 6 + .../Components/McpInputActionIntent.cs | 79 ++++++ .../Components/McpInputActionIntent.cs.meta | 2 + .../DCL/McpServer/Core/McpInputAction.cs | 44 ++++ .../DCL/McpServer/Core/McpInputAction.cs.meta | 2 + .../McpServer/Systems/McpInputActionSystem.cs | 160 ++++++++++++ .../Systems/McpInputActionSystem.cs.meta | 2 + .../DCL/McpServer/Systems/McpServerPlugin.cs | 6 + .../Tests/McpInputActionSystemShould.cs | 234 ++++++++++++++++++ .../Tests/McpInputActionSystemShould.cs.meta | 2 + .../Tests/PressInputActionToolShould.cs | 120 +++++++++ .../Tests/PressInputActionToolShould.cs.meta | 2 + .../DCL/McpServer/Tools/ClickEntityTool.cs | 40 +-- .../McpServer/Tools/PressInputActionTool.cs | 153 ++++++++++++ .../Tools/PressInputActionTool.cs.meta | 2 + docs/mcp-automation.md | 17 +- 18 files changed, 848 insertions(+), 25 deletions(-) create mode 100644 Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs create mode 100644 Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs.meta create mode 100644 Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs create mode 100644 Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs.meta create mode 100644 Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs create mode 100644 Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs.meta create mode 100644 Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs create mode 100644 Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs.meta create mode 100644 Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs create mode 100644 Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs.meta create mode 100644 Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs create mode 100644 Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs.meta diff --git a/.claude/skills/mcp-scene-iteration/SKILL.md b/.claude/skills/mcp-scene-iteration/SKILL.md index 8632b6991f3..1e6179ed585 100644 --- a/.claude/skills/mcp-scene-iteration/SKILL.md +++ b/.claude/skills/mcp-scene-iteration/SKILL.md @@ -123,6 +123,7 @@ Paths are relative to this skill's directory; requires curl + python3; pass `-p ## Interaction testing - `click_entity` presses a pointer button on a scene entity (get ids from `list_scene_entities`). The target needs a `PointerEvents` component and a collider; the aim is validated by a real camera-origin raycast, so occluders return `hit:false` + `blockedBy*` (reposition and retry) and the entity's `maxDistance` (default 10 m) applies — get close first. `upRayMissed: true` means the target moved between press and release (e.g. a door starting to swing) and the release was delivered with the press-frame hit. For GLTF entities whose collider sits away from the pivot, pass an explicit `x/y/z` aim point. The player must be standing on the scene's parcel — off-parcel clicks fail with "no running current scene". +- `press_input_action` sends an input action with no target, for scenes that poll input globally (`inputSystem.isTriggered(InputAction.IA_PRIMARY, ...)` with no entity) — `click_entity` cannot reach those, because every one of its paths needs a collider to hit. `action` takes the wire spelling of the SDK action (`primary`, `secondary`, `pointer`, `action_3`…`action_6`, `jump`, `forward`, …); `press` holds it for `holdSec` (default 0.2) and releases it on a later scene tick, so a scene reading `isPressed` sees a real hold. Use `down`/`up` for a hold you drive yourself — a `down` with no matching `up` leaves the scene believing the button is held, and nothing sequences the two calls, so a pair sent back to back can land in one scene tick and cancel out. Prefer `press` whenever the hold length is known up front. Only one action is in flight at a time. - `walk` moves relative to the camera and requires an explicit direction: pass `directionY: 1` for forward (`directionX` strafes); omitting both errors with "directionX and directionY must not both be zero". - Collider checks beat pixels for physics (cross-examine): `look_at` straight at the target, `walk` forward, then compare `get_player_state` positions to prove passage or blockage. - Trigger areas fire `onTriggerEnter` immediately after `reload_scene` if the player is already standing inside one — reposition the player outside all triggers before testing enter/exit sequencing (and treat post-reload trigger logs as stale state, not gameplay). diff --git a/Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DynamicWorldContainer.cs b/Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DynamicWorldContainer.cs index 4b71e20537b..6efe7114679 100644 --- a/Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DynamicWorldContainer.cs +++ b/Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DynamicWorldContainer.cs @@ -863,6 +863,7 @@ await MapRendererContainer bootstrapContainer.DiagnosticsContainer, exposedGlobalDataContainer.ExposedCameraData, staticContainer.EntityCollidersGlobalCache, + exposedGlobalDataContainer.GlobalInputEvents, coroutineRunner, globalWorld, localSceneDevelopment)); diff --git a/Explorer/Assets/DCL/Interaction/Systems/PrepareGlobalInputEventsSystem.cs b/Explorer/Assets/DCL/Interaction/Systems/PrepareGlobalInputEventsSystem.cs index 0958d26414f..668f5bf9755 100644 --- a/Explorer/Assets/DCL/Interaction/Systems/PrepareGlobalInputEventsSystem.cs +++ b/Explorer/Assets/DCL/Interaction/Systems/PrepareGlobalInputEventsSystem.cs @@ -8,6 +8,12 @@ namespace DCL.Interaction.PlayerOriginated.Systems { + /// + /// Refills from scratch each frame, for the scene worlds to drain in + /// their PreRendering group later in the same frame. Anything else that adds an entry — an automation + /// driver such as McpInputActionSystem — must therefore run after this system and within the same frame; + /// moving the clear, or this system's group, silently drops those entries. + /// [UpdateInGroup(typeof(PresentationSystemGroup))] [LogCategory(ReportCategory.INPUT)] public partial class PrepareGlobalInputEventsSystem : BaseUnityLoopSystem diff --git a/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs b/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs new file mode 100644 index 00000000000..4a5f4fb8f35 --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs @@ -0,0 +1,79 @@ +using Cysharp.Threading.Tasks; +using DCL.ECSComponents; +using DCL.McpServer.Core; + +namespace DCL.McpServer.Components +{ + /// + /// Present on the player entity while an agent-requested global input action awaits delivery. + /// McpInputActionSystem publishes the requested edge(s) into the same per-frame GlobalInputEvents buffer + /// the real key bindings feed, so the current scene receives an entity-less PBPointerEventsResult on its + /// root entity — the form an SDK7 scene reads through inputSystem.isTriggered / isPressed without any + /// entity being involved. A request the simulation never picks up is removed by the tool-side timeout. + /// + public struct McpInputActionIntent : IMcpEcsRequest + { + public readonly InputAction Action; + + /// + /// Pins delivery to one scene, matched by the definition id get_scene_state reports: the edge fails + /// instead of landing in whatever scene is current if the player moved after the request was made. + /// Null accepts the current scene as is. + /// + public readonly string? SceneId; + + /// The single edge to publish, or PetDown when asks for a release too. + public readonly PointerEventType EventType; + + /// + /// Seconds the button stays down before the matching release is published; null for a lone edge. + /// The release is owned by the system rather than by the caller, so an agent that disconnects mid-hold + /// cannot leave the scene believing the button is still held. + /// + public readonly float? HoldSeconds; + + public UniTaskCompletionSource? Completion { get; set; } + + /// UnityEngine.Time.time at which the press was published; null until it has been, which is also what tells the system a release is still owed. + public float? PressTime; + + /// Scene tick the press is taken to have been stamped with; the release waits for the scene to pass it. + public uint? PressTick; + + public McpInputActionIntent(InputAction action, string? sceneId, PointerEventType eventType, float? holdSeconds = null) + { + Action = action; + SceneId = sceneId; + EventType = eventType; + HoldSeconds = holdSeconds; + Completion = null; + PressTime = null; + PressTick = null; + } + } + + /// Wire-facing outcome of a global input action, serialized by the press_input_action tool. + public struct McpInputActionResult + { + /// + /// The edge was published to a running current scene. That is as far as the client can attest: what + /// the scene's JavaScript does with it, or whether it polls that action at all, is not observable here. + /// + public bool Delivered; + + public string? FailureReason; + + /// Definition id of the scene the edge was published to, when one was resolved. + public string? SceneId; + + /// How long the button was actually held, for a press that completed both legs. + public float HeldSeconds; + + /// + /// The press was published but its release was not (the scene stopped being current or stopped + /// running mid-hold, or a newer request preempted this one): the scene still sees the button as + /// held, exactly as it would for a real key whose scene was torn down mid-press. + /// + public bool ReleaseMissed; + } +} diff --git a/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs.meta b/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs.meta new file mode 100644 index 00000000000..7e89f0914ac --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Components/McpInputActionIntent.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: c364ee9c87664633929ef563944bef3e \ No newline at end of file diff --git a/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs b/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs new file mode 100644 index 00000000000..017107c6241 --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs @@ -0,0 +1,44 @@ +using DCL.ECSComponents; + +namespace DCL.McpServer.Core +{ + /// + /// Wire-facing spelling of , so a tool's schema and the parsing of its argument + /// derive from one enum (see ): "pointer" ↔ IA_POINTER, "action_5" ↔ + /// IA_ACTION_5. It mirrors member for member — all of which the production + /// input map in GlobalInteractionPlugin binds, so a scene may read any of them; a tool that accepts only + /// a subset narrows its schema through the allowed parameter of + /// rather than declaring an enum of its own. + /// + /// Members are declared in the protobuf's own order, so each lands on its counterpart's value and + /// converts by cast instead of a mapping table that could + /// drift. Nothing in the language enforces that alignment — PressInputActionToolShould does, by + /// checking every member against the protobuf member its name spells. + /// + /// + public enum McpInputAction : byte + { + POINTER, + PRIMARY, + SECONDARY, + ANY, + FORWARD, + BACKWARD, + RIGHT, + LEFT, + JUMP, + WALK, + ACTION_3, + ACTION_4, + ACTION_5, + ACTION_6, + MODIFIER, + } + + public static class McpInputActionExtensions + { + /// Valid by construction: the members are declared with their protobuf values. + public static InputAction ToInputAction(this McpInputAction action) => + (InputAction)action; + } +} diff --git a/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs.meta b/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs.meta new file mode 100644 index 00000000000..509ecd9648f --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Core/McpInputAction.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: 195b0bb006d84d05907a3edf61e2cec8 \ No newline at end of file diff --git a/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs b/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs new file mode 100644 index 00000000000..9f7ca97da72 --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs @@ -0,0 +1,160 @@ +using Arch.Core; +using Arch.SystemGroups; +using Arch.SystemGroups.DefaultSystemGroups; +using DCL.Diagnostics; +using DCL.ECSComponents; +using DCL.Interaction.PlayerOriginated; +using DCL.Interaction.PlayerOriginated.Systems; +using DCL.McpServer.Components; +using DCL.McpServer.Core; +using ECS.Abstract; +using ECS.SceneLifeCycle; +using SceneRunner.Scene; +using System.Diagnostics.CodeAnalysis; + +namespace DCL.McpServer.Systems +{ + /// + /// + /// Delivers an agent-requested global input action while an is + /// present on the player entity. The edge is published into the very + /// buffer the key bindings feed through , so the current + /// scene's WritePointerEventResultsSystem turns it into an entity-less PBPointerEventsResult on the + /// scene root entity — the shape an SDK7 scene reads with inputSystem.isTriggered / isPressed when no + /// entity is involved. Nothing is raycast and no collider has to qualify: unlike + /// , this path has no target. + /// + /// + /// The buffer is refilled from scratch every frame, so the entry must be added after + /// cleared it and before the scene worlds run their + /// PreRendering group later in the same frame. Publishing is as far as this system can see: the scene + /// writer drops the whole buffer for a frame in which an entity-targeted result was written instead + /// (the same suppression real input is subject to), so a request delivered concurrently with a + /// click_entity on a qualifying entity can be swallowed. + /// + /// + /// A press owns its release: the tool asks for a hold duration and the system publishes the PetUp + /// itself, so an agent that disconnects mid-hold cannot leave the scene believing the button is still + /// down. The release is withheld until the scene has advanced past the tick the press was stamped + /// with — the SDK keeps pointer results in a set keyed by that tick, and two edges sharing one key + /// collapse into an ambiguous button state. + /// + /// + [UpdateInGroup(typeof(PresentationSystemGroup))] + [UpdateAfter(typeof(PrepareGlobalInputEventsSystem))] + [LogCategory(ReportCategory.MCP)] + public partial class McpInputActionSystem : BaseUnityLoopSystem + { + private readonly IScenesCache scenesCache; + private readonly GlobalInputEvents globalInputEvents; + private readonly Entity playerEntity; + + internal McpInputActionSystem(World world, + IScenesCache scenesCache, + GlobalInputEvents globalInputEvents, + Entity playerEntity) : base(world) + { + this.scenesCache = scenesCache; + this.globalInputEvents = globalInputEvents; + this.playerEntity = playerEntity; + } + + protected override void Update(float t) + { + ref McpInputActionIntent intent = ref World.TryGetRef(playerEntity, out bool exists); + + if (!exists) + return; + + if (!TryResolve(in intent, out ISceneFacade? scene)) + return; + + if (intent.PressTime is { } pressTime) + Release(ref intent, scene, pressTime); + else + Publish(ref intent, scene); + } + + /// Picks the scene the edge must be delivered to, or completes the request with the reason no delivery is possible. + private bool TryResolve(in McpInputActionIntent intent, [NotNullWhen(true)] out ISceneFacade? scene) + { + scene = scenesCache.CurrentScene.Value; + + if (scene == null || !scene.SceneStateProvider.IsCurrent || scene.SceneStateProvider.IsNotRunningState()) + { + scene = null; + Fail(in intent, "no running current scene to deliver the input action to"); + return false; + } + + if (intent.SceneId != null && scene.SceneData.SceneEntityDefinition.id != intent.SceneId) + { + string reason = $"the request is pinned to scene '{intent.SceneId}' but the current scene is '{scene.Info.Name}' (did the player move?)"; + scene = null; + Fail(in intent, reason); + return false; + } + + return true; + } + + /// Publishes the requested edge; a press then waits out its hold, a lone edge completes here. + private void Publish(ref McpInputActionIntent intent, ISceneFacade scene) + { + globalInputEvents.Add(new IGlobalInputEvents.Entry(intent.Action, intent.EventType)); + + if (intent.HoldSeconds.HasValue) + { + intent.PressTime = UnityEngine.Time.time; + return; + } + + McpEcsRequest.CompleteAndRemove(World, playerEntity, intent, Delivered(scene)); + } + + /// Publishes the PetUp of a held press once both the hold and the tick gate have elapsed. + private void Release(ref McpInputActionIntent intent, ISceneFacade scene, float pressTime) + { + // The scene stamps the press between the frame it was published on and this one, so the tick the + // release is gated against is read a frame late: never too early, so the two edges cannot share it. + // The capturing frame fails the gate below by construction, which is what buys that frame. + intent.PressTick ??= scene.SceneStateProvider.TickNumber; + + float heldSeconds = UnityEngine.Time.time - pressTime; + + if (heldSeconds < intent.HoldSeconds || scene.SceneStateProvider.TickNumber <= intent.PressTick) + return; + + globalInputEvents.Add(new IGlobalInputEvents.Entry(intent.Action, PointerEventType.PetUp)); + + McpInputActionResult result = Delivered(scene); + result.HeldSeconds = heldSeconds; + McpEcsRequest.CompleteAndRemove(World, playerEntity, intent, result); + } + + /// + /// Completes the request with the reason it could not be delivered. A press already published counts + /// as delivered whatever rejects the rest of it — only its release is lost, and the scene goes on + /// seeing the button held. The intent is copied out before the structural removal, so the caller's + /// ref must not be touched afterwards. + /// + private void Fail(in McpInputActionIntent intent, string reason) + { + bool pressed = intent.PressTime.HasValue; + + McpEcsRequest.CompleteAndRemove(World, playerEntity, intent, new McpInputActionResult + { + Delivered = pressed, + ReleaseMissed = pressed, + FailureReason = reason, + }); + } + + private static McpInputActionResult Delivered(ISceneFacade scene) => + new () + { + Delivered = true, + SceneId = scene.SceneData.SceneEntityDefinition.id, + }; + } +} diff --git a/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs.meta b/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs.meta new file mode 100644 index 00000000000..3858d045a09 --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Systems/McpInputActionSystem.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: 5690de7803c64a68ac8b4a35ec98daf3 \ No newline at end of file diff --git a/Explorer/Assets/DCL/McpServer/Systems/McpServerPlugin.cs b/Explorer/Assets/DCL/McpServer/Systems/McpServerPlugin.cs index b09b1e74ab9..cce9e0208f1 100644 --- a/Explorer/Assets/DCL/McpServer/Systems/McpServerPlugin.cs +++ b/Explorer/Assets/DCL/McpServer/Systems/McpServerPlugin.cs @@ -4,6 +4,7 @@ using DCL.CharacterCamera; using DCL.Chat.MessageBus; using DCL.Diagnostics; +using DCL.Interaction.PlayerOriginated; using DCL.Interaction.Utility; using DCL.McpServer.Core; using DCL.McpServer.Tools; @@ -45,6 +46,7 @@ public class McpServerPlugin : IDCLGlobalPluginWithoutSettings private readonly Arch.Core.World globalWorld; private readonly IGlobalWorldActions globalWorldActions; private readonly IEntityCollidersGlobalCache entityCollidersGlobalCache; + private readonly GlobalInputEvents globalInputEvents; private readonly IWorldInfoHub worldInfoHub; private readonly IScenesCache scenesCache; @@ -72,6 +74,7 @@ public McpServerPlugin( DiagnosticsContainer diagnosticsContainer, ExposedCameraData exposedCameraData, IEntityCollidersGlobalCache entityCollidersGlobalCache, + GlobalInputEvents globalInputEvents, ICoroutineRunner coroutineRunner, Arch.Core.World globalWorld, bool localSceneDevelopment) @@ -91,6 +94,7 @@ public McpServerPlugin( this.reloadSceneController = reloadSceneController; this.exposedCameraData = exposedCameraData; this.entityCollidersGlobalCache = entityCollidersGlobalCache; + this.globalInputEvents = globalInputEvents; this.coroutineRunner = coroutineRunner; this.globalWorld = globalWorld; this.localSceneDevelopment = localSceneDevelopment; @@ -113,6 +117,7 @@ public void InjectToWorld(ref ArchSystemsWorldBuilder builder, { McpInputOverrideSystem.InjectToWorld(ref builder, arguments.PlayerEntity); McpPointerEventSystem.InjectToWorld(ref builder, scenesCache, entityCollidersGlobalCache, arguments.PlayerEntity); + McpInputActionSystem.InjectToWorld(ref builder, scenesCache, globalInputEvents, arguments.PlayerEntity); screenshotTool = new ScreenshotTool(coroutineRunner, globalWorld, arguments.PlayerEntity); @@ -133,6 +138,7 @@ public void InjectToWorld(ref ArchSystemsWorldBuilder builder, .Add(new GetEntityDetailsTool(worldInfoHub)) .Add(new TriggerEmoteTool(globalWorldActions)) .Add(new ClickEntityTool(globalWorld, arguments.PlayerEntity)) + .Add(new PressInputActionTool(globalWorld, arguments.PlayerEntity)) .Build(); server = new McpHttpServer(toolsRegistry, port); diff --git a/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs b/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs new file mode 100644 index 00000000000..925eb35af87 --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs @@ -0,0 +1,234 @@ +using Arch.Core; +using Cysharp.Threading.Tasks; +using DCL.ECSComponents; +using DCL.Interaction.PlayerOriginated; +using DCL.Ipfs; +using DCL.McpServer.Components; +using DCL.McpServer.Systems; +using DCL.Utilities; +using ECS.SceneLifeCycle; +using ECS.TestSuite; +using NSubstitute; +using NUnit.Framework; +using SceneRunner.Scene; +using System.Collections.Generic; +using Utility.Multithreading; + +namespace DCL.McpServer.Tests +{ + public class McpInputActionSystemShould : UnitySystemTestBase + { + /// The three ways the current scene can be unusable, which the system rejects identically. + public enum SceneGuard + { + ABSENT, + NOT_CURRENT, + NOT_RUNNING, + } + + private Entity playerEntity; + private GlobalInputEvents globalInputEvents = null!; + private ISceneStateProvider sceneStateProvider = null!; + private IReadonlyReactiveProperty currentScene = null!; + private uint tick; + + [SetUp] + public void SetUp() + { + playerEntity = world.Create(); + globalInputEvents = new GlobalInputEvents(); + + tick = 100u; + sceneStateProvider = Substitute.For(); + sceneStateProvider.IsCurrent.Returns(true); + sceneStateProvider.State.Returns(new Atomic(SceneState.Running)); + sceneStateProvider.TickNumber.Returns(_ => tick); + + ISceneFacade sceneFacade = Substitute.For(); + sceneFacade.SceneStateProvider.Returns(sceneStateProvider); + sceneFacade.SceneData.SceneEntityDefinition.Returns(new SceneEntityDefinition("scene-here", new SceneMetadata())); + + currentScene = Substitute.For>(); + currentScene.Value.Returns(sceneFacade); + + IScenesCache scenesCache = Substitute.For(); + scenesCache.CurrentScene.Returns(currentScene); + + system = new McpInputActionSystem(world, scenesCache, globalInputEvents, playerEntity); + } + + [TestCase(null, PointerEventType.PetDown)] + [TestCase(null, PointerEventType.PetUp)] + [TestCase("scene-here", PointerEventType.PetDown)] + public void PublishALoneEdgeAndCompleteImmediately(string? sceneId, PointerEventType eventType) + { + // Arrange + UniTaskCompletionSource completion = AddIntent(eventType, sceneId: sceneId); + + // Act + system!.Update(0); + + // Assert + AssertPublished(0, eventType); + McpInputActionResult result = ResultOf(completion); + Assert.That(result.Delivered, Is.True); + Assert.That(result.SceneId, Is.EqualTo("scene-here")); + Assert.That(result.ReleaseMissed, Is.False); + Assert.That(world.Has(playerEntity), Is.False); + } + + [Test] + public void WithholdTheReleaseUntilBothTheHoldAndTheSceneTickHaveMovedOn() + { + // A press and its release sharing one scene tick collapse into an ambiguous button state in the SDK, + // which keys pointer results by that tick — so an elapsed hold alone must not let the release out. + + // Arrange + UniTaskCompletionSource completion = AddIntent(PointerEventType.PetDown, holdSeconds: 60f); + + // Act + system!.Update(0); // publish the press + system.Update(0); // stamp the tick the release is gated against + tick++; + system.Update(0); // the tick gate is open, the hold is not + + // Assert + Assert.That(globalInputEvents.Entries.Count, Is.EqualTo(1)); + Assert.That(completion.Task.Status, Is.EqualTo(UniTaskStatus.Pending)); + Assert.That(world.Has(playerEntity), Is.True); + + // Act: back the press time off so the hold is over, and put the scene back on the press tick. + world.Get(playerEntity).PressTime = UnityEngine.Time.time - 61f; + tick--; + system.Update(0); + + // Assert + Assert.That(globalInputEvents.Entries.Count, Is.EqualTo(1), "the release must not share the press tick"); + Assert.That(completion.Task.Status, Is.EqualTo(UniTaskStatus.Pending)); + + // Act + tick++; + system.Update(0); + + // Assert + AssertPublished(1, PointerEventType.PetUp); + McpInputActionResult result = ResultOf(completion); + Assert.That(result.Delivered, Is.True); + Assert.That(result.ReleaseMissed, Is.False); + Assert.That(result.HeldSeconds, Is.GreaterThan(60f)); + Assert.That(world.Has(playerEntity), Is.False); + } + + [Test] + public void ReportAPressWhoseReleaseTheSceneGuardRejected() + { + // Arrange + UniTaskCompletionSource completion = AddIntent(PointerEventType.PetDown, holdSeconds: 0f); + + system!.Update(0); + system.Update(0); + + // Act + sceneStateProvider.IsCurrent.Returns(false); + tick++; + system.Update(0); + + // Assert + Assert.That(globalInputEvents.Entries.Count, Is.EqualTo(1), "only the press reached the scene"); + McpInputActionResult result = ResultOf(completion); + Assert.That(result.Delivered, Is.True); + Assert.That(result.ReleaseMissed, Is.True); + Assert.That(result.FailureReason, Does.Contain("no running current scene")); + } + + [TestCase(SceneGuard.ABSENT)] + [TestCase(SceneGuard.NOT_CURRENT)] + [TestCase(SceneGuard.NOT_RUNNING)] + public void FailWhenThereIsNoRunningCurrentScene(SceneGuard guard) + { + // Arrange + switch (guard) + { + case SceneGuard.ABSENT: + currentScene.Value.Returns((ISceneFacade?)null); + break; + case SceneGuard.NOT_CURRENT: + sceneStateProvider.IsCurrent.Returns(false); + break; + case SceneGuard.NOT_RUNNING: + sceneStateProvider.State.Returns(new Atomic(SceneState.JavaScriptError)); + break; + } + + UniTaskCompletionSource completion = AddIntent(PointerEventType.PetDown); + + // Act + system!.Update(0); + + // Assert + Assert.That(globalInputEvents.Entries, Is.Empty); + McpInputActionResult result = ResultOf(completion); + Assert.That(result.Delivered, Is.False); + Assert.That(result.ReleaseMissed, Is.False); + Assert.That(result.FailureReason, Does.Contain("no running current scene")); + } + + [Test] + public void FailWhenPinnedSceneIsNotCurrent() + { + // Arrange + UniTaskCompletionSource completion = AddIntent(PointerEventType.PetDown, sceneId: "scene-elsewhere"); + + // Act + system!.Update(0); + + // Assert + Assert.That(globalInputEvents.Entries, Is.Empty); + Assert.That(ResultOf(completion).FailureReason, Does.Contain("pinned")); + } + + [Test] + public void LeaveTheBufferAloneWhenNoRequestIsPending() + { + // The buffer belongs to the real input pipeline; an idle update of this system must not touch it. + + // Arrange + globalInputEvents.Add(new IGlobalInputEvents.Entry(InputAction.IaJump, PointerEventType.PetDown)); + + // Act + system!.Update(0); + + // Assert + Assert.That(globalInputEvents.Entries.Count, Is.EqualTo(1)); + } + + private UniTaskCompletionSource AddIntent( + PointerEventType eventType, + float? holdSeconds = null, + string? sceneId = null) + { + var completion = new UniTaskCompletionSource(); + + world.Add(playerEntity, new McpInputActionIntent(InputAction.IaAction5, sceneId, eventType, holdSeconds) + { + Completion = completion, + }); + + return completion; + } + + private void AssertPublished(int index, PointerEventType eventType) + { + IReadOnlyList entries = globalInputEvents.Entries; + Assert.That(entries.Count, Is.GreaterThan(index)); + Assert.That(entries[index].InputAction, Is.EqualTo(InputAction.IaAction5)); + Assert.That(entries[index].PointerEventType, Is.EqualTo(eventType)); + } + + private static McpInputActionResult ResultOf(UniTaskCompletionSource completion) + { + Assert.That(completion.Task.Status, Is.EqualTo(UniTaskStatus.Succeeded)); + return completion.Task.GetAwaiter().GetResult(); + } + } +} diff --git a/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs.meta b/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs.meta new file mode 100644 index 00000000000..a9f18f7317b --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tests/McpInputActionSystemShould.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: dff0c49d035f427abfe51431b5c21f61 \ No newline at end of file diff --git a/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs b/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs new file mode 100644 index 00000000000..c6045a4cb2d --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs @@ -0,0 +1,120 @@ +using Arch.Core; +using Cysharp.Threading.Tasks; +using DCL.ECSComponents; +using DCL.McpServer.Components; +using DCL.McpServer.Core; +using DCL.McpServer.Tools; +using Newtonsoft.Json.Linq; +using NUnit.Framework; +using System; +using System.Collections.Generic; +using System.Threading; + +namespace DCL.McpServer.Tests +{ + public class PressInputActionToolShould + { + private World world = null!; + private Entity playerEntity; + private PressInputActionTool tool = null!; + private CancellationTokenSource cts = null!; + + [SetUp] + public void SetUp() + { + world = World.Create(); + playerEntity = world.Create(); + tool = new PressInputActionTool(world, playerEntity); + cts = new CancellationTokenSource(); + } + + [TearDown] + public void TearDown() + { + // Accepted calls stay awaiting a system this suite does not run; cancelling unwinds them. + cts.Cancel(); + cts.Dispose(); + world.Dispose(); + } + + [Test] + public void OfferEveryInputActionOnTheWire() + { + // A scene may read any action, and the production key map binds all of them, so the tool narrows + // nothing — unlike click_entity, which passes a subset of the same enum. + + // Arrange + var wireNames = new List(); + + foreach (JToken value in tool.InputSchema["properties"]!["action"]!["enum"]!) + wireNames.Add(value.Value()!); + + // Assert + Assert.That(wireNames, Is.EqualTo(McpWireEnum.WIRE_NAMES)); + } + + [Test] + public void SpellEveryProtobufInputActionMemberOnTheWire() + { + // McpInputAction converts to InputAction by cast, so it has to stay a faithful renaming: same member + // count, and each member carrying the value of the protobuf member its own name spells. A member + // added or renumbered upstream fails here instead of quietly sending a scene the wrong action. + + // Arrange + var wireMembers = (McpInputAction[])Enum.GetValues(typeof(McpInputAction)); + + // Assert + Assert.That(wireMembers.Length, Is.EqualTo(Enum.GetValues(typeof(InputAction)).Length)); + + foreach (McpInputAction wireMember in wireMembers) + { + // "ACTION_5" ↔ "IaAction5": the protobuf spelling of the same name. + string protobufName = wireMember.ToInputAction().ToString().ToUpperInvariant(); + Assert.That(protobufName, Is.EqualTo($"IA{wireMember.ToString().Replace("_", string.Empty)}"), + $"{wireMember} maps to {wireMember.ToInputAction()}, which is a different action"); + } + } + + [TestCase("{}", "action is required")] + [TestCase("{'action':'ia_action_5'}", "action is required")] + [TestCase("{'action':'action_5','eventType':'tap'}", "eventType must be one of")] + public void RefuseArgumentsItCannotActOn(string arguments, string expectedFragment) + { + // Act: a refusal is answered before the tool's first await, so the task is already done. + UniTask call = tool.ExecuteAsync(JObject.Parse(arguments), cts.Token); + + // Assert + Assert.That(call.Status, Is.EqualTo(UniTaskStatus.Succeeded), "the tool was expected to refuse"); + McpToolResult result = call.GetAwaiter().GetResult(); + Assert.That(result.Payload["isError"]!.Value(), Is.True); + Assert.That(result.Payload["content"]![0]!["text"]!.Value(), Does.Contain(expectedFragment)); + Assert.That(world.Has(playerEntity), Is.False); + } + + [TestCase("{'action':'action_5','eventType':'down'}", InputAction.IaAction5, PointerEventType.PetDown, null)] + [TestCase("{'action':'primary','eventType':'up'}", InputAction.IaPrimary, PointerEventType.PetUp, null)] + [TestCase("{'action':'action_5','holdSec':900}", InputAction.IaAction5, PointerEventType.PetDown, 30f)] + public void InstallTheRequestedLegWithItsHoldClamped(string arguments, InputAction action, PointerEventType eventType, float? holdSeconds) + { + // Act: an accepted call is left awaiting the system this suite does not run, so the request it + // installed on the player entity is what the test reads. + tool.ExecuteAsync(JObject.Parse(arguments), cts.Token).Forget(); + + // Assert + McpInputActionIntent intent = world.Get(playerEntity); + Assert.That(intent.Action, Is.EqualTo(action)); + Assert.That(intent.EventType, Is.EqualTo(eventType)); + Assert.That(intent.HoldSeconds, Is.EqualTo(holdSeconds)); + } + + [Test] + public void PinTheRequestToASceneWhenAsked() + { + // Act + tool.ExecuteAsync(new JObject { ["action"] = "action_5", ["sceneId"] = "scene-here" }, cts.Token).Forget(); + + // Assert + Assert.That(world.Get(playerEntity).SceneId, Is.EqualTo("scene-here")); + } + } +} diff --git a/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs.meta b/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs.meta new file mode 100644 index 00000000000..3ed7941c7ec --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tests/PressInputActionToolShould.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: 8a5313d754354c958dacca15bd9556cc \ No newline at end of file diff --git a/Explorer/Assets/DCL/McpServer/Tools/ClickEntityTool.cs b/Explorer/Assets/DCL/McpServer/Tools/ClickEntityTool.cs index f09ed29b1bb..bb79eff1575 100644 --- a/Explorer/Assets/DCL/McpServer/Tools/ClickEntityTool.cs +++ b/Explorer/Assets/DCL/McpServer/Tools/ClickEntityTool.cs @@ -20,14 +20,6 @@ namespace DCL.McpServer.Tools /// public class ClickEntityTool : McpTool { - /// Wire-facing subset of : only the three pointer buttons make sense for a click. - private enum PointerButton : byte - { - POINTER, - PRIMARY, - SECONDARY, - } - /// Wire-facing gesture kinds: a full click, or a single press/release leg. private enum ClickKind : byte { @@ -44,6 +36,22 @@ private enum ClickKind : byte private const float MIN_TIMEOUT_SEC = 0.5f; private const float MAX_TIMEOUT_SEC = 15f; + /// + /// The actions a click can carry: the three pointer buttons plus the four action buttons, because a + /// PointerEvents entry may name any of them as its button. The movement actions and IA_ANY are left + /// out — they are not buttons a cursor clicks with; press_input_action sends those with no target. + /// + private static readonly McpInputAction[] ALLOWED_BUTTONS = + { + McpInputAction.POINTER, + McpInputAction.PRIMARY, + McpInputAction.SECONDARY, + McpInputAction.ACTION_3, + McpInputAction.ACTION_4, + McpInputAction.ACTION_5, + McpInputAction.ACTION_6, + }; + private readonly World world; private readonly Entity playerEntity; @@ -53,7 +61,8 @@ private enum ClickKind : byte "Press and release a pointer button on a scene entity so its PointerEvents fire exactly like a real click. " + "The click runs through the real reticle pipeline: occluders and the entity's maxDistance apply, and a miss " + "returns hit:false with the blocking entity. Ids come from list_scene_entities. For entities whose collider " - + "sits away from their pivot (e.g. GLTF meshes), pass an explicit x/y/z world point to aim at."; + + "sits away from their pivot (e.g. GLTF meshes), pass an explicit x/y/z world point to aim at. Every path " + + "here needs a collider to land on: to drive a scene that polls input globally, use press_input_action."; protected override McpJsonSchema DescribeInput(McpJsonSchema schema) => schema.Integer("entityId", "Target entity id in the current scene world (from list_scene_entities). Omit only when x/y/z are given, then the ray decides the target.") @@ -61,7 +70,7 @@ protected override McpJsonSchema DescribeInput(McpJsonSchema schema) => .Number("y") .Number("z") .String("sceneId", "Pin the click to this scene (id from get_scene_state): it fails instead of landing in another scene if the player moved.") - .Enum("button", "Which input action to press. Default pointer (left click / IA_POINTER).") + .Enum("button", "Which input action to press on the entity. Default pointer (left click / IA_POINTER).", ALLOWED_BUTTONS) .Enum("eventType", "click = down, then up on the next scene tick. Default click.") .Number("timeoutSec", "Seconds to wait for delivery. Default 3, max 15."); @@ -84,15 +93,10 @@ public override async UniTask ExecuteAsync(JObject arguments, Can if (!hasEntityId && !hasAimPoint) return McpToolResult.Error("Provide entityId, or a full x/y/z world aim point, or both."); - if (!arguments.TryGetEnum("button", PointerButton.POINTER, out PointerButton pointerButton)) - return McpToolResult.Error("button must be one of: pointer, primary, secondary."); + if (!arguments.TryGetEnum("button", McpInputAction.POINTER, out McpInputAction wireButton, ALLOWED_BUTTONS)) + return McpToolResult.Error($"button must be one of: {string.Join(", ", McpWireEnum.WireNamesOf(ALLOWED_BUTTONS))}."); - InputAction button = pointerButton switch - { - PointerButton.PRIMARY => InputAction.IaPrimary, - PointerButton.SECONDARY => InputAction.IaSecondary, - _ => InputAction.IaPointer, - }; + InputAction button = wireButton.ToInputAction(); if (!arguments.TryGetEnum("eventType", ClickKind.CLICK, out ClickKind kind)) return McpToolResult.Error("eventType must be one of: click, down, up."); diff --git a/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs b/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs new file mode 100644 index 00000000000..f216356d5aa --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs @@ -0,0 +1,153 @@ +using Arch.Core; +using Cysharp.Threading.Tasks; +using DCL.ECSComponents; +using DCL.McpServer.Components; +using DCL.McpServer.Core; +using DCL.McpServer.Utils; +using JetBrains.Annotations; +using Newtonsoft.Json.Linq; +using System; +using System.Threading; +using UnityEngine; + +namespace DCL.McpServer.Tools +{ + /// + /// Sends an edge to the current scene with no entity and no aim involved, by + /// way of an that McpInputActionSystem publishes: it lands on the + /// scene root entity the way a key press does, which is what a scene polling inputSystem.isTriggered / + /// isPressed every frame reads. This is the counterpart of click_entity, whose every path requires a + /// collider the reticle can qualify — so a game that never registers a PointerEvents component is + /// unreachable through it. A press owns its release: the system emits the PetUp even if this call is + /// cancelled, so a dropped connection cannot leave the scene with a button stuck down. + /// + public class PressInputActionTool : McpTool + { + /// Wire-facing gesture kinds: a press that releases itself, or a single down/up leg. + private enum PressKind : byte + { + /// Down, then up once holdSec has elapsed and the scene has advanced a tick. + PRESS, + + /// Down-only leg; parsed from the wire and exposed via the schema through reflection over this enum. + [UsedImplicitly] + DOWN, + UP, + } + + private const float DEFAULT_HOLD_SEC = 0.2f; + private const float MIN_HOLD_SEC = 0.1f; + private const float MAX_HOLD_SEC = 30f; + + private const float DEFAULT_TIMEOUT_SEC = 3f; + private const float MIN_TIMEOUT_SEC = 0.5f; + private const float MAX_TIMEOUT_SEC = 15f; + + private static readonly string ACTION_NAMES = string.Join(", ", McpWireEnum.WIRE_NAMES); + private static readonly string KIND_NAMES = string.Join(", ", McpWireEnum.WIRE_NAMES); + + private readonly World world; + private readonly Entity playerEntity; + + public override string Name => "press_input_action"; + + public override string Description => + "Send an SDK input action to the current scene without pointing at anything, so scenes that poll input " + + "globally (inputSystem.isTriggered / isPressed, with no entity argument) react. Use this for game " + + "controls; use click_entity when the scene registered the action on a specific entity. press holds " + + "the button for holdSec and releases it on a later scene tick, so a scene reading isPressed observes " + + "a real hold. A lone down leaves the scene believing the button is still held until a matching up."; + + protected override McpJsonSchema DescribeInput(McpJsonSchema schema) => + schema.Enum("action", "Which input action to send, e.g. primary (IA_PRIMARY) or action_5 (IA_ACTION_5).", isRequired: true) + .Enum("eventType", "press = down, then up after holdSec. Default press.") + .Number("holdSec", "How long a press keeps the button down. Default 0.2, min 0.1, max 30.") + .String("sceneId", "Pin the input to this scene (id from get_scene_state): it fails instead of landing in another scene if the player moved.") + .Number("timeoutSec", "Seconds to wait for the edge to be published. Default 3, max 15."); + + public override McpToolAnnotations Annotations => McpToolAnnotations.Mutating(destructive: false, idempotent: false); + + public PressInputActionTool(World world, Entity playerEntity) + { + this.world = world; + this.playerEntity = playerEntity; + } + + public override async UniTask ExecuteAsync(JObject arguments, CancellationToken ct) + { + if (!arguments.TryGetEnum("action", out McpInputAction action)) + return McpToolResult.Error($"action is required and must be one of: {ACTION_NAMES}."); + + if (!arguments.TryGetEnum("eventType", PressKind.PRESS, out PressKind kind)) + return McpToolResult.Error($"eventType must be one of: {KIND_NAMES}."); + + float holdSec = Mathf.Clamp(arguments.GetFloat("holdSec", DEFAULT_HOLD_SEC), MIN_HOLD_SEC, MAX_HOLD_SEC); + float timeoutSec = Mathf.Clamp(arguments.GetFloat("timeoutSec", DEFAULT_TIMEOUT_SEC), MIN_TIMEOUT_SEC, MAX_TIMEOUT_SEC); + string? sceneId = arguments["sceneId"]?.Type == JTokenType.String ? arguments["sceneId"]!.Value() : null; + + var intent = new McpInputActionIntent( + action.ToInputAction(), + sceneId, + kind == PressKind.UP ? PointerEventType.PetUp : PointerEventType.PetDown, + kind == PressKind.PRESS ? holdSec : null); + + // A press spends most of its budget holding the button down, so the timeout that guards a stuck + // simulation has to clear the hold itself before it can mean anything. + float budgetSec = kind == PressKind.PRESS ? holdSec + timeoutSec : timeoutSec; + + McpInputActionResult result; + + try + { + result = await McpEcsRequest.SendAsync(world, playerEntity, intent, PreemptedResult(world, playerEntity)) + .AttachExternalCancellation(ct) + .Timeout(TimeSpan.FromSeconds(budgetSec)); + } + catch (TimeoutException) + { + await McpEcsRequest.AbandonAsync(world, playerEntity); + return McpToolResult.Error($"press_input_action did not complete within {budgetSec}s (is the simulation paused?)."); + } + + var json = new JObject + { + ["delivered"] = result.Delivered, + ["action"] = McpWireEnum.ToWire(action), + ["eventType"] = McpWireEnum.ToWire(kind), + }; + + if (result.SceneId != null) + json["sceneId"] = result.SceneId; + + if (result.FailureReason != null) + json["reason"] = result.FailureReason; + + if (kind == PressKind.PRESS && result.Delivered && !result.ReleaseMissed) + json["heldSec"] = Math.Round(result.HeldSeconds, 2); + + if (result.ReleaseMissed) + json["releaseMissed"] = true; + + return McpToolResult.Json(json); + } + + /// + /// What the call being preempted reports. Only one input action is in flight at a time, so this call + /// drops whatever the previous one had going; when that was a press already down, the scene keeps + /// seeing the button held until the same action is sent again with eventType up, which is what + /// releaseMissed tells the preempted caller. It has to be read before SendAsync overwrites it. + /// + private static McpInputActionResult PreemptedResult(World world, Entity playerEntity) + { + bool heldPressDropped = world.TryGet(playerEntity, out McpInputActionIntent pending) + && pending.PressTime.HasValue; + + return new McpInputActionResult + { + Delivered = heldPressDropped, + ReleaseMissed = heldPressDropped, + FailureReason = "preempted by a newer press_input_action call", + }; + } + } +} diff --git a/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs.meta b/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs.meta new file mode 100644 index 00000000000..f71ce28cccb --- /dev/null +++ b/Explorer/Assets/DCL/McpServer/Tools/PressInputActionTool.cs.meta @@ -0,0 +1,2 @@ +fileFormatVersion: 2 +guid: 5db2dc3424014417abdb91e5b48511a7 \ No newline at end of file diff --git a/docs/mcp-automation.md b/docs/mcp-automation.md index 829379270bb..2a463efa8df 100644 --- a/docs/mcp-automation.md +++ b/docs/mcp-automation.md @@ -1,6 +1,6 @@ # MCP Automation Server -The Explorer can host an embedded [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server so coding agents (e.g. Claude Code) can **see** the running client (screenshots, player/scene state, scene console logs) and **control** it (teleport, move, walk, look, chat commands, scene reload) — closing the edit → reload → verify loop for SDK7 scene development without a human in the middle. +The Explorer can host an embedded [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server so coding agents (e.g. Claude Code) can **see** the running client (screenshots, player/scene state, scene console logs) and **control** it (teleport, move, walk, look, click, press input actions, chat commands, scene reload) — closing the edit → reload → verify loop for SDK7 scene development without a human in the middle. The server is compiled into all builds but stays dormant unless explicitly enabled at launch. @@ -80,7 +80,8 @@ The tables below are a human-readable overview. The authoritative argument contr | `send_chat` | `message` | Sends to Nearby chat; `/commands` run through the chat command pipeline | | `reload_scene` | `timeoutSec?` | Reloads the current scene (motion + skybox frozen during reload) | | `trigger_emote` | `urn` or `stop: true`, `loop?` | Plays or stops an avatar emote | -| `click_entity` | `entityId` and/or `x`,`y`,`z` aim point, `button?`, `eventType?`, `timeoutSec?` | Presses a pointer button on a scene entity exactly like a real click: a camera-origin raycast validates the aim (occluders and the entity's `maxDistance` apply), then the entity's pointer-event intent is filled so the scene receives an identical `PBPointerEventsResult`. `click` sends down + up on consecutive scene ticks. Returns `hit`, hover text, hit point/distance, or the blocking entity | +| `click_entity` | `entityId` and/or `x`,`y`,`z` aim point, `button?`, `eventType?`, `timeoutSec?` | Presses a pointer button on a scene entity exactly like a real click: a camera-origin raycast validates the aim (occluders and the entity's `maxDistance` apply), then the entity's pointer-event intent is filled so the scene receives an identical `PBPointerEventsResult`. `click` sends down + up on consecutive scene ticks. `button` accepts the three pointer buttons plus `action_3`…`action_6`. Returns `hit`, hover text, hit point/distance, or the blocking entity | +| `press_input_action` | `action`, `eventType?`, `holdSec?`, `sceneId?`, `timeoutSec?` | Sends an `InputAction` edge with **no target** — the shape a scene reads with `inputSystem.isTriggered` / `isPressed` and no entity argument, which no other tool can reach. `press` (default) holds the button for `holdSec` and releases it on a later scene tick; `down`/`up` send a single leg. Returns `delivered`, the scene it landed in and the measured `heldSec` | ## Structured output @@ -114,15 +115,17 @@ A user-invokable Claude Code skill wrapping this loop lives at `.claude/skills/m - **Verbose logs** — enabling the server registers a scene-console log handler, which turns on unconditional verbose logging for the session (same behavior as `--scene-console`). - **Scene entity dumps** — `list_scene_entities`/`get_entity_details` read the scene world without acquiring its sync lock (same as the existing `WorldInfoTool` debug tooling); treat results as a diagnostic snapshot. - **`click_entity` returns `hit:false` with `blockedBy*`** — another collider sits on the camera→target line; `move_to`/`look_at` to a clear vantage and retry. If the reason is "out of range", close within the entity's `maxDistance` (default 10 m) first. Entities whose collider sits away from the pivot (GLTF meshes) may need an explicit `x/y/z` aim point. +- **A scene ignores `click_entity` but is clearly input-driven** — it reads input globally instead of registering `PointerEvents` on an entity. Use `press_input_action`, which needs no target at all. +- **`press_input_action` reports `releaseMissed`** — the press reached the scene but its release did not, so the scene still believes the button is held: send the same `action` again with `eventType: "up"`. It happens when the scene stopped being current or stopped running mid-hold, or when a second call preempted the held press (only one input action is in flight at a time). ## Implementation map - `Explorer/Assets/DCL/McpServer/` — feature root, its own `DCL.McpServer` assembly. Two folders are folded into other assemblies via `.asmref` so they can reach code that assembly doesn't reference: - - `Core/` — protocol, transport and tool contract: `McpHttpServer` (`HttpListener` server + Origin validation), `McpJsonRpcDispatcher` (JSON-RPC 2.0 routing; `PROTOCOL_VERSION` `2025-06-18`), `McpTool` (abstract tool base), `McpToolsRegistry`, `McpToolResult`, `McpToolAnnotations` (behaviour hints), `McpJsonSchema` (typed schema builder). - - `Tools/` — one class per tool (16). - - `Components/` — ECS components for the input-driving tools: `McpMovementOverride`, `McpPointerEventIntent`. - - `Systems/` — **folded into `DCL.Plugins`** via `.asmref`: `McpServerPlugin` (builds the registry and hosts the server in `InjectToWorld`), `McpInputOverrideSystem` (held movement), `McpPointerEventSystem` (synthetic pointer press/release delivery; `ClickEntityTool` composes a click from two intents). + - `Core/` — protocol, transport and tool contract: `McpHttpServer` (`HttpListener` server + Origin validation), `McpJsonRpcDispatcher` (JSON-RPC 2.0 routing; `PROTOCOL_VERSION` `2025-06-18`), `McpTool` (abstract tool base), `McpToolsRegistry`, `McpToolResult`, `McpToolAnnotations` (behaviour hints), `McpJsonSchema` (typed schema builder), `McpInputAction` (wire spelling of the protobuf `InputAction`, shared by the two input tools). + - `Tools/` — one class per tool (17). + - `Components/` — ECS components for the input-driving tools: `McpMovementOverride`, `McpPointerEventIntent`, `McpInputActionIntent`. + - `Systems/` — **folded into `DCL.Plugins`** via `.asmref`: `McpServerPlugin` (builds the registry and hosts the server in `InjectToWorld`), `McpInputOverrideSystem` (held movement), `McpPointerEventSystem` (synthetic pointer press/release delivery; `ClickEntityTool` composes a click from two intents), `McpInputActionSystem` (targetless input actions, published into the same `GlobalInputEvents` buffer the key bindings feed so the current scene's `WritePointerEventResultsSystem` writes them to its root entity). - `Utils/` — `SceneLogBuffer`, `JObjectExtensions`. - - `Tests/` — EditMode tests **folded into `DCL.EditMode.Tests`** via `.asmref`: dispatcher / registry / result routing and the pointer-click system. + - `Tests/` — EditMode tests **folded into `DCL.EditMode.Tests`** via `.asmref`: dispatcher / registry / result routing, the pointer-click system and the input-action system. - Gating: `FeatureId.MCP_SERVER` in `FeaturesRegistry` (resolved as `appArgs.HasFlag(MCP) || appArgs.HasFlag(MCP_PORT)`); `DynamicWorldContainer.CreateAsync` reads `FeaturesRegistry.Instance.IsEnabled(FeatureId.MCP_SERVER)` and adds `McpServerPlugin`. - Flags: `AppArgsFlags.MCP` / `AppArgsFlags.MCP_PORT`; log category: `ReportCategory.MCP`.