docs: document input pipeline architecture and headless keyboard barrier#53
Closed
mark-e-deyoung wants to merge 1 commit into
Closed
Conversation
…ooting Documents the root cause of VNC/xdotool keyboard events not reaching Windows apps: Wine's explorer.exe /desktop intercepts all X11 keyboard input. The desktop supervisor continuously restarts it. Adds three documented solutions: 1. Hybrid control mode (WINEBOT_ALLOW_HEADLESS_HYBRID=1) 2. Disable desktop supervisor (WINEBOT_SUPERVISE_EXPLORER=0) 3. Mouse-only VNC interaction workaround Also documents the HTTP 423 'Agent control denied by policy' error and its fix. Discovered during headless SMAC game automation where /proc/PID/mem confirmed VNC key events delivered to Xvfb but terran.exe state never changed. Root cause confirmed by killing explorer.exe /desktop and observing that xdotool key injection then reached app windows directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Superseded by commit a7184c8 which includes this documentation plus a new /input/key endpoint and comprehensive tests. |
mark-e-deyoung
added a commit
that referenced
this pull request
Jun 21, 2026
feat: add /input/key endpoint and fix headless keyboard input pipeline Adds a POST /input/key API endpoint for keyboard injection into Windows applications. The endpoint defaults to AHK Send injection, which operates inside the Wine process space and bypasses the X11 explorer.exe/desktop keyboard interception layer entirely. Key changes: - New /input/key endpoint in api/routers/input.py with AHK and xdotool backends - xdotool-to-AHK key syntax translation (_xdotool_to_ahk_keys) - KeyModel Pydantic model for request validation - WINEBOT_INPUT_KEY_BACKEND and WINEBOT_TIMEOUT_INPUT_KEY_SECONDS config fields - Expose WINEBOT_ALLOW_HEADLESS_HYBRID and WINEBOT_SUPERVISE_EXPLORER in docker-compose - Document keyboard barrier and solutions in docs/troubleshooting.md - 20 unit tests for key translation, e2e test for full pipeline - All existing tests pass (162 passed, 0 new failures) - Ruff and Mypy pass clean Closes PR #52 and #53 (superseded by this commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> @
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Headless WineBot containers cannot inject keyboard input into Windows applications via VNC or xdotool. Key events arrive at Xvfb but never reach the target app. This was a long-running issue with the project.
Root Cause (Definitive)
The Wine desktop shell (
explorer.exe /desktop) intercepts ALL X11 keyboard input. The container runs a supervisor that continuously restarts this desktop, making it impossible to run apps in standalone X11 window mode.Input pipeline:
Proof:
struct.pack(">BBHI", 4, 1, 0, keysym)) are acknowledged by x11vnc/proc/PID/memreads of the target app show NO state changes after key injectionexplorer.exe /desktopmakes xdotool key injection work immediatelyDocumentation Added
Testing
No code changes — documentation only. Existing test suite passes.
🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com