Add keyboard and joystick REST API

[chrisgleissner]

Summary

This PR resolves #670 by adding U64-class keyboard and joystick injection on CIA1 level through GET /v1/machine:input and POST /v1/machine:input. Additionally, it supports navigating the application's menu via keyboard.

The endpoint accepts validated JSON input batches, applies keyboard and joystick state through separate REST-owned lanes, and returns a REST-injected state snapshot without reporting physical USB input.

For a detailed specification of this change as well as a documentation of the two new REST endpoints please see #670 (comment)

Main Use Cases

• Low-level remote input: Provides C64 keyboard and joystick injection at the CIA-visible input level, enabling remote joystick control and avoiding the limitations of Kernal keyboard-buffer input used by only some software.

• Remote multiplayer games, demos, and support: Remotely control a U64 while streaming its audio/video output, e.g. via C64 Stream:

  • Play together with friends on the same Ultimate 64 and watch video / hear audio via a private Youtube / Twitch channel.
  • Remotely assist another Ultimate 64 user by controlling their keyboard / joystick once granted permission.
  • For Internet-facing use, expose only an input-specific gateway, not the U64 REST API directly, and protect it for example with a Cloudflare Tunnel plus Cloudflare Access policies or service credentials.
  • LAN play is best for latency-sensitive games (latency of joystick propagation ca. 15ms); Internet play is better suited to slower-paced interaction.

• Automated testing and diagnostics: Enables deterministic keyboard and joystick input for U64 E2E tests, regression runners, and reproducible bug reports:

  • Input sequences can be replayed as REST-level test cases and test expectations can be asserted against the audio/video streamed from the device, e.g. combined with OCR-based text extraction where useful.
  • Capturing input is outside this PR’s firmware scope, but the included relay scripts can already log REST calls in verbose mode.
  • Capturing output is easily possible by enabling audio/video streaming.
  • Keyboard injection supports interacting both with C64 as well as the Ultimate 64 menu system. This is useful for testing Ultimate 64 firmware features in all possible UI-driven interaction modes:
    1. Telnet: already works without this PR.
    2. UI Overlay mode: Possible with this PR but often not needed since internally similar to Telnet.
    3. UI Freeze mode: Only possible with changes of this PR. Useful since it internally differs from Telnet / UI Overlay mode and thus allows for improving test coverage and fixing bugs in other parts of the firmware.

• Controller adapters and external tooling:

  • Provides a hardware-agnostic REST input API for modern controllers, scripts, browser frontends, accessibility tools, CI/HIL (Hardware in the Loop) automation, and development utilities.
  • This lets new input devices (e.g. XBox 360 controllers as demonstrated by the script included in this PR) be supported on the client side by any user without adding firmware-specific USB support for each device type.

Demo

Demo 1: Anykey / U64 Menu / Mousetest V2 / Basic

The following video demonstrates how an Ultimate 64 Elite I is controlled from a Kubuntu 24.04 machine via REST calls:

https://youtu.be/QVBFeKT-8SE

Overview

1. The video starts with pressing every key of the C64, including the RESTORE key.
2. Around 00:30, you see how a large number of keys are pressed and held simultaneously.
3. The video then shows how a joystick on each port is being moved in all directions after which all of its 3 buttons (called fire, fire2, and fire3 as per the REST API) are pressed in sequence.
4. Correct handling of key presses in the Ultimate 64 menu system.
5. Another mouse test, this time with Mousetest V2.
6. Handling of key presses in C64 Basic.

Setup

• As joystick, I am using an XBox 360 USB controller, connected to the Kubuntu 24.04 machine
• As keyboard, I am using a Keychron C3 Pro wired keyboard.
• The input_tool.py script (part of this PR and useful for manual tests) intercepts both devices and transforms their signals to REST calls against the C64. You can see all REST calls made by that tool on the left of the video.
• Anykey 1.7 listens to key presses and joystick movements and visualizes them.
• Please note that the concrete keyboard/joystick I used are irrelevant and only mentioned for completeness. This PR exposes a REST API that is agnostic of any controller devices.

Demo 2: The Great Giana Sisters

The following video demonstrates how to play one of the best games of all time, The Great Giana Sisters:

https://youtu.be/aIj1hI3g45I

• On the left-hand side of the screen, you see input_tool.py tool started in verbose mode (-V flag) which relays the USB XBox 360 Controller and Keychron C3 Pro keyboard actions via REST to an Ultimate 64 Elite I.
• On the right-hand side, you see a screen capture of the Ultimate 64 Elite I, propagated via the Ultimate 64 audio/video stream to the C64 Stream OBS plugin.
• Most of the video demonstrates joystick injection. At the end, it also shows keyboard injection.

High-Level Changes

• Added the machine:input REST endpoint with shared errors, keyboard, and two-port joysticks response shape for GET and successful POST.
• Added REST keyboard injection for press, release, tap, and release-all behavior, including shifted physical C64 keys and RESTORE tap handling.
• Added REST joystick injection for both C64 joystick ports, including diagonals and unusual direction combinations.
• Added fire2 / fire3 support through POTX/POTY state while keeping digital joystick bits 5..7 high.
• Added host tests and U64 E2E tooling for validation and manual exercising of keyboard and joystick injection.

Implementation Details

• Joystick writes go through a small output combiner so USB/HID joystick state and REST joystick state are merged at one U64 output point instead of racing direct writes to C64_JOY1_SWOUT.
• REST keyboard state is kept as a separate keyboard matrix source and combined with existing USB/UI matrix state at the existing matrix application point.
• On U64-II, REST keyboard state temporarily forces MATRIX_WASD_TO_JOY off at the keyboard matrix owner and restores the configured value when REST keyboard state clears, matching the existing keyboard-scan workaround that keeps the C64 keyboard usable.
• Tap input uses a persistent layer plus an overlay/timer layer so tap expiry cannot release an overlapping persistent press.
• If the Ultimate 64 menu is open, key strokes are injected via keyboard_usb.cc.
• fire2 / fire3 use POTX/POTY and leave joystick digital bits 5..7 high; U64-visible POT lows are mirrored through the first paddle register pair so port 2 extra buttons are observable by C64 software.
• U2 and U2+ builds wire the route but return HTTP 501 because v1 injection is only supported on U64-class hardware.
• Added safety fixes to harden JSON parsing, handle zero-length REST bodies without entering the JSON parser, and prevent software-injected joystick state from incorrectly blocking local keyboard scanning.
• The port 2 POT mirroring exists because live U64 E2E showed the REST state snapshot could report port 2 fire2 / fire3 while C64-side POT reads still remained released without that mirror.

Tests

• Unit tests: make -C software/api/tests passed, and make -C software/io/usb/tests passed.
• Automated E2E: python3 tools/api/input_test.py --host u64 passed against a deployed U64 build with input_test: OK (44 checks).
• Manual E2E: python3 tools/api/input_tool.py --self-test --host u64 --no-gamepad passed with input_tool self-test: OK; interactive typing/gamepad exercise was not run in this non-interactive session.
• Manual keyboard note: input_tool.py is best used with a US keyboard and uses positional mapping from the host keyboard to C64 keys.

Sample run of input_test.py:

./input_test.py 
[01] input snapshot has stable empty response shape ... OK
[02] POST accepts 64 event batch ... OK
[03] bad content-type is rejected without mutation ... OK
[04] missing JSON body is rejected without mutation ... OK
[05] malformed JSON is rejected without mutation ... OK
[06] unknown root field is rejected without mutation ... OK
[07] late invalid event keeps whole batch atomic ... OK
[08] joystick port 2 fire keeps Anykey buttons 2 and 3 released ... OK
[09] joystick port 2 fire2 lights only Anykey button 2 ... OK
[10] joystick port 2 fire3 lights only Anykey button 3 ... OK
[11] joystick port 1 up press is visible on CIA reads ... OK
[12] joystick port 1 all inputs and idempotent release are visible on CIA reads ... OK
[13] joystick port 2 diagonal and fire are visible on CIA reads ... OK
[14] joystick partial release is visible on CIA reads ... OK
[15] joystick fire2/fire3 round-trip through REST state ... OK
[16] joystick release_all then press in same batch is visible on CIA reads ... OK
[17] joystick unusual combination is visible on CIA reads ... OK
[18] joystick tap does not release persistent input ... OK
[19] joystick tap auto releases ... OK
[20] invalid joystick batch does not mutate state ... OK
[21] joystick release_all clears both ports ... OK
[22] machine reset clears keyboard and joystick REST state ... OK
[23] keyboard single letter reaches the live C64 matrix ... OK
[24] keyboard shifted pair reaches the live C64 matrix ... OK
[25] keyboard batch applies multiple presses atomically ... OK
[26] keyboard ordered batch and idempotent release ... OK
[27] keyboard release_all can be followed by press in same batch ... OK
[28] keyboard accepts eight simultaneous inputs ... OK
[29] keyboard tap does not release persistent key ... OK
[30] keyboard release_all clears state ... OK
[31] keyboard restore tap auto releases ... OK
[32] keyboard special-key taps snapshot correctly and auto release ... OK
[33] keyboard tap is visible in the live hardware snapshot and auto releases ... OK
[34] keyboard single-tap batch is consumed by BASIC in order ... OK
[35] keyboard cursor-left tap is visible in the live hardware snapshot and auto releases ... OK
[36] keyboard tap batch drains through the live matrix path ... OK
[37] keyboard long repeated tap train drains fully without sticky state ... OK
[38] keyboard 10 Hz mixed alphabet echo has no missed presses ... OK
[39] keyboard 20 Hz alternating ab echo has no missed presses ... OK
[40] keyboard 5 Hz alternating ab echo has no missed presses ... OK
[41] invalid keyboard batch does not mutate state ... OK
[42] menu editor keeps separate-batch shift active across POSTs ... OK
[43] menu editor repeats held printable keys and stops after release ... OK
[44] menu editor repeats held cursor keys and stops after release ... OK
input_test: OK (44 checks)

Concurrent Use Test:

• Concurrent hard-wired / REST joystick use: Started Anykey and interacted concurrently with hard-wired joystick on port 1 as well as REST joystick on the same port. Repeated the same for REST joystick wired to port 2. Interactions from the REST joystick did not affect the hard-wired joystick.
• Concurrent hard-wired / REST keyboard use: Started Anykey and interacted concurrently with native U64 Elite keyboard as well as REST keyboard. Interactions of one did not affect the other.
  • This test was performed both from the Basic prompt (to verify Kernal keyboard handling) and Anykey
  • It is possible to hold the SHIFT (or other modifier keys) on keyboard 1 (e.g. native) and then press another key (e.g. A) on keyboard 2 (e.g. REST). The effects of both key presses are combined on CIA1 level, but the firmware knows which bit changes were contributed via REST and is able to undo them (without affecting other CIA1 contributors) when a key is released.

[Copilot]

Pull request overview

Adds a new REST API surface for injecting keyboard matrix and two-port joystick state into U64-class devices via GET/POST /v1/machine:input, plus host-side tooling and tests to validate/drive the feature.

Changes:

• Introduces /v1/machine:input route with JSON batch validation and atomic application of keyboard/joystick events.
• Adds REST-owned keyboard matrix/tap overlay support and a joystick output combiner that merges USB (mouse/joystick) and REST joystick state.
• Adds host tools (input_test.py, input_tool.py) and C++ host tests for validation and state behavior.

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/api/input_tool.py	Interactive Linux input capture + REST injection tool (manual/E2E exercising).
tools/api/input_test.py	Host-driven contract/E2E validation for the new REST input endpoint.
target/u64ii/riscv/ultimate/Makefile	Wires new input route + joystick output module into U64-II build.
target/u64/nios2/ultimate/Makefile	Wires new input route + joystick output module into U64 build.
target/u2plus/nios/ultimate/Makefile	Builds input route but returns 501 for non-U64 hardware.
target/u2plus_L/riscv/ultimate/Makefile	Builds input route but returns 501 for non-U64 hardware.
target/u2/riscv/ultimate/Makefile	Builds input route but returns 501 for non-U64 hardware.
software/io/usb/usb_hid.cc	Routes joy1 SWOUT updates through new JoystickOutput combiner on U64.
software/io/usb/tests/Makefile	Updates test include paths for new dependencies.
software/io/usb/keyboard_usb.h	Adds REST keyboard state/tap overlay API and storage.
software/io/usb/keyboard_usb.cc	Implements REST keyboard matrix state, tap queue/overlay, and timer tick.
software/io/c64/keyboard_c64.h	Adds helper to detect when joystick activity should block keyboard scan.
software/io/c64/keyboard_c64.cc	Prevents REST-injected joystick activity from starving local keyboard scan.
software/io/c64/joystick_output.h	New output combiner interface for USB + REST joystick state.
software/io/c64/joystick_output.cc	Implements merged joystick output + POT mapping for fire2/fire3.
software/api/tests/Makefile	Adds build/run targets for new API validation + state host tests.
software/api/tests/input_api_validation_test.cpp	Unit tests for JSON schema/validation of input batches.
software/api/tests/input_api_state_test.cpp	Host tests for REST keyboard/joystick state overlay behavior.
software/api/routes.cc	Treats zero-length bodies as “no body” for multipart writers (prevents parsing).
software/api/route_machine.cc	Clears REST-injected input state on reset/reboot (U64).
software/api/route_input.cc	New /v1/machine:input GET/POST endpoint with apply + snapshot logic.
software/api/json.h	Whitespace-only formatting tweak.
software/api/json.cc	Hardens JSON conversion against empty/invalid token ranges and OOM.
software/api/input_api.h	New header-only JSON validation + keyboard/joystick mapping tables.
.gitignore	Ignores newly added host-test binaries.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chrisgleissner]

Hi @GideonZ ,

I hope you are doing well.

This PR is ready for review. I added a "Use Case" chapter to the PR description in order to highlight how this new feature can benefit U64 users and developers.

If there is anything you'd like me to change, please let me know.

Best wishes
Christian

[chrisgleissner]

Thanks @GideonZ !