Add debugger to machine code monitor

[chrisgleissner]

This PR adds a debugger to the Machine Code Monitor introduced by PR 358.

Overview

The debugger is available from the Assembly view and works across Telnet, UI Overlay, and UI Freeze modes. It adds interactive stepping, breakpoints, live CPU status and target prediction.

Please note that all features described below are fully implemented and tested on a U64 Elite I. This PR is in Draft mode nevertheless until the underlying Machine Code Monitor work will have been merged to master, at which point I'll merge master into this PR's branch, resolve any merge conflicts, and remove the Draft status.

The following chapters list features, show a demo as well as screenshots, and then cover the design, test approach, and known limitations.

Looking for U2 Testers

I only own an Ultimate 64 (and Commodore 64 Ultimate).

Thus, if anyone owns a U2 cartridge, I'd very much appreciate it if you could test this new feature on your device and let me know what works and what doesn't. Please have a look at the Known Limitations chapter first.

Many thanks.

Features

In the following overview, keyboard shortcuts are bolded.

• Stepping and execution

  • Supports RAM and ROM. (see *1)
  • Step Over, also known as Debug.
  • Step InTo, also known as Trace.
  • Step Out.
  • Continue, also known as Go.
  • Run to Kursor.

• Live CPU status

  • The debug footer shows PC, AC, XR, YR, SP, processor flags, and the IRQ/NMI vectors at $0314/$0315 and $0318/$0319.
  • Important values and active processor flags (NV-BDIZC) are highlighted.

• Target prediction

  • Shows the target of JMP, JSR, RTS, and branch opcodes.
  • Branch targets are highlighted when the branch is about to be taken.
  • This makes it clear where the next debug action will go before stepping.

• Breakpoints

  • Supports RAM and ROM. (see *1)
  • Supports up to 10 non-persistent breakpoints, aligned with the existing bookmark UX.
  • R toggles a breakpoint on the current cursor line.
  • C=+R opens the breakpoint list.
  • The breakpoint list supports:
    • 0..9: Jump to a breakpoint slot.
    • L: Change the breakpoint Label.
    • S: Set a breakpoint at the current cursor line.
    • E: Enable or disable a breakpoint.
    • DEL: Delete a breakpoint.

• Free exploration while debugging

  • Debug mode works together with the existing monitor views, including Memory, ASCII, Screen Code, Binary, and Assembly.
  • Edit mode can be active at the same time as Debug mode.
  • Memory can be inspected or edited without losing debug context.

• Multi-mode debugging

  • Supports Telnet, UI Overlay, and UI Freeze modes.

• Convenient reset

  • C=+X resets the C64 from inside Debug mode and returns to a clean monitor state.

Notes:

• (*1): ROM stepping / breakpoints only works on U64 since it relies on placing a temporary BRK opcode in the in-memory mirror of the ROM file. On a U2, such a mirror does not exist and hardware ROM can of course not be modified.

Demo

The demo shows a small program cycling the background colour, followed by stepping through KERNAL and BASIC:

https://youtu.be/ECsqq5HKPlE

Screenshots

Debugger

The following screenshot shows the debugger paused in the Kernal SCNKEY subroutine, which checks for pressed keys:

• The Dbg flag in the top right shows that debug mode is active. At $EA87, the start of the subroutine, [KEY] marks a breakpoint.
• The CPU is currently stopped at $EA98. Pressing D would take the highlighted branch to $EAFB, shown by the white target address. Without highlighting, the branch would not be taken.
• The reversed cursor row can be moved to inspect other memory while staying in debug mode. Because of that, the next instruction to execute is also marked with >....<. Here, the cursor row and next instruction happen to be the same.
• At the bottom, above the CPU/VIC bank footer, the two-line debug footer shows the current CPU state and updates after each debug step.

Entering and leaving Debug mode

• Press D in Assembly view to enter Debug mode.
• Leave Debug mode with either C=+D or RUN/STOP.
• Edit mode and Debug mode are independent, so both can be active at the same time.

Debug footer

The debug footer shows:

• The next opcode address.
• AC, XR, YR, and SP.
• Processor flags.
• IRQ and NMI vectors.
• Predicted jump, branch, and return targets where applicable.

The next opcode to be executed is marked with >.....< and is also reflected by the PC value in the footer. This lets the developer move freely around memory while still keeping track of the active debug state.

Breakpoints

Breakpoints are shown on the right side of assembly opcode lines as [BRKx], where x is the breakpoint index from 0 to 9. Custom labels are shown as [LABL]. Lines with breakpoints are also highlighted.

The breakpoint list popup is intentionally similar to the existing bookmarks popup. It supports jumping to breakpoints, deleting breakpoints, enabling or disabling breakpoints, and assigning custom labels.

Debug help page

Some rarely used monitor shortcuts have debug-specific meanings while Debug mode is active. These mappings are shown at the top of the help screen.

Design

The following chapter discusses key design considerations.

Breakpoints

• The machine-code monitor implements debugging by planting temporary 6510 BRK instructions because the FPGA core does not expose hardware breakpoints or direct 6510 register access usable by the application-hosted monitor.
• MemoryBackend::create_debug_session() is the only seam between the monitor and platform-specific debug backends. Host tests use fake backends; firmware builds use the real U64 and U2 implementations.
• The monitor UI talks only to a DebugSession abstraction (over, trace, step_out, go). The shared BRK engine owns trap installation, sentinel polling, register capture, resume, cleanup, breakpoint orchestration and patch tracking.
• Breakpoints save the original byte, write $00 (i.e. the BRK opcode), let the CPU trap, read the register snapshot from the BRK catcher stub, then restore the saved byte. RAM breakpoints work on both U64 and U2.
• BRK patches are recorded with their address, original byte and CPU port, then restored inside a stopped-session window. Reserved addresses, including trampoline memory and IRQ/NMI/BRK vectors, reject patches.
• The debugger borrows the cassette buffer for its catcher, resume trampoline, one-shot NMI trampoline and scratch state, and temporarily repoints the RAM BRK vector at $0316/$0317. Any changes are temporary and cleaned up on leaving debug mode.

Stepping

• Stepping is implemented by prediction rather than hardware single-step: the pure, host-testable 6502 predictor decodes the current instruction, plants BRKs at possible next instruction addresses, resumes the CPU, then captures the trap.
• D steps over JSR, T traces into it, O runs to the caller-side return, and G installs enabled breakpoints and free-runs. If G starts on a breakpoint, it first steps past it to avoid immediately re-trapping.
• Step Out uses a return-target stack populated when Trace enters a JSR, rather than relying only on live stack inspection.

ROM Support

• On U64, BASIC and KERNAL ROM breakpoints are supported by patching the volatile FPGA ROM image buffers, not persistent ROM storage. The same engine path is used; the U64 leaf decides whether an address maps to RAM or a visible ROM image.
• U2 does not support visible ROM image patching, so ROM breakpoints are refused cleanly. RAM breakpoints and register capture still use the same shared engine.

Cleanup and Modes

• Cleanup restores every patched byte, borrowed stub byte and vector on every Debug exit path, including normal exit, timeout, cancel, reset, monitor close, RUN/STOP, C=+O and C=+X.
• Overlay cleanup stages the resume path before restoring live code so the running CPU never executes a half-restored state.
• Freeze mode temporarily unfreezes during a step and refreezes afterwards. Overlay and Telnet modes do not use that path.
• C=+X clears transient debug state, resets through the backend, and either reopens the monitor or exits cleanly depending on the mode.

Implementation

Debug mode is implemented as a modal layer on top of the Assembly view. Normal monitor behaviour is unchanged.

The main source code files are:

• machine_monitor.cc

  • Handles UI, key routing, and Debug/Edit composition.

• monitor_debug.{h,cc}

  • Contains the debug data model, footer formatting, and help formatting.

• monitor_breakpoints.{h,cc}

  • Contains the 10-slot non-persistent breakpoint table.

• monitor_debug_session.h

  • Defines the backend abstraction.

• monitor_debug_brk_session.cc

  • Implements the shared BRK trampoline, sentinel handling, patch management, return-target tracking, run-window depth, and cleanup.

• monitor_debug_u64.cc and monitor_debug_u2.cc

  • Provide the U64 and U2 hardware hooks.

Testing

Hardware testing

• Tested on an Ultimate 64 Elite I.
• The underlying Machine Code Monitor branch was also tested on an Ultimate II.
• Debug stepping through ROM is not supported on an Ultimate II because it requires patching ROM code.

Test feedback for other devices would be greatly appreciated.

Unit tests

software/test/monitor/machine_monitor_debug_test.cc

Coverage includes:

• Predictor classification.
• Breakpoint table behaviour.
• Footer layout.
• Debug key handling.
• Continue, Trace, Step Over, Step Out, and Stop Debugging.
• C=+X reset and monitor re-entry.
• Modal and popup gating.
• Debug/Edit composition.
• Cleanup on all exit paths.
• Freeze and overlay behaviour.
• Timeout recovery.
• Step Out target validation.
• BASIC/KERNAL visible-ROM stepping on U64.

End-to-end tests

./tools/developer/machine-code-monitor/monitor_debug_test.py --host <u64-ip>

The E2E test drives deployed firmware through Telnet and uses the U64 REST API for memory setup. It asserts screen contents after each step.

Output:

./monitor_debug_test.py 
[01] Debug: ASM Edit at $2000 writes RAM and refreshes ASM/Hex ... OK
[02] Debug: setup at $C000 ... OK
[03] Debug: A switches to Assembly view ... OK
[04] Debug: D enters Debug mode without executing ... OK
[05] Debug: clear stale breakpoint slots before exercising popup flows ... OK
[06] Debug: footer rows show CPU labels ... OK
[07] Debug: CPU table sits above normal CPU/VIC footer ... OK
[08] Debug: unknown context renders blanks ... OK
[09] Debug: help keeps normal shortcuts and uses RSTOP labels ... OK
[10] Debug: R toggles a breakpoint (set + clear) ... OK
[11] Debug: C=+R opens the breakpoint list popup ... OK
[12] Debug: breakpoint opcode line shows [BRKx] before 3-char source ... OK
[13] Debug: C=+R shows the live breakpoint list ... OK
[14] Debug: breakpoint label replaces [BRKx] on the ASM page ... OK
[15] Debug: visible memory source indicators are 3 chars ... OK
[16] Debug: RETURN preserves Assembly navigation ... OK
[17] Debug: RETURN does not execute target code ... OK
[18] Debug: ESC/RUNSTOP leaves Edit before Dbg so debugging can continue ... OK
[19] Debug: C=+D leaves Debug mode while keeping Edit active ... OK
[20] Debug: returning to ASM after stepping elsewhere follows the current debug PC ... OK
[21] Debug: C=+X resets the machine and keeps Debug open with blank context ... OK
[22] Debug: C=+B opens bookmarks in Debug mode ... OK
[23] Debug: ESC leaves Debug mode ... OK
[24] Debug: B retains Binary view (not stolen by breakpoint) ... OK
[25] Debug: C=+D leaves Debug mode ... OK
[26] Debug: D over without captured context executes from cursor ... OK
[27] Debug: flag/control-flow program loads at $C200 ... OK
[28] Debug: flag capture and branch fall-through stay truthful across steps ... OK
[29] Debug: unsafe-target/refusal fixtures load at $C240/$C260/$C280 ... OK
[30] Debug: BRK, RTS, and RTI refuse no-context Over without fabricating state ... OK
[31] Debug: O outside a traced subroutine says NOT IN SUBROUTINE ... OK
[32] Debug: Over on RTS without active JSR frame says NOT IN SUBROUTINE (not PATCH FAILED) ... OK
[33] Debug: undocumented NOP is decoded by Undc but not debug-stepped ... OK
[34] Debug: traced RTS lands on the caller continuation address ... OK
[35] Debug: RTI restores the stacked target PC and flags truthfully ... OK
[36] Debug: page-cross branch and indirect-JMP fixtures load ... OK
[37] Debug: taken branch across a page stops at the real target ... OK
[38] Debug: not-taken branch across a page falls through correctly ... OK
[39] Debug: Over on JMP ($xxFF) follows the real 6502 page-wrap target ... OK
[40] Debug: nested Out fixtures load with stack-changing outer frame ... OK
[41] Debug: nested Out unwinds inner then outer caller frames truthfully ... OK
[42] Debug: Step Out breaks after active JSR, not nearby RTS ... OK
[43] Debug: KERNAL ROM Step Into from $E000 ... OK
[44] Debug: KERNAL ROM Step Over on visible JSR ... OK
[45] Debug: KERNAL ROM Step Into on visible JSR ... OK
[46] Debug: BASIC ROM Step Over on visible JSR ... OK
[47] Debug: BASIC ROM breakpoint set/hit/remove/step ... OK
[48] Debug: KERNAL ROM breakpoint set/hit/remove/step ... OK
[49] Debug: KERNAL $E000 G continues safely from BASIC $BC9B ... OK
[50] Debug: BASIC $BCF2 manual ROM breakpoint is reachable ... OK
[51] Debug: deep KERNAL/BASIC trace fixture loads at $C700 ... OK
[52] Debug: deep trace uses D/T/G/O across RAM, KERNAL, and BASIC ... OK
[53] Debug: load test program at $C000 (via REST) ... OK
[54] Debug: G with BP at $C006 stops with A=$AA X=$BB Y=$CC ... OK
[55] Debug: Over from $C006 steps NOP and stops at JSR ($C007) ... OK
[56] Debug: Trace into JSR enters subroutine at $C00D ... OK
[57] Debug: Out from inside the subroutine returns to $C00A ... OK
[58] Debug: cleanup restores user bytes at the breakpoint ... OK
[59] Debug: load side-effect stepping program ... OK
[60] Debug: Over STA writes $C180 ... OK
[61] Debug: Over JSR runs subroutine side effect and stops after call ... OK
[62] Debug: Over taken branch skips skipped-store side effect ... OK
[63] Debug: Over JMP reaches target and later store mutates $C183 ... OK
[64] Debug: Trace JSR enters before subroutine side effect, then Over executes it ... OK
[65] Debug: step-mode JSR/RTS keep the stack balanced and return address correct ... OK
[66] Debug: Stop Debugging + Exit resumes current $2000 context ... OK
[67] Debug: breakpoint re-entry loop loads at $C300 ... OK
[68] Debug: repeated G from the current breakpoint skips once and re-arms cleanly ... OK

==== debug_e2e_test summary ====
  target  : u64
  passed  : 68
  skipped : 0
  failed  : 0
debug_e2e_test: OK (68 checks, 0 skipped)

Soak test

./tools/developer/machine-code-monitor/monitor_debug_soak.py --host <u64-ip>

This long-running test loops high-risk scenarios such as freeze/refreeze handling, parking trampoline cleanup, and breakpoint re-entry.

Known Limitations

• Breakpoint placement and disassembly use the monitor-selected CPU bank view, not necessarily the live $01 value used by the running 6510. If these diverge in banked regions, stepping may decode or patch the wrong visible target and time out without corruption.
• There are no conditional breakpoints, watchpoints, or CPU history. It stops only at instruction boundaries and uses a bounded per-step trap timeout.
• U2 ROM breakpoints are not supported since it accesses the C64 ROM directly. We thus cannot place a temporary BRK opcode in the volatile ROM shadow as this feature only exists on a U64.

Related Work

• This branch was created from feature/151-machine-code-monitor, which has since been merged into the test-merge branch on Gideon's 1541ultimate repo.
• Once test-merge has been merged into master, master will be merged into this PR branch.