[Graph] Re-add SM110 support, and use Toolkit 12.8 for SM90,100,120

docs/source/user_guide/building_cudagraph_conditional_fatbin.md

@@ -1,38 +1,34 @@
-# Building the CUDA graph conditional fatbin
+# Advanced: Building the CUDA graph conditional fatbin
 
-The `graph_do_while` feature uses a tiny CUDA kernel that calls
-`cudaGraphSetConditional` (a device-side function from NVIDIA's
-`libcudadevrt.a`) to control CUDA graph conditional while nodes. These
-conditional nodes require SM 9.0+ (Hopper or later); on older GPUs,
-`graph_do_while` falls back to a host-side loop automatically.
+The `graph_do_while` feature uses a tiny CUDA kernel that calls `cudaGraphSetConditional` (a device-side function from NVIDIA's `libcudadevrt.a`) to control CUDA graph conditional while nodes. These conditional nodes require SM 9.0+ (Hopper or later); on older GPUs, `graph_do_while` falls back to a host-side loop automatically.
 
 There are three distinct phases:
 
-1. **Fatbin generation** (rare, manual) — A developer runs
-   `scripts/build_condition_kernel_fatbin.py`, which compiles the kernel and
-   device-links it with `libcudadevrt.a` to resolve `cudaGraphSetConditional`.
-   The output is a self-contained fatbin, committed to git as a C header.
-   Requires `nvcc` and the CUDA toolkit.
-2. **Quadrants build** (CI / developers) — The C header is `#include`d as a
-   plain byte array. No CUDA toolkit needed.
-3. **Runtime** (end users) — The fatbin is loaded via `cuModuleLoadData`.
-   No CUDA toolkit needed.
+1. **Fatbin generation** (rare, manual) — A developer runs `scripts/build_condition_kernel_fatbin.py` (and the sibling `build_checkpoint_gate_fatbin.py` / `build_checkpoint_yield_check_fatbin.py`), which compiles the kernel and device-links it with `libcudadevrt.a` to resolve `cudaGraphSetConditional`. Each script emits **one fatbin per build toolkit** (see Prerequisites), all committed to git in a single C header. Requires the CUDA toolkits listed below.
+2. **Quadrants build** (CI / developers) — The C header is `#include`d as plain byte arrays. No CUDA toolkit needed.
+3. **Runtime** (end users) — The fatbins are loaded via `cuModuleLoadData`. No CUDA toolkit needed.
 
-This page documents phase 1: regenerating the pre-built fatbin.
+This page documents phase 1: regenerating the pre-built fatbins.
 
 ## When to regenerate
 
 You only need to regenerate the fatbin if:
 
-- The condition kernel source (`quadrants/runtime/cuda/graph_do_while_cond.cu`)
-  changes.
+- The condition kernel source (`quadrants/runtime/cuda/graph_do_while_cond.cu`) changes.
 - You need to add support for a new SM architecture.
 
 ## Prerequisites
 
-- CUDA toolkit with `nvcc` (13.0 or later, required for SM 110 support;
-  earlier toolkits will fail with `Unsupported gpu architecture 'compute_110'`).
-- The `nvcc` binary must be on your `PATH`, or set `CUDA_HOME`.
+Each SM architecture is compiled with the oldest suitable CUDA toolkit, and the per-toolkit results are bundled into one header. This is what keeps each cubin loadable on the widest range of drivers: a SASS cubin can only be loaded by a driver whose CUDA version is `>=` the toolkit that produced it, so building e.g. sm_120 (RTX 5090) with a newer toolkit than necessary would make it reject older drivers with `CUDA_ERROR_INVALID_IMAGE`. The per-arch toolkit requirement lives in the `SM_TOOLKIT` mapping in `scripts/_fatbin_common.py`, where each SM maps to a version spec — `==X.Y` (exact) or `>=X.Y` (minimum):
+
+- **`==12.8` (matched exactly)** — every architecture except sm_110 (pre-Hopper sm_60 / sm_70 / sm_80, Hopper sm_90, Blackwell sm_100 / sm_120). 12.8 is the oldest toolkit that covers all shipping Blackwell parts, and is the wide-compatibility anchor, so it is matched exactly rather than "or newer".
+- **`>=13.0`** — sm_110 (Thor) only; earlier toolkits fail with `Unsupported gpu architecture 'compute_110'`. This is a minimum: the oldest available toolkit `>=` 13.0 is used (e.g. 13.1 is fine). There is no old-driver constraint pulling it lower, since Thor is new hardware.
+
+**All** required toolkits must be installed; the scripts raise `MissingToolkitError` (listing what is missing and what was discovered) before compiling anything if any is absent. Toolkits are auto-discovered from `/usr/local/cuda-*/bin/nvcc`, `nvcc` on `PATH`, and `CUDA_HOME` / `CUDA_PATH`. To point at toolkits in non-standard locations, set `QUADRANTS_NVCC_CANDIDATES` to an `os.pathsep`-separated list of nvcc binaries and/or CUDA root directories, e.g.:
+
+```bash
+export QUADRANTS_NVCC_CANDIDATES=/opt/cuda-12.8:/opt/cuda-13.1/bin/nvcc
+```
 
 ## Regenerating
 
@@ -42,34 +38,43 @@ Run the script from the repo root:
 python scripts/build_condition_kernel_fatbin.py
 ```
 
-This will:
+This will, for each build toolkit:
+
+1. Compile `quadrants/runtime/cuda/graph_do_while_cond.cu` with relocatable device code for that toolkit's target SM architectures.
+2. Device-link the result with `libcudadevrt.a` to resolve the `cudaGraphSetConditional` extern.
 
-1. Compile `quadrants/runtime/cuda/graph_do_while_cond.cu` with relocatable device
-   code for each target SM architecture.
-2. Device-link the result with `libcudadevrt.a` to resolve the
-   `cudaGraphSetConditional` extern.
-3. Write the fatbin as a C byte array to
-   `quadrants/runtime/cuda/graph_do_while_cond_fatbin.h`.
+It then writes all the per-toolkit fatbins as C byte arrays (plus a small lookup table) to `quadrants/runtime/cuda/graph_do_while_cond_fatbin.h`.
 
-After regenerating, commit the updated header. Quadrants must be rebuilt to
-pick up the new fatbin.
+Regenerate the checkpoint kernels the same way with `scripts/build_checkpoint_gate_fatbin.py` and `scripts/build_checkpoint_yield_check_fatbin.py`. The generator emits a compact byte-array layout; run `pre-commit run -a` (clang-format) on the regenerated headers so they match the committed formatting before committing them. Quadrants must be rebuilt to pick up the new fatbins.
+
+## Verifying the committed fatbins
+
+`tests/python/test_fatbin_arch_coverage.py` holds an explicit, hand-maintained `EXPECTED_LAYOUT` for each committed `*_fatbin.h` — the architectures and the CUDA toolkit expected in every blob (e.g. `sm_90 / sm_100 / sm_120` built with CUDA 12.8 in blob 0 and `sm_110` built with CUDA 13.0+ in blob 1 for the condition kernel). It deliberately does *not* re-derive this from `scripts/_fatbin_common.py`, so a regression in the `SM_TOOLKIT` mapping is caught rather than silently baked into a new header. For every embedded cubin it checks both the SASS arch and the build toolkit that `cuobjdump` reports, so it directly asserts the load-bearing fact behind #2942: the sm_120 (RTX 5090) cubin is built with the wide-compatibility CUDA 12.8, not a newer toolkit that would reintroduce `CUDA_ERROR_INVALID_IMAGE` on 570-series drivers. It also drift-guards each `SM_VERSIONS` list against the arch set actually baked into the checked-in header.
+
+The test shells out to `cuobjdump`, so it is skipped unless `cuobjdump` is on `PATH`; the toolkit providing it must also be at least as new as the newest one used to build the fatbins (CUDA 13.0+, for sm_110), otherwise the affected header is skipped rather than failed. Run it after regenerating:
+
+```bash
+# Make a recent CUDA toolkit's cuobjdump visible on PATH, e.g.:
+export PATH=/usr/local/cuda/bin:$PATH
+python -m pytest tests/python/test_fatbin_arch_coverage.py -v
+```
 
 ## Adding a new SM architecture
 
-Edit the `SM_VERSIONS` list in `scripts/build_condition_kernel_fatbin.py` to
-add the new SM version number (e.g., `130`), then regenerate.
+Add the new SM version number to the `SM_VERSIONS` list in the relevant `scripts/build_*_fatbin.py`, and add an entry for it to `SM_TOOLKIT` in `scripts/_fatbin_common.py`: use `"==X.Y"` to pin it to an exact toolkit (the oldest one that covers it, as for the `==12.8` archs) or `">=X.Y"` for a minimum (as done for `110: ">=13.0"`). Then make sure a matching toolkit is installed and regenerate. (Every SM in `SM_VERSIONS` must have an `SM_TOOLKIT` entry, or the scripts raise `KeyError`.)
 
 ## Files
 
 | File | Purpose |
 |------|---------|
+| `scripts/_fatbin_common.py` | Shared multi-toolkit build + header generation; arch → toolkit mapping |
 | `quadrants/runtime/cuda/graph_do_while_cond.cu` | CUDA C source for the condition kernel |
-| `scripts/build_condition_kernel_fatbin.py` | Regeneration script |
+| `scripts/build_condition_kernel_fatbin.py` | Regeneration script (condition kernel) |
 | `quadrants/runtime/cuda/graph_do_while_cond_fatbin.h` | Generated C header (checked into git) |
+| `scripts/build_checkpoint_gate_fatbin.py`, `scripts/build_checkpoint_yield_check_fatbin.py` | Regeneration scripts (checkpoint kernels) |
+| `quadrants/runtime/cuda/checkpoint_gate_fatbin.h`, `quadrants/runtime/cuda/checkpoint_yield_check_fatbin.h` | Generated C headers (checked into git) |
+| `tests/python/test_fatbin_arch_coverage.py` | Regression test: committed headers match an explicit per-blob arch + toolkit layout (needs `cuobjdump`) |
 
 ## How it's used at runtime
 
-`GraphManager::ensure_condition_kernel_loaded()` in
-`quadrants/runtime/cuda/graph_manager.cpp` loads the fatbin via
-`cuModuleLoadData`. If the fatbin does not contain SASS for the current GPU's
-SM architecture, loading fails with a clear error pointing to this script.
+`GraphManager::ensure_condition_kernel_loaded()` in `quadrants/runtime/cuda/graph_manager.cpp` loads the fatbins via `GraphManager::load_first_matching_fatbin()`, which tries each bundled per-toolkit fatbin with `cuModuleLoadData` and keeps the first that loads (a device only loads the fatbin containing SASS for its SM). If none contain SASS for the current GPU's SM architecture, loading fails with a clear error pointing to the relevant regeneration script. The checkpoint kernels load the same way from `graph_manager_checkpoint.cpp`.

docs/source/user_guide/supported_systems.md

@@ -17,11 +17,13 @@ We test the following systems in our CI servers:
 
 ### GPUs
 
-- CUDA GPUs, Pascal or later (>=sm_60)
+- CUDA GPUs, `sm_60` (Pascal) through `sm_120` (Blackwell / Thor) — i.e. `>=sm_60` and `<=sm_120`
 - Metal GPUs
 - AMD GPUs
 - Vulkan-compatible GPUs (e.g. Intel Arc)
 
+If you have a newer NVIDIA GPU (above `sm_120`), please [open an issue on the Quadrants repo](https://github.com/Genesis-Embodied-AI/quadrants/issues).
+
 ### Backend / OS matrix
 
 Which backends are available on each supported platform. `qd.cpu` and `qd.vulkan` run on every OS; the other GPU backends are platform-specific because they wrap vendor drivers (CUDA on NVIDIA, ROCm on AMD, Metal on Apple).
@@ -35,10 +37,10 @@ Which backends are available on each supported platform. `qd.cpu` and `qd.vulkan
 | Windows ARM64 | yes | no | no | n/a | yes |
 
 Notes:
-- `qd.cuda` requires an NVIDIA driver + CUDA runtime on the host; quadrants links against the CUDA runtime discovered at import time. NVIDIA ships CUDA for Linux ARM64 and Windows ARM64, but quadrants does not support them yet.
-- `qd.amdgpu` currently wires up the Linux x64 ROCm path only. AMD's HIP SDK also ships on Windows and on some Linux ARM64 targets, but quadrants does not support them yet. AMDGPU codegen always runs in wave64 mode, so `qd.simt.subgroup` primitives operate over 64 lanes regardless of whether the GPU is CDNA (Instinct, wave64-default) or RDNA (Radeon, wave32-default — quadrants overrides it to wave64).
+- `qd.cuda` requires an NVIDIA driver and CUDA runtime installed on the host. NVIDIA ships CUDA for Linux ARM64 and Windows ARM64, but quadrants does not support them yet.
+- `qd.amdgpu` currently wires up the Linux x64 ROCm path only. AMD's GPU toolchain also ships on Windows and on some Linux ARM64 targets, but quadrants does not support them yet. On AMD GPUs, a subgroup is always 64 threads wide, so [`qd.simt.subgroup`](./subgroup.md) primitives operate over 64 lanes on both CDNA (Instinct) and RDNA (Radeon) hardware.
 - `qd.metal` is only available on Apple hardware and is the recommended GPU backend there.
-- `qd.vulkan` on macOS ships a bundled MoltenVK dylib inside the wheel, so no separate MoltenVK install is required.
+- `qd.vulkan` on macOS bundles a copy of MoltenVK (a Vulkan-to-Metal translation layer) inside the wheel, so no separate install is required.
 
 ### Python backend