From 3d3327dc7ce314d823cf5b37a4d9b328ba63b78f Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 10:59:28 -0400 Subject: [PATCH 01/41] address doc CI issues --- docs/source/user_guide/graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index b1c239b080..b12fd4f6a8 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -4,7 +4,7 @@ Graphs reduce kernel launch overhead by capturing a sequence of GPU operations i ## Backend support -`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires CUDA SM 9.0+ / Hopper for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop that copies the condition value GPU → host each iteration (causing a pipeline stall). `qd.checkpoint` gating runs entirely on the device on every GPU backend; only the CPU backend uses host-side gating. +`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. | Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | | --- | --- | --- | --- | --- | --- | --- | @@ -44,7 +44,7 @@ my_kernel(x, y) # first call: builds and caches the graph my_kernel(x, y) # subsequent calls: replays the cached graph ``` -This works the same way on CUDA and AMDGPU. The cache is keyed per (compiled-kernel-specialization, launch-id), so different template instantiations (different field bindings, etc.) get their own cached graph. +This works the same way on CUDA and AMDGPU. ### Restrictions From dde771bc1626fae328897b7b8c65c44ee0b361e0 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:24:49 -0700 Subject: [PATCH 02/41] [CI] Fold reading-order (no-forward-reference) check into doc-quality Add RULE 3 to the doc-quality agent prompt: within a single doc, information must be ordered so a first-time reader can follow it top-to-bottom without jumping ahead (no forward references). Includes carve-outs for roadmaps, optional "see below" pointers, backward references, cross-doc references, and conventional preamble ordering, and hands pure undefined-term cases to Rule 1. Adds the [order] tag and reworks the violation cap so one rule cannot crowd out the others. --- .github/workflows/check_doc_quality.yml | 34 ++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/.github/workflows/check_doc_quality.yml b/.github/workflows/check_doc_quality.yml index 5e163d8b52..7bd82eae70 100644 --- a/.github/workflows/check_doc_quality.yml +++ b/.github/workflows/check_doc_quality.yml @@ -61,7 +61,7 @@ jobs: /tmp/changed_docs.txt contains a newline-separated list of Markdown (.md) doc files (paths relative to the repo root) that were added or modified in this PR. For EACH file in that list, read the FULL current contents of the file (not just the diff) and review it against - the two rules below. + the three rules below. Target audience: an *end user* of Quadrants. That is, someone who writes GPU programs in Python using the Quadrants library (`@qd.kernel`, `@qd.func`, ndarrays, fields, etc.). @@ -118,16 +118,42 @@ jobs: VIOLATION: internal / non-end-user material (subject to the carve-outs above) appears outside of such a clearly-marked advanced / under-the-hood section. + RULE 3 — Reading order (no forward references). + Within a single doc, information should be ordered so a first-time reader can follow it + top-to-bottom in one pass, without jumping ahead. A FORWARD REFERENCE is when a passage + depends on a concept, parameter, behavior, code element, or result that is only introduced + LATER in the SAME file, such that the reader cannot understand the current passage without + first reading the later one. + The following are NOT order violations: + - A brief overview / roadmap near the top that previews upcoming sections ("this guide + covers A, then B") — signposting the structure, not a dependency: the reader does not + need to understand A or B yet to keep reading. + - Optional "for more detail, see
below" pointers, where the current passage is + fully understandable WITHOUT following them. The test is: MUST read ahead to understand + (violation) vs more depth merely available later (fine). + - Backward references (to something earlier in the same file) are always fine. + - References to OTHER docs/files; this rule is about ordering WITHIN a single file only. + - Conventional preamble ordering (e.g. Prerequisites -> Installation -> Usage). + A plain term that is simply never defined belongs to Rule 1, not here; Rule 3 covers + ordering dependencies (concepts, examples, parameters, results) where the explanation DOES + exist in the file but appears too late. Do not double-flag the same issue under both rules. + VIOLATION: a passage cannot be understood by a first-time reader at the point they reach + it, because what it relies on is only introduced later in the same file. + Read each listed file in full before judging it. You may open any other file in the repo (e.g. linked docs) to verify whether a term is defined elsewhere. Do NOT modify any files. Be precise and conservative: only flag clear violations, not borderline cases. Judge each - file independently. Stop after finding 10 violations in total. + file independently against all three rules. Stop after reporting 10 violations in total, + but do not let one rule crowd out the others: if more than one rule has violations, make + sure each such rule is represented in your list before you stop. If there are NO violations, your final output must start with the word PASS. If there ARE violations, your final output must start with the word FAIL, followed by a list of violations (up to 10), one per line, in the format: - : [term|scope]: - where [term] is a Rule 1 violation and [scope] is a Rule 2 violation. + : [term|scope|order]: + where [term] is a Rule 1 violation, [scope] is a Rule 2 violation, and [order] is a Rule 3 + violation (for [order], cite both line numbers — what is used at line N is only introduced + at line M). PROMPT )" --model claude-4.6-opus-high-thinking --mode ask --output-format text --trust) From aafcc41c71b4703d0d0e9d8a1a267e1d887e4bd1 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:25:17 -0700 Subject: [PATCH 03/41] [Doc] Document reading-order rule in doc-quality check description The doc-quality check now enforces a third rule (reading order / no forward references); describe it and its carve-outs in contributing.md. --- docs/source/user_guide/contributing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/contributing.md b/docs/source/user_guide/contributing.md index bc7231fda0..f7826be26c 100644 --- a/docs/source/user_guide/contributing.md +++ b/docs/source/user_guide/contributing.md @@ -180,9 +180,9 @@ The agent reports up to 5 violations, each annotated with the host file's hotnes ### Doc quality check (`check_doc_quality.yml`) -Uses an AI agent to review documentation changes for an end-user audience (someone writing Quadrants kernels in Python, not a compiler engineer). For each `docs/**/*.md` file added or modified in the PR, it reads the entire current file (not just the diff) and checks two things: (1) **undefined terms** — a term a typical user is unlikely to know (specialized or internal jargon, project-specific abbreviations) must be defined at its first use in that file, either inline or via a link to a doc that defines it; (2) **end-user relevance** — internal / implementation / contributor-only material must be confined to a clearly-marked section whose heading contains "Advanced", "Under the hood", "Internals", or "Implementation". +Uses an AI agent to review documentation changes for an end-user audience (someone writing Quadrants kernels in Python, not a compiler engineer). For each `docs/**/*.md` file added or modified in the PR, it reads the entire current file (not just the diff) and checks three things: (1) **undefined terms** — a term a typical user is unlikely to know (specialized or internal jargon, project-specific abbreviations) must be defined at its first use in that file, either inline or via a link to a doc that defines it; (2) **end-user relevance** — internal / implementation / contributor-only material must be confined to a clearly-marked section whose heading contains "Advanced", "Under the hood", "Internals", or "Implementation"; (3) **reading order** — information must be ordered so a first-time reader can follow the file top-to-bottom in one pass, without forward references (a passage that can only be understood by reading something introduced later in the same file). -The following do not count as violations: references to public APIs that the author links to their docs and/or labels as public; brief reader-directed pointers suggesting the reader could contribute upstream or file an issue; the core public API vocabulary a user already knows (e.g. `@qd.kernel`, `@qd.func`, `qd.Template`, fields, ndarrays). The agent reports up to 10 violations. This check is delayed by 30 minutes, to avoid running repeatedly if multiple commits pushed with a short delay between each. +The following do not count as violations: references to public APIs that the author links to their docs and/or labels as public; brief reader-directed pointers suggesting the reader could contribute upstream or file an issue; the core public API vocabulary a user already knows (e.g. `@qd.kernel`, `@qd.func`, `qd.Template`, fields, ndarrays); and, for the reading-order check, an overview/roadmap near the top, optional "see below" pointers (where the current passage still reads fine on its own), and backward references. The agent reports up to 10 violations. This check is delayed by 30 minutes, to avoid running repeatedly if multiple commits pushed with a short delay between each. ### PR change report (`pr_change_report.yml`) From 899600ef3501bf4cbb9bc2090b1c67d597a8e2e6 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 24 Jun 2026 09:27:39 -0700 Subject: [PATCH 04/41] [CI] Simplify doc-quality RULE 3: drop reworked cap and double-flag note Both added cognitive load to the agent for little gain. Revert to the original "stop after 10 violations" cap and remove the Rule 1/Rule 3 delineation note. --- .github/workflows/check_doc_quality.yml | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/.github/workflows/check_doc_quality.yml b/.github/workflows/check_doc_quality.yml index 7bd82eae70..4fbb9f8aed 100644 --- a/.github/workflows/check_doc_quality.yml +++ b/.github/workflows/check_doc_quality.yml @@ -134,18 +134,13 @@ jobs: - Backward references (to something earlier in the same file) are always fine. - References to OTHER docs/files; this rule is about ordering WITHIN a single file only. - Conventional preamble ordering (e.g. Prerequisites -> Installation -> Usage). - A plain term that is simply never defined belongs to Rule 1, not here; Rule 3 covers - ordering dependencies (concepts, examples, parameters, results) where the explanation DOES - exist in the file but appears too late. Do not double-flag the same issue under both rules. VIOLATION: a passage cannot be understood by a first-time reader at the point they reach it, because what it relies on is only introduced later in the same file. Read each listed file in full before judging it. You may open any other file in the repo (e.g. linked docs) to verify whether a term is defined elsewhere. Do NOT modify any files. Be precise and conservative: only flag clear violations, not borderline cases. Judge each - file independently against all three rules. Stop after reporting 10 violations in total, - but do not let one rule crowd out the others: if more than one rule has violations, make - sure each such rule is represented in your list before you stop. + file independently. Stop after finding 10 violations in total. If there are NO violations, your final output must start with the word PASS. If there ARE violations, your final output must start with the word FAIL, followed by a list From 57f07bd4011f6fa509e5efb754be74bbfeb72b0d Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 09:01:45 -0400 Subject: [PATCH 05/41] fix doc ci failures --- docs/source/user_guide/graph.md | 38 ++++++++++++++++----------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index b12fd4f6a8..2ad0cbd182 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -2,20 +2,6 @@ Graphs reduce kernel launch overhead by capturing a sequence of GPU operations into a graph, then replaying it in a single launch. -## Backend support - -`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. - -| Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | -| --- | --- | --- | --- | --- | --- | --- | -| `graph=True` | hardware accelerated | hardware accelerated | hardware accelerated | runs (no acceleration) | runs (no acceleration) | runs (no acceleration) | -| `qd.graph_do_while` | hardware accelerated | host fallback | host fallback | host fallback | host fallback | host fallback | -| `qd.checkpoint` | GPU-side | GPU-side | GPU-side | GPU-side | GPU-side | host-side | - -AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). - -Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now — see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). - ## Basic usage Add `graph=True` to a `@qd.kernel` decorator: @@ -49,9 +35,9 @@ This works the same way on CUDA and AMDGPU. ### Restrictions - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. -- **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint. Autodiff kernels use the normal launch path. +- **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint (backward). [Autodiff](autodiff.md) kernels use the normal launch path. - **Device-resident ndarrays.** Graph mode bakes device pointers into the cached graph, so all ndarray arguments must be on the GPU. Passing a host-resident ndarray raises an error. -- **`qd_stream` is incompatible** with `graph=True`. Choose one or the other. +- [streams](streams.md) are incompatible** with graph. ### Passing different arguments @@ -179,7 +165,7 @@ Therefore on unsupported platforms, you might consider creating a second impleme ## Checkpoints with `qd.checkpoint` *(experimental)* -> **Experimental.** `qd.checkpoint`, `qd.GraphStatus`, and `kernel.resume(from_checkpoint=...)` are experimental APIs. The shape of the public surface (the context-manager signature, the `@qd.kernel(checkpoints=True)` flag, the `GraphStatus` fields, the host-side resume loop, the error messages, and the cross-backend lowering details) may change in any future release without a deprecation cycle. +> **Experimental.** `qd.checkpoint`, `qd.GraphStatus`, and `kernel.resume(from_checkpoint=...)` are experimental APIs, and may change in the near future, or be removed, or replaced. `qd.checkpoint` lets a graph kernel break partway through, surface a reason to the host, let the host fix things up, and resume from the same location on the next launch. An example use-case is an algorithm implemented as a graph that may need to allocate additional memory partway through, where the operations in the graph are in-place, and therefore cannot be rerun without changing/corrupting the output, and therefore for which simply retrying the whole graph from the start is not an option. @@ -268,7 +254,21 @@ while status.yielded: arr[i] = arr[i] + 1 ``` -The restriction is by design: each top-level statement inside a checkpoint becomes its own GPU task / graph node, so silently wrapping bare statements would hide a sequence of N field writes ballooning into N kernel launches. Forcing the user to write the `for`-wrap themselves keeps the lowering visible and gives a single obvious place to fuse multiple writes into one task by sharing a single wrapper. +The restriction is by design: each top-level statement inside a checkpoint becomes its own GPU task / graph node, so silently wrapping bare statements would hide a sequence of N field writes ballooning into N kernel launches. + +## Backend support + +`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. + +| Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | +| --- | --- | --- | --- | --- | --- | --- | +| `graph=True` | hardware accelerated | hardware accelerated | hardware accelerated | runs (no acceleration) | runs (no acceleration) | runs (no acceleration) | +| `qd.graph_do_while` | hardware accelerated | host fallback | host fallback | host fallback | host fallback | host fallback | +| `qd.checkpoint` | GPU-side | GPU-side | GPU-side | GPU-side | GPU-side | host-side | + +AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). + +Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now — see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). ## Performance @@ -397,7 +397,7 @@ If you do want fixed-size for loops to run optimally on unsupported hardware pla - on graph-do-while-supported hardware: handle adding the additional increment kernel - on graph-do-while-unsupported hardware: handle running the loop entirely on the host-side, to avoid adding a gpu pipeline stall -In practice, for our own kernels, i.e. in genesis-world, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. +In practice, for our own kernels, i.e. in [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world), they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. ### A while loop, conditional on a device-side scalar tensor, that has been optimized into a fixed-size for loop From cc8ea8a57cd45810bbb7faf9edb5e2f6727c3ef5 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 14:13:39 -0400 Subject: [PATCH 06/41] doc tewaks for ci --- docs/source/user_guide/graph.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2ad0cbd182..84ee6d594f 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -75,7 +75,7 @@ solve(x, counter) The argument to `qd.graph_do_while()` must be the name of a scalar `qd.i32` ndarray parameter. The loop body repeats while this value is non-zero. -- On CUDA SM 9.0+ (Hopper), this uses CUDA conditional while nodes — the entire iteration runs on the GPU with no host involvement. +- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses CUDA conditional while nodes — the entire iteration runs on the GPU with no host involvement. - On older CUDA GPUs, AMDGPU, and non-GPU backends, it falls back to a host-side do-while loop (see the [backend support table](#backend-support)). ### Patterns @@ -151,7 +151,7 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Caveats -On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** (HIP has no conditional / while node API as of ROCm 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** ([HIP](https://rocm.docs.amd.com/projects/HIP/en/latest/what_is_hip.html) has no conditional / while node API as of [ROCm](https://www.amd.com/en/products/software/rocm.html) 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside @@ -393,11 +393,9 @@ k1(a, count) The recommendation is to use the graph do while here anyway, if you need it for any platform, in order to ensure the code is compact and maintainable. -If you do want fixed-size for loops to run optimally on unsupported hardware platforms, we could add a specializd `qd.graph_range_for` function. This would: -- on graph-do-while-supported hardware: handle adding the additional increment kernel -- on graph-do-while-unsupported hardware: handle running the loop entirely on the host-side, to avoid adding a gpu pipeline stall +If you do want fixed-size for loops to run optimally on unsupported hardware platforms, please raise an issue, and we can look into this. -In practice, for our own kernels, i.e. in [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world), they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. +In practice, for our own [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world) kernels, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. ### A while loop, conditional on a device-side scalar tensor, that has been optimized into a fixed-size for loop From ee7754cf7146c43eff29ccc2a60cf3f8d4ea03ce Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 14:15:51 -0400 Subject: [PATCH 07/41] Apply suggestion from @graphite-app[bot] Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com> --- docs/source/user_guide/graph.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index a51b74e215..2e7c2ee95d 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -37,7 +37,9 @@ This works the same way on CUDA and AMDGPU. - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. - **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint (backward). [Autodiff](autodiff.md) kernels use the normal launch path. - **Device-resident ndarrays.** Graph mode bakes device pointers into the cached graph, so all ndarray arguments must be on the GPU. Passing a host-resident ndarray raises an error. -- [streams](streams.md) are incompatible** with graph. + +- [streams](streams.md) are **incompatible** with graph. + ### Passing different arguments From 04f2f0e77eb12cd19bf07e5cdfcd84f8806a2066 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 16:33:04 -0400 Subject: [PATCH 08/41] address doc CI --- docs/source/user_guide/graph.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2e7c2ee95d..e2e6dffa9c 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -149,7 +149,7 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Restrictions -- The counter ndarray may be swapped between calls: the cached graph reads each counter through an indirection slot that is refreshed on every launch, so passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. +- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. ### Caveats @@ -242,7 +242,7 @@ while status.yielded: ### Restrictions -- Must be used inside `@qd.kernel(graph=True, checkpoints=True)`. Without the flag, `qd.checkpoint(...)` raises `QuadrantsSyntaxError` at compile time with a fix-it pointing at `checkpoints=True`. +- Must be used inside `@qd.kernel(graph=True, checkpoints=True)`. Without the flag, `qd.checkpoint(...)` raises `QuadrantsSyntaxError` at compile time. - `cp_id` must be an int literal or an `IntEnum` value, and must be unique across the kernel. - `yield_on=` must be a kernel parameter that is a 0-d `qd.types.ndarray(qd.i32, ndim=0)`; expressions are not supported. - Checkpoints cannot be nested inside other checkpoints. Checkpoints inside a `qd.graph_do_while` body are fine. @@ -465,4 +465,4 @@ In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable - if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform - - for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast + - advanced: for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast From a1d925b3f96e06869ca1c51ebfc26891a91464ef Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Fri, 26 Jun 2026 16:54:49 -0400 Subject: [PATCH 09/41] address doc ci issues --- docs/source/user_guide/graph.md | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index e2e6dffa9c..9b3ca47fb5 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -289,9 +289,9 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): for i in range(c.shape[0]): fn_b(c, i) ``` -We have three top-level for loops, which we call 'offloaded tasks'. Each offloaded task is compiled into a separate GPU kernel. When we call `k1` from python, the c++ host-side code launches three gpu kernels. +We have three top-level for loops, which we call 'offloaded tasks'. Each offloaded task is compiled into a separate GPU kernel. When we call `k1` from python, three gpu kernels are launched, from the host side. -We can migrate it to graph by adding `graph=True`: +We can migrate this qd.kernel to graph by adding `graph=True`: ``` @qd.kernel(graph=True) def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): @@ -304,8 +304,8 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): ``` Results: -- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 kernels -- on other platforms, there is no change: we still launch 3 gpu kernels: no change: not better, not worse +- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate kernels +- on other platforms, there is no change: we still launch 3 separate kernels: no change: not better, not worse ### A while loop, conditional on a device-side scalar tensor @@ -328,11 +328,10 @@ So, we have: - the kernel contains device code, which will run on the gpu - each iteration, we copy the value of cond, from the gpu to the host, and check the value - this causes a gpu pipeline stall: - - first we wait for the entire default stream gpu work to complete/drain - - then we wait for the value of cond to copy from the gpu to the host - - then we run the python code to check the value of cond - - if we continue the loop, we now have to run through the python and c++ machinery to prepare the gpu kernel launch - - then launch the gpu kernels inside k1 + - first Quadrants wait for the entire default stream gpu work to complete/drain + - then Quadrants wait for the value of cond to copy from the gpu to the host + - then Quadrants run the python code to check the value of cond + - if we continue the loop, Quadrants now launches the gpu kernels inside k1 - together, these steps can cause a noticeable delay, reducing throughput speed After migrating to graph with graph do while we have: @@ -351,8 +350,6 @@ Now: - on supported hardware, the cond evaluation takes place on the gpu - and we avoid the gpu pipeline stall - on unsupported hardware, we still incur the pipeline stall, as before - - note that there will be some small acceleration, because the condition evaluation and kernel launch will take place entirely from c++, bypassing python - - no worse, incrementally better ### A fixed-size for loop @@ -371,8 +368,8 @@ for _ in range(num_its): In this case, we have `num_its` launches of the three gpu kernels in k1 - there is nothing on the host side that waits for anything to finish on the gpu-side - there is kernel launch latency associated with: - - running k1 from host-side python - - launching the gpu kernels for each of fn_1, fn_2, fn_3 from host-side c++ + - running k1 from host-side + - launching the gpu kernels for each of fn_1, fn_2, fn_3, also from host-side After migrating to graph we have something like: ``` @@ -464,5 +461,4 @@ The effect in reality is situation dependent: In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable -- if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform - - advanced: for example it could somehow run MAX_ITER iterations anyway, similar to the earlier hand-rolled version, but via the graph abstraction, hence allowing the code to be compact, cross-platform, and also optimally fast +- if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform, or raising an issue From 04f1f092c039bdf8cd7634ec07a6006d1bad5b93 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 09:47:47 -0400 Subject: [PATCH 10/41] address some comments --- docs/source/user_guide/graph.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 9b3ca47fb5..7276f37fac 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -32,6 +32,8 @@ my_kernel(x, y) # subsequent calls: replays the cached graph This works the same way on CUDA and AMDGPU. +Note that graph=True is a pre-requisite for all other options presented in this doc. + ### Restrictions - **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. @@ -149,11 +151,12 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Restrictions -- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) replays the cached graph without rebuilding it. +- The counter ndarray may be swapped between calls: passing a different ndarray (or alternating between several) does not trigger a recompile. ### Caveats -On platforms without native device-side conditional graph nodes — currently CUDA pre-SM 9.0 and **AMDGPU** ([HIP](https://rocm.docs.amd.com/projects/HIP/en/latest/what_is_hip.html) has no conditional / while node API as of [ROCm](https://www.amd.com/en/products/software/rocm.html) 7.2) — the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +On GPU backends without native device-side conditional graph nodes (see [Backend Support](#backend-support) below): +— the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside @@ -304,8 +307,8 @@ def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): ``` Results: -- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate kernels -- on other platforms, there is no change: we still launch 3 separate kernels: no change: not better, not worse +- on hardware-accelerated platforms, we only launch a single graph from the host, rather than 3 separate device kernels +- on other platforms, there is no change: we still launch 3 separate device kernels: no change: not better, not worse ### A while loop, conditional on a device-side scalar tensor From 1e2e53bbab2fc6c5eef11dcfb1bb409570444829 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:05:37 -0700 Subject: [PATCH 11/41] Add Advanced section to graph doc explaining kernel launch latency Explain what a GPU kernel launch is, break down launch latency into python/c++/gpu-driver sources, and describe why graph_do_while helps even on unsupported hardware (host-side loop driven in c++, avoiding per-iteration python latency). --- docs/source/user_guide/graph.md | 59 +++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 7276f37fac..74ac8b807b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -465,3 +465,62 @@ In this case, our recommendation is: - use graph do while anyway, if you need it on any platform - this will ensure your code is compact and maintainable - if you need optimum 100% performance on unsupported platforms, then consider PRing onto quadrants an optimized graph implementation for your target platform, or raising an issue + +## Advanced + +The rest of this guide treats "kernel launch latency" as a cost to be reduced, without pinning down what it actually is. +This section unpacks it: what launching a GPU kernel involves, where the latency comes from, and why `graph_do_while` +can still speed things up even on hardware that has no native device-side loop support. + +### What a GPU kernel launch is + +When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead +Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver +enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has +finished — usually before it has even started. Host and GPU run asynchronously. + +Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So +the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch +carries a fixed overhead that is independent of how much data the kernel processes — that per-launch overhead is the +"kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; +when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total +runtime. + +### Where kernel launch latency comes from + +It helps to split the latency of a single launch into three parts: + +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime: the + call into the `@qd.kernel` wrapper, argument type checking, the cache lookup that maps this call to an + already-compiled kernel, packing the arguments into a launch context (argument buffer), and crossing the Python→C++ + boundary. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python + `while`/`for` loop that calls a kernel N times pays it N times. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled + kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API + for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on + the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued + entirely from C++ pays it. + +Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches +into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the +graph is built, and then only a single graph replay per call. + +### Why graph_do_while helps even without hardware support + +Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, +every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads +`cond[()]` and evaluates the `if` before deciding whether to loop again. + +On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — +the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while +loop (see [Backend support](#backend-support)) — it is still typically faster than the hand-written Python version, +because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the +runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide +whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. + +In terms of the three-way split above, the fallback still pays the C++ side and GPU/driver side costs each iteration +(including the GPU→host copy of the condition value and the pipeline stall described in [Caveats](#caveats)), but it +eliminates the Python side cost per iteration. So `graph_do_while` on unsupported hardware lands between the two +extremes: slower than the fully-on-device path, but faster than an equivalent pure-Python `while` loop — while letting +you write the loop once and have it run on every backend. From e323513901898baba9b6791e7642a866d4994a27 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:08:54 -0700 Subject: [PATCH 12/41] Trim python-side latency bullet in graph doc Advanced section Drop the detailed enumeration of per-call python work; too much detail. --- docs/source/user_guide/graph.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 74ac8b807b..7317d3b9bc 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -490,11 +490,9 @@ runtime. It helps to split the latency of a single launch into three parts: -- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime: the - call into the `@qd.kernel` wrapper, argument type checking, the cache lookup that maps this call to an - already-compiled kernel, packing the arguments into a launch context (argument buffer), and crossing the Python→C++ - boundary. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python - `while`/`for` loop that calls a kernel N times pays it N times. +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. + This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` + loop that calls a kernel N times pays it N times. - **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. From e78e40ee803bcb6d078e44c4f8d49bee54d43745 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:10:41 -0700 Subject: [PATCH 13/41] Trim c++-side latency bullet in graph doc Advanced section Drop the detailed enumeration of per-task c++ runtime work; too much info. --- docs/source/user_guide/graph.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 7317d3b9bc..d6c1094dd9 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -493,9 +493,8 @@ It helps to split the latency of a single launch into three parts: - **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` loop that calls a kernel N times pays it N times. -- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary: selecting the compiled - kernel, copying the (usually tiny) argument buffer to the device, recording the launch, and invoking the driver API - for each offloaded task. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python + side, but non-zero, and paid per offloaded task. - **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued entirely from C++ pays it. From deb8969e3fe9b3aee8e08dc91dc193b5f14a47ed Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:11:13 -0700 Subject: [PATCH 14/41] Trim gpu/driver-side latency bullet in graph doc Advanced section Drop the "irreducible floor" sentence; doesn't add signal. --- docs/source/user_guide/graph.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index d6c1094dd9..3ebfa3049b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -496,8 +496,7 @@ It helps to split the latency of a single launch into three parts: - **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. - **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on - the GPU itself before the kernel begins running on the hardware. This is the irreducible floor: even a launch issued - entirely from C++ pays it. + the GPU itself before the kernel begins running on the hardware. Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the From b6f97326223bdffa21f54de9abb5526198f8067a Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:11:51 -0700 Subject: [PATCH 15/41] Use 'launch' instead of undefined 'replay' in graph doc Advanced section --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 3ebfa3049b..48e9f8a999 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -500,7 +500,7 @@ It helps to split the latency of a single launch into three parts: Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the -graph is built, and then only a single graph replay per call. +graph is built, and then only a single graph launch per call. ### Why graph_do_while helps even without hardware support From 18016887461ed83f29d0d5fb537b594542733bb3 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:14:09 -0700 Subject: [PATCH 16/41] Add latency hiding section to graph doc Advanced section Explain that launches overlap with execution, so reducing launch latency only helps throughput when kernels are small enough that launch latency is exposed rather than hidden behind execution. --- docs/source/user_guide/graph.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 48e9f8a999..29af230d5e 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -502,6 +502,21 @@ Graphs attack all three. Capturing a sequence of launches into a graph and repla into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the graph is built, and then only a single graph launch per call. +### Latency hiding + +Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next +launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the +GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. + +This is why reducing launch latency does not always improve overall throughput. When each kernel runs long enough that +its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — +the launches were already hidden behind execution. + +It matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, +so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that +regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases +throughput. + ### Why graph_do_while helps even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, From b192ca8b715b3664a2d86697c2ce18742d2c4e20 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:15 -0700 Subject: [PATCH 17/41] Reword 'This is why' to 'So' in graph doc latency hiding section --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 29af230d5e..2d46f7111b 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -508,7 +508,7 @@ Because host and GPU run asynchronously, kernel launches overlap with kernel exe launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. -This is why reducing launch latency does not always improve overall throughput. When each kernel runs long enough that +So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. From b58a08bda0583a44a4079fa7c3eff9b72ec4dee9 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:29 -0700 Subject: [PATCH 18/41] Clarify subject in graph doc latency hiding section ('It' -> 'Kernel launch latency') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 2d46f7111b..9f626f6209 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -512,7 +512,7 @@ So reducing launch latency does not always improve overall throughput. When each its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. -It matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, +Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases throughput. From 80eb935a46e20c700357a5e4c189889cc9b714c5 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:15:59 -0700 Subject: [PATCH 19/41] Soften claim in graph doc latency hiding section ('directly increases' -> 'can directly increase') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 9f626f6209..0ceacaeb86 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -514,7 +514,7 @@ the launches were already hidden behind execution. Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that -regime, cutting launch latency — for example by collapsing many launches into a single graph — directly increases +regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. ### Why graph_do_while helps even without hardware support From d634672d14baaac57a6fac454597fc38bfb7e86c Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:16:50 -0700 Subject: [PATCH 20/41] Soften unsupported-hardware claim in graph doc graph_do_while section Reframe from 'typically faster' to 'launch latency can be slightly reduced'. --- docs/source/user_guide/graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 0ceacaeb86..be51d91f93 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -525,8 +525,8 @@ every iteration pays the full Python-side launch latency: Python re-enters the k On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while -loop (see [Backend support](#backend-support)) — it is still typically faster than the hand-written Python version, -because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the +loop (see [Backend support](#backend-support)) — the launch latency can be slightly reduced compared to the hand-written +Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. From c14a85e3268b8cd7b7218e5e72ecbf5c0fea90d2 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:17:25 -0700 Subject: [PATCH 21/41] Remove closing summary paragraph from graph doc graph_do_while section --- docs/source/user_guide/graph.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index be51d91f93..fe4813256c 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -529,9 +529,3 @@ loop (see [Backend support](#backend-support)) — the launch latency can be sli Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. - -In terms of the three-way split above, the fallback still pays the C++ side and GPU/driver side costs each iteration -(including the GPU→host copy of the condition value and the pipeline stall described in [Caveats](#caveats)), but it -eliminates the Python side cost per iteration. So `graph_do_while` on unsupported hardware lands between the two -extremes: slower than the fully-on-device path, but faster than an equivalent pure-Python `while` loop — while letting -you write the loop once and have it run on every backend. From cfde9ce8ab02f58805cc2f45c9d6fb18cb2fa6d8 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Thu, 2 Jul 2026 07:17:52 -0700 Subject: [PATCH 22/41] Soften graph_do_while subsection heading in graph doc ('helps' -> 'can help slightly') --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index fe4813256c..52aea87424 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -517,7 +517,7 @@ so the GPU sits idle waiting for the next launch and the launch latency is *expo regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. -### Why graph_do_while helps even without hardware support +### Why graph_do_while can help slightly even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads From a0007e8ff6e6aa7619c386474656aad8daaca2e7 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 10:06:11 -0400 Subject: [PATCH 23/41] Unwrap Advanced section paragraphs in graph doc to fix line-wrapping check Co-authored-by: Cursor --- docs/source/user_guide/graph.md | 54 ++++++++------------------------- 1 file changed, 12 insertions(+), 42 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 52aea87424..fcf2d83783 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -468,64 +468,34 @@ In this case, our recommendation is: ## Advanced -The rest of this guide treats "kernel launch latency" as a cost to be reduced, without pinning down what it actually is. -This section unpacks it: what launching a GPU kernel involves, where the latency comes from, and why `graph_do_while` -can still speed things up even on hardware that has no native device-side loop support. +The rest of this guide treats "kernel launch latency" as a cost to be reduced, without pinning down what it actually is. This section unpacks it: what launching a GPU kernel involves, where the latency comes from, and why `graph_do_while` can still speed things up even on hardware that has no native device-side loop support. ### What a GPU kernel launch is -When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead -Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver -enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has -finished — usually before it has even started. Host and GPU run asynchronously. +When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has finished — usually before it has even started. Host and GPU run asynchronously. -Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So -the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch -carries a fixed overhead that is independent of how much data the kernel processes — that per-launch overhead is the -"kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; -when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total -runtime. +Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch carries a fixed overhead that is independent of how much data the kernel processes — that per-launch overhead is the "kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total runtime. ### Where kernel launch latency comes from It helps to split the latency of a single launch into three parts: -- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. - This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` - loop that calls a kernel N times pays it N times. -- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python - side, but non-zero, and paid per offloaded task. -- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on - the GPU itself before the kernel begins running on the hardware. +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` loop that calls a kernel N times pays it N times. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. -Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches -into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the -graph is built, and then only a single graph launch per call. +Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the graph is built, and then only a single graph launch per call. ### Latency hiding -Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next -launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the -GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. +Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. -So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that -its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — -the launches were already hidden behind execution. +So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. -Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, -so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that -regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase -throughput. +Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. ### Why graph_do_while can help slightly even without hardware support -Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, -every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads -`cond[()]` and evaluates the `if` before deciding whether to loop again. +Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads `cond[()]` and evaluates the `if` before deciding whether to loop again. -On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — -the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while -loop (see [Backend support](#backend-support)) — the launch latency can be slightly reduced compared to the hand-written -Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the -runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide -whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. +On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while loop (see [Backend support](#backend-support)) — the launch latency can be slightly reduced compared to the hand-written Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. From d128eae96f4ecb6053041db7ae19f918a2f8276b Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 10:09:45 -0400 Subject: [PATCH 24/41] Fix qd.type.NDArray typo to qd.types.NDArray in graph doc There is no qd.type module; the type namespace is qd.types. This matches the correct usage already present at line 407 and elsewhere in the docs. Co-authored-by: Cursor --- docs/source/user_guide/graph.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index fcf2d83783..a69ea76afa 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -284,7 +284,7 @@ The hardware support for each feature (graph and graph do while) was documented Before migrating to graph, we have for example: ``` @qd.kernel -def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): +def k1(a: qd.types.NDArray, b: qd.types.NDArray, c: qd.types.NDArray): for i in range(a.shape[0]): fn_a(a, i) for i in range(b.shape[0]): @@ -297,7 +297,7 @@ We have three top-level for loops, which we call 'offloaded tasks'. Each offload We can migrate this qd.kernel to graph by adding `graph=True`: ``` @qd.kernel(graph=True) -def k1(a: qd.type.NDArray, b: qd.type.NDArray, c: qd.type.NDArray): +def k1(a: qd.types.NDArray, b: qd.types.NDArray, c: qd.types.NDArray): for i in range(a.shape[0]): fn_a(a, i) for i in range(b.shape[0]): @@ -315,7 +315,7 @@ Results: Before migrating to graph we have for example: ``` @qd.kernel -def k1(a: qd.type.NDArray, cond: qd.type.NDArray): +def k1(a: qd.types.NDArray, cond: qd.types.NDArray): fn_1(a, cond) fn_2(a, cond) fn_3(a, cond) @@ -341,7 +341,7 @@ After migrating to graph with graph do while we have: ``` @qd.kernel(graph=True) -def k1(a: qd.type.NDArray, cond: qd.type.NDArray): +def k1(a: qd.types.NDArray, cond: qd.types.NDArray): while qd.graph_do_while(condition=cond): fn_1(a, cond) fn_2(a, cond) @@ -359,7 +359,7 @@ Now: Before migrating to graph we have for example: ``` @qd.kernel -def k1(a: qd.type.NDArray): +def k1(a: qd.types.NDArray): fn_1(a) # assume these each launch a single offloaded task (gpu kernel) fn_2(a) fn_3(a) @@ -377,7 +377,7 @@ In this case, we have `num_its` launches of the three gpu kernels in k1 After migrating to graph we have something like: ``` @qd.kernel(graph=True) -def k1(a: qd.type.NDArray, count: qd.type.NDArray): +def k1(a: qd.types.NDArray, count: qd.types.NDArray): while qd.graph_do_while(count): fn_1(a) fn_2(a) @@ -436,7 +436,7 @@ When we migrate this to graph do while, we get: ``` @qd.kernel(graph=True) -def k1(a: qd.type.NDArray, cond: qd.type.NDArray): +def k1(a: qd.types.NDArray, cond: qd.types.NDArray): while qd.graph_do_while(condition=cond): for j in range(a.shape[0]): # off-loaded task (gpu kernel) .... # no need for cond check From 6642def1b30694b69d2ef8f91e71e9a617c1b3e7 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 10:15:11 -0400 Subject: [PATCH 25/41] Replace non-ASCII em-dashes with ASCII hyphens in graph doc Co-authored-by: Cursor --- docs/source/user_guide/graph.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index a69ea76afa..93c23cc3f0 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -20,7 +20,7 @@ def my_kernel( The top level for-loops will be compiled into a single graph. The parallelism is the same as before, but the launch latency much reduced. -The kernel is used normally — no other API changes are needed: +The kernel is used normally - no other API changes are needed: ```python x = qd.ndarray(qd.f32, shape=(1024,)) @@ -45,7 +45,7 @@ Note that graph=True is a pre-requisite for all other options presented in this ### Passing different arguments -You can pass different ndarrays to the same kernel on subsequent calls. The cached graph is replayed with the updated arguments — no graph rebuild occurs: +You can pass different ndarrays to the same kernel on subsequent calls. The cached graph is replayed with the updated arguments - no graph rebuild occurs: ```python x1 = qd.ndarray(qd.f32, shape=(1024,)) @@ -79,7 +79,7 @@ solve(x, counter) The argument to `qd.graph_do_while()` must be the name of a scalar `qd.i32` ndarray parameter. The loop body repeats while this value is non-zero. -- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses CUDA conditional while nodes — the entire iteration runs on the GPU with no host involvement. +- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses CUDA conditional while nodes - the entire iteration runs on the GPU with no host involvement. - On older CUDA GPUs, AMDGPU, and non-GPU backends, it falls back to a host-side do-while loop (see the [backend support table](#backend-support)). ### Patterns @@ -156,13 +156,13 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Caveats On GPU backends without native device-side conditional graph nodes (see [Backend Support](#backend-support) below): -— the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +The value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside - launch new kernels for next loop iteration, if not finished yet -Note: the basic `graph=True` path (without `graph_do_while`) does **not** stall the host like this on either CUDA or AMDGPU — the entire kernel sequence runs as a single GPU-side graph replay. +Note: the basic `graph=True` path (without `graph_do_while`) does **not** stall the host like this on either CUDA or AMDGPU - the entire kernel sequence runs as a single GPU-side graph replay. Therefore on unsupported platforms, you might consider creating a second implementation, which works differently. e.g.: - fixed number of loop iterations, so no dependency on gpu data for kernel launch; combined perhaps with: @@ -213,7 +213,7 @@ When the body of a checkpoint writes a non-zero value into `yield_on[()]`: 1. Everything after the yielding checkpoint in the same launch is skipped. 2. `qd.checkpoint` will exit any surrounding `qd.graph_do_while`. -The framework never writes into your `yield_on` buffer — you own it end-to-end. That means: +The framework never writes into your `yield_on` buffer - you own it end-to-end. That means: - Before the **first** launch, initialize it to `0` (a freshly allocated `qd.ndarray` is not guaranteed to be zeroed). - :warning: Before each **resume** launch, reset it to `0` (otherwise the body of the same checkpoint sees the stale non-zero value and yields again on the same condition, looping forever). @@ -222,8 +222,8 @@ The framework never writes into your `yield_on` buffer — you own it end-to-end Kernels annotated with `checkpoints=True` return a `qd.GraphStatus` from every launch (including from `kernel.resume(...)`). The status carries two fields: -- `status.yielded` — `True` iff a checkpoint's `yield_on=` flag was non-zero during this launch. -- `status.checkpoint` — the `cp_id` label of the yielding checkpoint (or `None` when `yielded` is `False`). +- `status.yielded` - `True` iff a checkpoint's `yield_on=` flag was non-zero during this launch. +- `status.checkpoint` - the `cp_id` label of the yielding checkpoint (or `None` when `yielded` is `False`). Resume by calling `kernel.resume(..., from_checkpoint=label)`. Everything before `label` in source order is skipped on the resume launch; everything from `label` onward runs normally. The canonical host loop: @@ -273,7 +273,7 @@ The restriction is by design: each top-level statement inside a checkpoint becom AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). -Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now — see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). +Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now - see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). ## Performance @@ -472,15 +472,15 @@ The rest of this guide treats "kernel launch latency" as a cost to be reduced, w ### What a GPU kernel launch is -When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has finished — usually before it has even started. Host and GPU run asynchronously. +When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has finished - usually before it has even started. Host and GPU run asynchronously. -Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch carries a fixed overhead that is independent of how much data the kernel processes — that per-launch overhead is the "kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total runtime. +Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch carries a fixed overhead that is independent of how much data the kernel processes - that per-launch overhead is the "kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total runtime. ### Where kernel launch latency comes from It helps to split the latency of a single launch into three parts: -- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call — so a Python `while`/`for` loop that calls a kernel N times pays it N times. +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call - so a Python `while`/`for` loop that calls a kernel N times pays it N times. - **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. - **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. @@ -490,12 +490,12 @@ Graphs attack all three. Capturing a sequence of launches into a graph and repla Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. -So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing — the launches were already hidden behind execution. +So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing - the launches were already hidden behind execution. -Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency — for example by collapsing many launches into a single graph — can directly increase throughput. +Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency - for example by collapsing many launches into a single graph - can directly increase throughput. ### Why graph_do_while can help slightly even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads `cond[()]` and evaluates the `if` before deciding whether to loop again. -On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely — the ideal case. But even on hardware *without* that support — where `graph_do_while` falls back to a host-side do-while loop (see [Backend support](#backend-support)) — the launch latency can be slightly reduced compared to the hand-written Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again — all in C++, with no trip back through the Python interpreter between iterations. +On hardware with native device-side conditional graph nodes, `graph_do_while` removes the host from the loop entirely - the ideal case. But even on hardware *without* that support - where `graph_do_while` falls back to a host-side do-while loop (see [Backend support](#backend-support)) - the launch latency can be slightly reduced compared to the hand-written Python version, because the fallback loop is driven entirely in the C++ runtime rather than in Python. A single Python call enters the runtime, and the runtime then repeats internally: launch the loop body's tasks, read back the condition flag, and decide whether to iterate again - all in C++, with no trip back through the Python interpreter between iterations. From 2590b48694f0e48e933f8eb9def53470def08969 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 10:24:50 -0400 Subject: [PATCH 26/41] Replace deleted em-dash with ASCII hyphen in graph doc caveats Co-authored-by: Cursor --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 93c23cc3f0..fc052e7a77 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -156,7 +156,7 @@ Note that `qd.func`'s are inlined, so you can freely factorize these structures ### Caveats On GPU backends without native device-side conditional graph nodes (see [Backend Support](#backend-support) below): -The value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: +- the value of the `graph_do_while` parameter will be copied from the GPU to the host each iteration, in order to check whether we should continue iterating. This causes a GPU pipeline stall. For nested loops this host round-trip happens once per iteration of each loop level, and each loop-body task is replayed individually, so deeply nested loops on these backends pay correspondingly more host overhead (they remain correct, just slower than the CUDA SM 9.0+ native path). At the end of each loop iteration: - wait for GPU async queue to finish processing - copy condition value to hostside - evaluate condition value on hostside From 171422bb183e7c798b4bb3a7aede0b368c650ed1 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 11:57:42 -0400 Subject: [PATCH 27/41] add links that doc check was compalining missing --- docs/source/user_guide/graph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index fc052e7a77..3f9ec853d4 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -263,7 +263,7 @@ The restriction is by design: each top-level statement inside a checkpoint becom ## Backend support -`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via CUDA graphs) and AMDGPU (via HIP graphs); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. +`graph=True` and `graph_do_while` run on every backend. They are *hardware accelerated* on CUDA (via [CUDA graphs](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html)) and AMDGPU (via [HIP graphs](https://rocm.docs.amd.com/projects/HIP/en/latest/how-to/hip_runtime_api/hipgraph.html)); `graph_do_while` additionally requires [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus) for its hardware-accelerated path. On other backends, `graph=True` is silently ignored and the kernel runs via the normal launch path, and `graph_do_while` falls back to a host-side do-while loop. `qd.checkpoint` gating runs entirely on the device on every GPU backend. | Feature | `qd.cuda` SM 9.0+ | `qd.cuda` < SM 9.0 | `qd.amdgpu` | `qd.metal` | `qd.vulkan` | `qd.cpu` | | --- | --- | --- | --- | --- | --- | --- | @@ -271,7 +271,7 @@ The restriction is by design: each top-level statement inside a checkpoint becom | `qd.graph_do_while` | hardware accelerated | host fallback | host fallback | host fallback | host fallback | host fallback | | `qd.checkpoint` | GPU-side | GPU-side | GPU-side | GPU-side | GPU-side | host-side | -AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of ROCm 7.2). +AMDGPU `graph_do_while` falls back to the host-side loop because HIP does not currently expose conditional / while graph nodes (as of [ROCm](https://www.amd.com/en/products/software/rocm.html) 7.2). Nested and sibling `graph_do_while` loops (and mixing `graph_do_while` with top-level `for`-loops) are **experimental** for now - see [Nested loops and mixing with for-loops](#nested-loops-and-mixing-with-for-loops). From 5179cceba166b840e03b89da51daffd734828081 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 14:36:45 -0400 Subject: [PATCH 28/41] reference cuda conditional graph nodes --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 3f9ec853d4..056387b777 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -79,7 +79,7 @@ solve(x, counter) The argument to `qd.graph_do_while()` must be the name of a scalar `qd.i32` ndarray parameter. The loop body repeats while this value is non-zero. -- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses CUDA conditional while nodes - the entire iteration runs on the GPU with no host involvement. +- On [CUDA SM 9.0+](https://developer.nvidia.com/cuda/gpus), this uses [CUDA conditional while nodes](https://developer.nvidia.com/blog/dynamic-control-flow-in-cuda-graphs-with-conditional-nodes/) - the entire iteration runs on the GPU with no host involvement. - On older CUDA GPUs, AMDGPU, and non-GPU backends, it falls back to a host-side do-while loop (see the [backend support table](#backend-support)). ### Patterns From 523a99962555255fbf1c035984808f80af11d184 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 15:32:55 -0400 Subject: [PATCH 29/41] remove fix-it syntax --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 056387b777..cf07bd0970 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -249,7 +249,7 @@ while status.yielded: - `cp_id` must be an int literal or an `IntEnum` value, and must be unique across the kernel. - `yield_on=` must be a kernel parameter that is a 0-d `qd.types.ndarray(qd.i32, ndim=0)`; expressions are not supported. - Checkpoints cannot be nested inside other checkpoints. Checkpoints inside a `qd.graph_do_while` body are fine. -- The body of a `with qd.checkpoint(...)` block cannot contain bare top-level statements (assignments, augmented assignments, or bare call/expression statements). Every top-level statement must be inside a `for`-loop (or other control-flow construct). A docstring as the first statement is allowed. Bare statements raise `QuadrantsSyntaxError` at compile time with a fix-it pointing at the explicit one-iteration `for`-wrap: +- The body of a `with qd.checkpoint(...)` block cannot contain bare top-level statements (assignments, augmented assignments, or bare call/expression statements). Every top-level statement must be inside a `for`-loop (or other control-flow construct). A docstring as the first statement is allowed. Bare statements raise `QuadrantsSyntaxError` at compile time. ```python with qd.checkpoint(0, yield_on=flag): From 4d6717fa8816704934a52a7fbd4bafa86f0f5877 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 17:55:53 -0400 Subject: [PATCH 30/41] for real-world use-cases --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index cf07bd0970..909e337d74 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -397,7 +397,7 @@ The recommendation is to use the graph do while here anyway, if you need it for If you do want fixed-size for loops to run optimally on unsupported hardware platforms, please raise an issue, and we can look into this. -In practice, for our own [genesis-world](https://github.com/Genesis-Embodied-AI/genesis-world) kernels, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. +In practice, for real-world use-cases, they largely fall under the do while formulation, see the previous section. However, also have some that used to be do while, but have been migrated to an optimized fixed-size, see next section. ### A while loop, conditional on a device-side scalar tensor, that has been optimized into a fixed-size for loop From a6613d436bde5b81839c7990d2c916ab6aa35450 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Mon, 13 Jul 2026 17:57:06 -0400 Subject: [PATCH 31/41] Why using graph_do_while is relevant even without hardware support --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 909e337d74..bcda788064 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -494,7 +494,7 @@ So reducing launch latency does not always improve overall throughput. When each Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency - for example by collapsing many launches into a single graph - can directly increase throughput. -### Why graph_do_while can help slightly even without hardware support +### Why using graph_do_while is relevant even without hardware support Recall the earlier example **A while loop, conditional on a device-side scalar tensor**. Written as a plain Python loop, every iteration pays the full Python-side launch latency: Python re-enters the kernel wrapper, and Python itself reads `cond[()]` and evaluates the `if` before deciding whether to loop again. From 1d9f65e83396fba98c1cae70ff5e76c3f4b32e6a Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 06:28:52 -0700 Subject: [PATCH 32/41] Move general kernel-launch/latency material into new performance.md Per review feedback, the execution-model background (what a GPU kernel launch is, where launch latency comes from, latency hiding) is general rather than graph-specific. Extract it into a new performance.md landing page under the Performance section, with signposts to graph/streams/ fastcache/perf_dispatch. graph.md keeps the graph_do_while-specific subsection and links back for the background. --- docs/source/user_guide/graph.md | 26 +------------------- docs/source/user_guide/index.md | 1 + docs/source/user_guide/performance.md | 34 +++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 25 deletions(-) create mode 100644 docs/source/user_guide/performance.md diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index bcda788064..58f021189d 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -468,31 +468,7 @@ In this case, our recommendation is: ## Advanced -The rest of this guide treats "kernel launch latency" as a cost to be reduced, without pinning down what it actually is. This section unpacks it: what launching a GPU kernel involves, where the latency comes from, and why `graph_do_while` can still speed things up even on hardware that has no native device-side loop support. - -### What a GPU kernel launch is - -When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has finished - usually before it has even started. Host and GPU run asynchronously. - -Each top-level `for`-loop in a kernel is a separate offloaded task, and each offloaded task becomes one GPU kernel. So the three-loop `k1` in the earlier **qd.kernel** section launches three GPU kernels every time you call it. Every launch carries a fixed overhead that is independent of how much data the kernel processes - that per-launch overhead is the "kernel launch latency" this guide keeps referring to. When a kernel crunches a lot of data the overhead is negligible; when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total runtime. - -### Where kernel launch latency comes from - -It helps to split the latency of a single launch into three parts: - -- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call - so a Python `while`/`for` loop that calls a kernel N times pays it N times. -- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. -- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. - -Graphs attack all three. Capturing a sequence of launches into a graph and replaying it collapses many separate launches into a single graph launch, so on hardware-accelerated backends you pay the Python and C++ per-task costs once when the graph is built, and then only a single graph launch per call. - -### Latency hiding - -Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. - -So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing - the launches were already hidden behind execution. - -Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency - for example by collapsing many launches into a single graph - can directly increase throughput. +For background on what a GPU kernel launch is, where kernel launch latency comes from, and when reducing it actually helps throughput, see [Performance](performance.md). This section covers one graph-specific subtlety on top of that background. ### Why using graph_do_while is relevant even without hardware support diff --git a/docs/source/user_guide/index.md b/docs/source/user_guide/index.md index 0930813419..bbccc8420c 100644 --- a/docs/source/user_guide/index.md +++ b/docs/source/user_guide/index.md @@ -70,6 +70,7 @@ algorithms :maxdepth: 1 :titlesonly: +performance fastcache graph streams diff --git a/docs/source/user_guide/performance.md b/docs/source/user_guide/performance.md new file mode 100644 index 0000000000..e0aecbdedd --- /dev/null +++ b/docs/source/user_guide/performance.md @@ -0,0 +1,34 @@ +# Performance + +This page covers the execution model that the other performance guides build on: what happens when you launch a GPU kernel, where the time goes, and when reducing that time actually speeds up your program. The other pages in this section - graphs, streams, fastcache, and performance dispatch - are specific tools for reducing that cost. + +## What a GPU kernel launch is + +When you call a `@qd.kernel` from Python, the numerical work does not run inline in the calling thread. Instead Quadrants *launches* the work onto the GPU: it hands a compiled kernel plus its arguments to the GPU driver, the driver enqueues it onto a stream (an ordered queue of GPU work), and the launch call returns to the host before the GPU has finished - usually before it has even started. Host and GPU run asynchronously. + +Each top-level `for`-loop in a kernel is a separate *offloaded task*, and each offloaded task becomes one GPU kernel. So a kernel with three top-level for-loops launches three GPU kernels every time you call it. Every launch carries a fixed overhead that is independent of how much data the kernel processes - that per-launch overhead is what we call *kernel launch latency*. When a kernel crunches a lot of data the overhead is negligible; when it does little work, or when you relaunch small kernels many times in a loop, launch latency can dominate the total runtime. + +## Where kernel launch latency comes from + +It helps to split the latency of a single launch into three parts: + +- **Python side.** Everything that happens in the Python interpreter before control reaches Quadrants' C++ runtime. This is the most expensive of the three per call, and it is paid once per *Python* call - so a Python `while`/`for` loop that calls a kernel N times pays it N times. +- **C++ side.** Work inside the Quadrants runtime (C++) once control has crossed the boundary. Cheaper than the Python side, but non-zero, and paid per offloaded task. +- **GPU / driver side.** The GPU driver's own cost to place the kernel on a stream, plus the fixed scheduling latency on the GPU itself before the kernel begins running on the hardware. + +## Latency hiding + +Because host and GPU run asynchronously, kernel launches overlap with kernel execution: the host can be issuing the next launch while earlier kernels are still running on the GPU. As long as the host issues launches at least as fast as the GPU drains them, the queue never empties, the GPU stays busy, and the launch latency is fully *hidden* behind execution. + +So reducing launch latency does not always improve overall throughput. When each kernel runs long enough that its execution time exceeds the launch overhead, the GPU is the bottleneck and shaving launch latency changes nothing - the launches were already hidden behind execution. + +Kernel launch latency matters when kernels are relatively small. Then the GPU can finish a kernel before the host has issued the next one, so the GPU sits idle waiting for the next launch and the launch latency is *exposed* on the critical path. In that regime, cutting launch latency can directly increase throughput. + +## Reducing launch latency + +Quadrants provides several tools that attack launch latency in different ways: + +- [Graphs](graph.md) capture a sequence of launches into a graph and replay it as a single launch, so on backends with hardware graph support you pay the Python and C++ per-task costs once when the graph is built, rather than on every call. This is the most direct way to cut launch latency. +- [Streams](streams.md) let independent top-level for-loops run concurrently instead of serializing on the default stream, so the GPU can stay busy across otherwise-independent launches. +- [Fastcache](fastcache.md) reduces the time to load cached kernels when a new Python process starts, cutting first-call latency. +- [Performance dispatch](perf_dispatch.md) benchmarks multiple implementations of an operation at runtime and picks the fastest. From 88651248497375562c67f1cac172a320e921e827 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 08:20:03 -0700 Subject: [PATCH 33/41] Separate non-launch-latency tools in performance.md Per review, streams/fastcache/perf_dispatch do not reduce launch latency. Keep only graphs under "Reducing launch latency" and move the others to a new "Other performance tools" section. --- docs/source/user_guide/performance.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guide/performance.md b/docs/source/user_guide/performance.md index e0aecbdedd..8a15f52513 100644 --- a/docs/source/user_guide/performance.md +++ b/docs/source/user_guide/performance.md @@ -26,9 +26,12 @@ Kernel launch latency matters when kernels are relatively small. Then the GPU ca ## Reducing launch latency -Quadrants provides several tools that attack launch latency in different ways: +[Graphs](graph.md) are the main tool for reducing launch latency: they capture a sequence of launches into a single graph launch, so on backends with hardware graph support you pay the Python and C++ per-task costs once when the graph is built, rather than on every call. -- [Graphs](graph.md) capture a sequence of launches into a graph and replay it as a single launch, so on backends with hardware graph support you pay the Python and C++ per-task costs once when the graph is built, rather than on every call. This is the most direct way to cut launch latency. -- [Streams](streams.md) let independent top-level for-loops run concurrently instead of serializing on the default stream, so the GPU can stay busy across otherwise-independent launches. -- [Fastcache](fastcache.md) reduces the time to load cached kernels when a new Python process starts, cutting first-call latency. +## Other performance tools + +The rest of this section covers tools that improve performance in other ways, not by reducing launch latency: + +- [Streams](streams.md) let independent top-level for-loops run concurrently instead of serializing on the default stream. +- [Fastcache](fastcache.md) reduces the time to load cached kernels when a new Python process starts. - [Performance dispatch](perf_dispatch.md) benchmarks multiple implementations of an operation at runtime and picks the fastest. From 38c8ad1ecb82a00f62527f4a042fa4d5cd20f7fd Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 08:36:18 -0700 Subject: [PATCH 34/41] Clarify graphs do not remove per-call Python overhead in performance.md Cached graph replay collapses per-task C++/driver launch cost, but the Python kernel wrapper still runs on every call; only graph_do_while cuts the per-call Python cost by turning many iterations into one call. --- docs/source/user_guide/performance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/performance.md b/docs/source/user_guide/performance.md index 8a15f52513..8827d0b365 100644 --- a/docs/source/user_guide/performance.md +++ b/docs/source/user_guide/performance.md @@ -26,7 +26,7 @@ Kernel launch latency matters when kernels are relatively small. Then the GPU ca ## Reducing launch latency -[Graphs](graph.md) are the main tool for reducing launch latency: they capture a sequence of launches into a single graph launch, so on backends with hardware graph support you pay the Python and C++ per-task costs once when the graph is built, rather than on every call. +[Graphs](graph.md) are the main tool for reducing launch latency. On backends with hardware graph support, the kernel's whole sequence of GPU tasks is issued as a single graph launch rather than one launch per task, collapsing the per-task C++ and GPU/driver launch overhead. The per-call Python overhead is still paid on every call; to cut that too, move the loop itself into the graph with `graph_do_while` so that many iterations become a single call. ## Other performance tools From ddb03e6afc5f49fc749f384986d73589248e5a1f Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 08:54:29 -0700 Subject: [PATCH 35/41] Fix contradictory intro claim in performance.md Intro said all four tools reduce launch latency, contradicting the "Other performance tools" section; reframe them as performance tools built on the execution model. --- docs/source/user_guide/performance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/performance.md b/docs/source/user_guide/performance.md index 8827d0b365..7ad2035985 100644 --- a/docs/source/user_guide/performance.md +++ b/docs/source/user_guide/performance.md @@ -1,6 +1,6 @@ # Performance -This page covers the execution model that the other performance guides build on: what happens when you launch a GPU kernel, where the time goes, and when reducing that time actually speeds up your program. The other pages in this section - graphs, streams, fastcache, and performance dispatch - are specific tools for reducing that cost. +This page covers the execution model that the other performance guides build on: what happens when you launch a GPU kernel, where the time goes, and when reducing that time actually speeds up your program. The other pages in this section - graphs, streams, fastcache, and performance dispatch - are performance tools built on this execution model. ## What a GPU kernel launch is From cb8bca0bd29ed0702709e40e86c44bbb4659e60c Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 11:32:48 -0700 Subject: [PATCH 36/41] Reject qd.stream_parallel() inside graph=True kernels Previously stream_parallel groups inside a graph kernel were silently serialized rather than rejected, so users could combine two mutually incompatible features without warning. Raise a QuadrantsSyntaxError at compile time instead, and document that streams (both qd_stream and qd.stream_parallel) are incompatible with graphs. --- docs/source/user_guide/streams.md | 2 +- python/quadrants/lang/ast/ast_transformer.py | 6 +++++ tests/python/test_streams.py | 23 ++++++++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/source/user_guide/streams.md b/docs/source/user_guide/streams.md index a8db331bcc..ca8cf0573e 100644 --- a/docs/source/user_guide/streams.md +++ b/docs/source/user_guide/streams.md @@ -133,5 +133,5 @@ with qd.create_stream() as s: ## Limitations -- **Not compatible with graphs.** Do not pass `qd_stream` to a kernel decorated with `graph=True` (if you do, a `RuntimeError` will be raised). +- **Not compatible with graphs.** Streams cannot be combined with `graph=True` kernels: passing `qd_stream` to a graph kernel raises a `RuntimeError`, and using `qd.stream_parallel()` inside a graph kernel raises a `QuadrantsSyntaxError`. - **Not compatible with autodiff.** Do not pass `qd_stream` to a kernel that uses reverse-mode or forward-mode differentiation, or inside a `qd.ad.Tape` context (if you do, a `RuntimeError` will be raised). diff --git a/python/quadrants/lang/ast/ast_transformer.py b/python/quadrants/lang/ast/ast_transformer.py index 0297fffa8c..5020ab583d 100644 --- a/python/quadrants/lang/ast/ast_transformer.py +++ b/python/quadrants/lang/ast/ast_transformer.py @@ -1621,6 +1621,12 @@ def build_With(ctx: ASTTransformerFuncContext, node: ast.With) -> None: ) if not ctx.is_kernel: raise QuadrantsSyntaxError("qd.stream_parallel() can only be used inside @qd.kernel, not @qd.func") + kernel = ctx.global_context.current_kernel + if kernel is not None and kernel.use_graph: + raise QuadrantsSyntaxError( + "qd.stream_parallel() is not compatible with graph=True kernels. Streams and graphs cannot be " + "combined; remove graph=True or the qd.stream_parallel() block." + ) ctx.ast_builder.begin_stream_parallel() build_stmts(ctx, node.body) ctx.ast_builder.end_stream_parallel() diff --git a/tests/python/test_streams.py b/tests/python/test_streams.py index b89a3b4a42..10edee29c5 100644 --- a/tests/python/test_streams.py +++ b/tests/python/test_streams.py @@ -495,6 +495,29 @@ def bad_kernel(): bad_kernel() +@test_utils.test() +def test_stream_parallel_rejects_graph(): + """qd.stream_parallel() is incompatible with graph=True kernels.""" + from quadrants.lang.exception import QuadrantsSyntaxError + + N = 64 + a = qd.field(qd.f32, shape=(N,)) + b = qd.field(qd.f32, shape=(N,)) + + with pytest.raises(QuadrantsSyntaxError, match="not compatible with graph=True"): + + @qd.kernel(graph=True) + def bad_kernel(): + with qd.stream_parallel(): + for i in range(N): + a[i] = 1.0 + with qd.stream_parallel(): + for j in range(N): + b[j] = 2.0 + + bad_kernel() + + @test_utils.test() def test_stream_with_ndarray(): N = 1024 From 2482be308a66c90d04b61a20905490332acccee8 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 12:41:27 -0700 Subject: [PATCH 37/41] Link qd.ad.Tape/autodiff in streams.md limitations The doc-quality check reviews the full file once touched; qd.ad.Tape was flagged as an undefined term. Link it (and "autodiff") to autodiff.md. --- docs/source/user_guide/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/streams.md b/docs/source/user_guide/streams.md index ca8cf0573e..26fb80e69a 100644 --- a/docs/source/user_guide/streams.md +++ b/docs/source/user_guide/streams.md @@ -134,4 +134,4 @@ with qd.create_stream() as s: ## Limitations - **Not compatible with graphs.** Streams cannot be combined with `graph=True` kernels: passing `qd_stream` to a graph kernel raises a `RuntimeError`, and using `qd.stream_parallel()` inside a graph kernel raises a `QuadrantsSyntaxError`. -- **Not compatible with autodiff.** Do not pass `qd_stream` to a kernel that uses reverse-mode or forward-mode differentiation, or inside a `qd.ad.Tape` context (if you do, a `RuntimeError` will be raised). +- **Not compatible with [autodiff](autodiff.md).** Do not pass `qd_stream` to a kernel that uses reverse-mode or forward-mode differentiation, or inside a [`qd.ad.Tape`](autodiff.md) context (if you do, a `RuntimeError` will be raised). From 6251c9f67c4a3947b1cc937f16a0d0f899cafce2 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 13:27:13 -0700 Subject: [PATCH 38/41] Link graph=True/graphs to graph.md in streams.md limitations Doc-quality check flagged graph=True as an undefined term once streams.md was touched; link it (and "graphs") to graph.md. --- docs/source/user_guide/streams.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/streams.md b/docs/source/user_guide/streams.md index 26fb80e69a..5dc8b0cd36 100644 --- a/docs/source/user_guide/streams.md +++ b/docs/source/user_guide/streams.md @@ -133,5 +133,5 @@ with qd.create_stream() as s: ## Limitations -- **Not compatible with graphs.** Streams cannot be combined with `graph=True` kernels: passing `qd_stream` to a graph kernel raises a `RuntimeError`, and using `qd.stream_parallel()` inside a graph kernel raises a `QuadrantsSyntaxError`. +- **Not compatible with [graphs](graph.md).** Streams cannot be combined with [`graph=True`](graph.md) kernels: passing `qd_stream` to a graph kernel raises a `RuntimeError`, and using `qd.stream_parallel()` inside a graph kernel raises a `QuadrantsSyntaxError`. - **Not compatible with [autodiff](autodiff.md).** Do not pass `qd_stream` to a kernel that uses reverse-mode or forward-mode differentiation, or inside a [`qd.ad.Tape`](autodiff.md) context (if you do, a `RuntimeError` will be raised). From ee16ecfdb67fec9c3bac481975868203ccd126fd Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Tue, 14 Jul 2026 14:31:53 -0700 Subject: [PATCH 39/41] Reword graph return-value restriction to drop confusing "struct" The restriction applies to any return value (result_buffer_size > 0), not just struct returns; "struct" is the internal representation and misleads readers. Say "No return values" instead. --- docs/source/user_guide/graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/graph.md b/docs/source/user_guide/graph.md index 58f021189d..663bbcacb7 100644 --- a/docs/source/user_guide/graph.md +++ b/docs/source/user_guide/graph.md @@ -36,7 +36,7 @@ Note that graph=True is a pre-requisite for all other options presented in this ### Restrictions -- **No struct return values.** Kernels that return values (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. +- **No return values.** Kernels that return a value (e.g. `-> qd.i32`) cannot use graphs. An error is raised if `graph=True` is set on such a kernel. - **Primal kernels only.** The `graph=True` flag is applied to the primal (forward) kernel only, not its adjoint (backward). [Autodiff](autodiff.md) kernels use the normal launch path. - **Device-resident ndarrays.** Graph mode bakes device pointers into the cached graph, so all ndarray arguments must be on the GPU. Passing a host-resident ndarray raises an error. From 008566ac43dace401c0d6804def49705e6594899 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 15 Jul 2026 05:04:21 -0700 Subject: [PATCH 40/41] Cross-reference launch-latency docs between performance.md and parallelization.md parallelization.md's "Does GPU kernel launch latency matter?" section already covered latency hiding and practical launch-tuning levers (fewer/simpler params, field vs ndarray args). Link performance.md to it for those levers, and link it back to performance.md for the execution-model background, per review feedback. --- docs/source/user_guide/parallelization.md | 2 ++ docs/source/user_guide/performance.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/docs/source/user_guide/parallelization.md b/docs/source/user_guide/parallelization.md index 9d6ca98dc8..7bdae8d21f 100644 --- a/docs/source/user_guide/parallelization.md +++ b/docs/source/user_guide/parallelization.md @@ -88,6 +88,8 @@ Reducing the number and complexity kernel parameters reduces the kernel launch l - field args incur less launch latency than ndarray args - global fields incur no parameter-related launch latency +For the underlying execution model - what a launch actually involves, where the latency comes from, and when reducing it helps - see [Performance](performance.md). + ## Global memory In CUDA, there are 3 main types of memory: diff --git a/docs/source/user_guide/performance.md b/docs/source/user_guide/performance.md index 7ad2035985..7361ff063f 100644 --- a/docs/source/user_guide/performance.md +++ b/docs/source/user_guide/performance.md @@ -28,6 +28,8 @@ Kernel launch latency matters when kernels are relatively small. Then the GPU ca [Graphs](graph.md) are the main tool for reducing launch latency. On backends with hardware graph support, the kernel's whole sequence of GPU tasks is issued as a single graph launch rather than one launch per task, collapsing the per-task C++ and GPU/driver launch overhead. The per-call Python overhead is still paid on every call; to cut that too, move the loop itself into the graph with `graph_do_while` so that many iterations become a single call. +Independently of graphs, you can also shrink the launch latency itself by reducing the number and complexity of kernel parameters - field arguments are cheaper than ndarray arguments, and global fields incur no parameter-related launch latency. See [Parallelization](parallelization.md#does-gpu-kernel-launch-latency-matter) for these launch-tuning guidelines. + ## Other performance tools The rest of this section covers tools that improve performance in other ways, not by reducing launch latency: From 122102d64e285d412e007e2d926d2e6b29b547b5 Mon Sep 17 00:00:00 2001 From: Hugh Perkins Date: Wed, 15 Jul 2026 05:10:31 -0700 Subject: [PATCH 41/41] Link qd.static to its doc at first use in parallelization.md The cross-reference commit brought parallelization.md into the doc-quality check's scope, which flags qd.static as an undefined term at first use. Link it to static.md. --- docs/source/user_guide/parallelization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/parallelization.md b/docs/source/user_guide/parallelization.md index 7bdae8d21f..6f0ca06f5c 100644 --- a/docs/source/user_guide/parallelization.md +++ b/docs/source/user_guide/parallelization.md @@ -3,7 +3,7 @@ Each top-level for-loop will be parallelized, within a kernel. Under the hood, each top-level for-loop will be launched as a separate GPU kernel. A top-level for-loop can be encapsulated in one or more of the following, and still be parallelized: -- `if` statements where the conditional is `qd.static` +- `if` statements where the conditional is [`qd.static`](static.md) - inline functions (`@qd.func`) Note that adding a non-static `if` over the top of a for-loop will lead to the for-loop NOT being parallelized.