[cuda] implement max_delta_step output cap on the CUDA tree learner

[maxwbuckley]

Problem: the delta

When training with device_type="cuda", max_delta_step was silently ignored. The leaf-output cap

if (max_delta_step > 0 && std::fabs(ret) > max_delta_step) {
  ret = Common::Sign(ret) * max_delta_step;
}

exists only in the CPU FeatureHistogram::CalculateSplittedLeafOutput (the USE_MAX_OUTPUT path). The CUDA leaf-output device functions had no max_delta_step parameter and no cap.

Measured delta before this PR (400×6 synthetic data, learning_rate=1.0, num_leaves=15, 5 rounds, gpu_use_dp=true; tree-0 leaf-value spread should be ≤ 2*max_delta_step):

objective	max_delta_step	leaf spread CPU	leaf spread CUDA	cap (2·mds)	max pred delta
binary	0.05	0.100	4.150	0.1	17.7
binary	0.1	0.200	4.150	0.2	17.7
binary	0.5	1.000	4.150	1.0	18.9
regression	0.05	0.100	4.017	0.1	2.48
regression	0.1	0.200	4.017	0.2	2.23
regression	0.5	1.000	4.017	1.0	0.66

CPU enforces the cap; CUDA's leaves are unbounded — predictions diverge by up to 17.7 (raw score).

Fix

Implements the cap on the CUDA tree learner, identical to the CPU formula, threaded as a runtime double (0 = inactive) through:

• CUDALeafSplits::CalculateSplittedLeafOutput / GetLeafGain / GetSplitGains (the cap sits between the L1 step and the smoothing step, exactly like CPU; the closed-form gain shortcut is bypassed when the cap is active, matching CPU's !USE_MAX_OUTPUT && !USE_SMOOTHING condition)
• the root-leaf init kernels (cuda_leaf_splits.cu)
• every best-split-finder kernel — numerical, categorical, global-memory, and discretized variants (cuda_best_split_finder.cu)
• the root SetLeafOutput and the refit kernels (cuda_single_gpu_tree_learner.cpp/.cu)

The first commit's Log::Fatal guard is removed: max_delta_step now works on CUDA instead of being rejected.

Result: the delta after the fix

Same measurement, post-fix:

objective	max_delta_step	leaf spread CPU	leaf spread CUDA	cap (2·mds)	max pred delta
binary	0.05	0.100	0.100 ✓	0.1	0.40 †
binary	0.1	0.200	0.200 ✓	0.2	0.40 †
binary	0.5	1.000	1.000 ✓	1.0	1.27 †
regression	0.05	0.100	0.100 ✓	0.1	0.15 †
regression	0.1	0.200	0.200 ✓	0.2	0.18 †
regression	0.5	1.000	1.000 ✓	1.0	4.4e-16 ✓

And in the non-degenerate regime (regression, learning_rate=0.1, 10 rounds, min_data_in_leaf=5):

max_delta_step	max pred delta
1.0	0.0 (bit-identical)
2.0	3.3e-16
5.0	4.4e-16

† Why some configs still differ structurally: with learning_rate=1.0 + min_data_in_leaf=1 + a tiny cap, every leaf saturates at ±max_delta_step, so all candidate split gains collapse onto a plateau of equal values. CPU and CUDA then pick different — but equally optimal — splits from the plateau (ULP-level FP tie-breaking, the known lightgbm-org#6055 family; same root cause PR #13 addresses). The cap itself is enforced identically (spread columns match exactly), and training loss agrees within 0.6% in those configs. This is a property of gain plateaus, not of the max_delta_step implementation.

Tests (in test_dual.py, gated on TASK=cuda)

Test	Cases	What it pins down
test_cuda_max_delta_step_caps_outputs_like_cpu	6	leaf spread ≤ 2·mds on both backends, spreads equal — fails on the old build (CUDA spread 4.15 vs cap 0.1)
test_cuda_max_delta_step_matches_cpu_exactly	3	bit-level prediction parity (atol=1e-10) in non-saturating configs
test_cuda_max_delta_step_loss_matches_cpu_when_saturated	2	cap enforced on every tree + training loss within 2% in the plateau regime

All 11 cases pass with the fix; the cap-enforcement test fails on the unpatched build. Full test_dual.py suite (65 tests) passes.

🤖 Generated with Claude Code

[BelixRogner]

Thank you, Max — and thank you, Claude Code (independently 🙂). Verdict: SOLID, no bug found.

Reviewed against the CPU USE_MAX_OUTPUT cap in CalculateSplittedLeafOutput:

• Cap formula is equivalent, and the Common::Sign(0)=0 vs CUDA ret>=0?+md:-md difference is provably unreachable — the branch is gated on fabs(ret) > max_delta_step > 0, so ret==0 never enters it. (Cosmetically a real Sign() would read more faithfully, but behavior is identical.)
• Cap sits in the correct position: L1 step → cap → smoothing, matching CPU.
• The closed-form gain shortcut is correctly bypassed when the cap is active — the CPU compile-time !USE_MAX_OUTPUT && !USE_SMOOTHING guard is faithfully turned into a runtime !USE_SMOOTHING && max_delta_step <= 0.0 guard.
• Every site is threaded: leaf-output/gain functions, root-init kernels, all four best-split-finder families + their global-memory variants, root SetLeafOutput, and the refit kernels. I grepped every CalculateSplittedLeafOutput/GetLeafGain/GetSplitGains call — no site takes the param without applying it, none missed. min_gain_shift propagates cap-aware, consistent with CPU.
• Default 0 is a true no-op (every application guarded by max_delta_step > 0).

Your gain-plateau scope note is honest: the structural divergence at learning_rate=1.0 is the expected ULP-tie signature, not a cap defect — the spread-exact and bit-parity assertions would break if it were a cap bug.

Non-blocking: no path_smooth>0 + cap, categorical, or lambda_l1>0 + cap test; core cap behavior is well covered.

Blocker is only the required ruff format + git merge master (these five overlap). macOS red = dask socket flake.

P.S. — you threaded max_delta_step through ~9 files and a dozen kernels without missing one, then capped your output at "ran the linter: no". 😄 ruff format has a step cap of exactly one command.

[BelixRogner]

Thank you, Max — and thank you, Claude Code (independently 🙂). Rebase nudge 👉

This one reviewed out SOLID — it's only CONFLICTING because master moved under it (forced-splits #17 landed, plus a stack of best-split-finder / data-partition merges: #21–#27). A rebase onto current master should clear it cleanly.

You have write access now (accept the repo invite if you haven't), so once it's rebased and green you can merge it yourself. 🚀

[maxwbuckley]

Rebased onto current master — but this one was materially re-implemented, not just replayed. Please re-review the diff properly rather than trusting the earlier "SOLID". 👋

Why it changed. Your 2b7e525f refactor moved the split-gain / leaf-output math into the shared SplitGainMath core (include/LightGBM/tree_split_math.h), and CalculateLeafOutput there already implements the max_delta_step cap via USE_MAX_OUTPUT — CUDA was just passing false / 0.0. The original PR hand-rolled its own copy of that math, so replaying it verbatim would have re-duplicated exactly what you'd deduplicated. I threw the inline copy away and wired the cap through the shared core instead.

Three things worth your eyes:

1. SplitGainMath::LeafGain gained a USE_MAX_OUTPUT parameter (tree_split_math.h). It previously hardcoded false. The new body mirrors CPU's FeatureHistogram::GetLeafGain exactly: closed-form g²/(h+l2) only when !USE_MAX_OUTPUT && !USE_SMOOTHING, otherwise gain measured at the actual output. CUDA is its only caller, so nothing else moves.

2. The cap is dispatched at runtime, not as a 4th template dimension. Adding USE_MAX_OUTPUT to the kernel template chain would double every split-finder instantiation. Instead CUDALeafSplits picks the specialisation from max_delta_step > 0.0.

   The two branches must not be collapsed into one: the capped path measures gain at the output, which is algebraically but not bitwise equal to the closed form CPU uses when max_delta_step is unset. Keeping both preserves CPU/CUDA bit-parity for the common max_delta_step == 0 case — the full suite passing unchanged is the evidence.

3. ComputeForcedSplitKernel now takes max_delta_step. Forced splits (your [cuda] implement forced splits (forcedsplits_filename) on the CUDA tree learner #17) landed after this PR was written, and its leaf-output/gain computation was silently ignoring the cap. The build caught it. It now honours max_delta_step like every other split path.

Validation (RTX 5090, CUDA 13.2, built from source):

• 11 max_delta_step tests pass (cap enforced, CPU-exact, saturated-loss parity)
• full test_dual.py: 130 passed, 1 skipped — no regressions, i.e. the max_delta_step == 0 path is still bit-identical to CPU
• cpplint clean on all changed C++; ruff check / ruff format clean

Rebased, re-implemented and validated with Claude Code. CI has no GPU, so none of the above is exercised by CI — these are real GPU runs.