diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 000000000000..5b592e14a86c --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,158 @@ +# ExaBoost CUDA roadmap + +Ideas from the hybrid-growth + benchmark session (2026-07-13/14, PRs #32/#33), roughly +priority-ordered within groups. Measurements refer to an RTX 5090, per-tree/per-100-tree +figures from the profiles in the PR discussions. + +## Performance + +- [ ] **Graphs L1 — device-driven level loop.** Wrap the per-level pipeline in a CUDA + conditional-WHILE graph node; a level-controller kernel builds the next level's pair + descriptors on-device, updates grid dims via device-updatable node params, and sets the + loop condition; ONE host sync per tree. Est. ~1.6-1.7x on small-tree workloads + (fraud-class: ~344 us/tree turnaround idle + ~234 us launch API measured), ~1.3x mid + (covtype/year), ~nil on roofline-bound data (higgs/epsilon). Depth-limited exact regime + first; the budget-limited selective mode keeps its host loop until the selection + + tie-break replication is ported to device and re-verified against the quant md5 gates. +- [ ] **Graphs L2 — per-parent dependency chaining.** A child pair only depends on *its + parent's* partition + histogram, not on its level. Launch each pair's chain from the + device (fire-and-forget device graph launch) when its parent finishes — a + self-scheduling tree that removes phase-boundary tail effects on unbalanced trees. + Needs: device-atomic budget reservation + a global final-level top-K join, per-node + (not per-level) task/scratch regions, and an order-independent split log (the host + rebuild already renumbers canonically, absorbing completion-order nondeterminism). + Est. +10-20% over L1 on unbalanced small trees. +- [ ] **Static planner (auto-tuner tier 0).** At Init the dataset shape (rows, + features, actual bins/feature) and config (num_leaves, max_depth, num_class, + iterations) determine: expected level geometry (leaf sizes ~ rows/2^level) -> grid + configs, small-leaf threshold, speculative bounds; histogram partition packing and + quant bit thresholds from the real bin histogram; pipeline selection; and a compact + per-tree histogram layout for feature_fraction runs (static version of the 161fe88b + dead-entry mask). Precedent: CPU LightGBM's row/col-wise chooser, EFB, multi-val bin + packing. Decides only provable shape functions; supplies priors for the ambiguous + constants below (GPU cost models are brittle: the construct-floor cap gained + year/higgs 35% and regressed covtype 45% -- only measurement caught it). +- [ ] **Runtime auto-tuner ("JIT optimizer") tier 1 — online policy tuning.** Boosting + runs thousands of near-identical trees: measure per-tree wall time (CUDA events) + + feedback stats (churn, level widths, imbalance) and bandit-tune the existing + dataset-dependent knobs: batched-construct grid sizing (EXABOOST_BATCH_CONSTRUCT_FLOOR + -- covtype is latency-bound, year/higgs merge-bound), small-leaf construct threshold + (EXABOOST_SMALL_LEAF_ROWS), hist pipeline count, and the selective-growth speculation + policy (e.g. gain-margin gating for unbalanced trees). Speculation is model-invariant + by monotonicity, and quant-mode integer histograms keep md5 locks valid under any + schedule retuning -- the tuner cannot break exactness gates. Hysteresis vs noise; + decision logging. +- [ ] **Runtime auto-tuner tier 2 -- NVRTC shape-specialized kernels.** JIT-compile + construct/find kernels at Dataset construction with columns / per-feature bin counts + baked in (precedent: LightGBM's OpenCL backend JIT-compiled with #defined bin counts). + Star case: numerai's ~5.5-bin features (5 quintiles + a missing-marker bin on ~half + of them) waste >90% of the fixed 12288-entry shared histogram; a specialized kernel + packs ~10x more features per partition -> fewer + partitions, less shared->global merge traffic. One-time ~0.5s compile amortized over + thousands of trees; needs AOT fallback. +- [ ] **Runtime auto-tuner tier 3 -- persisted tuning cache**: store best-found configs + keyed by dataset-shape signature (FFTW-wisdom style) so retrains skip exploration. +- [ ] **Selective-growth churn reduction.** covtype 64/12 applies 2.09x the final split + count (52% displaced-then-pruned). Smarter speculation — e.g. only apply candidates + with a selection margin / hysteresis — to cut wasted search+apply. +- [x] **Wide-shape batched search** (161fe88b): multi-block batched find/sync for + num_tasks > 1024 (bit-identical reduction), compact-view batched construct, + bin-used-mask skipping of dead histogram entries, learner gather from the compact + matrix. numerai 4096/13: 39.4 -> 27.2s; example shape per-tree 41.6 -> 34.8ms. +- [x] **Numerai first-train setup** (22331b18): Booster create 13.9s -> 2.3s. Dense CUDA + row data now builds directly from column bins (tiled transpose, no zero-fills) and the + host multi-val bin is skipped for dense non-multi-val datasets (-14.6 GB peak RSS). + Remaining: ~0.8s pageable 15 GB H2D (pinned staging ring / transpose into pinned) and + ~1.2s host transpose (device-side transpose candidate); sparse row-wise CUDA datasets + still build the host multi-val bin. Kill switch EXABOOST_FAST_ROWDATA=0. +- [x] **Native small-int ingestion (int8/int16)** (9c0f5ffa): int8/int16 numpy + matrices (row- or col-major) pass zero-copy through the C API and are binned by + per-column LUTs via Bin::PushBlock; bins identical to the float path by construction + (md5-verified CUDA quant + CPU deterministic). numerai fed int8: construct 38.9 -> + 15.8s, peak RSS 86.4 -> 43.9 GB. Measured separately from the cross-library matrix + (which stays float32-fed for fairness). Follow-ups: EFB FindGroups is now the + dominant construct cost (~9s of 15.8s on 2748 features); pandas int frames still + convert to float32 (only raw numpy passes through); CSR/CSC + streaming push and + uint8 still use the double path. +- [x] **4-bit packed CUDA row/compact data for <=16-bin datasets** (354ceef3): row + matrix AND per-tree compact matrix packed two cells/byte; all construct/fill/gather + kernels unpack via IS_4BIT variants. numerai: 2000-tree train 87.6 -> 66.3s (-24%), + device memory 19.3 -> 11.5 GB, Booster create 2.25 -> 1.44s, gather source at 1.47 + TB/s. Kill switch EXABOOST_ROWDATA_4BIT=0; verify env checks unpacked equality. + Both follow-ups landed in 45eea4c6 (numerai per-tree 34.4 -> 30.7 ms, -10.6%; + 2000-tree ~60s): (a) float2 grad/hess interleave (EXABOOST_GH_INTERLEAVE, + compile-time-template dispatch -- a runtime branch cost ~1 ms in both modes), + modest (-0.5..1 ms) since grad/hess was the smaller half of scattered traffic; + (b) split kernels read the packed compact matrix directly + (EXABOOST_SPLIT_PACKED_READ), dropping the per-tree 1.5 GB row-to-col gather + (-4.2 ms/tree); classic per-split consumers lazily materialize the old view once + per tree. Next on this path: CUDAFillCompactData4BitKernel is now the #2 kernel + (5.3 ms/tree); the construct kernel (22.7 ms, ~74% of tree time) remains + latency-bound on the packed-bin row gathers themselves. Measurement hygiene: + desktop wall clocks drift up to ~15% between sessions -- trust same-session A/B + per-tree medians (or lock clocks) for future numerai perf work. +- [ ] **GPU-native dataset construction, default for CUDA (CuPy/cuDF ingestion, + QuantileDMatrix parity and beyond).** Whenever device_type=cuda and the input is + dense: sample -> host GreedyFindBin (identical boundaries, md5-comparable), then + stream row chunks through a pinned staging ring and bin ON DEVICE (LUT for small + ints, boundary search for floats) directly into the packed 4-bit partitioned + structures -- raw data never fully resident (numerai f32 is 60 GB vs 32 GB VRAM). + Accept __cuda_array_interface__ (CuPy; cuDF rides along) to skip the upload + entirely. Est. numerai construct: ~1s (CuPy int8), ~2-3s (numpy int8), ~5-6s + (numpy f32) vs 15.8/37.4s today; frees ~40 GB host RSS. Host bins become lazy + (linear_trees, save_binary, refit fall back). 22331b18/354ceef3 already removed + the host multi-val-bin dependency, so CUDARowData/compact are the only consumers. +- [ ] **EFB density precheck.** FindGroups spends ~9s on numerai's 2748 dense + features and (almost certainly) bundles nothing -- bundling exploits sparse + mutual exclusivity. Detect dense/no-op cases cheaply and skip; the GPU-native + constructor skips it by design. +- [ ] **Latency-bound construct on tiny-bin wide data**: post-161fe88b numerai construct + is scattered-read latency-bound (19ms/tree) -- candidate for NVRTC shape + specialization (auto-tuner tier 2) or layout changes. +- [ ] **Quant one-sync parity.** The quantized path still uses the two-sync level flow; + extend the one-sync speculative pipeline to it. Also investigate the per-tree gradient + discretization cost on many-tree/small-tree configs (numerai-quant is slower than + non-quant today). +- [ ] **FP32 gain math in the find kernels** (priority up: Felix accepts + nondeterminism for speed). find ~= 136 us/tree, FP64-bound and leaf-size-independent; + consumer Blackwell runs FP64 at 1/64 the FP32 rate, so this is the one genuinely + FLOPs-bound kernel. FP32/mixed gains behind a flag; verify quality via the benchmark + harness, not md5. +- [ ] **FP32-atomic histogram mode** (priority up, same rationale). Non-quant entries + are FP64 pairs (16 B/bin); an fp32 pair (8 B/bin) matches quant bandwidth with ZERO + per-tree discretization overhead -- candidate fastest mode for many-tree configs + (numerai-quant regressed to 155s vs 86s non-quant partly on discretizer cost). + Nondeterministic and cancellation-bias risk: judge by measured quality per dataset. + +## Correctness / determinism + +- [ ] **Deterministic non-quant CUDA mode** (deprioritized: determinism is a + verification tool here, not a production requirement -- quant mode already provides + it for md5 gates). Fixed-point integer histogram accumulation (XGBoost-style) would + eliminate float-atomic run-to-run jitter (measured +-2.6pp on covtype multiclass). + Related: `deterministic=true` silently no-ops on CUDA — make it work or warn; it also + doesn't pin the timing-based col/row-wise auto-choice on CPU (bimodal md5s; pin + force_col_wise in gates). + +## Upstream (lightgbm-org/LightGBM) bugs found (documented here for reference; +## we do not contribute upstream) + +- Quantized CUDA int32 histogram-index overflow on wide data (fixed here in + 6f8402f5; upstream segfaults at scale and silently corrupts below it). +- Quantized multiclass per-tree random-offset buffer overrun (fixed here in + 1b28ba03). +- CUDA-vs-CPU growth parity: upstream CUDA over-grows trees ~3.5x vs its own CPU at + identical params (854 vs 237 leaves/tree on covtype; lambda_l2 sweep shows it is + systematic). +- Latent race in the classic loop: child leaf-splits structs point into per-split + scratch that the next split overwrites; masked only by per-split syncs (fixed here + via point_structs_at_main + copy-event ordering). + +## Benchmark + +- [ ] Airline (115M rows): kt.ijs.si down; retry, find a mirror, or substitute the + benchm-ml 10M-row variant. +- [ ] Optional: xgboost-lossguide as a seventh benchmark config (leaf-wise + apples-to-apples column). +- [ ] Harness: host-memory guard for runs near RAM limits (xgboost/numerai peaked at + 102 GB and OOM-killed the host session; its curve cell is skipped). diff --git a/include/LightGBM/bin.h b/include/LightGBM/bin.h index b4a4fc198b50..5b29c95c888a 100644 --- a/include/LightGBM/bin.h +++ b/include/LightGBM/bin.h @@ -297,6 +297,25 @@ class Bin { */ virtual void Push(int tid, data_size_t idx, uint32_t value) = 0; + /*! \brief Sentinel for pre-encoded bins that must not be pushed (most frequent bin) */ + static constexpr uint32_t kSkipBin = std::numeric_limits::max(); + + /*! + * \brief Push a block of pre-encoded bins for consecutive records, skipping ``kSkipBin`` entries. + * Values must already be encoded exactly as FeatureGroup::PushData would push them. + * \param tid Thread id + * \param start_idx Index of the first record + * \param count Number of records + * \param bins Encoded bin values of the records + */ + virtual void PushBlock(int tid, data_size_t start_idx, data_size_t count, const uint32_t* bins) { + for (data_size_t i = 0; i < count; ++i) { + if (bins[i] != kSkipBin) { + Push(tid, start_idx + i, bins[i]); + } + } + } + virtual void CopySubrow(const Bin* full_bin, const data_size_t* used_indices, data_size_t num_used_indices) = 0; /*! * \brief Get bin iterator of this bin for specific feature diff --git a/include/LightGBM/c_api.h b/include/LightGBM/c_api.h index ad96e46a1a97..fa2c66e73a3c 100644 --- a/include/LightGBM/c_api.h +++ b/include/LightGBM/c_api.h @@ -38,6 +38,7 @@ typedef void* ByteBufferHandle; /*!< \brief Handle of ByteBuffer. */ #define C_API_DTYPE_INT32 (2) /*!< \brief int32. */ #define C_API_DTYPE_INT64 (3) /*!< \brief int64. */ #define C_API_DTYPE_INT8 (4) /*!< \brief int8. */ +#define C_API_DTYPE_INT16 (5) /*!< \brief int16. */ #define C_API_PREDICT_NORMAL (0) /*!< \brief Normal prediction, with transform (if needed). */ #define C_API_PREDICT_RAW_SCORE (1) /*!< \brief Predict raw score. */ @@ -407,7 +408,7 @@ LIGHTGBM_C_EXPORT int LGBM_DatasetCreateFromCSC(const void* col_ptr, /*! * \brief Create dataset from dense matrix. * \param data Pointer to the data space - * \param data_type Type of ``data`` pointer, can be ``C_API_DTYPE_FLOAT32`` or ``C_API_DTYPE_FLOAT64`` + * \param data_type Type of ``data`` pointer, can be ``C_API_DTYPE_FLOAT32``, ``C_API_DTYPE_FLOAT64``, ``C_API_DTYPE_INT8`` or ``C_API_DTYPE_INT16`` * \param nrow Number of rows * \param ncol Number of columns * \param is_row_major 1 for row-major, 0 for column-major @@ -429,7 +430,7 @@ LIGHTGBM_C_EXPORT int LGBM_DatasetCreateFromMat(const void* data, * \brief Create dataset from array of dense matrices. * \param nmat Number of dense matrices * \param data Pointer to the data space - * \param data_type Type of ``data`` pointer, can be ``C_API_DTYPE_FLOAT32`` or ``C_API_DTYPE_FLOAT64`` + * \param data_type Type of ``data`` pointer, can be ``C_API_DTYPE_FLOAT32``, ``C_API_DTYPE_FLOAT64``, ``C_API_DTYPE_INT8`` or ``C_API_DTYPE_INT16`` * \param nrow Number of rows * \param ncol Number of columns * \param is_row_major Pointer to the data layouts. 1 for row-major, 0 for column-major diff --git a/include/LightGBM/cuda/cuda_column_data.hpp b/include/LightGBM/cuda/cuda_column_data.hpp index a4bc196548c3..772507d51fa9 100644 --- a/include/LightGBM/cuda/cuda_column_data.hpp +++ b/include/LightGBM/cuda/cuda_column_data.hpp @@ -113,6 +113,35 @@ class CUDAColumnData { // Needed when feature_fraction = 1.0 / no compaction (rare path). void RestoreOriginalColumnView(); + // ===== Per-tree packed compact column view (4-bit) ===== + // Register the histogram constructor's 4-bit packed compact matrix as this + // tree's column source: column c's bin for row r is + // (packed_column_data(c)[r * packed_column_stride(c)] >> packed_column_shift(c)) & 0xf. + // Only the batched apply path (CUDAHybridApplyDescriptor, bit_type 4) reads + // it; callers needing a plain per-column buffer must first re-register one + // via SetCompactColumnView (see the tree learner's lazy classic fallback). + // Also nulls compact_column_host_view_ so a stale GetColumnData pointer can + // never be consumed silently. + void SetCompactPackedColumnView(const std::vector& column_to_compact_slot, + const uint8_t* packed_buf, + const std::vector& slot_base_byte, + const std::vector& slot_row_stride, + const std::vector& slot_shift); + + bool packed_column_view_active() const { return packed_column_view_active_; } + + const uint8_t* packed_column_data(const int column_index) const { + return packed_column_ptr_[column_index]; + } + + int packed_column_stride(const int column_index) const { + return packed_column_stride_[column_index]; + } + + uint8_t packed_column_shift(const int column_index) const { + return packed_column_shift_[column_index]; + } + // Skip per-column allocation in Init? Used when caller will provide compact view. bool init_skipped_per_column_alloc_ = false; @@ -160,6 +189,13 @@ class CUDAColumnData { // or nullptr for non-sampled columns. Populated by SetCompactColumnView so that // GetColumnData has a valid host-readable device pointer for the split column. std::vector compact_column_host_view_; + // Per-tree packed compact view (see SetCompactPackedColumnView): host-side + // per-column device base pointer / per-row byte stride / nibble shift. Only + // consumed host-side when building batched apply descriptors. + bool packed_column_view_active_ = false; + std::vector packed_column_ptr_; + std::vector packed_column_stride_; + std::vector packed_column_shift_; CUDAVector cuda_column_bit_type_; CUDAVector cuda_feature_min_bin_; diff --git a/include/LightGBM/cuda/cuda_row_data.hpp b/include/LightGBM/cuda/cuda_row_data.hpp index a1f3b5600a26..3503f331b008 100644 --- a/include/LightGBM/cuda/cuda_row_data.hpp +++ b/include/LightGBM/cuda/cuda_row_data.hpp @@ -68,6 +68,16 @@ class CUDARowData { uint8_t bit_type() const { return bit_type_; } + /*! \brief whether cuda_data_uint8_t_ stores two bin values per byte (see + * InitDense4BitData for the layout invariant); bit_type() stays 8 */ + bool is_4bit_packed() const { return is_4bit_packed_; } + + /*! \brief prefix over partitions of the packed per-row byte widths + * (ceil(num_columns_in_partition / 2)); only valid when is_4bit_packed() */ + const int* cuda_packed_partition_byte_offsets() const { return cuda_packed_partition_byte_offsets_.RawData(); } + + const std::vector& host_packed_partition_byte_offsets() const { return packed_partition_byte_offsets_; } + uint8_t row_ptr_bit_type() const { return row_ptr_bit_type_; } const int* cuda_feature_partition_column_index_offsets() const { return cuda_feature_partition_column_index_offsets_.RawData(); } @@ -89,6 +99,36 @@ class CUDARowData { private: void DivideCUDAFeatureGroups(const Dataset* train_data, TrainingShareStates* share_state); + /*! \brief Initialize the dense row-wise data. Prefers the fast build directly from + * the Dataset's column bins (EXABOOST_FAST_ROWDATA, default on); host_data (the + * row-wise multi-val bin) may be nullptr when Dataset::GetShareStates skipped its build. */ + template + void InitDenseData(const Dataset* train_data, const BIN_TYPE* host_data, CUDAVector* cuda_data); + + /*! \brief Initialize the dense row-wise data in 4-bit packed form (two bin + * values per byte; every column has <= 16 bins). See the implementation for + * the per-partition alignment invariant. */ + void InitDense4BitData(const Dataset* train_data, const uint8_t* host_data); + + /*! \brief Packed variant of BuildDensePartitionedFromColumns: writes nibbles. */ + void BuildDensePacked4BitFromColumns(const std::vector& column_data, + const std::vector& column_bit_types, uint8_t* out_data) const; + + /*! \brief Pack an 8-bit partitioned row-major buffer into the 4-bit layout. */ + void Pack4BitFromPartitioned(const uint8_t* unpacked, uint8_t* packed) const; + + /*! \brief Collect the raw per-column bin data in multi-val column order. + * Returns false (fast build not applicable) on multi-val or sparse groups. */ + bool CollectDenseColumnData(const Dataset* train_data, + std::vector* column_data, + std::vector* column_bit_types) const; + + /*! \brief Build the partitioned row-major buffer directly from column bins + * with a cache-tiled parallel transpose, bypassing the host multi-val bin. */ + template + void BuildDensePartitionedFromColumns(const std::vector& column_data, + const std::vector& column_bit_types, BIN_TYPE* out_data) const; + template void GetDenseDataPartitioned(const BIN_TYPE* row_wise_data, std::vector* partitioned_data); @@ -126,6 +166,10 @@ class CUDARowData { uint8_t row_ptr_bit_type_; /*! \brief is sparse row wise data */ bool is_sparse_; + /*! \brief cuda_data_uint8_t_ stores two bin values per byte (all columns <= 16 bins) */ + bool is_4bit_packed_ = false; + /*! \brief prefix over partitions of packed per-row byte widths (size P+1) */ + std::vector packed_partition_byte_offsets_; /*! \brief start column index of each feature partition */ std::vector feature_partition_column_index_offsets_; /*! \brief histogram offset of each column */ @@ -182,6 +226,8 @@ class CUDARowData { CUDAVector cuda_partition_ptr_uint64_t_; /*! \brief start column index of each feature partition */ CUDAVector cuda_feature_partition_column_index_offsets_; + /*! \brief prefix over partitions of packed per-row byte widths (4-bit mode) */ + CUDAVector cuda_packed_partition_byte_offsets_; /*! \brief histogram offset of each column */ CUDAVector cuda_column_hist_offsets_; /*! \brief hisotgram offset of each partition */ diff --git a/include/LightGBM/cuda/cuda_tree.hpp b/include/LightGBM/cuda/cuda_tree.hpp index 0de00a9e4d40..29b7cbc977ba 100644 --- a/include/LightGBM/cuda/cuda_tree.hpp +++ b/include/LightGBM/cuda/cuda_tree.hpp @@ -26,6 +26,47 @@ __device__ int8_t GetMissingTypeCUDA(int8_t decision_type); __device__ bool IsZeroCUDA(double fval); +/*! \brief Per-split descriptor for the batched tree-structure update of the hybrid + * level-batched growth phase: one SplitBatchKernel launch applies all numerical + * splits of a level (splits indexed by blockIdx.x). All fields are host-known + * before the level is applied; split VALUES (sums, gains, leaf outputs) come from + * the per-leaf device CUDASplitInfo pointer. */ +struct CUDATreeBatchSplit { + /*! \brief the leaf being split (left child keeps this index) */ + int leaf_index; + /*! \brief tree num_leaves at the time of this split (== the right child's leaf + * index; the new internal node index is num_leaves_at_split - 1) */ + int num_leaves_at_split; + int real_feature_index; + double real_threshold; + /*! \brief MissingType of the split feature, stored as int for POD-ness */ + int missing_type; + const CUDASplitInfo* split_info; +}; + +/*! \brief One split of a host-side tree rebuild (RebuildFromHostSplits): all the + * values SplitKernel would read from the device CUDASplitInfo, captured on the + * host. Used by the selective (grow-then-prune) hybrid growth, which replays the + * final greedy split sequence host-side after pruning displaced splits. */ +struct CUDATreeHostSplit { + /*! \brief the leaf being split, in FINAL (classic leaf-wise) numbering */ + int leaf_index; + int real_feature_index; + double real_threshold; + /*! \brief MissingType of the split feature, stored as int for POD-ness */ + int missing_type; + int inner_feature_index; + uint32_t threshold_in_bin; + double gain; + uint8_t default_left; + double left_sum_hessians; + double right_sum_hessians; + data_size_t left_count; + data_size_t right_count; + double left_value; + double right_value; +}; + class CUDATree : public Tree { public: /*! @@ -33,9 +74,29 @@ class CUDATree : public Tree { * \param max_leaves The number of max leaves * \param track_branch_features Whether to keep track of ancestors of leaf nodes * \param is_linear Whether the tree has linear models at each leaf + * \param pooled_device_buffer Optional externally owned device slab of at least + * PooledDeviceBufferSize(max_leaves) bytes. When given, all per-tree + * device arrays are carved from it as non-owning views instead of ~17 + * individual cudaMallocs, and ToHost() reads them back with ONE fused + * D2H copy. The caller must guarantee exclusive use of the slab until + * ToHost() has been called (the tree learner trains one tree at a time + * and calls ToHost() before returning it). Only cuda_leaf_value_ is + * needed after ToHost(); it is materialized into owned memory there. */ explicit CUDATree(int max_leaves, bool track_branch_features, bool is_linear, - const int gpu_device_id, const bool has_categorical_feature); + const int gpu_device_id, const bool has_categorical_feature, + uint8_t* pooled_device_buffer = nullptr); + + /*! \brief bytes needed by the pooled per-tree device arrays (16 arrays of + * max_leaves entries + the batched split descriptors), 8-byte aligned */ + static size_t PooledDeviceBufferSize(const int max_leaves) { + const size_t L = static_cast(max_leaves); + const size_t num_batch_splits = L / 2 + 2; + return 5 * sizeof(double) * L + // threshold, internal_weight/value, leaf_value/weight + sizeof(CUDATreeBatchSplit) * num_batch_splits + // batched split descriptors (8-byte aligned) + 10 * 4 * L + // int / uint32_t / data_size_t / float arrays + ((sizeof(int8_t) * L + 7) / 8) * 8; // decision_type, padded + } explicit CUDATree(const Tree* host_tree); @@ -47,6 +108,27 @@ class CUDATree : public Tree { const MissingType missing_type, const CUDASplitInfo* cuda_split_info); + /*! \brief Apply all numerical splits of one level in a single kernel launch + * (device tree-structure updates) plus the usual host-side mirror updates + * (num_leaves_, leaf_depth_, branch features). Equivalent to calling Split() + * once per entry in order: splits[k].num_leaves_at_split must equal the tree's + * num_leaves before split k is applied (right children take consecutive + * indices). The level's splits touch disjoint leaves, so the device updates + * are safe to run concurrently. */ + void SplitBatch(const std::vector& splits); + + /*! \brief Build the final tree structure host-side from a replayed split + * sequence (selective grow-then-prune hybrid growth). The device tree arrays + * were never written during growth; this replays SplitKernel's per-field + * arithmetic on the host mirrors (bit-identical: same IEEE double operations + * on the same captured CUDASplitInfo values), uploads the final leaf values + * to the device (AddPredictionToScore / Shrinkage / renewal kernels read + * them), and then performs ToHost()'s cleanup (host vector shrink, leaf + * value materialization, device array release, stream destroy). The caller + * must NOT call ToHost() afterwards. leaf_value_[0] must hold the root + * output before the call (it seeds the internal_value_ chain). */ + void RebuildFromHostSplits(const std::vector& splits); + int SplitCategorical( const int leaf_index, const int real_feature_index, @@ -111,7 +193,12 @@ class CUDATree : public Tree { void SyncLeafOutputFromCUDAToHost(); private: - void InitCUDAMemory(); + /*! \brief shared tail of ToHost() and RebuildFromHostSplits(): shrink the host + * vectors to num_leaves_, materialize the retained device leaf values and + * release every other per-tree device array + the per-tree stream */ + void ShrinkHostVectorsAndReleaseDevice(); + + void InitCUDAMemory(uint8_t* pooled_device_buffer); void InitCUDA(); @@ -121,6 +208,8 @@ class CUDATree : public Tree { const MissingType missing_type, const CUDASplitInfo* cuda_split_info); + void LaunchSplitBatchKernel(const int num_splits); + void LaunchSplitCategoricalKernel( const int leaf_index, const int real_feature_index, @@ -161,6 +250,10 @@ class CUDATree : public Tree { CUDAVector cuda_bitset_inner_; CUDAVector cuda_cat_boundaries_; CUDAVector cuda_cat_boundaries_inner_; + /*! \brief device copy of one level's batched split descriptors (SplitBatch) */ + CUDAVector cuda_batch_splits_; + /*! \brief whether the per-tree arrays are views into a pooled slab */ + bool pooled_ = false; cudaStream_t cuda_stream_; diff --git a/include/LightGBM/cuda/cuda_utils.hu b/include/LightGBM/cuda/cuda_utils.hu index cb37f3c77ffb..d7aaba044199 100644 --- a/include/LightGBM/cuda/cuda_utils.hu +++ b/include/LightGBM/cuda/cuda_utils.hu @@ -69,6 +69,14 @@ void CopyFromHostToCUDADevice(T* dst_ptr, const T* src_ptr, size_t size, const c CUDASUCCESS_OR_FATAL_OUTER(cudaMemcpy(void_dst_ptr, void_src_ptr, size_in_bytes, cudaMemcpyHostToDevice)); } +template +void CopyFromHostToCUDADeviceAsync(T* dst_ptr, const T* src_ptr, size_t size, cudaStream_t stream, const char* file, const int line) { + void* void_dst_ptr = reinterpret_cast(dst_ptr); + const void* void_src_ptr = reinterpret_cast(src_ptr); + size_t size_in_bytes = size * sizeof(T); + CUDASUCCESS_OR_FATAL_OUTER(cudaMemcpyAsync(void_dst_ptr, void_src_ptr, size_in_bytes, cudaMemcpyHostToDevice, stream)); +} + template void InitCUDAMemoryFromHostMemory(T** dst_ptr, const T* src_ptr, size_t size, const char* file, const int line) { AllocateCUDAMemory(dst_ptr, size, file, line); @@ -133,10 +141,12 @@ class CUDAVector { CUDAVector() { size_ = 0; data_ = nullptr; + owns_ = true; } explicit CUDAVector(size_t size) { size_ = size; + owns_ = true; AllocateCUDAMemory(&data_, size_, __FILE__, __LINE__); } @@ -154,9 +164,10 @@ class CUDAVector { const size_t size_for_old_content = std::min(size_, size); CopyFromCUDADeviceToCUDADevice(new_data, data_, size_for_old_content, __FILE__, __LINE__); } - DeallocateCUDAMemory(&data_, __FILE__, __LINE__); + ReleaseStorage(); data_ = new_data; size_ = size; + owns_ = true; } void InitFromHostVector(const std::vector& host_vector) { @@ -170,12 +181,37 @@ class CUDAVector { } void Clear() { - if (size_ > 0 && data_ != nullptr) { - DeallocateCUDAMemory(&data_, __FILE__, __LINE__); - } + ReleaseStorage(); size_ = 0; + owns_ = true; + } + + /*! \brief Point this vector at externally owned device memory (e.g. a pooled + * slab shared by short-lived objects). A view is never freed by this class; + * any mutating reallocation (Resize/PushBack) copies the view's content into + * fresh owned memory first, and Materialize() does so explicitly. */ + void SetView(T* data, size_t size) { + ReleaseStorage(); + data_ = data; + size_ = size; + owns_ = false; } + /*! \brief Convert a view into owned storage (copying the content); no-op on + * already-owned storage. */ + void Materialize() { + if (owns_ || data_ == nullptr || size_ == 0) { + return; + } + T* new_data = nullptr; + AllocateCUDAMemory(&new_data, size_, __FILE__, __LINE__); + CopyFromCUDADeviceToCUDADevice(new_data, data_, size_, __FILE__, __LINE__); + data_ = new_data; + owns_ = true; + } + + bool IsView() const { return !owns_; } + void PushBack(const T* values, size_t len) { T* new_data = nullptr; AllocateCUDAMemory(&new_data, size_ + len, __FILE__, __LINE__); @@ -183,17 +219,25 @@ class CUDAVector { CopyFromCUDADeviceToCUDADevice(new_data, data_, size_, __FILE__, __LINE__); } CopyFromCUDADeviceToCUDADevice(new_data + size_, values, len, __FILE__, __LINE__); - DeallocateCUDAMemory(&data_, __FILE__, __LINE__); + ReleaseStorage(); size_ += len; data_ = new_data; + owns_ = true; } size_t Size() const { return size_; } + /*! \brief O(1) exchange of the underlying device buffers (double-buffering) */ + void Swap(CUDAVector* other) { + std::swap(data_, other->data_); + std::swap(size_, other->size_); + std::swap(owns_, other->owns_); + } + ~CUDAVector() { - DeallocateCUDAMemory(&data_, __FILE__, __LINE__); + ReleaseStorage(); } std::vector ToHost() { @@ -230,8 +274,19 @@ class CUDAVector { } private: + /*! \brief free owned storage; views are simply detached */ + void ReleaseStorage() { + if (owns_) { + DeallocateCUDAMemory(&data_, __FILE__, __LINE__); + } else { + data_ = nullptr; + } + } + T* data_; size_t size_; + /*! \brief false when data_ is a view into externally owned memory */ + bool owns_; }; template diff --git a/include/LightGBM/dataset.h b/include/LightGBM/dataset.h index 72256da32da0..2792be8fb2b6 100644 --- a/include/LightGBM/dataset.h +++ b/include/LightGBM/dataset.h @@ -589,6 +589,21 @@ class Dataset { } } + /*! + * \brief Push a dense block of rows of a small integer type via per-column + * value->bin lookup tables, bypassing the per-value double conversion + * and bin binary search. Bit-identical to pushing every value through + * PushOneRow. + * \param data Pointer to the block's values + * \param nrow Number of rows in the block + * \param ncol Number of columns + * \param is_row_major 1 for row-major data, 0 for column-major + * \param start_row Dataset row index of the block's first row + */ + template + void PushDenseSmallIntRows(const T* data, int32_t nrow, int32_t ncol, + int is_row_major, data_size_t start_row); + inline void PushOneRow(int tid, data_size_t row_idx, const std::vector& feature_values) { for (size_t i = 0; i < feature_values.size() && i < static_cast(num_total_features_); ++i) { this->PushOneValue(tid, row_idx, i, feature_values[i]); @@ -683,7 +698,8 @@ class Dataset { TrainingShareStates* GetShareStates( score_t* gradients, score_t* hessians, const std::vector& is_feature_used, bool is_constant_hessian, - bool force_col_wise, bool force_row_wise, const int num_grad_quant_bins) const; + bool force_col_wise, bool force_row_wise, const int num_grad_quant_bins, + bool is_cuda_tree_learner = false) const; LIGHTGBM_EXPORT void FinishLoad(); @@ -1022,6 +1038,13 @@ class Dataset { void CreateCUDAColumnData(); + #ifdef USE_CUDA + /*! \brief Whether the CUDA row data can be built directly from column bins, + * making the host multi-val bin build in GetShareStates unnecessary + * (EXABOOST_FAST_ROWDATA). */ + bool CanSkipHostMultiValBinForCUDA() const; + #endif // USE_CUDA + void CopySubrowHostPart(const Dataset* fullset, const data_size_t* used_indices, data_size_t num_used_indices, bool need_meta_data); std::string data_filename_; diff --git a/include/LightGBM/feature_group.h b/include/LightGBM/feature_group.h index 055c620a2a71..661cb8b2096c 100644 --- a/include/LightGBM/feature_group.h +++ b/include/LightGBM/feature_group.h @@ -267,6 +267,32 @@ class FeatureGroup { } } + /*! + * \brief Encode a BinMapper::ValueToBin result of the subfeature the way PushData + * would push it, or ``Bin::kSkipBin`` if it would not be pushed. Together + * with PushTargetBin this lets callers push precomputed bins in blocks; + * must stay in sync with PushData. + * \param sub_feature_idx Index of the subfeature + * \param bin bin value of record + */ + inline uint32_t EncodeBinForPush(int sub_feature_idx, uint32_t bin) const { + if (bin == bin_mappers_[sub_feature_idx]->GetMostFreqBin()) { + return Bin::kSkipBin; + } + if (bin_mappers_[sub_feature_idx]->GetMostFreqBin() == 0) { + bin -= 1; + } + if (is_multi_val_) { + return bin + 1; + } + return bin + bin_offsets_[sub_feature_idx]; + } + + /*! \brief The Bin that PushData writes the subfeature's records into */ + inline Bin* PushTargetBin(int sub_feature_idx) { + return is_multi_val_ ? multi_bin_data_[sub_feature_idx].get() : bin_data_.get(); + } + void ReSize(int num_data) { if (!is_multi_val_) { bin_data_->ReSize(num_data); diff --git a/python-package/lightgbm/basic.py b/python-package/lightgbm/basic.py index 0be4ea1b5371..cf74804ccc3a 100644 --- a/python-package/lightgbm/basic.py +++ b/python-package/lightgbm/basic.py @@ -58,6 +58,7 @@ "ctypes._Pointer[ctypes.c_float]", "ctypes._Pointer[ctypes.c_double]", "ctypes._Pointer[ctypes.c_int8]", + "ctypes._Pointer[ctypes.c_int16]", ] _ctypes_float_array = Union[ "ctypes.Array[ctypes._Pointer[ctypes.c_float]]", @@ -173,7 +174,7 @@ def _get_sample_count(total_nrow: int, params: str) -> int: def _np2d_to_np1d(mat: np.ndarray) -> Tuple[np.ndarray, int]: dtype: "np.typing.DTypeLike" - if mat.dtype in (np.float32, np.float64, np.int8): + if mat.dtype in (np.float32, np.float64, np.int8, np.int16): dtype = mat.dtype else: dtype = np.float32 @@ -627,6 +628,7 @@ def _choose_param_value(main_param_name: str, params: Dict[str, Any], default_va _C_API_DTYPE_INT32 = 2 _C_API_DTYPE_INT64 = 3 _C_API_DTYPE_INT8 = 4 +_C_API_DTYPE_INT16 = 5 """Macro definition of data order in matrix""" _C_API_IS_COL_MAJOR = 0 @@ -675,7 +677,7 @@ def _convert_from_sliced_object(data: np.ndarray) -> np.ndarray: def _c_float_array(data: np.ndarray) -> Tuple[_ctypes_float_ptr, int, np.ndarray]: - """Get pointer of float/int8 numpy array / list.""" + """Get pointer of float/small-int numpy array / list.""" if _is_1d_list(data): data = np.asarray(data) if _is_numpy_1d_array(data): @@ -691,8 +693,11 @@ def _c_float_array(data: np.ndarray) -> Tuple[_ctypes_float_ptr, int, np.ndarray elif data.dtype == np.int8: ptr_data = data.ctypes.data_as(ctypes.POINTER(ctypes.c_int8)) type_data = _C_API_DTYPE_INT8 + elif data.dtype == np.int16: + ptr_data = data.ctypes.data_as(ctypes.POINTER(ctypes.c_int16)) + type_data = _C_API_DTYPE_INT16 else: - raise TypeError(f"Expected np.float32, np.float64 or np.int8, met type({data.dtype})") + raise TypeError(f"Expected np.float32, np.float64, np.int8 or np.int16, met type({data.dtype})") else: raise TypeError(f"Unknown type({type(data).__name__})") return (ptr_data, type_data, data) # return `data` to avoid the temporary copy is freed diff --git a/src/c_api.cpp b/src/c_api.cpp index 33c21e4874f0..c2e47d6b8057 100644 --- a/src/c_api.cpp +++ b/src/c_api.cpp @@ -945,6 +945,12 @@ RowFunctionFromDenseMatrix(const void* data, int num_row, int num_col, int data_ std::function>(int row_idx)> RowPairFunctionFromDenseMatrix(const void* data, int num_row, int num_col, int data_type, int is_row_major); +template +void SampleDenseSmallInt(const void** data, const std::vector& sample_indices, + const int32_t* nrow, int32_t ncol, const int* is_row_major, + std::vector>* sample_values, + std::vector>* sample_idx); + std::function>(int row_idx)> RowPairFunctionFromDenseRows(const void** data, int num_col, int data_type); @@ -1372,20 +1378,26 @@ int LGBM_DatasetCreateFromMats(int32_t nmat, std::vector> sample_values(ncol); std::vector> sample_idx(ncol); - int offset = 0; - int j = 0; - for (size_t i = 0; i < sample_indices.size(); ++i) { - auto idx = sample_indices[i]; - while ((idx - offset) >= nrow[j]) { - offset += nrow[j]; - ++j; - } + if (data_type == C_API_DTYPE_INT8) { + SampleDenseSmallInt(data, sample_indices, nrow, ncol, is_row_major, &sample_values, &sample_idx); + } else if (data_type == C_API_DTYPE_INT16) { + SampleDenseSmallInt(data, sample_indices, nrow, ncol, is_row_major, &sample_values, &sample_idx); + } else { + int offset = 0; + int j = 0; + for (size_t i = 0; i < sample_indices.size(); ++i) { + auto idx = sample_indices[i]; + while ((idx - offset) >= nrow[j]) { + offset += nrow[j]; + ++j; + } - auto row = get_row_fun[j](static_cast(idx - offset)); - for (size_t k = 0; k < row.size(); ++k) { - if (std::fabs(row[k]) > kZeroThreshold || std::isnan(row[k])) { - sample_values[k].emplace_back(row[k]); - sample_idx[k].emplace_back(static_cast(i)); + auto row = get_row_fun[j](static_cast(idx - offset)); + for (size_t k = 0; k < row.size(); ++k) { + if (std::fabs(row[k]) > kZeroThreshold || std::isnan(row[k])) { + sample_values[k].emplace_back(row[k]); + sample_idx[k].emplace_back(static_cast(i)); + } } } } @@ -1407,17 +1419,22 @@ int LGBM_DatasetCreateFromMats(int32_t nmat, } int32_t start_row = 0; for (int j = 0; j < nmat; ++j) { - OMP_INIT_EX(); - #pragma omp parallel for num_threads(OMP_NUM_THREADS()) schedule(static) - for (int i = 0; i < nrow[j]; ++i) { - OMP_LOOP_EX_BEGIN(); - const int tid = omp_get_thread_num(); - auto one_row = get_row_fun[j](i); - ret->PushOneRow(tid, start_row + i, one_row); - OMP_LOOP_EX_END(); + if (data_type == C_API_DTYPE_INT8) { + ret->PushDenseSmallIntRows(reinterpret_cast(data[j]), nrow[j], ncol, is_row_major[j], start_row); + } else if (data_type == C_API_DTYPE_INT16) { + ret->PushDenseSmallIntRows(reinterpret_cast(data[j]), nrow[j], ncol, is_row_major[j], start_row); + } else { + OMP_INIT_EX(); + #pragma omp parallel for num_threads(OMP_NUM_THREADS()) schedule(static) + for (int i = 0; i < nrow[j]; ++i) { + OMP_LOOP_EX_BEGIN(); + const int tid = omp_get_thread_num(); + auto one_row = get_row_fun[j](i); + ret->PushOneRow(tid, start_row + i, one_row); + OMP_LOOP_EX_END(); + } + OMP_THROW_EX(); } - OMP_THROW_EX(); - start_row += nrow[j]; } ret->FinishLoad(); @@ -2939,11 +2956,57 @@ RowFunctionFromDenseMatrix(const void* data, int num_row, int num_col, int data_ return RowFunctionFromDenseMatrix_helper(data, num_row, num_col, is_row_major); } else if (data_type == C_API_DTYPE_INT8) { return RowFunctionFromDenseMatrix_helper(data, num_row, num_col, is_row_major); + } else if (data_type == C_API_DTYPE_INT16) { + return RowFunctionFromDenseMatrix_helper(data, num_row, num_col, is_row_major); } Log::Fatal("Unknown data type in RowFunctionFromDenseMatrix"); return nullptr; } +// Column-parallel bin-mapper sampler for dense small-int matrices. Produces exactly +// what the row-function sampling loop would: ints are never NaN and pass the +// kZeroThreshold check iff nonzero, and per-column appends stay in sample order. +template +void SampleDenseSmallInt(const void** data, const std::vector& sample_indices, + const int32_t* nrow, int32_t ncol, const int* is_row_major, + std::vector>* sample_values, + std::vector>* sample_idx) { + const size_t n = sample_indices.size(); + std::vector mat_of(n); + std::vector row_of(n); + int offset = 0; + int j = 0; + for (size_t i = 0; i < n; ++i) { + auto idx = sample_indices[i]; + while ((idx - offset) >= nrow[j]) { + offset += nrow[j]; + ++j; + } + mat_of[i] = j; + row_of[i] = idx - offset; + } + // contiguous column stripes per thread, sample-index outer loop for locality + #pragma omp parallel num_threads(OMP_NUM_THREADS()) + { + const int num_threads = omp_get_num_threads(); + const int tid = omp_get_thread_num(); + const int col_lo = static_cast(static_cast(ncol) * tid / num_threads); + const int col_hi = static_cast(static_cast(ncol) * (tid + 1) / num_threads); + for (size_t i = 0; i < n; ++i) { + const T* mat = reinterpret_cast(data[mat_of[i]]); + const size_t r = static_cast(row_of[i]); + for (int col = col_lo; col < col_hi; ++col) { + const T value = is_row_major[mat_of[i]] ? mat[r * ncol + col] + : mat[static_cast(nrow[mat_of[i]]) * col + r]; + if (value != 0) { + (*sample_values)[col].emplace_back(static_cast(value)); + (*sample_idx)[col].emplace_back(static_cast(i)); + } + } + } + } +} + std::function>(int row_idx)> RowPairFunctionFromDenseMatrix(const void* data, int num_row, int num_col, int data_type, int is_row_major) { auto inner_function = RowFunctionFromDenseMatrix(data, num_row, num_col, data_type, is_row_major); diff --git a/src/io/cuda/cuda_column_data.cpp b/src/io/cuda/cuda_column_data.cpp index efbe5a1a5723..a3c0bb162f34 100644 --- a/src/io/cuda/cuda_column_data.cpp +++ b/src/io/cuda/cuda_column_data.cpp @@ -13,6 +13,10 @@ namespace LightGBM { +// Defined in cuda_column_data.cu: warms the lazy CUDA module load during +// Dataset construction so the first Train() call doesn't pay it (see there). +extern void WarmupCUDAKernelModule(); + CUDAColumnData::CUDAColumnData(const data_size_t num_data, const int gpu_device_id) { num_threads_ = OMP_NUM_THREADS(); num_data_ = num_data; @@ -62,6 +66,7 @@ void CUDAColumnData::Init(const int num_columns, const std::vector& feature_mfb_is_zero, const std::vector& feature_mfb_is_na, const std::vector& feature_to_column) { + WarmupCUDAKernelModule(); num_columns_ = num_columns; column_bit_type_ = column_bit_type; feature_max_bin_ = feature_max_bin; @@ -239,6 +244,29 @@ void CUDAColumnData::SetCompactColumnView(const std::vector& column_to_comp // Keep a host-readable mirror so GetColumnData() can return the split column's // device pointer in the skip-allocation path (data_by_column_ stays null there). compact_column_host_view_ = view; + packed_column_view_active_ = false; +} + +void CUDAColumnData::SetCompactPackedColumnView(const std::vector& column_to_compact_slot, + const uint8_t* packed_buf, + const std::vector& slot_base_byte, + const std::vector& slot_row_stride, + const std::vector& slot_shift) { + packed_column_ptr_.assign(num_columns_, nullptr); + packed_column_stride_.assign(num_columns_, 0); + packed_column_shift_.assign(num_columns_, 0); + for (int c = 0; c < num_columns_; ++c) { + if (c < static_cast(column_to_compact_slot.size()) && column_to_compact_slot[c] >= 0) { + const int slot = column_to_compact_slot[c]; + packed_column_ptr_[c] = packed_buf + slot_base_byte[slot]; + packed_column_stride_[c] = slot_row_stride[slot]; + packed_column_shift_[c] = slot_shift[slot]; + } + } + // no per-column plain buffer exists this tree: null the host view so any + // GetColumnData consumer fails loudly instead of reading last tree's bytes + compact_column_host_view_.assign(num_columns_, nullptr); + packed_column_view_active_ = true; } void CUDAColumnData::RestoreOriginalColumnView() { diff --git a/src/io/cuda/cuda_column_data.cu b/src/io/cuda/cuda_column_data.cu index 83f278b88c6e..4be12ea32c70 100644 --- a/src/io/cuda/cuda_column_data.cu +++ b/src/io/cuda/cuda_column_data.cu @@ -13,6 +13,21 @@ namespace LightGBM { +__global__ void CUDAWarmupKernel() {} + +// Force the lazy CUDA module load (the whole library's device code is one +// device-linked module) while the Dataset is being constructed. Without this, +// the first kernel launch of the process pays the full module load (~37 ms on +// a 34 MB library), and that first launch happens inside the tree learner's +// first Train() call, i.e. inside any training wall-clock measurement. Dataset +// construction is the natural warm spot: it already initializes the CUDA +// context, and it runs before the booster/learner exists. +void WarmupCUDAKernelModule() { + CUDAWarmupKernel<<<1, 1>>>(); + CUDASUCCESS_OR_FATAL(cudaGetLastError()); + CUDASUCCESS_OR_FATAL(cudaDeviceSynchronize()); +} + __global__ void CopySubrowKernel_ColumnData( uint8_t* const* in_cuda_data_by_column, const uint8_t* cuda_column_bit_type, diff --git a/src/io/cuda/cuda_row_data.cpp b/src/io/cuda/cuda_row_data.cpp index b2865c21e54c..c0db03ca2319 100644 --- a/src/io/cuda/cuda_row_data.cpp +++ b/src/io/cuda/cuda_row_data.cpp @@ -8,10 +8,75 @@ #include +#include +#include +#include +#include +#include #include namespace LightGBM { +namespace { + +bool FastRowDataEnabled() { + const char* env = std::getenv("EXABOOST_FAST_ROWDATA"); + return env == nullptr || std::string(env) != std::string("0"); +} + +bool FastRowDataVerifyEnabled() { + const char* env = std::getenv("EXABOOST_FAST_ROWDATA_VERIFY"); + return env != nullptr && std::string(env) == std::string("1"); +} + +bool RowData4BitEnabled() { + const char* env = std::getenv("EXABOOST_ROWDATA_4BIT"); + return env == nullptr || std::string(env) != std::string("0"); +} + +bool RowData4BitVerifyEnabled() { + const char* env = std::getenv("EXABOOST_ROWDATA_4BIT_VERIFY"); + return env != nullptr && std::string(env) == std::string("1"); +} + +// Read one bin value out of a raw host column (bit type 4/8/16/32). +inline uint8_t FetchColumnBin(const void* column_data, uint8_t column_bit_type, data_size_t row) { + if (column_bit_type == 4) { + const uint8_t* in = reinterpret_cast(column_data); + return (in[row >> 1] >> ((row & 1) << 2)) & 0xf; + } else if (column_bit_type == 8) { + return reinterpret_cast(column_data)[row]; + } else if (column_bit_type == 16) { + return static_cast(reinterpret_cast(column_data)[row]); + } else { + return static_cast(reinterpret_cast(column_data)[row]); + } +} + +// Scatter one column's rows [start, end) into its slot of a row-major partition: +// out_data points at (partition base + local column index), row_stride is the +// number of columns in the partition. +template +void TransposeColumnTile(const void* column_data, data_size_t start, data_size_t end, + int row_stride, BIN_TYPE* out_data) { + const COLUMN_TYPE* in_data = reinterpret_cast(column_data); + for (data_size_t row = start; row < end; ++row) { + out_data[static_cast(row) * row_stride] = static_cast(in_data[row]); + } +} + +template +void TransposeColumnTile4Bit(const void* column_data, data_size_t start, data_size_t end, + int row_stride, BIN_TYPE* out_data) { + const uint8_t* in_data = reinterpret_cast(column_data); + for (data_size_t row = start; row < end; ++row) { + out_data[static_cast(row) * row_stride] = + static_cast((in_data[row >> 1] >> ((row & 1) << 2)) & 0xf); + } +} + +} // anonymous namespace + CUDARowData::CUDARowData(const Dataset* train_data, const TrainingShareStates* train_share_state, const int gpu_device_id, @@ -52,98 +117,119 @@ void CUDARowData::Init(const Dataset* train_data, TrainingShareStates* train_sha const void* host_row_ptr = nullptr; row_ptr_bit_type_ = 0; const void* host_data = train_share_state->GetRowWiseData(&bit_type_, &total_size, &is_sparse_, &host_row_ptr, &row_ptr_bit_type_); - if (bit_type_ == 8) { - if (!is_sparse_) { - std::vector partitioned_data; - GetDenseDataPartitioned(reinterpret_cast(host_data), &partitioned_data); - cuda_data_uint8_t_.InitFromHostVector(partitioned_data); + if (host_data == nullptr) { + // Dataset::GetShareStates skipped the host multi-val bin build (EXABOOST_FAST_ROWDATA): + // the row-wise data is known to be dense; recover the bin bit width from the + // per-column bin counts, exactly as MultiValBin::CreateMultiValDenseBin would. + const std::vector& column_hist_offsets = train_share_state->column_hist_offsets(); + uint32_t max_bin_per_column = 0; + for (size_t i = 0; i + 1 < column_hist_offsets.size(); ++i) { + max_bin_per_column = std::max(max_bin_per_column, column_hist_offsets[i + 1] - column_hist_offsets[i]); + } + is_sparse_ = false; + bit_type_ = max_bin_per_column <= 256 ? 8 : (max_bin_per_column <= 65536 ? 16 : 32); + } + // 4-bit packed row data: eligible when the data is dense uint8 and EVERY + // column fits in <= 16 bins (bin values index the per-column histogram span, + // so they are < the column's hist-offset delta <= 16, i.e. nibble-sized). + // Large-bin partitions are impossible under that cap, so the global-memory + // histogram kernels (which do not unpack) are unreachable. + is_4bit_packed_ = false; + if (!is_sparse_ && bit_type_ == 8 && large_bin_partitions_.empty() && + num_feature_partitions_ > 0 && RowData4BitEnabled()) { + const std::vector& column_hist_offsets = train_share_state->column_hist_offsets(); + uint32_t max_bin_per_column = 0; + for (size_t i = 0; i + 1 < column_hist_offsets.size(); ++i) { + max_bin_per_column = std::max(max_bin_per_column, column_hist_offsets[i + 1] - column_hist_offsets[i]); + } + is_4bit_packed_ = max_bin_per_column <= 16; + } + if (!is_sparse_) { + if (is_4bit_packed_) { + InitDense4BitData(train_data, reinterpret_cast(host_data)); + } else if (bit_type_ == 8) { + InitDenseData(train_data, reinterpret_cast(host_data), &cuda_data_uint8_t_); + } else if (bit_type_ == 16) { + InitDenseData(train_data, reinterpret_cast(host_data), &cuda_data_uint16_t_); + } else if (bit_type_ == 32) { + InitDenseData(train_data, reinterpret_cast(host_data), &cuda_data_uint32_t_); } else { - if (row_ptr_bit_type_ == 16) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint8_t_, - &cuda_row_ptr_uint16_t_, - &cuda_partition_ptr_uint16_t_); - } else if (row_ptr_bit_type_ == 32) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint8_t_, - &cuda_row_ptr_uint32_t_, - &cuda_partition_ptr_uint32_t_); - } else if (row_ptr_bit_type_ == 64) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint8_t_, - &cuda_row_ptr_uint64_t_, - &cuda_partition_ptr_uint64_t_); - } else { - Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); - } + Log::Fatal("Unknown bit type = %d", bit_type_); + } + } else if (bit_type_ == 8) { + if (row_ptr_bit_type_ == 16) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint8_t_, + &cuda_row_ptr_uint16_t_, + &cuda_partition_ptr_uint16_t_); + } else if (row_ptr_bit_type_ == 32) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint8_t_, + &cuda_row_ptr_uint32_t_, + &cuda_partition_ptr_uint32_t_); + } else if (row_ptr_bit_type_ == 64) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint8_t_, + &cuda_row_ptr_uint64_t_, + &cuda_partition_ptr_uint64_t_); + } else { + Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); } } else if (bit_type_ == 16) { - if (!is_sparse_) { - std::vector partitioned_data; - GetDenseDataPartitioned(reinterpret_cast(host_data), &partitioned_data); - cuda_data_uint16_t_.InitFromHostVector(partitioned_data); + if (row_ptr_bit_type_ == 16) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint16_t_, + &cuda_row_ptr_uint16_t_, + &cuda_partition_ptr_uint16_t_); + } else if (row_ptr_bit_type_ == 32) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint16_t_, + &cuda_row_ptr_uint32_t_, + &cuda_partition_ptr_uint32_t_); + } else if (row_ptr_bit_type_ == 64) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint16_t_, + &cuda_row_ptr_uint64_t_, + &cuda_partition_ptr_uint64_t_); } else { - if (row_ptr_bit_type_ == 16) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint16_t_, - &cuda_row_ptr_uint16_t_, - &cuda_partition_ptr_uint16_t_); - } else if (row_ptr_bit_type_ == 32) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint16_t_, - &cuda_row_ptr_uint32_t_, - &cuda_partition_ptr_uint32_t_); - } else if (row_ptr_bit_type_ == 64) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint16_t_, - &cuda_row_ptr_uint64_t_, - &cuda_partition_ptr_uint64_t_); - } else { - Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); - } + Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); } } else if (bit_type_ == 32) { - if (!is_sparse_) { - std::vector partitioned_data; - GetDenseDataPartitioned(reinterpret_cast(host_data), &partitioned_data); - cuda_data_uint32_t_.InitFromHostVector(partitioned_data); + if (row_ptr_bit_type_ == 16) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint32_t_, + &cuda_row_ptr_uint16_t_, + &cuda_partition_ptr_uint16_t_); + } else if (row_ptr_bit_type_ == 32) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint32_t_, + &cuda_row_ptr_uint32_t_, + &cuda_partition_ptr_uint32_t_); + } else if (row_ptr_bit_type_ == 64) { + InitSparseData( + reinterpret_cast(host_data), + reinterpret_cast(host_row_ptr), + &cuda_data_uint32_t_, + &cuda_row_ptr_uint64_t_, + &cuda_partition_ptr_uint64_t_); } else { - if (row_ptr_bit_type_ == 16) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint32_t_, - &cuda_row_ptr_uint16_t_, - &cuda_partition_ptr_uint16_t_); - } else if (row_ptr_bit_type_ == 32) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint32_t_, - &cuda_row_ptr_uint32_t_, - &cuda_partition_ptr_uint32_t_); - } else if (row_ptr_bit_type_ == 64) { - InitSparseData( - reinterpret_cast(host_data), - reinterpret_cast(host_row_ptr), - &cuda_data_uint32_t_, - &cuda_row_ptr_uint64_t_, - &cuda_partition_ptr_uint64_t_); - } else { - Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); - } + Log::Fatal("Unknown data ptr bit type %d", row_ptr_bit_type_); } } else { Log::Fatal("Unknown bit type = %d", bit_type_); @@ -327,6 +413,258 @@ void CUDARowData::GetDenseDataPartitioned(const BIN_TYPE* row_wise_data, std::ve }); } +bool CUDARowData::CollectDenseColumnData(const Dataset* train_data, + std::vector* column_data, + std::vector* column_bit_types) const { + const int num_feature_groups = train_data->num_feature_groups(); + for (int group_index = 0; group_index < num_feature_groups; ++group_index) { + if (train_data->IsMultiGroup(group_index)) { + // multi-val group columns need per-feature most_freq_bin remapping, + // which only the multi-val bin path implements + return false; + } + uint8_t bit_type = 0; + bool is_sparse = false; + BinIterator* bin_iterator = nullptr; + const void* one_column_data = train_data->GetColWiseData(group_index, -1, &bit_type, &is_sparse, &bin_iterator); + if (bin_iterator != nullptr) { + delete bin_iterator; + } + if (is_sparse || one_column_data == nullptr) { + return false; + } + column_data->push_back(one_column_data); + column_bit_types->push_back(bit_type); + } + CHECK_EQ(static_cast(column_data->size()), feature_partition_column_index_offsets_.back()); + return true; +} + +template +void CUDARowData::BuildDensePartitionedFromColumns( + const std::vector& column_data, + const std::vector& column_bit_types, + BIN_TYPE* out_data) const { + // For a non-multi-val dense group the multi-val bin stores the raw group bin + // value unchanged (FeatureGroupIterator: min_bin == 1, most_freq_bin == 0, so + // Get() is the identity), so the columns can be transposed directly. + // Cache-tiled: per 512-row tile, each source column is read sequentially and + // scattered into the row-major partition tile, which stays L2-resident. + constexpr data_size_t kRowTileSize = 512; + Threading::For(0, num_data_, kRowTileSize, + [this, &column_data, &column_bit_types, out_data] (int /*thread_index*/, data_size_t start, data_size_t end) { + for (data_size_t tile_start = start; tile_start < end; tile_start += kRowTileSize) { + const data_size_t tile_end = std::min(tile_start + kRowTileSize, end); + for (size_t i = 0; i + 1 < feature_partition_column_index_offsets_.size(); ++i) { + const int partition_column_start = feature_partition_column_index_offsets_[i]; + const int partition_column_end = feature_partition_column_index_offsets_[i + 1]; + const int num_columns_in_cur_partition = partition_column_end - partition_column_start; + BIN_TYPE* partition_out_data = out_data + static_cast(num_data_) * static_cast(partition_column_start); + for (int column_index = partition_column_start; column_index < partition_column_end; ++column_index) { + BIN_TYPE* column_out_data = partition_out_data + (column_index - partition_column_start); + const void* one_column_data = column_data[column_index]; + const uint8_t column_bit_type = column_bit_types[column_index]; + if (column_bit_type == 4) { + TransposeColumnTile4Bit(one_column_data, tile_start, tile_end, num_columns_in_cur_partition, column_out_data); + } else if (column_bit_type == 8) { + TransposeColumnTile(one_column_data, tile_start, tile_end, num_columns_in_cur_partition, column_out_data); + } else if (column_bit_type == 16) { + TransposeColumnTile(one_column_data, tile_start, tile_end, num_columns_in_cur_partition, column_out_data); + } else if (column_bit_type == 32) { + TransposeColumnTile(one_column_data, tile_start, tile_end, num_columns_in_cur_partition, column_out_data); + } else { + Log::Fatal("Unknown column bit type %d", static_cast(column_bit_type)); + } + } + } + } + }); +} + +template +void CUDARowData::InitDenseData(const Dataset* train_data, const BIN_TYPE* host_data, CUDAVector* cuda_data) { + const bool fast_enabled = FastRowDataEnabled(); + std::vector column_data; + std::vector column_bit_types; + const bool use_fast_build = fast_enabled && CollectDenseColumnData(train_data, &column_data, &column_bit_types); + const size_t total_size = static_cast(feature_partition_column_index_offsets_.back()) * static_cast(num_data_); + if (use_fast_build) { + Log::Debug("CUDARowData: fast dense row data build from column bins (disable with EXABOOST_FAST_ROWDATA=0)"); + // new[] leaves the staging buffer uninitialized on purpose; the transpose writes every element + std::unique_ptr buffer(new BIN_TYPE[total_size]); + BuildDensePartitionedFromColumns(column_data, column_bit_types, buffer.get()); + if (FastRowDataVerifyEnabled()) { + if (host_data == nullptr) { + Log::Fatal("EXABOOST_FAST_ROWDATA_VERIFY=1 requires the host multi-val bin, but its build was skipped."); + } + std::vector reference; + GetDenseDataPartitioned(host_data, &reference); + CHECK_EQ(reference.size(), total_size); + if (std::memcmp(reference.data(), buffer.get(), total_size * sizeof(BIN_TYPE)) != 0) { + Log::Fatal("EXABOOST_FAST_ROWDATA_VERIFY: fast dense row data differs from the multi-val bin path."); + } + Log::Info("EXABOOST_FAST_ROWDATA_VERIFY: fast dense row data matches the multi-val bin path (%d columns, %d rows).", + feature_partition_column_index_offsets_.back(), num_data_); + } + cuda_data->InitFromHostMemory(buffer.get(), total_size); + return; + } + if (host_data == nullptr) { + Log::Fatal("The host multi-val bin build was skipped but the fast dense row data build is not applicable."); + } + if (fast_enabled) { + // fallback repack from the host multi-val bin, but without the redundant zero-fill + std::unique_ptr buffer(new BIN_TYPE[total_size]); + GetDenseDataPartitionedToBuffer(host_data, buffer.get()); + cuda_data->InitFromHostMemory(buffer.get(), total_size); + } else { + std::vector partitioned_data; + GetDenseDataPartitioned(host_data, &partitioned_data); + cuda_data->InitFromHostVector(partitioned_data); + } +} + +void CUDARowData::BuildDensePacked4BitFromColumns( + const std::vector& column_data, + const std::vector& column_bit_types, + uint8_t* out_data) const { + // Same cache-tiled transpose as BuildDensePartitionedFromColumns, but the + // output is the packed layout. Columns of a partition are processed in order + // by the same thread, so the even column assigns its byte (initializing it) + // and the odd column ORs its high nibble in -- no read-modify-write hazard. + constexpr data_size_t kRowTileSize = 512; + Threading::For(0, num_data_, kRowTileSize, + [this, &column_data, &column_bit_types, out_data] (int /*thread_index*/, data_size_t start, data_size_t end) { + for (data_size_t tile_start = start; tile_start < end; tile_start += kRowTileSize) { + const data_size_t tile_end = std::min(tile_start + kRowTileSize, end); + for (size_t i = 0; i + 1 < feature_partition_column_index_offsets_.size(); ++i) { + const int partition_column_start = feature_partition_column_index_offsets_[i]; + const int partition_column_end = feature_partition_column_index_offsets_[i + 1]; + const int packed_width = packed_partition_byte_offsets_[i + 1] - packed_partition_byte_offsets_[i]; + uint8_t* partition_out_data = out_data + + static_cast(num_data_) * static_cast(packed_partition_byte_offsets_[i]); + for (int column_index = partition_column_start; column_index < partition_column_end; ++column_index) { + const int local_column = column_index - partition_column_start; + uint8_t* column_out_data = partition_out_data + (local_column >> 1); + const void* one_column_data = column_data[column_index]; + const uint8_t column_bit_type = column_bit_types[column_index]; + if ((local_column & 1) == 0) { + // even column initializes the byte (high nibble 0 covers odd-width padding) + for (data_size_t row = tile_start; row < tile_end; ++row) { + column_out_data[static_cast(row) * packed_width] = + FetchColumnBin(one_column_data, column_bit_type, row); + } + } else { + for (data_size_t row = tile_start; row < tile_end; ++row) { + column_out_data[static_cast(row) * packed_width] |= + static_cast(FetchColumnBin(one_column_data, column_bit_type, row) << 4); + } + } + } + } + } + }); +} + +void CUDARowData::Pack4BitFromPartitioned(const uint8_t* unpacked, uint8_t* packed) const { + Threading::For(0, num_data_, 512, + [this, unpacked, packed] (int /*thread_index*/, data_size_t start, data_size_t end) { + for (size_t i = 0; i + 1 < feature_partition_column_index_offsets_.size(); ++i) { + const int partition_column_start = feature_partition_column_index_offsets_[i]; + const int num_columns = feature_partition_column_index_offsets_[i + 1] - partition_column_start; + const int packed_width = packed_partition_byte_offsets_[i + 1] - packed_partition_byte_offsets_[i]; + const uint8_t* in_partition = unpacked + + static_cast(num_data_) * static_cast(partition_column_start); + uint8_t* out_partition = packed + + static_cast(num_data_) * static_cast(packed_partition_byte_offsets_[i]); + for (data_size_t row = start; row < end; ++row) { + const uint8_t* in_row = in_partition + static_cast(row) * num_columns; + uint8_t* out_row = out_partition + static_cast(row) * packed_width; + for (int j = 0; j < num_columns; j += 2) { + const uint8_t lo = in_row[j]; + const uint8_t hi = j + 1 < num_columns ? in_row[j + 1] : 0; + out_row[j >> 1] = static_cast(lo | (hi << 4)); + } + } + } + }); +} + +void CUDARowData::InitDense4BitData(const Dataset* train_data, const uint8_t* host_data) { + // Packed layout invariant: each partition's packed row width is + // ceil(num_columns_in_partition / 2) bytes, i.e. the column count is padded + // to an even number PER PARTITION, so every (row, partition) segment starts + // byte-aligned even when partitions start at odd global column offsets. + // Within a partition, column j lives in byte (j >> 1), nibble (j & 1) + // (low nibble = even column, matching DenseBin). + packed_partition_byte_offsets_.clear(); + packed_partition_byte_offsets_.emplace_back(0); + for (size_t i = 0; i + 1 < feature_partition_column_index_offsets_.size(); ++i) { + const int num_columns = feature_partition_column_index_offsets_[i + 1] - feature_partition_column_index_offsets_[i]; + packed_partition_byte_offsets_.emplace_back(packed_partition_byte_offsets_.back() + ((num_columns + 1) >> 1)); + } + const size_t packed_total = static_cast(packed_partition_byte_offsets_.back()) * static_cast(num_data_); + const size_t unpacked_total = static_cast(feature_partition_column_index_offsets_.back()) * static_cast(num_data_); + std::vector column_data; + std::vector column_bit_types; + const bool use_fast_build = FastRowDataEnabled() && CollectDenseColumnData(train_data, &column_data, &column_bit_types); + std::unique_ptr packed(new uint8_t[packed_total]); + if (use_fast_build) { + BuildDensePacked4BitFromColumns(column_data, column_bit_types, packed.get()); + } else { + if (host_data == nullptr) { + Log::Fatal("The host multi-val bin build was skipped but the fast dense row data build is not applicable."); + } + std::unique_ptr staging(new uint8_t[unpacked_total]); + GetDenseDataPartitionedToBuffer(host_data, staging.get()); + Pack4BitFromPartitioned(staging.get(), packed.get()); + } + if (RowData4BitVerifyEnabled()) { + // build the exact 8-bit representation the non-packed path would produce and + // compare element-wise after unpacking (the layouts differ by design) + std::unique_ptr reference(new uint8_t[unpacked_total]); + if (use_fast_build) { + BuildDensePartitionedFromColumns(column_data, column_bit_types, reference.get()); + } else { + GetDenseDataPartitionedToBuffer(host_data, reference.get()); + } + std::vector thread_mismatch(num_threads_, 0); + Threading::For(0, num_data_, 512, + [this, &reference, &packed, &thread_mismatch] (int thread_index, data_size_t start, data_size_t end) { + for (size_t i = 0; i + 1 < feature_partition_column_index_offsets_.size(); ++i) { + const int partition_column_start = feature_partition_column_index_offsets_[i]; + const int num_columns = feature_partition_column_index_offsets_[i + 1] - partition_column_start; + const int packed_width = packed_partition_byte_offsets_[i + 1] - packed_partition_byte_offsets_[i]; + const uint8_t* ref_partition = reference.get() + + static_cast(num_data_) * static_cast(partition_column_start); + const uint8_t* packed_partition = packed.get() + + static_cast(num_data_) * static_cast(packed_partition_byte_offsets_[i]); + for (data_size_t row = start; row < end; ++row) { + const uint8_t* ref_row = ref_partition + static_cast(row) * num_columns; + const uint8_t* packed_row = packed_partition + static_cast(row) * packed_width; + for (int j = 0; j < num_columns; ++j) { + const uint8_t unpacked_value = (packed_row[j >> 1] >> ((j & 1) << 2)) & 0xf; + if (unpacked_value != ref_row[j]) { + thread_mismatch[thread_index] = 1; + } + } + } + } + }); + for (int t = 0; t < num_threads_; ++t) { + if (thread_mismatch[t] != 0) { + Log::Fatal("EXABOOST_ROWDATA_4BIT_VERIFY: packed row data differs from the 8-bit representation."); + } + } + Log::Info("EXABOOST_ROWDATA_4BIT_VERIFY: 4-bit packed row data matches the 8-bit representation (%d columns, %d rows).", + feature_partition_column_index_offsets_.back(), num_data_); + } + Log::Debug("CUDARowData: 4-bit packed dense row data engaged (%d partitions, %d -> %d bytes per row; disable with EXABOOST_ROWDATA_4BIT=0)", + num_feature_partitions_, feature_partition_column_index_offsets_.back(), packed_partition_byte_offsets_.back()); + cuda_data_uint8_t_.InitFromHostMemory(packed.get(), packed_total); + cuda_packed_partition_byte_offsets_.InitFromHostVector(packed_partition_byte_offsets_); +} + template void CUDARowData::GetSparseDataPartitioned( const BIN_TYPE* row_wise_data, diff --git a/src/io/cuda/cuda_tree.cpp b/src/io/cuda/cuda_tree.cpp index fa0f1d561caf..377fd29466c5 100644 --- a/src/io/cuda/cuda_tree.cpp +++ b/src/io/cuda/cuda_tree.cpp @@ -8,10 +8,15 @@ #include +#include +#include +#include + namespace LightGBM { CUDATree::CUDATree(int max_leaves, bool track_branch_features, bool is_linear, - const int gpu_device_id, const bool has_categorical_feature): + const int gpu_device_id, const bool has_categorical_feature, + uint8_t* pooled_device_buffer): Tree(max_leaves, track_branch_features, is_linear), num_threads_per_block_add_prediction_to_score_(1024) { is_cuda_tree_ = true; @@ -24,7 +29,7 @@ num_threads_per_block_add_prediction_to_score_(1024) { cuda_cat_boundaries_.Resize(max_leaves); cuda_cat_boundaries_inner_.Resize(max_leaves); } - InitCUDAMemory(); + InitCUDAMemory(pooled_device_buffer); } CUDATree::CUDATree(const Tree* host_tree): @@ -42,28 +47,65 @@ CUDATree::~CUDATree() { } } -void CUDATree::InitCUDAMemory() { - cuda_left_child_.Resize(static_cast(max_leaves_)); - cuda_right_child_.Resize(static_cast(max_leaves_)); - cuda_split_feature_inner_.Resize(static_cast(max_leaves_)); - cuda_split_feature_.Resize(static_cast(max_leaves_)); - cuda_leaf_depth_.Resize(static_cast(max_leaves_)); - cuda_leaf_parent_.Resize(static_cast(max_leaves_)); - cuda_threshold_in_bin_.Resize(static_cast(max_leaves_)); - cuda_threshold_.Resize(static_cast(max_leaves_)); - cuda_decision_type_.Resize(static_cast(max_leaves_)); - cuda_leaf_value_.Resize(static_cast(max_leaves_)); - cuda_internal_weight_.Resize(static_cast(max_leaves_)); - cuda_internal_value_.Resize(static_cast(max_leaves_)); - cuda_leaf_weight_.Resize(static_cast(max_leaves_)); - cuda_leaf_count_.Resize(static_cast(max_leaves_)); - cuda_internal_count_.Resize(static_cast(max_leaves_)); - cuda_split_gain_.Resize(static_cast(max_leaves_)); - SetCUDAMemory(cuda_leaf_value_.RawData(), 0.0f, 1, __FILE__, __LINE__); - SetCUDAMemory(cuda_leaf_weight_.RawData(), 0.0f, 1, __FILE__, __LINE__); - SetCUDAMemory(cuda_leaf_parent_.RawData(), -1, 1, __FILE__, __LINE__); +void CUDATree::InitCUDAMemory(uint8_t* pooled_device_buffer) { + if (pooled_device_buffer != nullptr) { + // carve every per-tree array from the caller-owned slab as non-owning views: + // no per-tree cudaMalloc, and the whole set reads back with one D2H copy in + // ToHost(). Layout must match PooledDeviceBufferSize() and the fused ToHost + // readback: 8-byte types first, then 4-byte, then decision_type (int8). + // Fresh cudaMalloc'd memory was never zeroed either, so reusing a slab with + // stale content across trees is equivalent (every read is write-first, + // except the three first-element memsets below). + pooled_ = true; + const size_t L = static_cast(max_leaves_); + uint8_t* ptr = pooled_device_buffer; + const auto take = [&ptr](size_t bytes) { uint8_t* p = ptr; ptr += bytes; return p; }; + cuda_threshold_.SetView(reinterpret_cast(take(sizeof(double) * L)), L); + cuda_internal_weight_.SetView(reinterpret_cast(take(sizeof(double) * L)), L); + cuda_internal_value_.SetView(reinterpret_cast(take(sizeof(double) * L)), L); + cuda_leaf_value_.SetView(reinterpret_cast(take(sizeof(double) * L)), L); + cuda_leaf_weight_.SetView(reinterpret_cast(take(sizeof(double) * L)), L); + const size_t num_batch_splits = L / 2 + 2; + cuda_batch_splits_.SetView(reinterpret_cast( + take(sizeof(CUDATreeBatchSplit) * num_batch_splits)), num_batch_splits); + cuda_left_child_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_right_child_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_split_feature_inner_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_split_feature_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_leaf_depth_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_leaf_parent_.SetView(reinterpret_cast(take(sizeof(int) * L)), L); + cuda_threshold_in_bin_.SetView(reinterpret_cast(take(sizeof(uint32_t) * L)), L); + cuda_leaf_count_.SetView(reinterpret_cast(take(sizeof(data_size_t) * L)), L); + cuda_internal_count_.SetView(reinterpret_cast(take(sizeof(data_size_t) * L)), L); + cuda_split_gain_.SetView(reinterpret_cast(take(sizeof(float) * L)), L); + cuda_decision_type_.SetView(reinterpret_cast(take(sizeof(int8_t) * L)), L); + } else { + cuda_left_child_.Resize(static_cast(max_leaves_)); + cuda_right_child_.Resize(static_cast(max_leaves_)); + cuda_split_feature_inner_.Resize(static_cast(max_leaves_)); + cuda_split_feature_.Resize(static_cast(max_leaves_)); + cuda_leaf_depth_.Resize(static_cast(max_leaves_)); + cuda_leaf_parent_.Resize(static_cast(max_leaves_)); + cuda_threshold_in_bin_.Resize(static_cast(max_leaves_)); + cuda_threshold_.Resize(static_cast(max_leaves_)); + cuda_decision_type_.Resize(static_cast(max_leaves_)); + cuda_leaf_value_.Resize(static_cast(max_leaves_)); + cuda_internal_weight_.Resize(static_cast(max_leaves_)); + cuda_internal_value_.Resize(static_cast(max_leaves_)); + cuda_leaf_weight_.Resize(static_cast(max_leaves_)); + cuda_leaf_count_.Resize(static_cast(max_leaves_)); + cuda_internal_count_.Resize(static_cast(max_leaves_)); + cuda_split_gain_.Resize(static_cast(max_leaves_)); + } + // async memsets on the default stream, stream-ordered before every consumer + // (SetCUDAMemory would pay one full device sync each) + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_value_.RawData()), 0, sizeof(double))); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_weight_.RawData()), 0, sizeof(double))); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_parent_.RawData()), -1, sizeof(int))); CUDASUCCESS_OR_FATAL(cudaStreamCreate(&cuda_stream_)); - SynchronizeCUDADevice(__FILE__, __LINE__); + if (!pooled_) { + SynchronizeCUDADevice(__FILE__, __LINE__); + } } void CUDATree::InitCUDA() { @@ -102,6 +144,32 @@ int CUDATree::Split(const int leaf_index, return num_leaves_ - 1; } +void CUDATree::SplitBatch(const std::vector& splits) { + if (splits.empty()) { + return; + } + if (cuda_batch_splits_.Size() < splits.size()) { + // preallocate for the deepest possible level so this resizes at most once + cuda_batch_splits_.Resize(std::max(splits.size(), + static_cast(max_leaves_ / 2 + 2))); + } + // async H2D on cuda_stream_: ordered after any prior SplitBatchKernel still + // reading the buffer and before the launch below (same stream). The host + // vector is only rewritten after the caller's per-level full sync, and a + // pageable async H2D returns only once the data is staged, so reuse is safe. + CopyFromHostToCUDADeviceAsync(cuda_batch_splits_.RawData(), + splits.data(), splits.size(), cuda_stream_, __FILE__, __LINE__); + LaunchSplitBatchKernel(static_cast(splits.size())); + // host mirrors, in the exact order the per-split Split() loop would apply them + for (const CUDATreeBatchSplit& split : splits) { + CHECK_EQ(split.num_leaves_at_split, num_leaves_); + RecordBranchFeatures(split.leaf_index, num_leaves_, split.real_feature_index); + leaf_depth_[num_leaves_] = leaf_depth_[split.leaf_index] + 1; + leaf_depth_[split.leaf_index]++; + ++num_leaves_; + } +} + int CUDATree::SplitCategorical(const int leaf_index, const int real_feature_index, const MissingType missing_type, @@ -177,6 +245,39 @@ void CUDATree::ToHost() { leaf_depth_.resize(max_leaves_); const size_t num_leaves_size = static_cast(num_leaves_); + if (pooled_) { + // all per-tree arrays live in one contiguous slab: ONE D2H copy replaces the + // 16 individual synchronous copies below (each pays a full launch+sync + // round trip). The scatter into the host vectors is plain host memcpy. + const size_t L = static_cast(max_leaves_); + const size_t slab_bytes = PooledDeviceBufferSize(max_leaves_); + std::vector staging(slab_bytes); + // cuda_threshold_ is the first view carved from the slab (see InitCUDAMemory) + CopyFromCUDADeviceToHost(staging.data(), + reinterpret_cast(cuda_threshold_.RawData()), slab_bytes, __FILE__, __LINE__); + const uint8_t* p = staging.data(); + const auto scatter = [&p](void* dst, const size_t bytes, const size_t stride) { + if (bytes > 0) std::memcpy(dst, p, bytes); + p += stride; + }; + scatter(threshold_.data(), (num_leaves_size - 1) * sizeof(double), L * sizeof(double)); + scatter(internal_weight_.data(), (num_leaves_size - 1) * sizeof(double), L * sizeof(double)); + scatter(internal_value_.data(), (num_leaves_size - 1) * sizeof(double), L * sizeof(double)); + scatter(leaf_value_.data(), num_leaves_size * sizeof(double), L * sizeof(double)); + scatter(leaf_weight_.data(), num_leaves_size * sizeof(double), L * sizeof(double)); + p += sizeof(CUDATreeBatchSplit) * (L / 2 + 2); // skip the batched split descriptors + scatter(left_child_.data(), (num_leaves_size - 1) * sizeof(int), L * sizeof(int)); + scatter(right_child_.data(), (num_leaves_size - 1) * sizeof(int), L * sizeof(int)); + scatter(split_feature_inner_.data(), (num_leaves_size - 1) * sizeof(int), L * sizeof(int)); + scatter(split_feature_.data(), (num_leaves_size - 1) * sizeof(int), L * sizeof(int)); + scatter(leaf_depth_.data(), num_leaves_size * sizeof(int), L * sizeof(int)); + scatter(leaf_parent_.data(), (num_leaves_size - 1) * sizeof(int), L * sizeof(int)); + scatter(threshold_in_bin_.data(), (num_leaves_size - 1) * sizeof(uint32_t), L * sizeof(uint32_t)); + scatter(leaf_count_.data(), num_leaves_size * sizeof(data_size_t), L * sizeof(data_size_t)); + scatter(internal_count_.data(), (num_leaves_size - 1) * sizeof(data_size_t), L * sizeof(data_size_t)); + scatter(split_gain_.data(), (num_leaves_size - 1) * sizeof(float), L * sizeof(float)); + scatter(decision_type_.data(), (num_leaves_size - 1) * sizeof(int8_t), L * sizeof(int8_t)); + } else { CopyFromCUDADeviceToHost(left_child_.data(), cuda_left_child_.RawData(), num_leaves_size - 1, __FILE__, __LINE__); CopyFromCUDADeviceToHost(right_child_.data(), cuda_right_child_.RawData(), num_leaves_size - 1, __FILE__, __LINE__); CopyFromCUDADeviceToHost(split_feature_inner_.data(), cuda_split_feature_inner_.RawData(), num_leaves_size - 1, __FILE__, __LINE__); @@ -193,6 +294,7 @@ void CUDATree::ToHost() { CopyFromCUDADeviceToHost(internal_weight_.data(), cuda_internal_weight_.RawData(), num_leaves_size - 1, __FILE__, __LINE__); CopyFromCUDADeviceToHost(internal_count_.data(), cuda_internal_count_.RawData(), num_leaves_size - 1, __FILE__, __LINE__); CopyFromCUDADeviceToHost(leaf_depth_.data(), cuda_leaf_depth_.RawData(), num_leaves_size, __FILE__, __LINE__); + } if (num_cat_ > 0) { cuda_cat_boundaries_inner_.Resize(num_cat_ + 1); @@ -203,6 +305,10 @@ void CUDATree::ToHost() { cat_threshold_inner_ = cuda_bitset_inner_.ToHost(); } + ShrinkHostVectorsAndReleaseDevice(); +} + +void CUDATree::ShrinkHostVectorsAndReleaseDevice() { // Shrink host vectors to actual size before they're kept long-term in the // Booster's model list. With max_leaves_=8192 but actual num_leaves_~125 on // Numerai prod, this drops per-tree CPU memory from ~650 KB to ~10 KB. @@ -233,9 +339,13 @@ void CUDATree::ToHost() { // accumulated per-tree GPU memory OOMs a 32GB device after ~6k trees on // the Numerai prod config. cuda_leaf_value_ is the only field needed // post-train (AddPredictionToScore reads it); shrink to actual num_leaves_. + // For pooled trees the slab is reused by the next tree, so the retained leaf + // values must be materialized into owned memory in every case (Resize on a + // view copies-then-owns; Materialize covers num_leaves_ == max_leaves_). if (num_leaves_ > 0 && num_leaves_ < max_leaves_ && cuda_leaf_value_.Size() > 0) { cuda_leaf_value_.Resize(static_cast(num_leaves_)); } + cuda_leaf_value_.Materialize(); cuda_left_child_.Clear(); cuda_right_child_.Clear(); cuda_split_feature_inner_.Clear(); @@ -251,6 +361,7 @@ void CUDATree::ToHost() { cuda_leaf_weight_.Clear(); cuda_internal_count_.Clear(); cuda_split_gain_.Clear(); + cuda_batch_splits_.Clear(); // Destroy the per-tree CUDA stream — only used during construction. if (cuda_stream_ != nullptr) { @@ -259,6 +370,62 @@ void CUDATree::ToHost() { } } +void CUDATree::RebuildFromHostSplits(const std::vector& splits) { + // Host replay of SplitKernel, one split at a time in classic (greedy) order. + // The host mirrors are in their constructor state (the selective growth phase + // never calls Split/SplitBatch), so leaf_parent_[0] == -1 and leaf_depth_[0] + // == 0 hold; leaf_value_[0] holds the root output (set via SetLeafOutput). + num_leaves_ = 1; + for (const CUDATreeHostSplit& s : splits) { + const int new_node_index = num_leaves_ - 1; + const int leaf_index = s.leaf_index; + CHECK_LT(leaf_index, num_leaves_); + const int parent_index = leaf_parent_[leaf_index]; + if (parent_index >= 0) { + // if cur node is left child + if (left_child_[parent_index] == ~leaf_index) { + left_child_[parent_index] = new_node_index; + } else { + right_child_[parent_index] = new_node_index; + } + } + left_child_[new_node_index] = ~leaf_index; + right_child_[new_node_index] = ~num_leaves_; + leaf_parent_[leaf_index] = new_node_index; + leaf_parent_[num_leaves_] = new_node_index; + split_feature_inner_[new_node_index] = s.inner_feature_index; + split_feature_[new_node_index] = s.real_feature_index; + split_gain_[new_node_index] = static_cast(s.gain); + internal_weight_[new_node_index] = s.left_sum_hessians + s.right_sum_hessians; + leaf_weight_[leaf_index] = s.left_sum_hessians; + // save current leaf value to internal node before change (SplitKernel order) + internal_value_[new_node_index] = leaf_value_[leaf_index]; + leaf_value_[leaf_index] = std::isnan(s.left_value) ? 0.0f : s.left_value; + internal_count_[new_node_index] = s.left_count + s.right_count; + leaf_count_[leaf_index] = s.left_count; + leaf_value_[num_leaves_] = std::isnan(s.right_value) ? 0.0f : s.right_value; + leaf_weight_[num_leaves_] = s.right_sum_hessians; + leaf_count_[num_leaves_] = s.right_count; + leaf_depth_[num_leaves_] = leaf_depth_[leaf_index] + 1; + ++leaf_depth_[leaf_index]; + int8_t decision_type = 0; + SetDecisionType(&decision_type, false, kCategoricalMask); + SetDecisionType(&decision_type, s.default_left != 0, kDefaultLeftMask); + SetMissingType(&decision_type, static_cast(s.missing_type)); + decision_type_[new_node_index] = decision_type; + threshold_in_bin_[new_node_index] = s.threshold_in_bin; + threshold_[new_node_index] = s.real_threshold; + RecordBranchFeatures(leaf_index, num_leaves_, s.real_feature_index); + ++num_leaves_; + } + // upload the final leaf values into the (still max_leaves_-sized, possibly + // pooled-view) device array: AddPredictionToScore, Shrinkage and the leaf + // renewal kernels read them + CopyFromHostToCUDADevice(cuda_leaf_value_.RawData(), leaf_value_.data(), + static_cast(num_leaves_), __FILE__, __LINE__); + ShrinkHostVectorsAndReleaseDevice(); +} + void CUDATree::SyncLeafOutputFromHostToCUDA() { CopyFromHostToCUDADevice(cuda_leaf_value_.RawData(), leaf_value_.data(), leaf_value_.size(), __FILE__, __LINE__); } diff --git a/src/io/cuda/cuda_tree.cu b/src/io/cuda/cuda_tree.cu index 01068d027b84..7e0fa8a1d9e2 100644 --- a/src/io/cuda/cuda_tree.cu +++ b/src/io/cuda/cuda_tree.cu @@ -158,6 +158,112 @@ void CUDATree::LaunchSplitKernel(const int leaf_index, cuda_threshold_.RawData()); } +// Batched variant of SplitKernel for the hybrid level-batched growth phase: one +// block per split (blockIdx.x indexes CUDATreeBatchSplit descriptors), threadIdx.x +// plays the role of SplitKernel's global thread index. The level's splits act on +// disjoint leaves; the only shared cells are a sibling pair's parent child-pointers, +// where the compare-then-write is race-free because node indices (>= 0) can never +// equal leaf codes (~leaf < 0). +__global__ void SplitBatchKernel( // split information + const CUDATreeBatchSplit* batch_splits, + // tree structure + int* leaf_parent, + int* leaf_depth, + int* left_child, + int* right_child, + int* split_feature_inner, + int* split_feature, + float* split_gain, + double* internal_weight, + double* internal_value, + data_size_t* internal_count, + double* leaf_weight, + double* leaf_value, + data_size_t* leaf_count, + int8_t* decision_type, + uint32_t* threshold_in_bin, + double* threshold) { + const CUDATreeBatchSplit batch_split = batch_splits[blockIdx.x]; + const int leaf_index = batch_split.leaf_index; + const int num_leaves = batch_split.num_leaves_at_split; + const CUDASplitInfo* cuda_split_info = batch_split.split_info; + const int new_node_index = num_leaves - 1; + const int thread_index = static_cast(threadIdx.x); + const int parent_index = leaf_parent[leaf_index]; + if (thread_index == 0) { + if (parent_index >= 0) { + // if cur node is left child + if (left_child[parent_index] == ~leaf_index) { + left_child[parent_index] = new_node_index; + } else { + right_child[parent_index] = new_node_index; + } + } + left_child[new_node_index] = ~leaf_index; + right_child[new_node_index] = ~num_leaves; + leaf_parent[leaf_index] = new_node_index; + leaf_parent[num_leaves] = new_node_index; + } else if (thread_index == 1) { + // add new node + split_feature_inner[new_node_index] = cuda_split_info->inner_feature_index; + } else if (thread_index == 2) { + split_feature[new_node_index] = batch_split.real_feature_index; + } else if (thread_index == 3) { + split_gain[new_node_index] = static_cast(cuda_split_info->gain); + } else if (thread_index == 4) { + // save current leaf value to internal node before change + internal_weight[new_node_index] = cuda_split_info->left_sum_hessians + cuda_split_info->right_sum_hessians; + leaf_weight[leaf_index] = cuda_split_info->left_sum_hessians; + } else if (thread_index == 5) { + internal_value[new_node_index] = leaf_value[leaf_index]; + leaf_value[leaf_index] = isnan(cuda_split_info->left_value) ? 0.0f : cuda_split_info->left_value; + } else if (thread_index == 6) { + internal_count[new_node_index] = cuda_split_info->left_count + cuda_split_info->right_count; + } else if (thread_index == 7) { + leaf_count[leaf_index] = cuda_split_info->left_count; + } else if (thread_index == 8) { + leaf_value[num_leaves] = isnan(cuda_split_info->right_value) ? 0.0f : cuda_split_info->right_value; + } else if (thread_index == 9) { + leaf_weight[num_leaves] = cuda_split_info->right_sum_hessians; + } else if (thread_index == 10) { + leaf_count[num_leaves] = cuda_split_info->right_count; + } else if (thread_index == 11) { + // update leaf depth + leaf_depth[num_leaves] = leaf_depth[leaf_index] + 1; + leaf_depth[leaf_index]++; + } else if (thread_index == 12) { + decision_type[new_node_index] = 0; + SetDecisionTypeCUDA(&decision_type[new_node_index], false, kCategoricalMask); + SetDecisionTypeCUDA(&decision_type[new_node_index], cuda_split_info->default_left, kDefaultLeftMask); + SetMissingTypeCUDA(&decision_type[new_node_index], static_cast(batch_split.missing_type)); + } else if (thread_index == 13) { + threshold_in_bin[new_node_index] = cuda_split_info->threshold; + } else if (thread_index == 14) { + threshold[new_node_index] = batch_split.real_threshold; + } +} + +void CUDATree::LaunchSplitBatchKernel(const int num_splits) { + SplitBatchKernel<<>>( + cuda_batch_splits_.RawDataReadOnly(), + cuda_leaf_parent_.RawData(), + cuda_leaf_depth_.RawData(), + cuda_left_child_.RawData(), + cuda_right_child_.RawData(), + cuda_split_feature_inner_.RawData(), + cuda_split_feature_.RawData(), + cuda_split_gain_.RawData(), + cuda_internal_weight_.RawData(), + cuda_internal_value_.RawData(), + cuda_internal_count_.RawData(), + cuda_leaf_weight_.RawData(), + cuda_leaf_value_.RawData(), + cuda_leaf_count_.RawData(), + cuda_decision_type_.RawData(), + cuda_threshold_in_bin_.RawData(), + cuda_threshold_.RawData()); +} + __global__ void SplitCategoricalKernel( // split information const int leaf_index, const int real_feature_index, diff --git a/src/io/dataset.cpp b/src/io/dataset.cpp index 7a25f2cd3f21..124b81d187cc 100644 --- a/src/io/dataset.cpp +++ b/src/io/dataset.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -445,6 +446,88 @@ void Dataset::Construct(std::vector>* bin_mappers, gpu_device_id_ = io_config.gpu_device_id; } +template +void Dataset::PushDenseSmallIntRows(const T* data, int32_t nrow, int32_t ncol, + int is_row_major, data_size_t start_row) { + if (is_finish_load_) { + return; + } + const int num_cols = std::min(ncol, num_total_features_); + // per column: the Bin to push into and a fully encoded value->bin table + // (most-freq skip, offset and multi-val adjustment folded in) + constexpr int64_t kLutSize = int64_t(1) << (8 * sizeof(T)); + constexpr int64_t kLutOffset = kLutSize / 2; + std::vector lut(static_cast(num_cols) * kLutSize, Bin::kSkipBin); + std::vector targets(num_cols, nullptr); + #pragma omp parallel for num_threads(OMP_NUM_THREADS()) schedule(static) + for (int col = 0; col < num_cols; ++col) { + const int feature_idx = used_feature_map_[col]; + if (feature_idx < 0) { + continue; + } + const int sub_feature = feature2subfeature_[feature_idx]; + FeatureGroup* fg = feature_groups_[feature2group_[feature_idx]].get(); + targets[col] = fg->PushTargetBin(sub_feature); + const BinMapper* mapper = FeatureBinMapper(feature_idx); + uint32_t* col_lut = lut.data() + static_cast(col) * kLutSize; + for (int64_t v = 0; v < kLutSize; ++v) { + col_lut[v] = fg->EncodeBinForPush( + sub_feature, mapper->ValueToBin(static_cast(static_cast(v - kLutOffset)))); + } + } + // tiles keep the strided per-column reads of row-major data cache-resident + constexpr int32_t kTileRows = 512; + const int32_t num_tiles = (nrow + kTileRows - 1) / kTileRows; + OMP_INIT_EX(); + #pragma omp parallel for num_threads(OMP_NUM_THREADS()) schedule(static) + for (int32_t tile = 0; tile < num_tiles; ++tile) { + OMP_LOOP_EX_BEGIN(); + const int tid = omp_get_thread_num(); + const int32_t r0 = tile * kTileRows; + const int32_t r1 = std::min(nrow, r0 + kTileRows); + uint32_t bins[kTileRows]; + for (int col = 0; col < num_cols; ++col) { + Bin* bin_data = targets[col]; + if (bin_data == nullptr) { + continue; + } + const uint32_t* col_lut = lut.data() + static_cast(col) * kLutSize + kLutOffset; + if (is_row_major) { + for (int32_t r = r0; r < r1; ++r) { + bins[r - r0] = col_lut[data[static_cast(r) * ncol + col]]; + } + } else { + const T* col_ptr = data + static_cast(nrow) * col; + for (int32_t r = r0; r < r1; ++r) { + bins[r - r0] = col_lut[col_ptr[r]]; + } + } + bin_data->PushBlock(tid, start_row + r0, r1 - r0, bins); + } + if (has_raw_) { + for (int col = 0; col < num_cols; ++col) { + const int feature_idx = used_feature_map_[col]; + if (feature_idx < 0 || numeric_feature_map_[feature_idx] < 0) { + continue; + } + float* raw = raw_data_[numeric_feature_map_[feature_idx]].data(); + for (int32_t r = r0; r < r1; ++r) { + const T value = is_row_major ? data[static_cast(r) * ncol + col] + : data[static_cast(nrow) * col + r]; + raw[start_row + r] = static_cast(value); + } + } + } + OMP_LOOP_EX_END(); + } + OMP_THROW_EX(); +} + +template void Dataset::PushDenseSmallIntRows(const int8_t* data, int32_t nrow, int32_t ncol, + int is_row_major, data_size_t start_row); +template void Dataset::PushDenseSmallIntRows(const int16_t* data, int32_t nrow, int32_t ncol, + int is_row_major, data_size_t start_row); + void Dataset::FinishLoad() { if (is_finish_load_) { return; @@ -613,12 +696,52 @@ MultiValBin* Dataset::GetMultiBinFromAllFeatures(const std::vector& of return ret.release(); } +#ifdef USE_CUDA +bool Dataset::CanSkipHostMultiValBinForCUDA() const { + const char* fast_env = std::getenv("EXABOOST_FAST_ROWDATA"); + if (fast_env != nullptr && std::string(fast_env) == std::string("0")) { + return false; + } + const char* verify_env = std::getenv("EXABOOST_FAST_ROWDATA_VERIFY"); + if (verify_env != nullptr && std::string(verify_env) == std::string("1")) { + // verification needs the multi-val bin path as the reference + return false; + } + // CUDARowData's fast build handles dense non-multi-val groups only, and the + // skip is valid only when GetMultiBinFromAllFeatures would produce a dense + // multi-val bin; replicate its sparsity decision exactly + double sum_dense_ratio = 0; + int ncol = 0; + for (int gid = 0; gid < num_groups_; ++gid) { + if (feature_groups_[gid]->is_multi_val_) { + return false; + } + ++ncol; + for (int fid = 0; fid < feature_groups_[gid]->num_feature_; ++fid) { + sum_dense_ratio += 1.0f - feature_groups_[gid]->bin_mappers_[fid]->sparse_rate(); + } + uint8_t bit_type = 0; + bool is_sparse = false; + BinIterator* bin_iterator = nullptr; + GetColWiseData(gid, -1, &bit_type, &is_sparse, &bin_iterator); + if (bin_iterator != nullptr) { + delete bin_iterator; + } + if (is_sparse) { + return false; + } + } + sum_dense_ratio /= ncol; + return (1.0 - sum_dense_ratio) < MultiValBin::multi_val_bin_sparse_threshold; +} +#endif // USE_CUDA + template TrainingShareStates* Dataset::GetShareStates( score_t* gradients, score_t* hessians, const std::vector& is_feature_used, bool is_constant_hessian, bool force_col_wise, bool force_row_wise, - const int num_grad_quant_bins) const { + const int num_grad_quant_bins, bool is_cuda_tree_learner) const { Common::FunctionTimer fun_timer("Dataset::TestMultiThreadingMethod", global_timer); if (force_col_wise && force_row_wise) { @@ -647,8 +770,22 @@ TrainingShareStates* Dataset::GetShareStates( std::vector offsets; share_state->CalcBinOffsets( feature_groups_, &offsets, false); + #ifdef USE_CUDA + // On the CUDA path the host multi-val bin exists only to feed + // CUDARowData::Init, which can build the row-wise data directly from the + // column bins; skip the expensive (and memory-hungry) host build then. + const bool skip_multi_val_bin = is_cuda_tree_learner && + device_type_ == std::string("cuda") && CanSkipHostMultiValBinForCUDA(); + if (skip_multi_val_bin) { + Log::Debug("Dataset::GetShareStates: skipping host multi-val bin build for CUDA (disable with EXABOOST_FAST_ROWDATA=0)"); + } else { + share_state->SetMultiValBin(GetMultiBinFromAllFeatures(offsets), num_data_, + feature_groups_, false, false, num_grad_quant_bins); + } + #else share_state->SetMultiValBin(GetMultiBinFromAllFeatures(offsets), num_data_, feature_groups_, false, false, num_grad_quant_bins); + #endif // USE_CUDA share_state->is_col_wise = false; share_state->is_constant_hessian = is_constant_hessian; return share_state; @@ -732,19 +869,19 @@ template TrainingShareStates* Dataset::GetShareStates( score_t* gradients, score_t* hessians, const std::vector& is_feature_used, bool is_constant_hessian, bool force_col_wise, bool force_row_wise, - const int num_grad_quant_bins) const; + const int num_grad_quant_bins, bool is_cuda_tree_learner) const; template TrainingShareStates* Dataset::GetShareStates( score_t* gradients, score_t* hessians, const std::vector& is_feature_used, bool is_constant_hessian, bool force_col_wise, bool force_row_wise, - const int num_grad_quant_bins) const; + const int num_grad_quant_bins, bool is_cuda_tree_learner) const; template TrainingShareStates* Dataset::GetShareStates( score_t* gradients, score_t* hessians, const std::vector& is_feature_used, bool is_constant_hessian, bool force_col_wise, bool force_row_wise, - const int num_grad_quant_bins) const; + const int num_grad_quant_bins, bool is_cuda_tree_learner) const; void Dataset::CopyFeatureMapperFrom(const Dataset* dataset) { feature_groups_.clear(); diff --git a/src/io/dense_bin.hpp b/src/io/dense_bin.hpp index 5052b0ead1f3..928f5adfa10f 100644 --- a/src/io/dense_bin.hpp +++ b/src/io/dense_bin.hpp @@ -82,6 +82,15 @@ class DenseBin : public Bin { } } + void PushBlock(int tid, data_size_t start_idx, data_size_t count, const uint32_t* bins) override { + for (data_size_t i = 0; i < count; ++i) { + if (bins[i] != kSkipBin) { + // qualified call devirtualizes and inlines the per-record push + DenseBin::Push(tid, start_idx + i, bins[i]); + } + } + } + void ReSize(data_size_t num_data) override { if (num_data_ != num_data) { num_data_ = num_data; diff --git a/src/treelearner/cuda/cuda_best_split_finder.cpp b/src/treelearner/cuda/cuda_best_split_finder.cpp index c6e1926c684a..3e4e6203e599 100644 --- a/src/treelearner/cuda/cuda_best_split_finder.cpp +++ b/src/treelearner/cuda/cuda_best_split_finder.cpp @@ -51,6 +51,9 @@ CUDABestSplitFinder::~CUDABestSplitFinder() { gpuAssert(cudaStreamDestroy(cuda_streams_[1]), __FILE__, __LINE__); cuda_streams_.clear(); cuda_streams_.shrink_to_fit(); + if (pinned_leaf_best_split_info_ != nullptr) { + gpuAssert(cudaFreeHost(pinned_leaf_best_split_info_), __FILE__, __LINE__); + } } void CUDABestSplitFinder::InitFeatureMetaInfo(const Dataset* train_data) { @@ -309,6 +312,23 @@ void CUDABestSplitFinder::BeforeTrain(const std::vector& is_feature_used CopyFromHostToCUDADevice(cuda_is_feature_used_bytree_.RawData(), is_feature_used_bytree.data(), is_feature_used_bytree.size(), __FILE__, __LINE__); + // hybrid batched find: compact the task grid to this tree's feature sample + // (see host_used_task_indices_); only rebuilt/uploaded when sampling is active + host_used_task_indices_.clear(); + for (int task_index = 0; task_index < num_tasks_; ++task_index) { + if (is_feature_used_bytree[split_find_tasks_[task_index].inner_feature_index]) { + host_used_task_indices_.push_back(task_index); + } + } + num_used_tasks_ = static_cast(host_used_task_indices_.size()); + if (num_used_tasks_ < num_tasks_ && num_used_tasks_ > 0) { + if (cuda_used_task_indices_.Size() < static_cast(num_tasks_)) { + cuda_used_task_indices_.Resize(static_cast(num_tasks_)); + } + CopyFromHostToCUDADevice(cuda_used_task_indices_.RawData(), + host_used_task_indices_.data(), + host_used_task_indices_.size(), __FILE__, __LINE__); + } } void CUDABestSplitFinder::FindBestSplitsForLeaf( @@ -325,7 +345,8 @@ void CUDABestSplitFinder::FindBestSplitsForLeaf( const uint8_t smaller_num_bits_in_histogram_bins, const uint8_t larger_num_bits_in_histogram_bins, const bool smaller_leaf_below_max_depth, - const bool larger_leaf_below_max_depth) { + const bool larger_leaf_below_max_depth, + const bool synchronize) { const bool is_smaller_leaf_valid = (num_data_in_smaller_leaf > min_data_in_leaf_ && sum_hessians_in_smaller_leaf > min_sum_hessian_in_leaf_ && smaller_leaf_below_max_depth); @@ -342,10 +363,43 @@ void CUDABestSplitFinder::FindBestSplitsForLeaf( } global_timer.Start("CUDABestSplitFinder::LaunchSyncBestSplitForLeafKernel"); LaunchSyncBestSplitForLeafKernel(smaller_leaf_index, larger_leaf_index, is_smaller_leaf_valid, is_larger_leaf_valid); - SynchronizeCUDADevice(__FILE__, __LINE__); + if (synchronize) { + SynchronizeCUDADevice(__FILE__, __LINE__); + } global_timer.Stop("CUDABestSplitFinder::LaunchSyncBestSplitForLeafKernel"); } +void CUDABestSplitFinder::EnsureHybridLevelCapacity(const int num_pairs) { + const size_t needed = 2 * static_cast(num_tasks_) * static_cast(num_pairs); + if (cuda_best_split_info_.Size() < needed) { + cuda_best_split_info_.Resize(needed); + // (re)initialize the categorical-threshold pointers of every slot; the batched + // path never runs with categorical features, so this just nulls them so the + // sync kernel's wholesale struct copies stay safe + AllocateCatVectors(cuda_best_split_info_.RawData(), cuda_cat_threshold_feature_.RawData(), + cuda_cat_threshold_real_feature_.RawData(), needed); + } +} + +void CUDABestSplitFinder::FindBestSplitsForLevel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const score_t* grad_scale, + const score_t* hess_scale) { + if (num_pairs <= 0) { + return; + } + EnsureHybridLevelCapacity(num_pairs); + // order the batched find after the whole batched histogram phase of this level + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], hist_subtract_done_events_[0], 0)); + if (grad_scale != nullptr && hess_scale != nullptr) { + LaunchFindBestSplitsDiscretizedForLevelKernel(pair_descs, num_pairs, grad_scale, hess_scale); + } else { + LaunchFindBestSplitsForLevelKernel(pair_descs, num_pairs); + } + LaunchSyncBestSplitForLevelKernel(pair_descs, num_pairs); +} + const CUDASplitInfo* CUDABestSplitFinder::FindBestFromAllSplits( const int cur_num_leaves, const int smaller_leaf_index, diff --git a/src/treelearner/cuda/cuda_best_split_finder.cu b/src/treelearner/cuda/cuda_best_split_finder.cu index 483358d1e6a7..b8536b7a76ac 100644 --- a/src/treelearner/cuda/cuda_best_split_finder.cu +++ b/src/treelearner/cuda/cuda_best_split_finder.cu @@ -9,6 +9,8 @@ #include "cuda_best_split_finder.hpp" +#include + #include #include @@ -1811,16 +1813,18 @@ void CUDABestSplitFinder::LaunchFindBestSplitsForLeafKernelInner2(LaunchFindBest is_feature_used_by_larger_node = is_feature_used_by_larger_node_.RawData(); } // Order each leaf's FindBestSplits after the histogram it reads, on its own stream, - // instead of a device-wide sync: the smaller leaf reads the constructed histogram and - // the larger leaf reads the subtracted histogram (both produced on the histogram - // constructor's cuda_stream_). cudaStreamWaitEvent keeps the host from stalling and lets - // the two leaves run concurrently. construct_done is recorded whenever either leaf is - // valid, and subtract every split, so the events are current here. + // instead of a device-wide sync: cudaStreamWaitEvent keeps the host from stalling and + // lets the two leaves run concurrently. BOTH leaves wait on the subtract-done event + // (recorded after fix + subtract on the constructor's stream): the smaller leaf's + // histogram is only complete after FixHistogramKernel writes its most-frequent bin, + // and that runs AFTER the construct-done event is recorded, so waiting on + // construct_done alone raced with the fix kernel (nondeterministic splits when the + // per-pair pipelines overlap). if (is_smaller_leaf_valid) { - CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], hist_construct_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], hist_subtract_done_events_[active_hist_pipeline_], 0)); } if (is_larger_leaf_valid) { - CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[1], hist_subtract_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[1], hist_subtract_done_events_[active_hist_pipeline_], 0)); } if (!use_global_memory_) { if (is_smaller_leaf_valid) { @@ -1946,15 +1950,17 @@ template void CUDABestSplitFinder::LaunchFindBestSplitsDiscretizedForLeafKernelInner2(LaunchFindBestSplitsDiscretizedForLeafKernel_PARAMS) { if (!use_global_memory_) { // Order each leaf after the histogram it reads via its own stream (see the - // non-discretized path for the rationale); no device sync between the two leaves. + // non-discretized path for the rationale — both leaves wait on subtract_done, + // which is recorded after the fix kernel completes the smaller histogram); + // no device sync between the two leaves. if (is_smaller_leaf_valid) { - CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], hist_construct_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], hist_subtract_done_events_[active_hist_pipeline_], 0)); FindBestSplitsDiscretizedForLeafKernel <<>> (FindBestSplitsDiscretizedForLeafKernel_ARGS); } if (is_larger_leaf_valid) { - CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[1], hist_subtract_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[1], hist_subtract_done_events_[active_hist_pipeline_], 0)); FindBestSplitsDiscretizedForLeafKernel <<>> (FindBestSplitsDiscretizedForLeafKernel_ARGS); @@ -1969,6 +1975,385 @@ void CUDABestSplitFinder::LaunchFindBestSplitsDiscretizedForLeafKernelInner2(Lau #undef FindBestSplitsDiscretizedForLeafKernel_ARGS +// ----- Batched per-level find + sync (hybrid growth) ------------------------------- +// One find launch covers every (pair, leaf, task) of a level: blockIdx.x = task, +// blockIdx.y = pair, blockIdx.z = 0 (smaller leaf) / 1 (larger leaf). Each pair +// writes its own region of cuda_best_split_info (2 * num_tasks slots at offset +// pair * 2 * num_tasks) so pairs never conflict. Only the template combination the +// benchmarks exercise is instantiated (no rand / L1 / smoothing); other configs +// fall back to the per-pair path (see SupportsBatchedLevel). + +template +__global__ void FindBestSplitsForLevelKernel( + const int8_t* is_feature_used_bytree, + const int num_tasks, + const SplitFindTask* tasks, + const int* used_task_indices, + CUDARandom* cuda_randoms, + const CUDAHybridPairDescriptor* pair_descs, + const data_size_t min_data_in_leaf, + const double min_sum_hessian_in_leaf, + const double min_gain_to_split, + const double lambda_l1, + const double lambda_l2, + const double path_smooth, + CUDASplitInfo* cuda_best_split_info) { + const unsigned int pair_index = blockIdx.y; + const bool is_larger = (blockIdx.z == 1); + const CUDAHybridPairDescriptor* desc = pair_descs + pair_index; + // desc->{smaller,larger}_valid carries the HOST-known gates (max_depth); the + // min_data/min_hessian gates are evaluated on-device from the leaf struct so + // the speculative single-sync flow can enqueue this kernel before the child + // statistics are read back. In the classic flow the host flags already include + // the identical data/hessian conditions, so the device check is a no-op. + if (is_larger ? !desc->larger_valid : !desc->smaller_valid) { + return; + } + const CUDALeafSplitsStruct* leaf_splits = is_larger ? desc->larger_struct : desc->smaller_struct; + const data_size_t num_data = leaf_splits->num_data_in_leaf; + const double leaf_sum_hessians = leaf_splits->sum_of_hessians; + if (leaf_splits->leaf_index < 0 || num_data <= min_data_in_leaf || + leaf_sum_hessians <= min_sum_hessian_in_leaf) { + return; + } + // feature-sampled trees launch one block per USED task only; the output slot + // keeps the ORIGINAL task index (unused slots are masked out by the sync) + const unsigned int task_index = used_task_indices == nullptr ? + blockIdx.x : static_cast(used_task_indices[blockIdx.x]); + const SplitFindTask* task = tasks + task_index; + const double parent_gain = leaf_splits->gain; + const double sum_gradients = leaf_splits->sum_of_gradients; + const double sum_hessians = leaf_sum_hessians + 2 * kEpsilon; + const double parent_output = leaf_splits->leaf_value; + const unsigned int output_offset = pair_index * (2 * static_cast(num_tasks)) + + (is_larger ? task_index + num_tasks : task_index); + CUDASplitInfo* out = cuda_best_split_info + output_offset; + CUDARandom* cuda_random = USE_RAND ? + (is_larger ? cuda_randoms + task_index * 2 + 1 : cuda_randoms + task_index * 2) : nullptr; + if (is_feature_used_bytree[task->inner_feature_index]) { + // no categorical branch: the batched path is gated on !has_categorical_feature_ + const hist_t* hist_ptr = leaf_splits->hist_in_leaf + task->hist_offset * 2; + if (!task->reverse) { + FindBestSplitsForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients, sum_hessians, num_data, parent_output, + out); + } else { + FindBestSplitsForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients, sum_hessians, num_data, parent_output, + out); + } + } else { + out->is_valid = false; + } +} + +template +__global__ void FindBestSplitsDiscretizedForLevelKernel( + const int8_t* is_feature_used_bytree, + const int num_tasks, + const SplitFindTask* tasks, + const int* used_task_indices, + CUDARandom* cuda_randoms, + const CUDAHybridPairDescriptor* pair_descs, + const data_size_t min_data_in_leaf, + const double min_sum_hessian_in_leaf, + const double min_gain_to_split, + const double lambda_l1, + const double lambda_l2, + const double path_smooth, + const score_t* grad_scale, + const score_t* hess_scale, + CUDASplitInfo* cuda_best_split_info) { + const unsigned int pair_index = blockIdx.y; + const bool is_larger = (blockIdx.z == 1); + const CUDAHybridPairDescriptor* desc = pair_descs + pair_index; + if (is_larger ? !desc->larger_valid : !desc->smaller_valid) { + return; + } + const CUDALeafSplitsStruct* leaf_splits = is_larger ? desc->larger_struct : desc->smaller_struct; + const unsigned int task_index = used_task_indices == nullptr ? + blockIdx.x : static_cast(used_task_indices[blockIdx.x]); + const SplitFindTask* task = tasks + task_index; + const double parent_gain = leaf_splits->gain; + const int64_t sum_gradients_hessians = leaf_splits->sum_of_gradients_hessians; + const data_size_t num_data = is_larger ? desc->num_data_in_larger_leaf : desc->num_data_in_smaller_leaf; + const double parent_output = leaf_splits->leaf_value; + const unsigned int output_offset = pair_index * (2 * static_cast(num_tasks)) + + (is_larger ? task_index + num_tasks : task_index); + CUDASplitInfo* out = cuda_best_split_info + output_offset; + CUDARandom* cuda_random = USE_RAND ? + (is_larger ? cuda_randoms + task_index * 2 + 1 : cuda_randoms + task_index * 2) : nullptr; + const bool use_16bit_bin = is_larger ? (desc->larger_num_bits <= 16) : (desc->smaller_num_bits <= 16); + if (is_feature_used_bytree[task->inner_feature_index]) { + // no categorical branch: quantized training rejects categorical features + if (!task->reverse) { + if (use_16bit_bin) { + const int32_t* hist_ptr = + reinterpret_cast(leaf_splits->hist_in_leaf) + task->hist_offset; + FindBestSplitsDiscretizedForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients_hessians, num_data, parent_output, + *grad_scale, *hess_scale, out); + } else { + // 32-bit histogram is int64-per-bin; read as int64 (see the per-pair kernel) + const int64_t* hist_ptr = + reinterpret_cast(leaf_splits->hist_in_leaf) + task->hist_offset; + FindBestSplitsDiscretizedForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients_hessians, num_data, parent_output, + *grad_scale, *hess_scale, out); + } + } else { + if (use_16bit_bin) { + const int32_t* hist_ptr = + reinterpret_cast(leaf_splits->hist_in_leaf) + task->hist_offset; + FindBestSplitsDiscretizedForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients_hessians, num_data, parent_output, + *grad_scale, *hess_scale, out); + } else { + const int64_t* hist_ptr = + reinterpret_cast(leaf_splits->hist_in_leaf) + task->hist_offset; + FindBestSplitsDiscretizedForLeafKernelInner( + hist_ptr, task, cuda_random, + lambda_l1, lambda_l2, path_smooth, + min_data_in_leaf, min_sum_hessian_in_leaf, min_gain_to_split, + parent_gain, sum_gradients_hessians, num_data, parent_output, + *grad_scale, *hess_scale, out); + } + } + } else { + out->is_valid = false; + } +} + +// Batched reduction over the per-pair task regions: blockIdx.x = 0 (smaller) / 1 +// (larger), blockIdx.y = pair, blockIdx.z = task block (num_tasks may exceed one +// sync block for wide datasets). Each block reduces its slice of one leaf's +// 2 * num_tasks region exactly like SyncBestSplitForLeafKernel (same block size, +// same task slicing, same reduction order, hence bit-identical results) and +// writes that leaf's slot of the per-leaf best-split cache (block z writes slot +// leaf + z * num_leaves, exactly like the per-pair kernel); leaves of different +// pairs are disjoint, so no conflicts. Multi-block leaves are merged afterwards +// by SyncBestSplitForLevelKernelAllBlocks (the per-pair AllBlocks replica). +__global__ void SyncBestSplitForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + CUDASplitInfo* cuda_leaf_best_split_info, + const SplitFindTask* tasks, + const int8_t* is_feature_used_bytree, + const CUDASplitInfo* cuda_best_split_info, + const int num_tasks, + const int num_leaves, + const data_size_t min_data_in_leaf, + const double min_sum_hessian_in_leaf) { + __shared__ double shared_gain_buffer[WARPSIZE]; + __shared__ bool shared_found_buffer[WARPSIZE]; + __shared__ uint32_t shared_thread_index_buffer[WARPSIZE]; + const unsigned int pair_index = blockIdx.y; + const bool is_larger = (blockIdx.x == 1); + const unsigned int leaf_block_index = blockIdx.z; + const CUDAHybridPairDescriptor* desc = pair_descs + pair_index; + // leaf index and validity mirror the find kernel: the leaf index comes from + // the struct (written by the batched apply kernels; equal to the host value in + // the classic flow) and the host validity flags are ANDed with the on-device + // min_data/min_hessian gates so the speculative single-sync flow needs no + // host-side child statistics. + const CUDALeafSplitsStruct* leaf_splits = is_larger ? desc->larger_struct : desc->smaller_struct; + const int leaf_index = leaf_splits->leaf_index; + if (leaf_index < 0) { + return; + } + bool leaf_valid = is_larger ? (desc->larger_valid != 0) : (desc->smaller_valid != 0); + leaf_valid = leaf_valid && leaf_splits->num_data_in_leaf > min_data_in_leaf && + leaf_splits->sum_of_hessians > min_sum_hessian_in_leaf; + if (!leaf_valid) { + // mirror SetInvalidLeafSplitInfoKernel of the per-pair path (block-slot + // copies for z >= 1 stay stale, so the AllBlocks merge skips invalid leaves) + if (threadIdx.x == 0 && leaf_block_index == 0) { + cuda_leaf_best_split_info[leaf_index].is_valid = false; + } + return; + } + const uint32_t threadIdx_x = threadIdx.x; + const CUDASplitInfo* pair_split_info = cuda_best_split_info + + static_cast(pair_index) * (2 * static_cast(num_tasks)); + bool best_found = false; + double best_gain = kMinScore; + uint32_t shared_read_index = 0; + const int task_index = static_cast(leaf_block_index * blockDim.x + threadIdx_x); + const uint32_t read_index = is_larger ? static_cast(task_index + num_tasks) : static_cast(task_index); + // Feature-sampled trees launch the batched find over USED tasks only, leaving + // unused tasks' output slots stale; mask them to not-found here. This is + // bit-identical to the per-pair reduction, where those lanes hold the + // is_valid=false slots the (full-grid) find would have written: a not-found + // lane never propagates its gain/read-index through ReduceBestGain. + if (task_index < num_tasks && + (is_feature_used_bytree == nullptr || + is_feature_used_bytree[tasks[task_index].inner_feature_index])) { + best_found = pair_split_info[read_index].is_valid; + best_gain = pair_split_info[read_index].gain; + shared_read_index = read_index; + } + __syncthreads(); + const uint32_t best_read_index = ReduceBestGain(best_gain, best_found, shared_read_index, + shared_gain_buffer, shared_found_buffer, shared_thread_index_buffer); + if (threadIdx.x == 0) { + CUDASplitInfo* cuda_split_info = cuda_leaf_best_split_info + + static_cast(leaf_index) + leaf_block_index * static_cast(num_leaves); + const int best_task_index = best_read_index >= static_cast(num_tasks) ? + static_cast(best_read_index) - num_tasks : static_cast(best_read_index); + // when nothing was found the reduction returns thread 0's own lane, whose + // slot may be stale under task compaction: re-apply the mask before reading + const bool best_task_used = is_feature_used_bytree == nullptr || + is_feature_used_bytree[tasks[best_task_index].inner_feature_index]; + const CUDASplitInfo* best_split_info = pair_split_info + best_read_index; + if (best_task_used && best_split_info->is_valid) { + *cuda_split_info = *best_split_info; + cuda_split_info->inner_feature_index = tasks[best_task_index].inner_feature_index; + cuda_split_info->is_valid = true; + } else { + cuda_split_info->gain = kMinScore; + cuda_split_info->is_valid = false; + } + } +} + +// Cross-block merge for wide datasets (num_tasks > one sync block): one block per +// (leaf role, pair) sequentially folds block slots 1..num_blocks_per_leaf-1 into +// slot 0 with the exact comparison (and hence tie-break order: the LOWEST task +// block wins ties) of the per-pair SyncBestSplitForLeafKernelAllBlocks. +__global__ void SyncBestSplitForLevelKernelAllBlocks( + const CUDAHybridPairDescriptor* pair_descs, + CUDASplitInfo* cuda_leaf_best_split_info, + const unsigned int num_blocks_per_leaf, + const int num_leaves, + const data_size_t min_data_in_leaf, + const double min_sum_hessian_in_leaf) { + const unsigned int pair_index = blockIdx.y; + const bool is_larger = (blockIdx.x == 1); + const CUDAHybridPairDescriptor* desc = pair_descs + pair_index; + const CUDALeafSplitsStruct* leaf_splits = is_larger ? desc->larger_struct : desc->smaller_struct; + const int leaf_index = leaf_splits->leaf_index; + if (leaf_index < 0) { + return; + } + bool leaf_valid = is_larger ? (desc->larger_valid != 0) : (desc->smaller_valid != 0); + leaf_valid = leaf_valid && leaf_splits->num_data_in_leaf > min_data_in_leaf && + leaf_splits->sum_of_hessians > min_sum_hessian_in_leaf; + if (!leaf_valid) { + // the sync kernel's block-slot copies were skipped for this leaf + return; + } + CUDASplitInfo* leaf_split_info = cuda_leaf_best_split_info + leaf_index; + for (unsigned int block_index = 1; block_index < num_blocks_per_leaf; ++block_index) { + const unsigned int leaf_read_pos = static_cast(leaf_index) + + block_index * static_cast(num_leaves); + const CUDASplitInfo* other_split_info = cuda_leaf_best_split_info + leaf_read_pos; + if ((other_split_info->is_valid && leaf_split_info->is_valid && + other_split_info->gain > leaf_split_info->gain) || + (!leaf_split_info->is_valid && other_split_info->is_valid)) { + *leaf_split_info = *other_split_info; + } + } +} + +void CUDABestSplitFinder::LaunchFindBestSplitsForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs) { + const bool compact_tasks = num_used_tasks_ > 0 && num_used_tasks_ < num_tasks_; + if (num_used_tasks_ == 0) { + return; // no usable feature this tree; the sync masks every lane not-found + } + dim3 grid_dim(compact_tasks ? num_used_tasks_ : num_tasks_, num_pairs, 2); + FindBestSplitsForLevelKernel + <<>>( + cuda_is_feature_used_bytree_.RawData(), + num_tasks_, + cuda_split_find_tasks_.RawData(), + compact_tasks ? cuda_used_task_indices_.RawDataReadOnly() : nullptr, + cuda_randoms_.RawData(), + pair_descs, + min_data_in_leaf_, + min_sum_hessian_in_leaf_, + min_gain_to_split_, + lambda_l1_, + lambda_l2_, + path_smooth_, + cuda_best_split_info_.RawData()); +} + +void CUDABestSplitFinder::LaunchFindBestSplitsDiscretizedForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const score_t* grad_scale, + const score_t* hess_scale) { + const bool compact_tasks = num_used_tasks_ > 0 && num_used_tasks_ < num_tasks_; + if (num_used_tasks_ == 0) { + return; // no usable feature this tree; the sync masks every lane not-found + } + dim3 grid_dim(compact_tasks ? num_used_tasks_ : num_tasks_, num_pairs, 2); + FindBestSplitsDiscretizedForLevelKernel + <<>>( + cuda_is_feature_used_bytree_.RawData(), + num_tasks_, + cuda_split_find_tasks_.RawData(), + compact_tasks ? cuda_used_task_indices_.RawDataReadOnly() : nullptr, + cuda_randoms_.RawData(), + pair_descs, + min_data_in_leaf_, + min_sum_hessian_in_leaf_, + min_gain_to_split_, + lambda_l1_, + lambda_l2_, + path_smooth_, + grad_scale, + hess_scale, + cuda_best_split_info_.RawData()); +} + +void CUDABestSplitFinder::LaunchSyncBestSplitForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs) { + const int num_blocks_per_leaf = (num_tasks_ + NUM_TASKS_PER_SYNC_BLOCK - 1) / NUM_TASKS_PER_SYNC_BLOCK; + const bool compact_tasks = num_used_tasks_ < num_tasks_; + dim3 grid_dim(2, num_pairs, num_blocks_per_leaf); + SyncBestSplitForLevelKernel<<>>( + pair_descs, + cuda_leaf_best_split_info_.RawData(), + cuda_split_find_tasks_.RawData(), + compact_tasks ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr, + cuda_best_split_info_.RawData(), + num_tasks_, + num_leaves_, + min_data_in_leaf_, + min_sum_hessian_in_leaf_); + if (num_blocks_per_leaf > 1) { + // stream-ordered after the sync kernel above; same stream as the batched + // find, so no extra synchronization is needed + dim3 merge_grid_dim(2, num_pairs); + SyncBestSplitForLevelKernelAllBlocks<<>>( + pair_descs, + cuda_leaf_best_split_info_.RawData(), + static_cast(num_blocks_per_leaf), + num_leaves_, + min_data_in_leaf_, + min_sum_hessian_in_leaf_); + } +} + + // ----- Forced-split support ------------------------------------------------------- // Mirrors FeatureHistogram::GatherInfoForThresholdNumericalInner. Single-threaded so // the bin accumulation order matches the CPU loop and results are bit-identical. @@ -2363,6 +2748,34 @@ void CUDABestSplitFinder::SyncLeafBestSplitToHost( } } +void CUDABestSplitFinder::SyncAllLeafBestSplitsToHost(const int num_leaves, std::vector* out) const { + out->resize(static_cast(num_leaves)); + // synchronous D2H on the legacy default stream: implicitly waits for all + // preceding work on the (blocking) streams, so no explicit device sync needed. + // Staged through a PINNED buffer: this copy runs once per level on the + // hybrid critical path, and a pageable sync D2H pays an extra driver staging + // round trip; the host-to-host memcpy of a few KB afterwards is negligible. + if (pinned_leaf_best_split_info_size_ < static_cast(num_leaves)) { + if (pinned_leaf_best_split_info_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaFreeHost(pinned_leaf_best_split_info_)); + } + pinned_leaf_best_split_info_size_ = static_cast(num_leaves_ > num_leaves ? num_leaves_ : num_leaves); + CUDASUCCESS_OR_FATAL(cudaHostAlloc(reinterpret_cast(&pinned_leaf_best_split_info_), + pinned_leaf_best_split_info_size_ * sizeof(CUDASplitInfo), cudaHostAllocDefault)); + } + CopyFromCUDADeviceToHost(pinned_leaf_best_split_info_, cuda_leaf_best_split_info_.RawDataReadOnly(), + static_cast(num_leaves), __FILE__, __LINE__); + std::memcpy(reinterpret_cast(out->data()), pinned_leaf_best_split_info_, + static_cast(num_leaves) * sizeof(CUDASplitInfo)); + // the raw copy brings over device categorical-threshold pointers; scrub them so the + // host-side destructor never frees device memory (hybrid growth is numerical-only) + for (CUDASplitInfo& info : *out) { + info.num_cat_threshold = 0; + info.cat_threshold = nullptr; + info.cat_threshold_real = nullptr; + } +} + void CUDABestSplitFinder::LaunchFindBestFromAllSplitsKernel( const int cur_num_leaves, const int smaller_leaf_index, const int larger_leaf_index, diff --git a/src/treelearner/cuda/cuda_best_split_finder.hpp b/src/treelearner/cuda/cuda_best_split_finder.hpp index 9705b2f5e9e0..72b8d9101e23 100644 --- a/src/treelearner/cuda/cuda_best_split_finder.hpp +++ b/src/treelearner/cuda/cuda_best_split_finder.hpp @@ -13,6 +13,7 @@ #include #include +#include #include #include @@ -62,11 +63,15 @@ class CUDABestSplitFinder { * FindBestSplits kernels can be ordered after histogram construction/subtraction * via cudaStreamWaitEvent instead of a per-split device sync. Called once after * both objects are initialized. */ - void SetHistogramEvents(cudaEvent_t construct_done_event, cudaEvent_t subtract_done_event) { - hist_construct_done_event_ = construct_done_event; - hist_subtract_done_event_ = subtract_done_event; + void SetHistogramEvents(const cudaEvent_t* construct_done_events, const cudaEvent_t* subtract_done_events) { + hist_construct_done_events_ = construct_done_events; + hist_subtract_done_events_ = subtract_done_events; } + /*! \brief select which histogram pipeline's completion events the next + * FindBestSplitsForLeaf call waits on (hybrid level-batched growth) */ + void SetActiveHistPipeline(const int pipeline) { active_hist_pipeline_ = pipeline; } + void BeforeTrain(const std::vector& is_feature_used_bytree); void FindBestSplitsForLeaf( @@ -83,7 +88,48 @@ class CUDABestSplitFinder { const uint8_t smaller_num_bits_in_histogram_bins, const uint8_t larger_num_bits_in_histogram_bins, const bool smaller_leaf_below_max_depth, - const bool larger_leaf_below_max_depth); + const bool larger_leaf_below_max_depth, + const bool synchronize = true); + + /*! \brief whether the batched per-level find+sync path supports the current + * configuration (only the template combination the benchmarks exercise: + * no extra_trees / L1 / path smoothing, shared-memory histograms, no + * per-node feature selection, no categorical features). Wide shapes + * (num_tasks > one sync block) use the multi-block batched sync mirroring + * the per-pair two-stage reduction; EXABOOST_BATCH_WIDE=0 disables that. */ + bool SupportsBatchedLevel() const { + return !extra_trees_ && lambda_l1_ <= 0.0f && !use_smoothing_ && + !use_global_memory_ && !select_features_by_node_ && + !has_categorical_feature_ && + (num_tasks_ <= NUM_TASKS_PER_SYNC_BLOCK || ExaboostBatchWideEnabled()); + } + + /*! \brief one-line gate dump for EXABOOST_HYBRID_DIAG */ + std::string BatchedLevelGateDiag() const { + char buf[256]; + snprintf(buf, sizeof(buf), + "finder: extra_trees=%d l1=%f smoothing=%d global_mem=%d " + "by_node=%d categorical=%d num_tasks=%d (cap %d, batch_wide=%d)", + static_cast(extra_trees_), lambda_l1_, + static_cast(use_smoothing_), static_cast(use_global_memory_), + static_cast(select_features_by_node_), + static_cast(has_categorical_feature_), num_tasks_, + NUM_TASKS_PER_SYNC_BLOCK, static_cast(ExaboostBatchWideEnabled())); + return std::string(buf); + } + + /*! \brief Batched per-level best-split search for hybrid growth: one find launch + * and one sync launch cover every sibling pair of a level. Each pair writes its + * own region of the task output buffer and its own (disjoint) per-leaf slots of + * the best-split cache. Waits on the histogram constructor's batched + * subtract-done event before reading any histogram; grad_scale/hess_scale are + * null for non-quantized training. The caller is expected to synchronize via + * SyncAllLeafBestSplitsToHost afterwards. */ + void FindBestSplitsForLevel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const score_t* grad_scale, + const score_t* hess_scale); const CUDASplitInfo* FindBestFromAllSplits( const int cur_num_leaves, @@ -140,6 +186,18 @@ class CUDABestSplitFinder { uint32_t* larger_leaf_best_split_threshold, uint8_t* larger_leaf_best_split_default_left); + // Copy the whole device per-leaf best-split cache for leaves [0, num_leaves) to the + // host in a single transfer (synchronizes the device). Used by the hybrid + // level-batched growth phase, which needs every frontier leaf's candidate at once. + // The categorical-threshold pointers inside the copied structs are device pointers + // and must not be dereferenced on the host. + void SyncAllLeafBestSplitsToHost(const int num_leaves, std::vector* out) const; + + // Device pointer to a leaf's cached best split, for passing to CUDATree::Split. + const CUDASplitInfo* leaf_best_split_info_ptr(const int leaf_index) const { + return cuda_leaf_best_split_info_.RawDataReadOnly() + leaf_index; + } + private: void LaunchComputeForcedSplitKernel( const CUDALeafSplitsStruct* leaf_splits, @@ -206,6 +264,25 @@ class CUDABestSplitFinder { const bool is_smaller_leaf_valid, const bool is_larger_leaf_valid); + // ---- batched per-level launchers (hybrid growth) ---- + // Grow the task output buffer so every pair of a level has its own region of + // 2 * num_tasks_ CUDASplitInfo slots. No-op once capacity is reached. + void EnsureHybridLevelCapacity(const int num_pairs); + + void LaunchFindBestSplitsForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs); + + void LaunchFindBestSplitsDiscretizedForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const score_t* grad_scale, + const score_t* hess_scale); + + void LaunchSyncBestSplitForLevelKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs); + void LaunchFindBestFromAllSplitsKernel( const int cur_num_leaves, const int smaller_leaf_index, @@ -228,6 +305,10 @@ class CUDABestSplitFinder { // Host memory int num_features_; int num_leaves_; + /*! \brief pinned staging buffer of SyncAllLeafBestSplitsToHost (raw bytes; + * CUDASplitInfo is never constructed/destructed in it) */ + mutable CUDASplitInfo* pinned_leaf_best_split_info_ = nullptr; + mutable size_t pinned_leaf_best_split_info_size_ = 0; int max_num_bin_in_feature_; std::vector feature_hist_offsets_; std::vector feature_mfb_offsets_; @@ -251,11 +332,22 @@ class CUDABestSplitFinder { std::vector cuda_streams_; /*! \brief histogram constructor completion events (not owned); used to order the * per-leaf FindBestSplits kernels after histogram construction/subtraction. */ - cudaEvent_t hist_construct_done_event_ = nullptr; - cudaEvent_t hist_subtract_done_event_ = nullptr; + const cudaEvent_t* hist_construct_done_events_ = nullptr; + const cudaEvent_t* hist_subtract_done_events_ = nullptr; + int active_hist_pipeline_ = 0; // for best split find tasks std::vector split_find_tasks_; int num_tasks_; + /*! \brief hybrid batched find: per-tree list of the task indices whose feature + * is in this tree's feature_fraction sample. The batched find launches one + * block per USED task only (a 10x saving for wide sampled datasets); output + * slots keep the ORIGINAL task index so the batched sync reduces the exact + * same lanes as the per-pair kernels (unused lanes masked not-found, exactly + * as if the find had written is_valid=false there). Empty/full -> nullptr + * passed to the kernels (zero overhead without sampling). */ + std::vector host_used_task_indices_; + CUDAVector cuda_used_task_indices_; + int num_used_tasks_ = 0; // use global memory bool use_global_memory_; // number of total bins in the dataset diff --git a/src/treelearner/cuda/cuda_data_partition.cpp b/src/treelearner/cuda/cuda_data_partition.cpp index 3a5c9faf14be..9f9f99d1760a 100644 --- a/src/treelearner/cuda/cuda_data_partition.cpp +++ b/src/treelearner/cuda/cuda_data_partition.cpp @@ -8,7 +8,9 @@ #ifdef USE_CUDA #include +#include #include +#include #include #include "cuda_data_partition.hpp" @@ -60,6 +62,12 @@ CUDADataPartition::~CUDADataPartition() { CUDASUCCESS_OR_FATAL(cudaStreamDestroy(cuda_streams_[3])); cuda_streams_.clear(); cuda_streams_.shrink_to_fit(); + if (indices_copy_done_event_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaEventDestroy(indices_copy_done_event_)); + } + if (pinned_split_info_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaFreeHost(pinned_split_info_)); + } } void CUDADataPartition::Init() { @@ -79,7 +87,11 @@ void CUDADataPartition::Init() { cuda_hist_pool_.Resize(static_cast(num_leaves_)); CopyFromHostToCUDADevice(cuda_hist_pool_.RawData(), &cuda_hist_, 1, __FILE__, __LINE__); - cuda_split_info_buffer_.Resize(18); + // 18 ints per split; slot 0 serves the classic immediate mode, slots [0, num_leaves) + // serve the hybrid level-batched mode's deferred readback + cuda_split_info_buffer_.Resize(18 * static_cast(num_leaves_) + 18); + // one smaller-child count per split of a level (a level has < num_leaves splits) + cuda_level_smaller_counts_.Resize(static_cast(num_leaves_) + 1); cuda_leaf_output_.Resize(static_cast(num_leaves_)); @@ -88,6 +100,7 @@ void CUDADataPartition::Init() { gpuAssert(cudaStreamCreate(&cuda_streams_[1]), __FILE__, __LINE__); gpuAssert(cudaStreamCreate(&cuda_streams_[2]), __FILE__, __LINE__); gpuAssert(cudaStreamCreate(&cuda_streams_[3]), __FILE__, __LINE__); + CUDASUCCESS_OR_FATAL(cudaEventCreateWithFlags(&indices_copy_done_event_, cudaEventDisableTiming)); cuda_num_data_.InitFromHostVector(std::vector{num_data_}); use_bagging_ = false; @@ -98,10 +111,11 @@ void CUDADataPartition::BeforeTrain() { if (!use_bagging_) { LaunchFillDataIndicesBeforeTrain(); } - SetCUDAMemory(cuda_leaf_num_data_.RawData(), 0, static_cast(num_leaves_), __FILE__, __LINE__); - SetCUDAMemory(cuda_leaf_data_start_.RawData(), 0, static_cast(num_leaves_), __FILE__, __LINE__); - SetCUDAMemory(cuda_leaf_data_end_.RawData(), 0, static_cast(num_leaves_), __FILE__, __LINE__); - SynchronizeCUDADevice(__FILE__, __LINE__); + // async memsets on the default stream (SetCUDAMemory would pay one full + // device sync each); the synchronous copies below order after them + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_num_data_.RawData()), 0, sizeof(data_size_t) * static_cast(num_leaves_))); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_data_start_.RawData()), 0, sizeof(data_size_t) * static_cast(num_leaves_))); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_leaf_data_end_.RawData()), 0, sizeof(data_size_t) * static_cast(num_leaves_))); if (!use_bagging_) { CopyFromCUDADeviceToCUDADevice(cuda_leaf_num_data_.RawData(), cuda_num_data_.RawData(), 1, __FILE__, __LINE__); CopyFromCUDADeviceToCUDADevice(cuda_leaf_data_end_.RawData(), cuda_num_data_.RawData(), 1, __FILE__, __LINE__); @@ -109,7 +123,6 @@ void CUDADataPartition::BeforeTrain() { CopyFromHostToCUDADevice(cuda_leaf_num_data_.RawData(), &num_used_indices_, 1, __FILE__, __LINE__); CopyFromHostToCUDADevice(cuda_leaf_data_end_.RawData(), &num_used_indices_, 1, __FILE__, __LINE__); } - SynchronizeCUDADevice(__FILE__, __LINE__); CopyFromHostToCUDADevice(cuda_hist_pool_.RawData(), &cuda_hist_, 1, __FILE__, __LINE__); } @@ -138,7 +151,9 @@ void CUDADataPartition::Split( double* left_leaf_sum_of_gradients, double* right_leaf_sum_of_gradients, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data) { + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main, + const int deferred_slot) { CalcBlockDim(num_data_in_leaf); // CalcBlockDim is non-monotonic in num_data_in_leaf: the per-block data count is // rounded up to a power of two, so a *smaller* leaf can require *more* blocks than @@ -155,10 +170,21 @@ void CUDADataPartition::Split( // here, mirroring the resize logic in ResetTrainingData. if (grid_dim_ > max_num_split_indices_blocks_) { max_num_split_indices_blocks_ = grid_dim_; - cuda_block_data_to_left_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); - cuda_block_data_to_right_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); - SetCUDAMemory(cuda_block_data_to_left_offset_.RawData(), 0, static_cast(max_num_split_indices_blocks_) + 1, __FILE__, __LINE__); - SetCUDAMemory(cuda_block_data_to_right_offset_.RawData(), 0, static_cast(max_num_split_indices_blocks_) + 1, __FILE__, __LINE__); + // the batched level apply (SplitLevelBatched) may already have grown the + // buffers past this size; never shrink them back + if (cuda_block_data_to_left_offset_.Size() < static_cast(max_num_split_indices_blocks_) + 1) { + cuda_block_data_to_left_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); + cuda_block_data_to_right_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); + SetCUDAMemory(cuda_block_data_to_left_offset_.RawData(), 0, cuda_block_data_to_left_offset_.Size(), __FILE__, __LINE__); + SetCUDAMemory(cuda_block_data_to_right_offset_.RawData(), 0, cuda_block_data_to_right_offset_.Size(), __FILE__, __LINE__); + } + } + // the previous split's CopyDataIndicesKernel (stream 2) may still be reading the + // shared scratch this split's kernels overwrite; order after it + if (indices_copy_done_event_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], indices_copy_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[1], indices_copy_done_event_, 0)); + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[3], indices_copy_done_event_, 0)); } global_timer.Start("GenDataToLeftBitVector"); GenDataToLeftBitVector(num_data_in_leaf, @@ -188,7 +214,10 @@ void CUDADataPartition::Split( left_leaf_sum_of_gradients, right_leaf_sum_of_gradients, global_left_leaf_num_data, - global_right_leaf_num_data); + global_right_leaf_num_data, + point_structs_at_main, + deferred_slot, + leaf_data_start); global_timer.Stop("SplitInner"); } @@ -224,6 +253,207 @@ void CUDADataPartition::GenDataToLeftBitVector( } } +void CUDADataPartition::FinishSplitBatch(const int num_splits, std::vector* out) { + out->resize(static_cast(num_splits) * 18); + // synchronous D2H on the legacy default stream: implicitly waits for all + // preceding work on the (blocking) streams, so no explicit device sync + // needed. Staged through a PINNED buffer (once-per-level critical path; a + // pageable sync D2H pays an extra driver staging round trip). + const size_t num_ints = static_cast(num_splits) * 18; + if (pinned_split_info_size_ < num_ints) { + if (pinned_split_info_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaFreeHost(pinned_split_info_)); + } + const size_t capacity = std::max(num_ints, static_cast(num_leaves_ / 2 + 2) * 18); + CUDASUCCESS_OR_FATAL(cudaHostAlloc(reinterpret_cast(&pinned_split_info_), + capacity * sizeof(int), cudaHostAllocDefault)); + pinned_split_info_size_ = capacity; + } + CopyFromCUDADeviceToHost(pinned_split_info_, cuda_split_info_buffer_.RawData(), + num_ints, __FILE__, __LINE__); + std::memcpy(out->data(), pinned_split_info_, num_ints * sizeof(int)); +} + +void CUDADataPartition::SetLeafDataLayout(const std::vector& leaf_num_data, + const std::vector& leaf_data_start, + int num_leaves) { + if (num_leaves <= 0) { + return; + } + std::vector leaf_data_end(num_leaves); + for (int i = 0; i < num_leaves; ++i) { + leaf_data_end[i] = leaf_data_start[i] + leaf_num_data[i]; + } + CopyFromHostToCUDADevice(cuda_leaf_num_data_.RawData(), leaf_num_data.data(), + static_cast(num_leaves), __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_leaf_data_start_.RawData(), leaf_data_start.data(), + static_cast(num_leaves), __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_leaf_data_end_.RawData(), leaf_data_end.data(), + static_cast(num_leaves), __FILE__, __LINE__); +} + +void CUDADataPartition::SplitLevelBatched(const std::vector& splits) { + const int num_splits = static_cast(splits.size()); + if (num_splits <= 0) { + return; + } + CHECK_LE(num_splits, 65535); // grid y-dimension limit + global_timer.Start("CUDADataPartition::SplitLevelBatched"); + host_apply_descs_.resize(splits.size()); + data_size_t total_block_offset_slots = 0; + int max_num_blocks = 0; + for (int k = 0; k < num_splits; ++k) { + const CUDAHybridApplySplitInput& in = splits[k]; + CUDAHybridApplyDescriptor& desc = host_apply_descs_[k]; + const int split_feature_index = in.split_feature; + CHECK(!is_categorical_feature_[split_feature_index]); + // host-side split metadata math, mirroring LaunchGenDataToLeftBitVectorKernel + const bool missing_is_zero = static_cast(cuda_column_data_->feature_missing_is_zero(split_feature_index)); + const bool missing_is_na = static_cast(cuda_column_data_->feature_missing_is_na(split_feature_index)); + const bool mfb_is_zero = static_cast(cuda_column_data_->feature_mfb_is_zero(split_feature_index)); + const bool mfb_is_na = static_cast(cuda_column_data_->feature_mfb_is_na(split_feature_index)); + const bool is_single_feature_in_column = is_single_feature_in_column_[split_feature_index]; + const uint32_t default_bin = cuda_column_data_->feature_default_bin(split_feature_index); + const uint32_t most_freq_bin = cuda_column_data_->feature_most_freq_bin(split_feature_index); + const uint32_t min_bin = is_single_feature_in_column ? 1 : cuda_column_data_->feature_min_bin(split_feature_index); + const uint32_t max_bin = cuda_column_data_->feature_max_bin(split_feature_index); + uint32_t th = in.split_threshold + min_bin; + uint32_t t_zero_bin = min_bin + default_bin; + if (most_freq_bin == 0) { + --th; + --t_zero_bin; + } + uint8_t split_default_to_left = 0; + uint8_t split_missing_default_to_left = 0; + int default_leaf_index = in.right_leaf_index; + int missing_default_leaf_index = in.right_leaf_index; + if (most_freq_bin <= in.split_threshold) { + split_default_to_left = 1; + default_leaf_index = in.left_leaf_index; + } + if (missing_is_zero || missing_is_na) { + if (in.split_default_left) { + split_missing_default_to_left = 1; + missing_default_leaf_index = in.left_leaf_index; + } + } + const int column_index = cuda_column_data_->feature_to_column(split_feature_index); + if (cuda_column_data_->packed_column_view_active()) { + // 4-bit packed compact source (EXABOOST_SPLIT_PACKED_READ): read the + // split column's nibbles straight from the histogram constructor's packed + // compact matrix instead of a per-tree column-major gather + desc.column_data = cuda_column_data_->packed_column_data(column_index); + desc.packed_row_stride = cuda_column_data_->packed_column_stride(column_index); + desc.packed_shift = cuda_column_data_->packed_column_shift(column_index); + CHECK(desc.column_data != nullptr); + } else { + desc.column_data = cuda_column_data_->GetColumnData(column_index); + desc.packed_row_stride = 0; + desc.packed_shift = 0; + } + desc.best_split_info = in.best_split_info; + desc.smaller_leaf_splits = in.smaller_leaf_splits; + desc.larger_leaf_splits = in.larger_leaf_splits; + desc.leaf_data_start = in.leaf_data_start; + desc.num_data_in_leaf = in.num_data_in_leaf; + desc.block_offset_start = total_block_offset_slots; + desc.num_blocks = (in.num_data_in_leaf + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION - 1) / + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION; + desc.left_leaf_index = in.left_leaf_index; + desc.right_leaf_index = in.right_leaf_index; + desc.th = th; + desc.t_zero_bin = t_zero_bin; + desc.max_bin = max_bin; + desc.min_bin = min_bin; + desc.default_leaf_index = default_leaf_index; + desc.missing_default_leaf_index = missing_default_leaf_index; + desc.split_default_to_left = split_default_to_left; + desc.split_missing_default_to_left = split_missing_default_to_left; + desc.bit_type = cuda_column_data_->packed_column_view_active() ? + 4 : cuda_column_data_->column_bit_type(column_index); + desc.min_is_max = (min_bin < max_bin) ? 0 : 1; + desc.missing_is_zero = missing_is_zero ? 1 : 0; + desc.missing_is_na = missing_is_na ? 1 : 0; + desc.mfb_is_zero = mfb_is_zero ? 1 : 0; + desc.mfb_is_na = mfb_is_na ? 1 : 0; + desc.max_bin_to_left = (max_bin <= th) ? 1 : 0; + desc.use_min_bin = is_single_feature_in_column ? 0 : 1; + total_block_offset_slots += desc.num_blocks + 1; + if (desc.num_blocks > max_num_blocks) { + max_num_blocks = desc.num_blocks; + } + } + // The split kernels write each split leaf's partitioned indices into the leaf's + // own window of cuda_out_data_indices_in_leaf_ (mirroring the main array + // layout); instead of copying every window back we SWAP the two buffers after + // the launches, making the out buffer the new main index array. Regions owned + // by leaves that are NOT split this level (terminal leaves) must be carried + // over explicitly: append one copy descriptor per gap in the split regions' + // coverage of [0, root_num_data()). Terminal regions are typically a small + // fraction of the data, so this replaces a full-size copy with a sparse one. + int num_gaps = 0; + int max_gap_blocks = 0; + { + std::vector> regions; // (start, num) + regions.reserve(splits.size()); + for (const CUDAHybridApplySplitInput& in : splits) { + regions.emplace_back(in.leaf_data_start, in.num_data_in_leaf); + } + std::sort(regions.begin(), regions.end()); + data_size_t cursor = 0; + const data_size_t num_data_total = root_num_data(); + auto append_gap = [this, &num_gaps, &max_gap_blocks](const data_size_t start, const data_size_t num) { + CUDAHybridApplyDescriptor desc; + std::memset(&desc, 0, sizeof(desc)); + desc.leaf_data_start = start; + desc.num_data_in_leaf = num; + desc.num_blocks = (num + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION - 1) / + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION; + if (desc.num_blocks > max_gap_blocks) { + max_gap_blocks = desc.num_blocks; + } + host_apply_descs_.push_back(desc); + ++num_gaps; + }; + for (const std::pair& region : regions) { + if (region.first > cursor) { + append_gap(cursor, region.first - cursor); + } + cursor = region.first + region.second; + } + if (cursor < num_data_total) { + append_gap(cursor, num_data_total - cursor); + } + } + // per-split regions of the level-sized block offset buffers + if (cuda_block_data_to_left_offset_.Size() < static_cast(total_block_offset_slots)) { + cuda_block_data_to_left_offset_.Resize(static_cast(total_block_offset_slots)); + cuda_block_data_to_right_offset_.Resize(static_cast(total_block_offset_slots)); + SetCUDAMemory(cuda_block_data_to_left_offset_.RawData(), 0, cuda_block_data_to_left_offset_.Size(), __FILE__, __LINE__); + SetCUDAMemory(cuda_block_data_to_right_offset_.RawData(), 0, cuda_block_data_to_right_offset_.Size(), __FILE__, __LINE__); + } + const int num_descs = num_splits + num_gaps; + if (cuda_apply_descs_.Size() < static_cast(num_descs)) { + // preallocate for the deepest possible level (splits + as many gaps) so + // this resizes at most once + cuda_apply_descs_.Resize(static_cast(std::max(num_descs, num_leaves_ + 4))); + } + // single async H2D per level on cuda_streams_[0]: ordered after any batched + // apply kernel of a previous level still reading the descriptor buffer (same + // stream) and before the launches below. The host staging buffer is only + // rewritten after the next FinishSplitBatch full sync, and a pageable async + // H2D returns only once the data is staged, so reuse is safe. + CopyFromHostToCUDADeviceAsync( + cuda_apply_descs_.RawData(), host_apply_descs_.data(), + static_cast(num_descs), cuda_streams_[0], __FILE__, __LINE__); + LaunchSplitLevelBatchedKernels(num_splits, max_num_blocks, num_gaps, max_gap_blocks); + // the out buffer now holds every leaf's indices at the main layout positions: + // promote it to the main index array (the old main becomes the next scratch) + cuda_data_indices_.Swap(&cuda_out_data_indices_in_leaf_); + cur_num_leaves_ += num_splits; + global_timer.Stop("CUDADataPartition::SplitLevelBatched"); +} + void CUDADataPartition::SplitInner( const data_size_t num_data_in_leaf, const CUDASplitInfo* best_split_info, @@ -241,7 +471,10 @@ void CUDADataPartition::SplitInner( double* left_leaf_sum_of_gradients, double* right_leaf_sum_of_gradients, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data) { + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main, + const int deferred_slot, + const data_size_t leaf_data_start_for_copy) { LaunchSplitInnerKernel( num_data_in_leaf, best_split_info, @@ -258,7 +491,10 @@ void CUDADataPartition::SplitInner( left_leaf_sum_of_gradients, right_leaf_sum_of_gradients, global_left_leaf_num_data, - global_right_leaf_num_data); + global_right_leaf_num_data, + point_structs_at_main, + deferred_slot, + leaf_data_start_for_copy); ++cur_num_leaves_; } @@ -314,11 +550,12 @@ void CUDADataPartition::ResetTrainingData(const Dataset* train_data, const int n CalcBlockDim(num_data_); const int old_max_num_split_indices_blocks = max_num_split_indices_blocks_; max_num_split_indices_blocks_ = grid_dim_; - if (max_num_split_indices_blocks_ > old_max_num_split_indices_blocks) { + if (max_num_split_indices_blocks_ > old_max_num_split_indices_blocks && + cuda_block_data_to_left_offset_.Size() < static_cast(max_num_split_indices_blocks_) + 1) { cuda_block_data_to_left_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); cuda_block_data_to_right_offset_.Resize(static_cast(max_num_split_indices_blocks_) + 1); - SetCUDAMemory(cuda_block_data_to_left_offset_.RawData(), 0, static_cast(max_num_split_indices_blocks_) + 1, __FILE__, __LINE__); - SetCUDAMemory(cuda_block_data_to_right_offset_.RawData(), 0, static_cast(max_num_split_indices_blocks_) + 1, __FILE__, __LINE__); + SetCUDAMemory(cuda_block_data_to_left_offset_.RawData(), 0, cuda_block_data_to_left_offset_.Size(), __FILE__, __LINE__); + SetCUDAMemory(cuda_block_data_to_right_offset_.RawData(), 0, cuda_block_data_to_right_offset_.Size(), __FILE__, __LINE__); } cuda_data_indices_.Resize(static_cast(num_data_)); cuda_block_to_left_offset_.Resize(static_cast(num_data_)); @@ -340,6 +577,8 @@ void CUDADataPartition::ResetConfig(const Config* config, hist_t* cuda_hist) { cuda_leaf_num_data_.Resize(static_cast(num_leaves_)); cuda_hist_pool_.Resize(static_cast(num_leaves_)); cuda_leaf_output_.Resize(static_cast(num_leaves_)); + cuda_split_info_buffer_.Resize(18 * static_cast(num_leaves_) + 18); + cuda_level_smaller_counts_.Resize(static_cast(num_leaves_) + 1); } void CUDADataPartition::SetBaggingSubset(const Dataset* subset) { diff --git a/src/treelearner/cuda/cuda_data_partition.cu b/src/treelearner/cuda/cuda_data_partition.cu index 21a1cff76bdb..f95e493fe917 100644 --- a/src/treelearner/cuda/cuda_data_partition.cu +++ b/src/treelearner/cuda/cuda_data_partition.cu @@ -791,6 +791,8 @@ __global__ void SplitTreeStructureKernel(const int left_leaf_index, data_size_t* block_to_left_offset_buffer, data_size_t* block_to_right_offset_buffer, data_size_t* cuda_leaf_data_start, data_size_t* cuda_leaf_data_end, data_size_t* cuda_leaf_num_data, const data_size_t* cuda_data_indices, + const data_size_t* cuda_data_indices_main, + const bool point_structs_at_main, const CUDASplitInfo* best_split_info, // for leaf splits information update CUDALeafSplitsStruct* smaller_leaf_splits, @@ -862,7 +864,8 @@ __global__ void SplitTreeStructureKernel(const int left_leaf_index, } else if (global_thread_index == 5) { smaller_leaf_splits->leaf_value = best_split_info->left_value; } else if (global_thread_index == 6) { - smaller_leaf_splits->data_indices_in_leaf = cuda_data_indices; + smaller_leaf_splits->data_indices_in_leaf = point_structs_at_main ? + cuda_data_indices_main + cuda_leaf_data_start[left_leaf_index] : cuda_data_indices; } else if (global_thread_index == 7) { larger_leaf_splits->leaf_index = right_leaf_index; } else if (global_thread_index == 8) { @@ -879,7 +882,8 @@ __global__ void SplitTreeStructureKernel(const int left_leaf_index, } else if (global_thread_index == 12) { larger_leaf_splits->leaf_value = best_split_info->right_value; } else if (global_thread_index == 13) { - larger_leaf_splits->data_indices_in_leaf = cuda_data_indices + cuda_leaf_num_data[left_leaf_index]; + larger_leaf_splits->data_indices_in_leaf = point_structs_at_main ? + cuda_data_indices_main + cuda_leaf_data_start[right_leaf_index] : cuda_data_indices + cuda_leaf_num_data[left_leaf_index]; } else if (global_thread_index == 14) { cuda_split_info_buffer[6] = left_leaf_index; } else if (global_thread_index == 15) { @@ -904,7 +908,8 @@ __global__ void SplitTreeStructureKernel(const int left_leaf_index, } else if (global_thread_index == 5) { larger_leaf_splits->leaf_value = best_split_info->left_value; } else if (global_thread_index == 6) { - larger_leaf_splits->data_indices_in_leaf = cuda_data_indices; + larger_leaf_splits->data_indices_in_leaf = point_structs_at_main ? + cuda_data_indices_main + cuda_leaf_data_start[left_leaf_index] : cuda_data_indices; } else if (global_thread_index == 7) { smaller_leaf_splits->leaf_index = right_leaf_index; } else if (global_thread_index == 8) { @@ -921,7 +926,8 @@ __global__ void SplitTreeStructureKernel(const int left_leaf_index, } else if (global_thread_index == 12) { smaller_leaf_splits->leaf_value = best_split_info->right_value; } else if (global_thread_index == 13) { - smaller_leaf_splits->data_indices_in_leaf = cuda_data_indices + cuda_leaf_num_data[left_leaf_index]; + smaller_leaf_splits->data_indices_in_leaf = point_structs_at_main ? + cuda_data_indices_main + cuda_leaf_data_start[right_leaf_index] : cuda_data_indices + cuda_leaf_num_data[left_leaf_index]; } else if (global_thread_index == 14) { cuda_hist_pool[right_leaf_index] = cuda_hist + 2 * right_leaf_index * num_total_bin; smaller_leaf_splits->hist_in_leaf = cuda_hist_pool[right_leaf_index]; @@ -991,7 +997,10 @@ void CUDADataPartition::LaunchSplitInnerKernel( double* left_leaf_sum_of_gradients_ref, double* right_leaf_sum_of_gradients_ref, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data) { + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main, + const int deferred_slot, + const data_size_t leaf_data_start_for_copy) { int num_blocks_final_ref = grid_dim_ - 1; int num_blocks_final_aligned = 1; while (num_blocks_final_ref > 0) { @@ -1043,18 +1052,22 @@ void CUDADataPartition::LaunchSplitInnerKernel( global_timer.Start("CUDADataPartition::SplitTreeStructureKernel"); + int* split_info_ptr = cuda_split_info_buffer_.RawData() + + (deferred_slot >= 0 ? 18 * static_cast(deferred_slot) : 0); #define SPLIT_TREE_ARGS \ left_leaf_index, right_leaf_index, \ cuda_block_data_to_left_offset_.RawData(), \ cuda_block_data_to_right_offset_.RawData(), cuda_leaf_data_start_.RawData(), cuda_leaf_data_end_.RawData(), \ cuda_leaf_num_data_.RawData(), cuda_out_data_indices_in_leaf_.RawData(), \ + cuda_data_indices_.RawData(), \ + point_structs_at_main, \ best_split_info, \ smaller_leaf_splits, \ larger_leaf_splits, \ num_total_bin_, \ cuda_hist_, \ cuda_hist_pool_.RawData(), \ - cuda_leaf_output_.RawData(), cuda_split_info_buffer_.RawData() + cuda_leaf_output_.RawData(), split_info_ptr if (nccl_communicator_ != nullptr) { if (use_quantized_grad_) { @@ -1072,6 +1085,16 @@ void CUDADataPartition::LaunchSplitInnerKernel( #undef SPLIT_TREE_ARGS global_timer.Stop("CUDADataPartition::SplitTreeStructureKernel"); + if (deferred_slot >= 0) { + // deferred (level-batched) mode: no per-split synchronization. The copy's + // destination and size are host-known (the split leaf's region); split info + // is read back once per level via FinishSplitBatch. + CopyDataIndicesKernel<<>>( + num_data_in_leaf, cuda_out_data_indices_in_leaf_.RawData(), + cuda_data_indices_.RawData() + leaf_data_start_for_copy); + CUDASUCCESS_OR_FATAL(cudaEventRecord(indices_copy_done_event_, cuda_streams_[2])); + return; + } std::vector cpu_split_info_buffer(18); const double* cpu_sum_hessians_info = reinterpret_cast(cpu_split_info_buffer.data() + 8); global_timer.Start("CUDADataPartition::CopyFromCUDADeviceToHostAsync"); @@ -1084,6 +1107,7 @@ void CUDADataPartition::LaunchSplitInnerKernel( global_timer.Start("CUDADataPartition::CopyDataIndicesKernel"); CopyDataIndicesKernel<<>>( left_leaf_num_data + right_leaf_num_data, cuda_out_data_indices_in_leaf_.RawData(), cuda_data_indices_.RawData() + left_leaf_data_start); + CUDASUCCESS_OR_FATAL(cudaEventRecord(indices_copy_done_event_, cuda_streams_[2])); global_timer.Stop("CUDADataPartition::CopyDataIndicesKernel"); const data_size_t right_leaf_data_start = cpu_split_info_buffer[5]; *left_leaf_num_data_ref = left_leaf_num_data; @@ -1100,6 +1124,546 @@ void CUDADataPartition::LaunchSplitInnerKernel( } } +// ---- batched apply kernels for the hybrid level-batched growth ---- +// One launch per kernel family covers ALL splits of a level. Data-parallel +// kernels use grid (max_num_blocks, num_splits) with a fixed 1024-thread block; +// blocks beyond a split's own num_blocks exit early. The results are identical +// to the per-split kernels for any block size because the partition is stable +// (per-block stable scatter + exact per-block counts), so the batched path may +// use a fixed block size where CalcBlockDim would have chosen a smaller one. +// Per-split scratch regions: the uint16 bit vector and the out-indices buffer +// use the split leaf's own [leaf_data_start, +num_data_in_leaf) window; the +// block offset buffers use [block_offset_start, +num_blocks+1). + +__device__ __forceinline__ uint32_t HybridLoadBin( + const CUDAHybridApplyDescriptor& d, const data_size_t global_data_index) { + if (d.bit_type == 8) { + return static_cast(static_cast(d.column_data)[global_data_index]); + } else if (d.bit_type == 4) { + // 4-bit packed compact source: column_data already points at this column's + // byte of row 0; rows are packed_row_stride bytes apart, the bin is the + // packed_shift nibble (see SetCompactPackedColumnView) + const uint8_t packed = static_cast(d.column_data)[ + static_cast(global_data_index) * static_cast(d.packed_row_stride)]; + return static_cast(packed >> d.packed_shift) & 0xfu; + } else if (d.bit_type == 16) { + return static_cast(static_cast(d.column_data)[global_data_index]); + } else { + return static_cast(d.column_data)[global_data_index]; + } +} + +// runtime-branch replica of GenDataToLeftBitVectorKernel's templated decision +// (branches are block-uniform descriptor fields, so divergence-free); the +// (MISSING_IS_ZERO || MFB_IS_ZERO) condition mirrors the template verbatim +__device__ __forceinline__ uint16_t HybridGenBitVectorDecision( + const CUDAHybridApplyDescriptor& d, const uint32_t bin) { + if (!d.min_is_max) { + if ((d.missing_is_zero && !d.mfb_is_zero && bin == d.t_zero_bin) || + (d.missing_is_na && !d.mfb_is_na && bin == d.max_bin)) { + return d.split_missing_default_to_left; + } else if ((d.use_min_bin && (bin < d.min_bin || bin > d.max_bin)) || + (!d.use_min_bin && bin == 0)) { + if ((d.missing_is_na && d.mfb_is_na) || (d.missing_is_zero || d.mfb_is_zero)) { + return d.split_missing_default_to_left; + } else { + return d.split_default_to_left; + } + } else if (bin <= d.th) { + return 1; + } else { + return 0; + } + } else { + if (d.missing_is_zero && !d.mfb_is_zero && bin == d.t_zero_bin) { + return d.split_missing_default_to_left; + } else if (bin != d.max_bin) { + if ((d.missing_is_na && d.mfb_is_na) || (d.missing_is_zero && d.mfb_is_zero)) { + return d.split_missing_default_to_left; + } else { + return d.split_default_to_left; + } + } else { + if (d.missing_is_na && !d.mfb_is_na) { + return d.split_missing_default_to_left; + } else if (d.max_bin_to_left) { + return 1; + } else { + return 0; + } + } + } +} + +// runtime-branch replica of UpdateDataIndexToLeafIndexKernel's templated decision +__device__ __forceinline__ int HybridUpdateLeafIndexDecision( + const CUDAHybridApplyDescriptor& d, const uint32_t bin) { + if (!d.min_is_max) { + if ((d.missing_is_zero && !d.mfb_is_zero && bin == d.t_zero_bin) || + (d.missing_is_na && !d.mfb_is_na && bin == d.max_bin)) { + return d.missing_default_leaf_index; + } else if ((d.use_min_bin && (bin < d.min_bin || bin > d.max_bin)) || + (!d.use_min_bin && bin == 0)) { + if ((d.missing_is_na && d.mfb_is_na) || (d.missing_is_zero && d.mfb_is_zero)) { + return d.missing_default_leaf_index; + } else { + return d.default_leaf_index; + } + } else if (bin > d.th) { + return d.right_leaf_index; + } else { + return d.left_leaf_index; + } + } else { + if (d.missing_is_zero && !d.mfb_is_zero && bin == d.t_zero_bin) { + return d.missing_default_leaf_index; + } else if (bin != d.max_bin) { + if ((d.missing_is_na && d.mfb_is_na) || (d.missing_is_zero && d.mfb_is_zero)) { + return d.missing_default_leaf_index; + } else { + return d.default_leaf_index; + } + } else { + if (d.missing_is_na && !d.mfb_is_na) { + return d.missing_default_leaf_index; + } else if (d.max_bin_to_left) { + return d.left_leaf_index; + } else { + return d.right_leaf_index; + } + } + } +} + +// fused GenDataToLeftBitVector + UpdateDataIndexToLeafIndex: both derive from +// the SAME (data index, bin) load, so fusing halves the column/index reads that +// dominate the apply phase's memory traffic. The written values are identical +// to the two per-split kernels'. +__global__ void HybridGenBitVectorUpdateLeafIndexBatchKernel( + const CUDAHybridApplyDescriptor* descs, + const data_size_t* cuda_data_indices, + uint16_t* block_to_left_offset, + data_size_t* block_to_left_offset_buffer, + data_size_t* block_to_right_offset_buffer, + int* cuda_data_index_to_leaf_index) { + __shared__ uint16_t shared_mem_buffer[WARPSIZE]; + const CUDAHybridApplyDescriptor d = descs[blockIdx.y]; + if (static_cast(blockIdx.x) >= d.num_blocks) { + return; + } + uint16_t thread_to_left_offset_cnt = 0; + const data_size_t local_data_index = static_cast(blockIdx.x * blockDim.x + threadIdx.x); + if (local_data_index < d.num_data_in_leaf) { + const data_size_t global_data_index = cuda_data_indices[d.leaf_data_start + local_data_index]; + const uint32_t bin = HybridLoadBin(d, global_data_index); + thread_to_left_offset_cnt = HybridGenBitVectorDecision(d, bin); + cuda_data_index_to_leaf_index[global_data_index] = HybridUpdateLeafIndexDecision(d, bin); + } + __syncthreads(); + PrepareOffset(d.num_data_in_leaf, + block_to_left_offset + d.leaf_data_start + blockIdx.x * blockDim.x, + block_to_left_offset_buffer + d.block_offset_start, + block_to_right_offset_buffer + d.block_offset_start, + thread_to_left_offset_cnt, shared_mem_buffer); +} + +// per-split block-offset prefix sum + children leaf metadata update; one block +// per split, generalizing AggregateBlockOffsetKernel0 to arbitrary num_blocks. +// The split's region slot 0 is zeroed explicitly (a slot that held a non-zero +// value in a previous level's layout can become a region start in this one). +__global__ void HybridAggregateBlockOffsetBatchKernel( + const CUDAHybridApplyDescriptor* descs, + data_size_t* block_to_left_offset_buffer_base, + data_size_t* block_to_right_offset_buffer_base, + data_size_t* cuda_leaf_data_start, + data_size_t* cuda_leaf_data_end, + data_size_t* cuda_leaf_num_data, + data_size_t* level_smaller_counts) { + __shared__ uint32_t shared_mem_buffer[WARPSIZE]; + __shared__ uint32_t to_left_total_count; + const CUDAHybridApplyDescriptor d = descs[blockIdx.x]; + data_size_t* block_to_left_offset_buffer = block_to_left_offset_buffer_base + d.block_offset_start; + data_size_t* block_to_right_offset_buffer = block_to_right_offset_buffer_base + d.block_offset_start; + const data_size_t num_blocks = static_cast(d.num_blocks); + const data_size_t num_data_in_leaf = d.num_data_in_leaf; + const unsigned int blockDim_x = blockDim.x; + const unsigned int threadIdx_x = threadIdx.x; + const data_size_t num_blocks_plus_1 = num_blocks + 1; + const uint32_t num_blocks_per_thread = (num_blocks_plus_1 + blockDim_x - 1) / blockDim_x; + const uint32_t remain = num_blocks_plus_1 - ((num_blocks_per_thread - 1) * blockDim_x); + const uint32_t remain_offset = remain * num_blocks_per_thread; + uint32_t thread_start_block_index = 0; + uint32_t thread_end_block_index = 0; + if (threadIdx_x < remain) { + thread_start_block_index = threadIdx_x * num_blocks_per_thread; + thread_end_block_index = min(thread_start_block_index + num_blocks_per_thread, num_blocks_plus_1); + } else { + thread_start_block_index = remain_offset + (num_blocks_per_thread - 1) * (threadIdx_x - remain); + thread_end_block_index = min(thread_start_block_index + num_blocks_per_thread - 1, num_blocks_plus_1); + } + if (threadIdx.x == 0) { + block_to_left_offset_buffer[0] = 0; + block_to_right_offset_buffer[0] = 0; + } + __syncthreads(); + for (uint32_t block_index = thread_start_block_index + 1; block_index < thread_end_block_index; ++block_index) { + block_to_left_offset_buffer[block_index] += block_to_left_offset_buffer[block_index - 1]; + block_to_right_offset_buffer[block_index] += block_to_right_offset_buffer[block_index - 1]; + } + __syncthreads(); + uint32_t block_to_left_offset = 0; + uint32_t block_to_right_offset = 0; + if (thread_start_block_index < thread_end_block_index && thread_start_block_index > 1) { + block_to_left_offset = block_to_left_offset_buffer[thread_start_block_index - 1]; + block_to_right_offset = block_to_right_offset_buffer[thread_start_block_index - 1]; + } + block_to_left_offset = ShufflePrefixSum(block_to_left_offset, shared_mem_buffer); + __syncthreads(); + block_to_right_offset = ShufflePrefixSum(block_to_right_offset, shared_mem_buffer); + if (threadIdx_x == blockDim_x - 1) { + to_left_total_count = block_to_left_offset + block_to_left_offset_buffer[num_blocks]; + } + __syncthreads(); + const uint32_t to_left_thread_block_offset = block_to_left_offset; + const uint32_t to_right_thread_block_offset = block_to_right_offset + to_left_total_count; + for (uint32_t block_index = thread_start_block_index; block_index < thread_end_block_index; ++block_index) { + block_to_left_offset_buffer[block_index] += to_left_thread_block_offset; + block_to_right_offset_buffer[block_index] += to_right_thread_block_offset; + } + __syncthreads(); + if (threadIdx.x == 0) { + const int left_leaf_index = d.left_leaf_index; + const int right_leaf_index = d.right_leaf_index; + const data_size_t old_leaf_data_end = cuda_leaf_data_end[left_leaf_index]; + const data_size_t left_count = static_cast(to_left_total_count); + const data_size_t right_count = num_data_in_leaf - left_count; + cuda_leaf_data_end[left_leaf_index] = cuda_leaf_data_start[left_leaf_index] + left_count; + cuda_leaf_num_data[left_leaf_index] = left_count; + cuda_leaf_data_start[right_leaf_index] = cuda_leaf_data_end[left_leaf_index]; + cuda_leaf_data_end[right_leaf_index] = old_leaf_data_end; + cuda_leaf_num_data[right_leaf_index] = right_count; + // actual smaller-child size of this split, consumed by the speculative + // batched construct kernel's device row-grouping (single-sync flow) + level_smaller_counts[blockIdx.x] = left_count < right_count ? left_count : right_count; + } +} + +__global__ void HybridSplitInnerBatchKernel( + const CUDAHybridApplyDescriptor* descs, + const data_size_t* cuda_data_indices, + const data_size_t* block_to_left_offset_buffer_base, + const data_size_t* block_to_right_offset_buffer_base, + const uint16_t* block_to_left_offset, + data_size_t* out_data_indices_in_leaf) { + const CUDAHybridApplyDescriptor d = descs[blockIdx.y]; + if (static_cast(blockIdx.x) >= d.num_blocks) { + return; + } + const data_size_t num_data_in_leaf = d.num_data_in_leaf; + const unsigned int threadIdx_x = threadIdx.x; + const unsigned int blockDim_x = blockDim.x; + const unsigned int global_thread_index = blockIdx.x * blockDim_x + threadIdx_x; + const data_size_t* cuda_data_indices_in_leaf = cuda_data_indices + d.leaf_data_start; + const uint16_t* block_to_left_offset_ptr = block_to_left_offset + d.leaf_data_start + blockIdx.x * blockDim_x; + const uint32_t to_right_block_offset = block_to_right_offset_buffer_base[d.block_offset_start + blockIdx.x]; + const uint32_t to_left_block_offset = block_to_left_offset_buffer_base[d.block_offset_start + blockIdx.x]; + data_size_t* left_out_data_indices_in_leaf = out_data_indices_in_leaf + d.leaf_data_start + to_left_block_offset; + data_size_t* right_out_data_indices_in_leaf = out_data_indices_in_leaf + d.leaf_data_start + to_right_block_offset; + if (static_cast(global_thread_index) < num_data_in_leaf) { + const uint32_t thread_to_left_offset = (threadIdx_x == 0 ? 0 : block_to_left_offset_ptr[threadIdx_x - 1]); + const bool to_left = block_to_left_offset_ptr[threadIdx_x] > thread_to_left_offset; + if (to_left) { + left_out_data_indices_in_leaf[thread_to_left_offset] = cuda_data_indices_in_leaf[global_thread_index]; + } else { + const uint32_t thread_to_right_offset = threadIdx.x - thread_to_left_offset; + right_out_data_indices_in_leaf[thread_to_right_offset] = cuda_data_indices_in_leaf[global_thread_index]; + } + } +} + +// replica of SplitTreeStructureKernel with +// point_structs_at_main fixed to true (batched apply always points the child +// structs' data_indices_in_leaf at the main index array); one 32-thread block +// per split, threadIdx.x playing the original global thread index. +template +__global__ void HybridSplitTreeStructureBatchKernel( + const CUDAHybridApplyDescriptor* descs, + data_size_t* cuda_leaf_data_start, + data_size_t* cuda_leaf_num_data, + const data_size_t* cuda_data_indices_main, + const int num_total_bin, + hist_t* cuda_hist, hist_t** cuda_hist_pool, + double* cuda_leaf_output, + int* cuda_split_info_buffer_base) { + const CUDAHybridApplyDescriptor d = descs[blockIdx.x]; + const int left_leaf_index = d.left_leaf_index; + const int right_leaf_index = d.right_leaf_index; + const CUDASplitInfo* best_split_info = d.best_split_info; + CUDALeafSplitsStruct* smaller_leaf_splits = d.smaller_leaf_splits; + CUDALeafSplitsStruct* larger_leaf_splits = d.larger_leaf_splits; + int* cuda_split_info_buffer = cuda_split_info_buffer_base + 18 * blockIdx.x; + const unsigned int to_left_total_cnt = cuda_leaf_num_data[left_leaf_index]; + double* cuda_split_info_buffer_for_hessians = reinterpret_cast(cuda_split_info_buffer + 8); + const unsigned int global_thread_index = threadIdx.x; + if (global_thread_index == 0) { + cuda_leaf_output[left_leaf_index] = best_split_info->left_value; + } else if (global_thread_index == 1) { + cuda_leaf_output[right_leaf_index] = best_split_info->right_value; + } else if (global_thread_index == 2) { + cuda_split_info_buffer[0] = left_leaf_index; + } else if (global_thread_index == 3) { + cuda_split_info_buffer[1] = cuda_leaf_num_data[left_leaf_index]; + } else if (global_thread_index == 4) { + cuda_split_info_buffer[2] = cuda_leaf_data_start[left_leaf_index]; + } else if (global_thread_index == 5) { + cuda_split_info_buffer[3] = right_leaf_index; + } else if (global_thread_index == 6) { + cuda_split_info_buffer[4] = cuda_leaf_num_data[right_leaf_index]; + } else if (global_thread_index == 7) { + cuda_split_info_buffer[5] = cuda_leaf_data_start[right_leaf_index]; + } else if (global_thread_index == 8) { + cuda_split_info_buffer_for_hessians[0] = best_split_info->left_sum_hessians; + cuda_split_info_buffer_for_hessians[2] = best_split_info->left_sum_gradients; + } else if (global_thread_index == 9) { + cuda_split_info_buffer_for_hessians[1] = best_split_info->right_sum_hessians; + cuda_split_info_buffer_for_hessians[3] = best_split_info->right_sum_gradients; + } + + const bool left_is_smaller = cuda_leaf_num_data[left_leaf_index] < cuda_leaf_num_data[right_leaf_index]; + + if (left_is_smaller) { + if (global_thread_index == 0) { + hist_t* parent_hist_ptr = cuda_hist_pool[left_leaf_index]; + cuda_hist_pool[right_leaf_index] = parent_hist_ptr; + cuda_hist_pool[left_leaf_index] = cuda_hist + 2 * right_leaf_index * num_total_bin; + smaller_leaf_splits->hist_in_leaf = cuda_hist_pool[left_leaf_index]; + larger_leaf_splits->hist_in_leaf = cuda_hist_pool[right_leaf_index]; + } else if (global_thread_index == 1) { + smaller_leaf_splits->sum_of_gradients = best_split_info->left_sum_gradients; + if (USE_GRAD_DISCRETIZED) { + smaller_leaf_splits->sum_of_gradients_hessians = best_split_info->left_sum_of_gradients_hessians; + } + } else if (global_thread_index == 2) { + smaller_leaf_splits->sum_of_hessians = best_split_info->left_sum_hessians; + } else if (global_thread_index == 3) { + smaller_leaf_splits->num_data_in_leaf = to_left_total_cnt; + } else if (global_thread_index == 4) { + smaller_leaf_splits->gain = best_split_info->left_gain; + } else if (global_thread_index == 5) { + smaller_leaf_splits->leaf_value = best_split_info->left_value; + } else if (global_thread_index == 6) { + smaller_leaf_splits->data_indices_in_leaf = cuda_data_indices_main + cuda_leaf_data_start[left_leaf_index]; + } else if (global_thread_index == 7) { + larger_leaf_splits->leaf_index = right_leaf_index; + } else if (global_thread_index == 8) { + larger_leaf_splits->sum_of_gradients = best_split_info->right_sum_gradients; + if (USE_GRAD_DISCRETIZED) { + larger_leaf_splits->sum_of_gradients_hessians = best_split_info->right_sum_of_gradients_hessians; + } + } else if (global_thread_index == 9) { + larger_leaf_splits->sum_of_hessians = best_split_info->right_sum_hessians; + } else if (global_thread_index == 10) { + larger_leaf_splits->num_data_in_leaf = cuda_leaf_num_data[right_leaf_index]; + } else if (global_thread_index == 11) { + larger_leaf_splits->gain = best_split_info->right_gain; + } else if (global_thread_index == 12) { + larger_leaf_splits->leaf_value = best_split_info->right_value; + } else if (global_thread_index == 13) { + larger_leaf_splits->data_indices_in_leaf = cuda_data_indices_main + cuda_leaf_data_start[right_leaf_index]; + } else if (global_thread_index == 14) { + cuda_split_info_buffer[6] = left_leaf_index; + } else if (global_thread_index == 15) { + cuda_split_info_buffer[7] = right_leaf_index; + } else if (global_thread_index == 16) { + smaller_leaf_splits->leaf_index = left_leaf_index; + } + } else { + if (global_thread_index == 0) { + larger_leaf_splits->leaf_index = left_leaf_index; + } else if (global_thread_index == 1) { + larger_leaf_splits->sum_of_gradients = best_split_info->left_sum_gradients; + if (USE_GRAD_DISCRETIZED) { + larger_leaf_splits->sum_of_gradients_hessians = best_split_info->left_sum_of_gradients_hessians; + } + } else if (global_thread_index == 2) { + larger_leaf_splits->sum_of_hessians = best_split_info->left_sum_hessians; + } else if (global_thread_index == 3) { + larger_leaf_splits->num_data_in_leaf = to_left_total_cnt; + } else if (global_thread_index == 4) { + larger_leaf_splits->gain = best_split_info->left_gain; + } else if (global_thread_index == 5) { + larger_leaf_splits->leaf_value = best_split_info->left_value; + } else if (global_thread_index == 6) { + larger_leaf_splits->data_indices_in_leaf = cuda_data_indices_main + cuda_leaf_data_start[left_leaf_index]; + } else if (global_thread_index == 7) { + smaller_leaf_splits->leaf_index = right_leaf_index; + } else if (global_thread_index == 8) { + smaller_leaf_splits->sum_of_gradients = best_split_info->right_sum_gradients; + if (USE_GRAD_DISCRETIZED) { + smaller_leaf_splits->sum_of_gradients_hessians = best_split_info->right_sum_of_gradients_hessians; + } + } else if (global_thread_index == 9) { + smaller_leaf_splits->sum_of_hessians = best_split_info->right_sum_hessians; + } else if (global_thread_index == 10) { + smaller_leaf_splits->num_data_in_leaf = cuda_leaf_num_data[right_leaf_index]; + } else if (global_thread_index == 11) { + smaller_leaf_splits->gain = best_split_info->right_gain; + } else if (global_thread_index == 12) { + smaller_leaf_splits->leaf_value = best_split_info->right_value; + } else if (global_thread_index == 13) { + smaller_leaf_splits->data_indices_in_leaf = cuda_data_indices_main + cuda_leaf_data_start[right_leaf_index]; + } else if (global_thread_index == 14) { + cuda_hist_pool[right_leaf_index] = cuda_hist + 2 * right_leaf_index * num_total_bin; + smaller_leaf_splits->hist_in_leaf = cuda_hist_pool[right_leaf_index]; + } else if (global_thread_index == 15) { + larger_leaf_splits->hist_in_leaf = cuda_hist_pool[left_leaf_index]; + } else if (global_thread_index == 16) { + cuda_split_info_buffer[6] = right_leaf_index; + } else if (global_thread_index == 17) { + cuda_split_info_buffer[7] = left_leaf_index; + } + } +} + +// selective grow-then-prune: rewrite the data-index-to-leaf-index entries of the +// collapsed subtree windows (blockIdx.y indexes windows; blocks beyond a window's +// own row count exit early). The window rows in the main index array are exactly +// the subtree's rows (child regions nest inside the parent region). +__global__ void CollapseLeafWindowsKernel( + const CUDACollapseWindow* windows, + const data_size_t* cuda_data_indices, + int* cuda_data_index_to_leaf_index) { + const CUDACollapseWindow w = windows[blockIdx.y]; + const data_size_t local_index = static_cast(blockIdx.x * blockDim.x + threadIdx.x); + if (local_index < w.count) { + cuda_data_index_to_leaf_index[cuda_data_indices[w.start + local_index]] = w.target_leaf; + } +} + +void CUDADataPartition::CollapseLeafWindows(const std::vector& windows) { + const int num_windows = static_cast(windows.size()); + if (num_windows <= 0) { + return; + } + host_collapse_windows_ = windows; + if (cuda_collapse_windows_.Size() < windows.size()) { + cuda_collapse_windows_.Resize(std::max(windows.size(), + static_cast(num_leaves_ / 2 + 2))); + } + // async H2D on the apply stream: ordered before the kernel below and any later + // batched apply of the same level (same stream). The host staging buffer is + // only rewritten after the next full sync, and a pageable async H2D returns + // only once the data is staged, so reuse is safe. + CopyFromHostToCUDADeviceAsync( + cuda_collapse_windows_.RawData(), host_collapse_windows_.data(), + windows.size(), cuda_streams_[0], __FILE__, __LINE__); + data_size_t max_count = 0; + for (const CUDACollapseWindow& w : windows) { + max_count = std::max(max_count, w.count); + } + const int max_blocks = (max_count + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION - 1) / + SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION; + const dim3 grid(static_cast(max_blocks), static_cast(num_windows)); + CollapseLeafWindowsKernel<<>>( + cuda_collapse_windows_.RawDataReadOnly(), cuda_data_indices_.RawData(), + cuda_data_index_to_leaf_index_.RawData()); +} + +// selective grow-then-prune finalize: map every used row's leaf index from the +// hybrid numbering to the final classic numbering (pruned leaves map to their +// surviving ancestor's final leaf) +__global__ void RemapDataIndexToLeafIndexKernel( + const data_size_t num_data, + const data_size_t* cuda_data_indices, + const int* leaf_map, + int* cuda_data_index_to_leaf_index) { + const data_size_t i = static_cast(blockIdx.x * blockDim.x + threadIdx.x); + if (i < num_data) { + const data_size_t data_index = cuda_data_indices[i]; + cuda_data_index_to_leaf_index[data_index] = leaf_map[cuda_data_index_to_leaf_index[data_index]]; + } +} + +void CUDADataPartition::RemapDataIndexToLeafIndex(const std::vector& leaf_map) { + const data_size_t num_data_in_root = root_num_data(); + if (num_data_in_root <= 0 || leaf_map.empty()) { + return; + } + if (cuda_leaf_index_remap_.Size() < leaf_map.size()) { + cuda_leaf_index_remap_.Resize(std::max(leaf_map.size(), static_cast(num_leaves_))); + } + CopyFromHostToCUDADevice(cuda_leaf_index_remap_.RawData(), leaf_map.data(), + leaf_map.size(), __FILE__, __LINE__); + const int num_blocks = (num_data_in_root + FILL_INDICES_BLOCK_SIZE_DATA_PARTITION - 1) / + FILL_INDICES_BLOCK_SIZE_DATA_PARTITION; + RemapDataIndexToLeafIndexKernel<<>>( + num_data_in_root, cuda_data_indices_.RawData(), cuda_leaf_index_remap_.RawDataReadOnly(), + cuda_data_index_to_leaf_index_.RawData()); +} + +// region copy at identical offsets (src and dst share the main array layout); +// used to carry terminal (non-split) leaves' regions from the old main index +// array into the out buffer before it is swapped in as the new main array +__global__ void HybridCopyDataIndicesBatchKernel( + const CUDAHybridApplyDescriptor* descs, + const data_size_t* src_data_indices, + data_size_t* dst_data_indices) { + const CUDAHybridApplyDescriptor d = descs[blockIdx.y]; + if (static_cast(blockIdx.x) >= d.num_blocks) { + return; + } + const data_size_t local_data_index = static_cast(blockIdx.x * blockDim.x + threadIdx.x); + if (local_data_index < d.num_data_in_leaf) { + dst_data_indices[d.leaf_data_start + local_data_index] = + src_data_indices[d.leaf_data_start + local_data_index]; + } +} + +void CUDADataPartition::LaunchSplitLevelBatchedKernels(const int num_splits, const int max_num_blocks, + const int num_gaps, const int max_gap_blocks) { + const CUDAHybridApplyDescriptor* descs = cuda_apply_descs_.RawDataReadOnly(); + const dim3 data_grid(static_cast(max_num_blocks), static_cast(num_splits)); + constexpr int block_dim = SPLIT_INDICES_BLOCK_SIZE_DATA_PARTITION; + // out buffer of this level == new main array after the caller's swap + data_size_t* new_main_indices = cuda_out_data_indices_in_leaf_.RawData(); + // the level is bracketed by device synchronizations (best-split readback before, + // FinishSplitBatch after), but wait on the last recorded copy event anyway so a + // still-in-flight per-split CopyDataIndicesKernel can never race the scratch + if (indices_copy_done_event_ != nullptr) { + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent(cuda_streams_[0], indices_copy_done_event_, 0)); + } + HybridGenBitVectorUpdateLeafIndexBatchKernel<<>>( + descs, cuda_data_indices_.RawData(), cuda_block_to_left_offset_.RawData(), + cuda_block_data_to_left_offset_.RawData(), cuda_block_data_to_right_offset_.RawData(), + cuda_data_index_to_leaf_index_.RawData()); + HybridAggregateBlockOffsetBatchKernel<<>>( + descs, cuda_block_data_to_left_offset_.RawData(), cuda_block_data_to_right_offset_.RawData(), + cuda_leaf_data_start_.RawData(), cuda_leaf_data_end_.RawData(), cuda_leaf_num_data_.RawData(), + cuda_level_smaller_counts_.RawData()); + HybridSplitInnerBatchKernel<<>>( + descs, cuda_data_indices_.RawData(), cuda_block_data_to_left_offset_.RawData(), + cuda_block_data_to_right_offset_.RawData(), cuda_block_to_left_offset_.RawData(), + new_main_indices); + if (use_quantized_grad_) { + HybridSplitTreeStructureBatchKernel<<>>( + descs, cuda_leaf_data_start_.RawData(), cuda_leaf_num_data_.RawData(), + new_main_indices, num_total_bin_, cuda_hist_, cuda_hist_pool_.RawData(), + cuda_leaf_output_.RawData(), cuda_split_info_buffer_.RawData()); + } else { + HybridSplitTreeStructureBatchKernel<<>>( + descs, cuda_leaf_data_start_.RawData(), cuda_leaf_num_data_.RawData(), + new_main_indices, num_total_bin_, cuda_hist_, cuda_hist_pool_.RawData(), + cuda_leaf_output_.RawData(), cuda_split_info_buffer_.RawData()); + } + if (num_gaps > 0) { + // gap descriptors follow the split descriptors in cuda_apply_descs_ + const dim3 gap_grid(static_cast(max_gap_blocks), static_cast(num_gaps)); + HybridCopyDataIndicesBatchKernel<<>>( + descs + num_splits, cuda_data_indices_.RawData(), new_main_indices); + } + CUDASUCCESS_OR_FATAL(cudaEventRecord(indices_copy_done_event_, cuda_streams_[0])); +} + template __global__ void AddPredictionToScoreKernel( const data_size_t* data_indices_in_leaf, diff --git a/src/treelearner/cuda/cuda_data_partition.hpp b/src/treelearner/cuda/cuda_data_partition.hpp index f6dc38671725..2bbe5d93f1b9 100644 --- a/src/treelearner/cuda/cuda_data_partition.hpp +++ b/src/treelearner/cuda/cuda_data_partition.hpp @@ -27,6 +27,82 @@ namespace LightGBM { +/*! \brief Host-side per-split input for the batched (per-level) apply phase of the + * hybrid growth: everything the per-split Split() call would receive, restricted + * to numerical splits in deferred mode with point_structs_at_main semantics. The + * deferred split-info slot of split k is k (18 ints each, see FinishSplitBatch). */ +struct CUDAHybridApplySplitInput { + const CUDASplitInfo* best_split_info; + int left_leaf_index; + int right_leaf_index; + int split_feature; + uint32_t split_threshold; + uint8_t split_default_left; + data_size_t num_data_in_leaf; + data_size_t leaf_data_start; + CUDALeafSplitsStruct* smaller_leaf_splits; + CUDALeafSplitsStruct* larger_leaf_splits; +}; + +/*! \brief Device-side per-split descriptor for the batched apply kernels. One + * entry per split of a level; built host-side by SplitLevelBatched (mirroring the + * host metadata math of LaunchGenDataToLeftBitVectorKernel) and uploaded in a + * single H2D copy. Data-parallel kernels index splits via a grid dimension and + * size themselves by the level's largest split region; blocks beyond a split's + * own num_blocks exit early. Per-split scratch lives in per-split regions: + * the bit vector and out-indices scratch use the split leaf's own + * [leaf_data_start, +num_data_in_leaf) window (mirroring the main index array + * layout), the block offset buffers use [block_offset_start, +num_blocks+1). */ +struct CUDAHybridApplyDescriptor { + const void* column_data; + const CUDASplitInfo* best_split_info; + CUDALeafSplitsStruct* smaller_leaf_splits; + CUDALeafSplitsStruct* larger_leaf_splits; + data_size_t leaf_data_start; + data_size_t num_data_in_leaf; + data_size_t block_offset_start; + /*! \brief per-split grid size: ceil(num_data_in_leaf / 1024) (host-computed) */ + int num_blocks; + int left_leaf_index; + int right_leaf_index; + /*! \brief split decision parameters (see LaunchGenDataToLeftBitVectorKernel) */ + uint32_t th; + uint32_t t_zero_bin; + uint32_t max_bin; + uint32_t min_bin; + int default_leaf_index; + int missing_default_leaf_index; + uint8_t split_default_to_left; + uint8_t split_missing_default_to_left; + /*! \brief per-row byte stride of the 4-bit packed compact source (bit_type 4) */ + int packed_row_stride; + /*! \brief column storage width: 8, 16 or 32 bits, or 4 = 4-bit packed compact + * source (column_data points at the column's byte within the packed compact + * matrix; the bin is (byte >> packed_shift) & 0xf) */ + uint8_t bit_type; + /*! \brief nibble shift (0 or 4) of the 4-bit packed compact source */ + uint8_t packed_shift; + /*! \brief runtime versions of the GenDataToLeftBitVectorKernel template flags */ + uint8_t min_is_max; + uint8_t missing_is_zero; + uint8_t missing_is_na; + uint8_t mfb_is_zero; + uint8_t mfb_is_na; + uint8_t max_bin_to_left; + uint8_t use_min_bin; +}; + +/*! \brief One collapsed (pruned) subtree of the selective grow-then-prune hybrid + * growth: the window [start, start + count) of the main data index array holds + * exactly the subtree's rows (child regions nest inside the parent region), and + * every row's data-index-to-leaf-index entry is rewritten to target_leaf (the + * leaf index the collapsed node reverts to). */ +struct CUDACollapseWindow { + data_size_t start; + data_size_t count; + int target_leaf; +}; + class CUDADataPartition: public NCCLInfo { public: CUDADataPartition( @@ -68,7 +144,50 @@ class CUDADataPartition: public NCCLInfo { double* left_leaf_sum_of_gradients, double* right_leaf_sum_of_gradients, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data); + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main = false, + const int deferred_slot = -1); + + // Deferred split-info readback for the hybrid level-batched growth: one device + // synchronization + one transfer for all of a level's splits (18 ints per split, + // laid out by the deferred_slot passed to Split). Layout per slot: [0] left leaf, + // [1] left count, [2] left start, [4] right count, [5] right start, [6] right leaf, + // ints [8..15] reinterpreted as 4 doubles: left/right sum_hessians, left/right + // sum_gradients. + void FinishSplitBatch(const int num_splits, std::vector* out); + + /*! \brief Batched apply phase for the hybrid level-batched growth: applies ALL + * numerical splits of a level with one launch per kernel family (gen bit + * vector, update data-index-to-leaf-index, aggregate block offsets, split + * inner, split tree structure, copy indices) and ZERO host synchronization. + * Equivalent to calling Split(..., point_structs_at_main=true, deferred_slot=k) + * for k = 0..n-1: split k writes deferred split-info slot k, read back by the + * caller via FinishSplitBatch(n). Requires the level's splits to partition + * disjoint index regions (host-known leaf_data_start/num_data from the + * previous level's readback). */ + void SplitLevelBatched(const std::vector& splits); + + /*! \brief Selective grow-then-prune: rewrite the data-index-to-leaf-index + * entries of the given collapsed subtree windows to their target leaves. + * Enqueued on cuda_streams_[0] (ordered before the next batched apply and, + * via the legacy-stream semantics, before subsequent histogram work); the + * caller guarantees the device is otherwise idle (post-readback). */ + void CollapseLeafWindows(const std::vector& windows); + + /*! \brief Selective grow-then-prune finalize: remap the data-index-to-leaf- + * index entries of all rows of this tree (via the main index array, which + * covers exactly root_num_data() rows) through leaf_map (hybrid leaf index -> + * final classic leaf index). Async on the default stream; the caller + * synchronizes. */ + void RemapDataIndexToLeafIndex(const std::vector& leaf_map); + + /*! \brief Selective grow-then-prune finalize: upload the final per-leaf data + * layout (num_data / data_start / data_end) for the first num_leaves leaves, + * so post-training consumers (objective renewal, quantized leaf renewal) + * see the final classic numbering. */ + void SetLeafDataLayout(const std::vector& leaf_num_data, + const std::vector& leaf_data_start, + int num_leaves); void UpdateTrainScore(const Tree* tree, double* cuda_scores); @@ -104,9 +223,29 @@ class CUDADataPartition: public NCCLInfo { bool use_bagging() const { return use_bagging_; } + /*! \brief recorded on the apply stream after the last batched apply kernel of a + * level; the speculative single-sync search phase waits on it before reading + * the child leaf-splits structs the apply kernels write */ + cudaEvent_t apply_done_event() const { return indices_copy_done_event_; } + + /*! \brief device array of the current level's actual smaller-child sizes + * (min(left, right) count of split k), written by the batched aggregate + * kernel; consumed by the speculative batched construct kernel to derive its + * exact row grouping before the sizes are host-known */ + const data_size_t* level_smaller_leaf_counts() const { return cuda_level_smaller_counts_.RawDataReadOnly(); } + private: void CalcBlockDim(const data_size_t num_data_in_leaf); + /*! \brief launch the batched apply kernels of one level, in dependency order + * on cuda_streams_[0] (fused gen-bit-vector+update-leaf-index, aggregate + * block offsets, split inner into the out buffer, split tree structure, and + * a sparse gap copy carrying terminal leaves' regions into the out buffer); + * descriptors (splits first, then gaps) already uploaded. The caller swaps + * the out buffer in as the new main index array afterwards. */ + void LaunchSplitLevelBatchedKernels(const int num_splits, const int max_num_blocks, + const int num_gaps, const int max_gap_blocks); + void GenDataToLeftBitVector( const data_size_t num_data_in_leaf, const int split_feature_index, @@ -137,7 +276,10 @@ class CUDADataPartition: public NCCLInfo { double* left_leaf_sum_of_gradients, double* right_leaf_sum_of_gradients, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data); + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main, + const int deferred_slot, + const data_size_t leaf_data_start_for_copy); // kernel launch functions void LaunchFillDataIndicesBeforeTrain(); @@ -161,7 +303,10 @@ class CUDADataPartition: public NCCLInfo { double* left_leaf_sum_of_gradients, double* right_leaf_sum_of_gradients, data_size_t* global_left_leaf_num_data, - data_size_t* global_right_leaf_num_data); + data_size_t* global_right_leaf_num_data, + const bool point_structs_at_main, + const int deferred_slot, + const data_size_t leaf_data_start_for_copy); void LaunchGenDataToLeftBitVectorKernel( const data_size_t num_data_in_leaf, @@ -355,6 +500,12 @@ class CUDADataPartition: public NCCLInfo { // CUDA streams /*! \brief cuda streams used for asynchronizing kernel computing and memory copy */ std::vector cuda_streams_; + // recorded on cuda_streams_[2] after CopyDataIndicesKernel; consumed at the next + // Split() so back-to-back splits (hybrid level growth) don't race on the shared + // cuda_out_data_indices_in_leaf_ scratch while the copy is still in flight + cudaEvent_t indices_copy_done_event_ = nullptr; + // host staging for the batched apply descriptors (one entry per split of a level) + std::vector host_apply_descs_; // CUDA memory, held by this object @@ -383,10 +534,25 @@ class CUDADataPartition: public NCCLInfo { CUDAVector cuda_block_data_to_right_offset_; /*! \brief buffer for splitting data indices, will be copied back to cuda_data_indices_ after split */ CUDAVector cuda_out_data_indices_in_leaf_; + /*! \brief device copy of one level's batched apply descriptors */ + CUDAVector cuda_apply_descs_; + /*! \brief selective grow-then-prune: device copy of one level's collapsed + * subtree windows and host staging (see CollapseLeafWindows) */ + CUDAVector cuda_collapse_windows_; + std::vector host_collapse_windows_; + /*! \brief selective grow-then-prune: device copy of the hybrid-to-final leaf + * index map (see RemapDataIndexToLeafIndex) */ + CUDAVector cuda_leaf_index_remap_; // split tree structure algorithm related /*! \brief buffer to store split information, prepared to be copied to cpu */ CUDAVector cuda_split_info_buffer_; + /*! \brief pinned staging buffer of FinishSplitBatch */ + int* pinned_split_info_ = nullptr; + size_t pinned_split_info_size_ = 0; + /*! \brief per-split smaller-child size of the current level's batched apply + * (see level_smaller_leaf_counts()) */ + CUDAVector cuda_level_smaller_counts_; // dataset information /*! \brief number of data in training set, for initialization of cuda_leaf_num_data_ and cuda_leaf_data_end_ */ diff --git a/src/treelearner/cuda/cuda_histogram_constructor.cpp b/src/treelearner/cuda/cuda_histogram_constructor.cpp index 811fde64f8f8..62178605428c 100644 --- a/src/treelearner/cuda/cuda_histogram_constructor.cpp +++ b/src/treelearner/cuda/cuda_histogram_constructor.cpp @@ -10,6 +10,7 @@ #include "cuda_histogram_constructor.hpp" #include +#include #include namespace LightGBM { @@ -82,18 +83,72 @@ void CUDAHistogramConstructor::InitFeatureMetaInfo(const Dataset* train_data, co } else { num_total_bin_ = static_cast(feature_hist_offsets.back()); } + // register-accumulation construct body (batched compact path only): usable + // when EVERY feature fits the register bin cap (see kRegHistMaxBins == 8); + // EXABOOST_BATCH_REGHIST=0 disables it + static const bool reg_hist_enabled = []() { + const char* env = std::getenv("EXABOOST_BATCH_REGHIST"); + return env == nullptr || std::string(env) != "0"; + }(); + uint32_t max_num_bin = 0; + for (const uint32_t num_bin : feature_num_bins_) { + if (num_bin > max_num_bin) { + max_num_bin = num_bin; + } + } + construct_reg_bins_ = reg_hist_enabled && max_num_bin <= 8 && !feature_num_bins_.empty(); } +void LaunchInterleaveGradHessKernel( + const score_t* gradients, + const score_t* hessians, + float2* gradients_hessians, + data_size_t num_data); + void CUDAHistogramConstructor::BeforeTrain(const score_t* gradients, const score_t* hessians) { cuda_gradients_ = gradients; cuda_hessians_ = hessians; - cuda_hist_.SetValue(0); + // interleave (gradient, hessian) into float2 pairs for the dense construct + // kernels (one scattered 32B sector per row instead of two). Launched on the + // legacy default stream, so it orders before the construct kernels on the + // (blocking) histogram streams; a trivial streaming kernel (~0.05ms at 5M rows). + gh_interleave_valid_ = false; + if (!use_quantized_grad_ && GHInterleaveEnabled() && + gradients != nullptr && hessians != nullptr) { + if (cuda_gradients_hessians_.Size() < static_cast(num_data_)) { + cuda_gradients_hessians_.Resize(static_cast(num_data_)); + } + LaunchInterleaveGradHessKernel(gradients, hessians, + cuda_gradients_hessians_.RawData(), num_data_); + gh_interleave_valid_ = true; + } + // async memset on the legacy default stream: the construct kernels on the + // (blocking) histogram streams implicitly order after it, so no device sync + // is needed (SetValue would pay a full device sync on every tree). + // Only the leaf slots the previous tree actually used can be dirty (leaf k's + // histogram lives at slot k, and every tree assigns slots [0, num_leaves) + // consecutively), so zero just those: with num_leaves=1023 buffers trees of + // ~100 leaves this cuts >100MB (~65us of GPU memset) per tree. + const size_t num_slots_to_zero = num_dirty_leaves_ < 0 ? + static_cast(num_leaves_) : + std::min(static_cast(num_dirty_leaves_), static_cast(num_leaves_)); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_hist_.RawData()), 0, + num_slots_to_zero * static_cast(2 * num_total_bin_) * sizeof(hist_t))); } void CUDAHistogramConstructor::ZeroHistForLeaf(int /*leaf_index*/) { // No-op: BeforeTrain zeroes the entire cuda_hist_ buffer. } +void CUDAHistogramConstructor::ZeroHistSlots(const std::vector& slots) { + const size_t slot_size = static_cast(2 * num_total_bin_); + for (const int slot : slots) { + CUDASUCCESS_OR_FATAL(cudaMemsetAsync( + reinterpret_cast(cuda_hist_.RawData() + static_cast(slot) * slot_size), 0, + slot_size * sizeof(hist_t))); + } +} + void CUDAHistogramConstructor::SetFeatureUsedBytree(const std::vector& is_feature_used_bytree) { if (cuda_is_feature_used_bytree_.Size() != is_feature_used_bytree.size()) { cuda_is_feature_used_bytree_.Resize(is_feature_used_bytree.size()); @@ -101,6 +156,39 @@ void CUDAHistogramConstructor::SetFeatureUsedBytree(const std::vector& i CopyFromHostToCUDADevice(cuda_is_feature_used_bytree_.RawData(), is_feature_used_bytree.data(), is_feature_used_bytree.size(), __FILE__, __LINE__); + // per-tree bin-level used mask for the batched fix/subtract/construct-merge + // kernels: with feature_fraction sampling, ~ (1 - fraction) of every leaf + // histogram belongs to features no kernel of this tree will ever read, so the + // elementwise batched kernels skip them. nullptr (no sampling) keeps every + // kernel byte-identical to the unmasked behavior. + any_feature_unused_bytree_ = false; + const int mask_features = std::min(num_features_, static_cast(is_feature_used_bytree.size())); + for (int f = 0; f < mask_features; ++f) { + if (!is_feature_used_bytree[f]) { + any_feature_unused_bytree_ = true; + break; + } + } + if (any_feature_unused_bytree_) { + host_bin_used_bytree_.assign(static_cast(num_total_bin_), 0); + for (int f = 0; f < mask_features; ++f) { + if (!is_feature_used_bytree[f]) { + continue; + } + const uint32_t bin_start = feature_hist_offsets_[f]; + const uint32_t bin_end = f + 1 < static_cast(feature_hist_offsets_.size()) ? + feature_hist_offsets_[f + 1] : static_cast(num_total_bin_); + for (uint32_t bin = bin_start; bin < bin_end && bin < static_cast(num_total_bin_); ++bin) { + host_bin_used_bytree_[bin] = 1; + } + } + if (cuda_bin_used_bytree_.Size() < static_cast(num_total_bin_)) { + cuda_bin_used_bytree_.Resize(static_cast(num_total_bin_)); + } + CopyFromHostToCUDADevice(cuda_bin_used_bytree_.RawData(), + host_bin_used_bytree_.data(), + host_bin_used_bytree_.size(), __FILE__, __LINE__); + } } void LaunchDiagRead(cudaStream_t stream, const uint8_t* src, uint8_t* dst, int n); @@ -123,10 +211,26 @@ void LaunchFillCompactDataKernel( const size_t* slot_dst_byte, const int* slot_dst_stride, int total_compact_cols, + data_size_t num_data, + uint8_t* colmajor_out); + +// Implemented in cuda_histogram_constructor.cu — 4-bit packed source AND +// destination variant (one thread per destination byte = compact column pair). +void LaunchFillCompactData4BitKernel( + cudaStream_t stream, + const uint8_t* src_data, + uint8_t* compact_data, + const size_t* bs_src_nib0, + const size_t* bs_src_nib1, + const int* bs_src_stride_nib, + const size_t* bs_dst_byte, + const int* bs_dst_stride, + int total_byte_slots, data_size_t num_data); bool CUDAHistogramConstructor::BuildCompactView(const std::vector& is_feature_used_bytree) { use_compact_view_ = false; + compact_col_major_filled_ = false; // Gate: only support the standard dense path (uint8 bins, no large-bin partitions, no sparse). // This is what our Numerai workload uses; other paths fall back to the full kernel. if (cuda_row_data_->is_sparse() || cuda_row_data_->bit_type() != 8 || @@ -177,11 +281,60 @@ bool CUDAHistogramConstructor::BuildCompactView(const std::vector& is_fe const data_size_t num_data = cuda_row_data_->num_data(); + // 4-bit mode: when the row matrix is packed, the compact matrix is packed the + // same way (per-partition packed row width = ceil(used_columns / 2) bytes, + // column j of a partition in byte (j >> 1), nibble (j & 1)). + compact_is_4bit_ = cuda_row_data_->is_4bit_packed() && !cuda_row_data_->is_data_host_mapped(); + std::vector compact_packed_part_offsets; // [P+1] prefix of packed compact row widths + compact_packed_part_offsets.push_back(0); + for (int p = 0; p < num_partitions; ++p) { + const int used_in_p = compact_part_col_offsets[p + 1] - compact_part_col_offsets[p]; + compact_packed_part_offsets.push_back(compact_packed_part_offsets.back() + ((used_in_p + 1) >> 1)); + } + + // Host copies of the compact layout for the tree learner's column-view build + // (source column per slot + row-major-in-partition placement of each slot). + // 8-bit: slot byte includes the column-in-partition offset (col entry 0); + // 4-bit: slot byte is the partition's packed base and the logical + // column-in-partition travels separately (the gather kernel derives + // byte (col >> 1) / nibble (col & 1) from it). + compact_src_cols_host_.resize(total_compact); + compact_slot_byte_host_.resize(total_compact); + compact_slot_stride_host_.resize(total_compact); + compact_slot_col_host_.resize(total_compact); + for (int s = 0; s < total_compact; ++s) { + const int p = partition_for_compact_h[s]; + compact_src_cols_host_[s] = src_part_col_offsets[p] + src_local_col_for_compact_h[s]; + const int compact_part_start = compact_part_col_offsets[p]; + const int compact_col_in_p = s - compact_part_start; + if (compact_is_4bit_) { + compact_slot_byte_host_[s] = static_cast(compact_packed_part_offsets[p]) * static_cast(num_data); + compact_slot_stride_host_[s] = compact_packed_part_offsets[p + 1] - compact_packed_part_offsets[p]; + compact_slot_col_host_[s] = compact_col_in_p; + } else { + const int compact_stride = compact_part_col_offsets[p + 1] - compact_part_start; + compact_slot_byte_host_[s] = static_cast(compact_part_start) * static_cast(num_data) + + static_cast(compact_col_in_p); + compact_slot_stride_host_[s] = compact_stride; + compact_slot_col_host_[s] = 0; + } + } + // Allocate / resize compact buffers. - const size_t compact_data_bytes = static_cast(total_compact) * static_cast(num_data); + const size_t compact_data_bytes = compact_is_4bit_ ? + static_cast(compact_packed_part_offsets.back()) * static_cast(num_data) : + static_cast(total_compact) * static_cast(num_data); if (compact_data_uint8_t_.Size() < compact_data_bytes) { compact_data_uint8_t_.Resize(compact_data_bytes); } + if (compact_is_4bit_) { + if (compact_packed_partition_byte_offsets_.Size() < compact_packed_part_offsets.size()) { + compact_packed_partition_byte_offsets_.Resize(compact_packed_part_offsets.size()); + } + CopyFromHostToCUDADevice(compact_packed_partition_byte_offsets_.RawData(), + compact_packed_part_offsets.data(), + compact_packed_part_offsets.size(), __FILE__, __LINE__); + } // Upload metadata. // We need cuda copies of: @@ -283,6 +436,69 @@ bool CUDAHistogramConstructor::BuildCompactView(const std::vector& is_fe total_compact); CUDASUCCESS_OR_FATAL(cudaStreamSynchronize(cuda_stream_)); compact_is_col_major_ = false; // compact_data is now row-major-in-partition + compact_col_major_filled_ = true; // the col-major staging holds the same slots + } else if (compact_is_4bit_) { + // 4-bit fill: one thread per DESTINATION byte (a pair of adjacent compact + // slots), so no two threads share an output byte. Source positions are + // nibble indices: column c of a packed partition with byte base B and row + // width W sits at nibble 2*B + c + row * (2*W). + const std::vector& src_packed_offsets = cuda_row_data_->host_packed_partition_byte_offsets(); + int total_byte_slots = 0; + for (int p = 0; p < num_partitions; ++p) { + const int used_in_p = compact_part_col_offsets[p + 1] - compact_part_col_offsets[p]; + total_byte_slots += (used_in_p + 1) >> 1; + } + std::vector bs_src_nib0_h(total_byte_slots); + std::vector bs_src_nib1_h(total_byte_slots); + std::vector bs_src_stride_nib_h(total_byte_slots); + std::vector bs_dst_byte_h(total_byte_slots); + std::vector bs_dst_stride_h(total_byte_slots); + int byte_slot = 0; + for (int p = 0; p < num_partitions; ++p) { + const int compact_part_start = compact_part_col_offsets[p]; + const int used_in_p = compact_part_col_offsets[p + 1] - compact_part_start; + const int dst_packed_width = compact_packed_part_offsets[p + 1] - compact_packed_part_offsets[p]; + const size_t dst_part_byte = static_cast(compact_packed_part_offsets[p]) * static_cast(num_data); + const size_t src_part_nib = static_cast(src_packed_offsets[p]) * static_cast(num_data) * 2; + const int src_stride_nib = (src_packed_offsets[p + 1] - src_packed_offsets[p]) * 2; + for (int m = 0; m < ((used_in_p + 1) >> 1); ++m) { + const int s0 = compact_part_start + 2 * m; + bs_src_nib0_h[byte_slot] = src_part_nib + static_cast(src_local_col_for_compact_h[s0]); + bs_src_nib1_h[byte_slot] = (2 * m + 1) < used_in_p ? + src_part_nib + static_cast(src_local_col_for_compact_h[s0 + 1]) : + ~static_cast(0); + bs_src_stride_nib_h[byte_slot] = src_stride_nib; + bs_dst_byte_h[byte_slot] = dst_part_byte + static_cast(m); + bs_dst_stride_h[byte_slot] = dst_packed_width; + ++byte_slot; + } + } + CHECK_EQ(byte_slot, total_byte_slots); + if (cuda_bs_src_nib0_.Size() < static_cast(total_byte_slots)) { + cuda_bs_src_nib0_.Resize(total_byte_slots); + cuda_bs_src_nib1_.Resize(total_byte_slots); + cuda_bs_src_stride_nib_.Resize(total_byte_slots); + cuda_bs_dst_byte_.Resize(total_byte_slots); + cuda_bs_dst_stride_.Resize(total_byte_slots); + } + CopyFromHostToCUDADevice(cuda_bs_src_nib0_.RawData(), bs_src_nib0_h.data(), total_byte_slots, __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_bs_src_nib1_.RawData(), bs_src_nib1_h.data(), total_byte_slots, __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_bs_src_stride_nib_.RawData(), bs_src_stride_nib_h.data(), total_byte_slots, __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_bs_dst_byte_.RawData(), bs_dst_byte_h.data(), total_byte_slots, __FILE__, __LINE__); + CopyFromHostToCUDADevice(cuda_bs_dst_stride_.RawData(), bs_dst_stride_h.data(), total_byte_slots, __FILE__, __LINE__); + LaunchFillCompactData4BitKernel( + cuda_stream_, + cuda_row_data_->GetBin(), + compact_data_uint8_t_.RawData(), + cuda_bs_src_nib0_.RawData(), + cuda_bs_src_nib1_.RawData(), + cuda_bs_src_stride_nib_.RawData(), + cuda_bs_dst_byte_.RawData(), + cuda_bs_dst_stride_.RawData(), + total_byte_slots, + num_data); + CUDASUCCESS_OR_FATAL(cudaStreamSynchronize(cuda_stream_)); + compact_is_col_major_ = false; } else { // Build per-slot src/dst metadata host-side. Each compact slot has a fully // computed source byte offset and destination byte offset, so the kernel @@ -315,6 +531,10 @@ bool CUDAHistogramConstructor::BuildCompactView(const std::vector& is_fe CopyFromHostToCUDADevice(cuda_slot_src_stride_.RawData(), slot_src_stride_h.data(), total_compact, __FILE__, __LINE__); CopyFromHostToCUDADevice(cuda_slot_dst_byte_.RawData(), slot_dst_byte_h.data(), total_compact, __FILE__, __LINE__); CopyFromHostToCUDADevice(cuda_slot_dst_stride_.RawData(), slot_dst_stride_h.data(), total_compact, __FILE__, __LINE__); + // NOTE: a fused column-major second output here was measured SLOWER than + // the tree learner's separate tile-transposed gather from the compact + // matrix (the fill's slot-major warps write the column-major layout one + // 32-byte sector per byte); keep the fill single-output. LaunchFillCompactDataKernel( cuda_stream_, cuda_row_data_->GetBin(), @@ -324,7 +544,8 @@ bool CUDAHistogramConstructor::BuildCompactView(const std::vector& is_fe cuda_slot_dst_byte_.RawData(), cuda_slot_dst_stride_.RawData(), total_compact, - num_data); + num_data, + nullptr); CUDASUCCESS_OR_FATAL(cudaStreamSynchronize(cuda_stream_)); compact_is_col_major_ = false; } @@ -357,11 +578,20 @@ void CUDAHistogramConstructor::Init(const Dataset* train_data, TrainingShareStat CUDASUCCESS_OR_FATAL(cudaStreamCreate(&cuda_stream_)); // Lightweight (timing-disabled) events used to order the best split finder's // per-leaf kernels after histogram construction/subtraction without a device sync. - CUDASUCCESS_OR_FATAL(cudaEventCreateWithFlags(&construct_done_event_, cudaEventDisableTiming)); - CUDASUCCESS_OR_FATAL(cudaEventCreateWithFlags(&subtract_done_event_, cudaEventDisableTiming)); + // One (stream, event-pair) pipeline per concurrently-processed sibling pair. + pipeline_streams_[0] = cuda_stream_; + for (int p = 1; p < kNumHistPipelines; ++p) { + CUDASUCCESS_OR_FATAL(cudaStreamCreate(&pipeline_streams_[p])); + } + for (int p = 0; p < kNumHistPipelines; ++p) { + CUDASUCCESS_OR_FATAL(cudaEventCreateWithFlags(&construct_done_events_[p], cudaEventDisableTiming)); + CUDASUCCESS_OR_FATAL(cudaEventCreateWithFlags(&subtract_done_events_[p], cudaEventDisableTiming)); + } cuda_need_fix_histogram_features_.InitFromHostVector(need_fix_histogram_features_); cuda_need_fix_histogram_features_num_bin_aligned_.InitFromHostVector(need_fix_histogram_features_num_bin_aligend_); + cuda_hybrid_construct_dim_y_.Resize(1); + InitFixMFBMask(); if (cuda_row_data_->NumLargeBinPartition() > 0) { int grid_dim_x = 0, grid_dim_y = 0, block_dim_x = 0, block_dim_y = 0; @@ -379,7 +609,12 @@ void CUDAHistogramConstructor::Init(const Dataset* train_data, TrainingShareStat cuda_hist_buffer_.Resize(buffer_size); } } - hist_buffer_for_num_bit_change_.Resize(num_total_bin_ * 2); + // one int32 region of num_total_bin_ entries per histogram pipeline (the pairs + // of a level run concurrently on different pipeline streams and must not share + // the 64->32-bit compaction scratch space); sized in hist_t (8 byte) units + hist_buffer_for_num_bit_change_.Resize( + std::max(static_cast(num_total_bin_) * 2, + (static_cast(num_total_bin_) * kNumHistPipelines + 1) / 2)); } void CUDAHistogramConstructor::ConstructHistogramForLeaf( @@ -400,7 +635,62 @@ if ((global_num_data_in_smaller_leaf <= min_data_in_leaf_ || sum_hessians_in_sma // Record completion on cuda_stream_ instead of a device-wide sync. The best split // finder waits on this event before reading the smaller-leaf histogram, so the host // is not stalled here (see CUDABestSplitFinder::FindBestSplitsForLeaf). - CUDASUCCESS_OR_FATAL(cudaEventRecord(construct_done_event_, cuda_stream_)); + CUDASUCCESS_OR_FATAL(cudaEventRecord(construct_done_events_[active_pipeline_], current_stream())); +} + +void CUDAHistogramConstructor::ConstructHistogramsForLevel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const bool any_pair_needs_bit_change_copy, + const data_size_t* level_smaller_num_data) { + if (num_pairs <= 0) { + return; + } + if (use_quantized_grad_ && any_pair_needs_bit_change_copy) { + // one int32 region of num_total_bin_ entries per pair; sized in hist_t (8 byte) + // units so this allocates twice the strict need, but is only reached when a + // level actually contains a >16-bit parent with a <=16-bit larger child + const size_t needed = static_cast(num_pairs) * static_cast(num_total_bin_); + if (hist_buffer_for_num_bit_change_.Size() < needed) { + hist_buffer_for_num_bit_change_.Resize(needed); + } + } + global_timer.Start("CUDAHistogramConstructor::ConstructHistogramsForLevel"); + // The batched construct kernel routes each pair whose ACTUAL smaller leaf is + // tiny (on-device check against SmallLeafRowThreshold(); non-quantized only) + // to a direct global-atomic body that skips the shared-histogram zero+merge + // dominating small leaves. The quantized path never takes it: its integer + // shared-then-merge accumulation (and the covtype quant md5 lock) stays + // byte-identical. + LaunchConstructHistogramBatchedKernel(pair_descs, num_pairs, max_num_data_in_smaller_leaf, + level_smaller_num_data); + // (no construct_done event here: the batched find kernel only waits on the + // subtract event below, and the per-pair path re-records its own events) + if (!use_quantized_grad_ && SmallLeafConstructEnabled()) { + // fused fix + subtract: one launch, bit-identical to the sequential pair + LaunchFixSubtractHistogramSmallLeafBatchedKernel(pair_descs, num_pairs); + } else { + LaunchSubtractHistogramBatchedKernel(pair_descs, num_pairs, any_pair_needs_bit_change_copy); + } + // the best split finder's batched find kernel waits on this event before reading + // any of this level's histograms (construct/fix/subtract are stream-ordered here) + CUDASUCCESS_OR_FATAL(cudaEventRecord(subtract_done_events_[0], cuda_stream_)); + global_timer.Stop("CUDAHistogramConstructor::ConstructHistogramsForLevel"); +} + +void CUDAHistogramConstructor::InitFixMFBMask() { + // mask over the 2 * num_total_bin_ histogram entries: 1 at the most-frequent- + // bin gradient/hessian slots of the need-fix features (owned by the fused + // small-leaf kernel's fix blocks), 0 everywhere else (subtract blocks) + std::vector host_mask(static_cast(2 * num_total_bin_), 0); + for (const int feature_index : need_fix_histogram_features_) { + const size_t pos = (static_cast(feature_hist_offsets_[feature_index]) + + static_cast(feature_most_freq_bins_[feature_index])) << 1; + host_mask[pos] = 1; + host_mask[pos + 1] = 1; + } + cuda_fix_mfb_mask_.InitFromHostVector(host_mask); } void CUDAHistogramConstructor::SubtractHistogramForLeaf( @@ -416,7 +706,7 @@ void CUDAHistogramConstructor::SubtractHistogramForLeaf( // Record completion on cuda_stream_; the best split finder waits on this before // reading the larger-leaf (subtracted) histogram, replacing the per-split device sync // that previously separated the smaller- and larger-leaf FindBestSplits launches. - CUDASUCCESS_OR_FATAL(cudaEventRecord(subtract_done_event_, cuda_stream_)); + CUDASUCCESS_OR_FATAL(cudaEventRecord(subtract_done_events_[active_pipeline_], current_stream())); global_timer.Stop("CUDAHistogramConstructor::ConstructHistogramForLeaf::LaunchSubtractHistogramKernel"); } @@ -433,6 +723,30 @@ void CUDAHistogramConstructor::CalcConstructHistogramKernelDim( ((num_data_in_smaller_leaf + NUM_DATA_PER_THREAD - 1) / NUM_DATA_PER_THREAD + (*block_dim_y) - 1) / (*block_dim_y)); } +void CUDAHistogramConstructor::CalcConstructHistogramBatchedKernelDim( + int* grid_dim_x, + int* grid_dim_y, + int* block_dim_x, + int* block_dim_y, + const data_size_t max_num_data_in_smaller_leaf, + const int num_pairs) { + *block_dim_x = cuda_row_data_->max_num_column_per_partition(); + *block_dim_y = NUM_THREADS_PER_BLOCK / cuda_row_data_->max_num_column_per_partition(); + *grid_dim_x = cuda_row_data_->num_feature_partitions(); + // The per-leaf sizing forces min_grid_dim_y_ y-blocks to saturate the device + // for a SINGLE leaf; every active block, however, pays a fixed shared-hist + // zero + global-merge cost, which dominates small leaves. In the batched + // kernel the pair grid dimension already provides parallelism, so share the + // saturation floor across pairs and otherwise cap the y-grid at + // min_rows_per_thread rows per thread (identical to the per-leaf sizing for + // single-pair levels and for leaves large enough that the cap is inactive). + // The formula lives in HybridBatchedConstructGridDimY so the device replica + // used by the speculative single-sync flow computes the identical value. + *grid_dim_y = HybridBatchedConstructGridDimY( + max_num_data_in_smaller_leaf, num_pairs, *block_dim_y, min_grid_dim_y_, + BatchConstructMinRowsPerThread(), BatchConstructSaturationFloor()); +} + void CUDAHistogramConstructor::ResetTrainingData(const Dataset* train_data, TrainingShareStates* share_states) { num_data_ = train_data->num_data(); num_features_ = train_data->num_features(); @@ -440,6 +754,7 @@ void CUDAHistogramConstructor::ResetTrainingData(const Dataset* train_data, Trai cuda_hist_.Resize(static_cast(num_total_bin_ * 2 * num_leaves_)); cuda_hist_.SetValue(0); + num_dirty_leaves_ = -1; cuda_feature_num_bins_.InitFromHostVector(feature_num_bins_); cuda_feature_hist_offsets_.InitFromHostVector(feature_hist_offsets_); cuda_feature_most_freq_bins_.InitFromHostVector(feature_most_freq_bins_); @@ -449,6 +764,7 @@ void CUDAHistogramConstructor::ResetTrainingData(const Dataset* train_data, Trai cuda_need_fix_histogram_features_.InitFromHostVector(need_fix_histogram_features_); cuda_need_fix_histogram_features_num_bin_aligned_.InitFromHostVector(need_fix_histogram_features_num_bin_aligend_); + InitFixMFBMask(); } void CUDAHistogramConstructor::ResetConfig(const Config* config) { @@ -458,6 +774,7 @@ void CUDAHistogramConstructor::ResetConfig(const Config* config) { min_sum_hessian_in_leaf_ = config->min_sum_hessian_in_leaf; cuda_hist_.Resize(static_cast(num_total_bin_ * 2 * num_leaves_)); cuda_hist_.SetValue(0); + num_dirty_leaves_ = -1; } } // namespace LightGBM diff --git a/src/treelearner/cuda/cuda_histogram_constructor.cu b/src/treelearner/cuda/cuda_histogram_constructor.cu index 3d2235accded..f6dece0d6a3b 100644 --- a/src/treelearner/cuda/cuda_histogram_constructor.cu +++ b/src/treelearner/cuda/cuda_histogram_constructor.cu @@ -46,17 +46,25 @@ __global__ void CUDAFillCompactDataKernel( const size_t* __restrict__ slot_dst_byte, const int* __restrict__ slot_dst_stride, const int total_compact_cols, - const data_size_t num_data) { + const data_size_t num_data, + uint8_t* __restrict__ colmajor_out) { const int slot = blockIdx.x * blockDim.x + threadIdx.x; if (slot >= total_compact_cols) return; const size_t src_byte = slot_src_byte[slot]; const size_t src_stride = static_cast(slot_src_stride[slot]); const size_t dst_byte = slot_dst_byte[slot]; const size_t dst_stride = static_cast(slot_dst_stride[slot]); + const size_t colmajor_base = static_cast(slot) * static_cast(num_data); const data_size_t row_stride = static_cast(gridDim.y) * static_cast(blockDim.y); for (data_size_t row = blockIdx.y * blockDim.y + threadIdx.y; row < num_data; row += row_stride) { - compact_data[dst_byte + static_cast(row) * dst_stride] = - src_data[src_byte + static_cast(row) * src_stride]; + const uint8_t val = src_data[src_byte + static_cast(row) * src_stride]; + compact_data[dst_byte + static_cast(row) * dst_stride] = val; + if (colmajor_out != nullptr) { + // fused second output: the tree learner's column-major compact view + // (compact_col_buf[slot * num_data + row]), produced from the same source + // read so the full bin matrix streams through L2 only once per tree + colmajor_out[colmajor_base + static_cast(row)] = val; + } } } @@ -75,6 +83,7 @@ __global__ void CUDAFillCompactDataKernel( // slot_p_byte[s] = partition_byte_offset for that slot's source column // slot_p_stride[s] = partition row stride // slot_col_in_p[s] = column index within partition +template __global__ void CUDARowToColCompactKernel( const uint8_t* __restrict__ src_data, uint8_t* __restrict__ compact_col_buf, @@ -103,10 +112,20 @@ __global__ void CUDARowToColCompactKernel( const data_size_t row = row_block + ty; uint8_t val = 0; if (slot < num_compact_cols && row < num_data) { - const size_t base = slot_p_byte[slot] - + static_cast(row) * static_cast(slot_p_stride[slot]) - + static_cast(slot_col_in_p[slot]); - val = src_data[base]; + const int col_in_p = slot_col_in_p[slot]; + if (IS_4BIT) { + // packed source: slot_p_byte/slot_p_stride are the partition's packed + // byte base/row width; column j sits in byte (j >> 1), nibble (j & 1) + const size_t base = slot_p_byte[slot] + + static_cast(row) * static_cast(slot_p_stride[slot]) + + static_cast(col_in_p >> 1); + val = (src_data[base] >> ((col_in_p & 1) << 2)) & 0xf; + } else { + const size_t base = slot_p_byte[slot] + + static_cast(row) * static_cast(slot_p_stride[slot]) + + static_cast(col_in_p); + val = src_data[base]; + } } tile[ty][tx] = val; } @@ -132,7 +151,8 @@ void LaunchRowToColCompactKernel( const int* slot_p_stride, const int* slot_col_in_p, int num_compact_cols, - data_size_t num_data) { + data_size_t num_data, + bool src_is_4bit) { const int TX = 32; const int TY = 32; // Cap grid_y at 32k to stay under CUDA's 65535 limit; kernel strides over rows. @@ -140,9 +160,15 @@ void LaunchRowToColCompactKernel( if (grid_y > 32768) grid_y = 32768; dim3 block_dim(TX, TY); dim3 grid_dim((num_compact_cols + TX - 1) / TX, grid_y); - CUDARowToColCompactKernel<<>>( - src_data, compact_col_buf, slot_p_byte, slot_p_stride, slot_col_in_p, - num_compact_cols, num_data); + if (src_is_4bit) { + CUDARowToColCompactKernel<<>>( + src_data, compact_col_buf, slot_p_byte, slot_p_stride, slot_col_in_p, + num_compact_cols, num_data); + } else { + CUDARowToColCompactKernel<<>>( + src_data, compact_col_buf, slot_p_byte, slot_p_stride, slot_col_in_p, + num_compact_cols, num_data); + } } // Transpose column-major staging into row-major-in-partition compact_data. @@ -192,6 +218,34 @@ void LaunchTransposeColMajorToRowMajor( num_data, total_compact_cols); } +// Interleave the per-row gradient and hessian arrays into float2 pairs so the +// scattered per-row reads of the dense construct kernels touch one 32B sector +// per row instead of two. Bit-identical values, purely a layout change. +__global__ void InterleaveGradHessKernel( + const score_t* __restrict__ gradients, + const score_t* __restrict__ hessians, + float2* __restrict__ gradients_hessians, + const data_size_t num_data) { + const data_size_t i = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; + if (i < num_data) { + gradients_hessians[i] = make_float2(static_cast(gradients[i]), + static_cast(hessians[i])); + } +} + +// Host wrapper called from cuda_histogram_constructor.cpp. Legacy default +// stream: ordered before subsequently enqueued work on the blocking streams. +void LaunchInterleaveGradHessKernel( + const score_t* gradients, + const score_t* hessians, + float2* gradients_hessians, + data_size_t num_data) { + const int block_size = 1024; + const int num_blocks = static_cast((num_data + block_size - 1) / block_size); + InterleaveGradHessKernel<<>>( + gradients, hessians, gradients_hessians, num_data); +} + // Diagnostic kernel: read N bytes from a (possibly host-mapped) source pointer. __global__ void DiagReadKernel(const uint8_t* __restrict__ src, uint8_t* dst, int n) { const int i = blockIdx.x * blockDim.x + threadIdx.x; @@ -211,7 +265,8 @@ void LaunchFillCompactDataKernel( const size_t* slot_dst_byte, const int* slot_dst_stride, int total_compact_cols, - data_size_t num_data) { + data_size_t num_data, + uint8_t* colmajor_out) { const int TX = 32; const int TY = 32; // Cap grid_y at 32k so we stay well under CUDA's 65535 limit; the kernel @@ -228,7 +283,70 @@ void LaunchFillCompactDataKernel( slot_dst_byte, slot_dst_stride, total_compact_cols, - num_data); + num_data, + colmajor_out); +} + +// 4-bit variant of the compaction: both the SOURCE (full packed bin matrix) and +// the DESTINATION (packed compact matrix) hold two columns per byte. One thread +// owns one (destination byte, row) cell, i.e. a PAIR of adjacent compact slots, +// so no two threads read-modify-write the same output byte. Source positions are +// precomputed per byte-slot as NIBBLE indices (byte * 2 + nibble): the base +// nibble of each of the two source columns plus a per-row nibble stride +// (2 * packed source row width). bs_src_nib1 == SIZE_MAX marks the padding +// nibble of an odd-width partition (writes 0). +__global__ void CUDAFillCompactData4BitKernel( + const uint8_t* __restrict__ src_data, + uint8_t* __restrict__ compact_data, + const size_t* __restrict__ bs_src_nib0, + const size_t* __restrict__ bs_src_nib1, + const int* __restrict__ bs_src_stride_nib, + const size_t* __restrict__ bs_dst_byte, + const int* __restrict__ bs_dst_stride, + const int total_byte_slots, + const data_size_t num_data) { + const int slot = blockIdx.x * blockDim.x + threadIdx.x; + if (slot >= total_byte_slots) return; + const size_t src_nib0 = bs_src_nib0[slot]; + const size_t src_nib1 = bs_src_nib1[slot]; + const size_t stride_nib = static_cast(bs_src_stride_nib[slot]); + const size_t dst_byte = bs_dst_byte[slot]; + const size_t dst_stride = static_cast(bs_dst_stride[slot]); + const bool has_hi = src_nib1 != ~static_cast(0); + const data_size_t row_stride = static_cast(gridDim.y) * static_cast(blockDim.y); + for (data_size_t row = blockIdx.y * blockDim.y + threadIdx.y; row < num_data; row += row_stride) { + const size_t nib0 = src_nib0 + static_cast(row) * stride_nib; + const uint8_t lo = (src_data[nib0 >> 1] >> ((nib0 & 1) << 2)) & 0xf; + uint8_t hi = 0; + if (has_hi) { + const size_t nib1 = src_nib1 + static_cast(row) * stride_nib; + hi = (src_data[nib1 >> 1] >> ((nib1 & 1) << 2)) & 0xf; + } + compact_data[dst_byte + static_cast(row) * dst_stride] = static_cast(lo | (hi << 4)); + } +} + +// Host wrapper called from cuda_histogram_constructor.cpp. +void LaunchFillCompactData4BitKernel( + cudaStream_t stream, + const uint8_t* src_data, + uint8_t* compact_data, + const size_t* bs_src_nib0, + const size_t* bs_src_nib1, + const int* bs_src_stride_nib, + const size_t* bs_dst_byte, + const int* bs_dst_stride, + int total_byte_slots, + data_size_t num_data) { + const int TX = 32; + const int TY = 32; + int grid_y = (num_data + TY - 1) / TY; + if (grid_y > 32768) grid_y = 32768; + dim3 block_dim(TX, TY); + dim3 grid_dim((total_byte_slots + TX - 1) / TX, grid_y); + CUDAFillCompactData4BitKernel<<>>( + src_data, compact_data, bs_src_nib0, bs_src_nib1, bs_src_stride_nib, + bs_dst_byte, bs_dst_stride, total_byte_slots, num_data); } // Column-major-in-partition variant of the dense histogram kernel. @@ -291,37 +409,88 @@ __global__ void CUDAConstructHistogramDenseColMajorKernel( } } -template -__global__ void CUDAConstructHistogramDenseKernel( +// Shared body of the dense histogram kernel; the shared-memory histogram is +// declared by the calling __global__ kernel and passed in so a kernel that +// instantiates the helper more than once does not duplicate the allocation. +// Blocks whose row range lies beyond this leaf's data exit before touching +// shared/global memory (they would only add zeros); this keeps over-provisioned +// batched grids (sized for the level's largest pair) cheap. dim_y is the row +// grouping extent: gridDim.y * blockDim.y in the classic flow, or the +// device-computed effective value in the speculative single-sync flow (whose +// launch grid is only an upper bound). +/*! \brief bin cap of the register-accumulation construct body (USE_REG_BINS): + * active only when EVERY feature has at most this many bins (host-gated) */ +#define kRegHistMaxBins (8) + +// 4-bit packed dense bin matrix (IS_4BIT construct variants): the partition's +// packed row width is ceil(num_columns_in_partition / 2) bytes (columns padded +// to an even count PER PARTITION, so each row segment is byte-aligned); column +// j lives in byte (j >> 1), nibble (j & 1), low nibble = even column. The +// packed per-partition byte-width prefix comes in via +// packed_partition_byte_offsets (nullptr and unused in the 8-bit variants). +template +__device__ __forceinline__ uint32_t ReadDenseBin( + const BIN_TYPE* row_ptr, const unsigned int column_in_partition) { + if (IS_4BIT) { + const uint32_t packed = static_cast(row_ptr[column_in_partition >> 1]); + return (packed >> ((column_in_partition & 1) << 2)) & 0xfu; + } else { + return static_cast(row_ptr[column_in_partition]); + } +} + +// USE_GH2: read the per-row (gradient, hessian) pair from cuda_gh, the +// interleaved float2 copy of the two score_t arrays (see GHInterleaveEnabled): +// same bits, one scattered 32B sector per row instead of two. Compile-time so +// the non-interleaved instantiation stays byte-identical to the historical +// kernel (a runtime branch measurably raised the kernel's latency). +template +__device__ __forceinline__ void ConstructHistogramDenseInner( const CUDALeafSplitsStruct* smaller_leaf_splits, + HIST_TYPE* shared_hist, const score_t* cuda_gradients, const score_t* cuda_hessians, + const float2* cuda_gh, const BIN_TYPE* data, const uint32_t* column_hist_offsets, const uint32_t* column_hist_offsets_full, const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, const int8_t* is_feature_used_bytree, - const data_size_t num_data) { - const int dim_y = static_cast(gridDim.y * blockDim.y); + const data_size_t num_data, + const int dim_y, + const uint8_t* bin_used = nullptr) { const data_size_t num_data_in_smaller_leaf = smaller_leaf_splits->num_data_in_leaf; const data_size_t num_data_per_thread = (num_data_in_smaller_leaf + dim_y - 1) / dim_y; + const unsigned int blockIdx_y = blockIdx.y; + const data_size_t block_start = (static_cast(blockIdx_y) * blockDim.y) * num_data_per_thread; + if (block_start >= num_data_in_smaller_leaf) { + return; + } const data_size_t* data_indices_ref = smaller_leaf_splits->data_indices_in_leaf; - __shared__ HIST_TYPE shared_hist[SHARED_HIST_SIZE]; const unsigned int num_threads_per_block = blockDim.x * blockDim.y; const int partition_column_start = feature_partition_column_index_offsets[blockIdx.x]; const int partition_column_end = feature_partition_column_index_offsets[blockIdx.x + 1]; - const BIN_TYPE* data_ptr = data + static_cast(partition_column_start) * num_data; const int num_columns_in_partition = partition_column_end - partition_column_start; + const int row_stride = IS_4BIT ? + packed_partition_byte_offsets[blockIdx.x + 1] - packed_partition_byte_offsets[blockIdx.x] : + num_columns_in_partition; + const BIN_TYPE* data_ptr = data + static_cast( + IS_4BIT ? packed_partition_byte_offsets[blockIdx.x] : partition_column_start) * num_data; const uint32_t partition_hist_start = column_hist_offsets_full[blockIdx.x]; const uint32_t partition_hist_end = column_hist_offsets_full[blockIdx.x + 1]; const uint32_t num_items_in_partition = (partition_hist_end - partition_hist_start) << 1; const unsigned int thread_idx = threadIdx.x + threadIdx.y * blockDim.x; + // bin_used (per-tree feature_fraction bin mask, may be null) skips the zeroing + // and global merge of histogram entries belonging to features outside this + // tree's sample: only used columns accumulate into shared memory, and unused + // global entries are dead storage this tree, so both loops may skip them. for (unsigned int i = thread_idx; i < num_items_in_partition; i += num_threads_per_block) { - shared_hist[i] = 0.0f; + if (bin_used == nullptr || bin_used[partition_hist_start + (i >> 1)]) { + shared_hist[i] = 0.0f; + } } __syncthreads(); - const unsigned int blockIdx_y = blockIdx.y; - const data_size_t block_start = (static_cast(blockIdx_y) * blockDim.y) * num_data_per_thread; const data_size_t* data_indices_ref_this_block = data_indices_ref + block_start; data_size_t block_num_data = max(0, min(num_data_in_smaller_leaf - block_start, num_data_per_thread * static_cast(blockDim.y))); const int column_index = static_cast(threadIdx.x) + partition_column_start; @@ -331,21 +500,320 @@ __global__ void CUDAConstructHistogramDenseKernel( (is_feature_used_bytree == nullptr || is_feature_used_bytree[column_index]); if (feat_used) { HIST_TYPE* shared_hist_ptr = shared_hist + (column_hist_offsets[column_index] << 1); - for (data_size_t inner_data_index = static_cast(threadIdx.y); inner_data_index < block_num_data; inner_data_index += blockDim.y) { - const data_size_t data_index = data_indices_ref_this_block[inner_data_index]; - const score_t grad = cuda_gradients[data_index]; - const score_t hess = cuda_hessians[data_index]; - const uint32_t bin = static_cast(data_ptr[static_cast(data_index) * num_columns_in_partition + threadIdx.x]); - const uint32_t pos = bin << 1; - HIST_TYPE* pos_ptr = shared_hist_ptr + pos; - atomicAdd_block(pos_ptr, grad); - atomicAdd_block(pos_ptr + 1, hess); + if (USE_REG_BINS) { + // Few-bin datasets (every feature <= kRegHistMaxBins bins): accumulate the + // thread's rows into registers and flush once, instead of two same-address + // shared atomics per row. With ~7 bins and blockDim.y threads per column + // the per-row atomics serialize heavily; the register accumulation is + // contention-free. Float accumulation ORDER differs from the atomic + // per-row order, so this path is non-quantized-only (quality-parity, not + // bit-parity, is the contract for non-quantized training). + HIST_TYPE reg_grad[kRegHistMaxBins]; + HIST_TYPE reg_hess[kRegHistMaxBins]; +#pragma unroll + for (int b = 0; b < kRegHistMaxBins; ++b) { + reg_grad[b] = 0.0f; + reg_hess[b] = 0.0f; + } + for (data_size_t inner_data_index = static_cast(threadIdx.y); inner_data_index < block_num_data; inner_data_index += blockDim.y) { + const data_size_t data_index = data_indices_ref_this_block[inner_data_index]; + score_t grad, hess; + if (USE_GH2) { + const float2 gh = cuda_gh[data_index]; + grad = gh.x; + hess = gh.y; + } else { + grad = cuda_gradients[data_index]; + hess = cuda_hessians[data_index]; + } + const uint32_t bin = ReadDenseBin(data_ptr + static_cast(data_index) * row_stride, threadIdx.x); +#pragma unroll + for (int b = 0; b < kRegHistMaxBins; ++b) { + if (bin == static_cast(b)) { + reg_grad[b] += grad; + reg_hess[b] += hess; + } + } + } +#pragma unroll + for (int b = 0; b < kRegHistMaxBins; ++b) { + // (0, 0) sums are no-ops; skipping them saves most of the flush atomics + if (reg_grad[b] != 0.0f || reg_hess[b] != 0.0f) { + atomicAdd_block(shared_hist_ptr + (b << 1), reg_grad[b]); + atomicAdd_block(shared_hist_ptr + (b << 1) + 1, reg_hess[b]); + } + } + } else { + for (data_size_t inner_data_index = static_cast(threadIdx.y); inner_data_index < block_num_data; inner_data_index += blockDim.y) { + const data_size_t data_index = data_indices_ref_this_block[inner_data_index]; + score_t grad, hess; + if (USE_GH2) { + const float2 gh = cuda_gh[data_index]; + grad = gh.x; + hess = gh.y; + } else { + grad = cuda_gradients[data_index]; + hess = cuda_hessians[data_index]; + } + const uint32_t bin = ReadDenseBin(data_ptr + static_cast(data_index) * row_stride, threadIdx.x); + const uint32_t pos = bin << 1; + HIST_TYPE* pos_ptr = shared_hist_ptr + pos; + atomicAdd_block(pos_ptr, grad); + atomicAdd_block(pos_ptr + 1, hess); + } } } __syncthreads(); hist_t* feature_histogram_ptr = smaller_leaf_splits->hist_in_leaf + (partition_hist_start << 1); for (unsigned int i = thread_idx; i < num_items_in_partition; i += num_threads_per_block) { - atomicAdd_system(feature_histogram_ptr + i, shared_hist[i]); + if (bin_used == nullptr || bin_used[partition_hist_start + (i >> 1)]) { + atomicAdd_system(feature_histogram_ptr + i, shared_hist[i]); + } + } +} + +template +__global__ void CUDAConstructHistogramDenseKernel( + const CUDALeafSplitsStruct* smaller_leaf_splits, + const score_t* cuda_gradients, + const score_t* cuda_hessians, + const BIN_TYPE* data, + const uint32_t* column_hist_offsets, + const uint32_t* column_hist_offsets_full, + const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, + const int8_t* is_feature_used_bytree, + const data_size_t num_data) { + __shared__ HIST_TYPE shared_hist[SHARED_HIST_SIZE]; + ConstructHistogramDenseInner( + smaller_leaf_splits, shared_hist, cuda_gradients, cuda_hessians, nullptr, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + is_feature_used_bytree, num_data, static_cast(gridDim.y * blockDim.y)); +} + +// Computes the effective row-grouping extent (dim_y) of the speculative batched +// construct launch: the exact host sizing formula applied to the level's ACTUAL +// smaller-child sizes (written by the batched apply's aggregate kernel), so the +// row grouping -- and hence the float histograms -- are bit-identical to the +// classic host sizing. One tiny single-block launch per level; the construct +// blocks then read a single int. +__global__ void ComputeBatchedConstructDimYKernel( + const data_size_t* level_smaller_num_data, + const int num_pairs, + const int block_dim_y, + const int min_grid_dim_y, + const int min_rows_per_thread, + const int saturation_floor_total, + int* out_dim_y) { + __shared__ data_size_t shared_max[32]; + data_size_t thread_max = 0; + for (int i = static_cast(threadIdx.x); i < num_pairs; i += static_cast(blockDim.x)) { + const data_size_t n = level_smaller_num_data[i]; + if (n > thread_max) { + thread_max = n; + } + } + const uint32_t warp_id = threadIdx.x / warpSize; + const uint32_t lane = threadIdx.x % warpSize; + for (int offset = warpSize / 2; offset > 0; offset >>= 1) { + const data_size_t other = __shfl_down_sync(0xffffffffu, thread_max, offset); + if (other > thread_max) { + thread_max = other; + } + } + if (lane == 0) { + shared_max[warp_id] = thread_max; + } + __syncthreads(); + if (threadIdx.x == 0) { + data_size_t max_num_data_in_smaller_leaf = 0; + const uint32_t num_warps = (blockDim.x + warpSize - 1) / warpSize; + for (uint32_t w = 0; w < num_warps; ++w) { + if (shared_max[w] > max_num_data_in_smaller_leaf) { + max_num_data_in_smaller_leaf = shared_max[w]; + } + } + out_dim_y[0] = HybridBatchedConstructGridDimY( + max_num_data_in_smaller_leaf, num_pairs, block_dim_y, + min_grid_dim_y, min_rows_per_thread, saturation_floor_total) * block_dim_y; + } +} + +void CUDAHistogramConstructor::LaunchComputeBatchedConstructDimYKernel( + const data_size_t* level_smaller_num_data, + const int num_pairs, + const int block_dim_y) { + ComputeBatchedConstructDimYKernel<<<1, 128, 0, cuda_stream_>>>( + level_smaller_num_data, num_pairs, block_dim_y, min_grid_dim_y_, + BatchConstructMinRowsPerThread(), BatchConstructSaturationFloor(), + cuda_hybrid_construct_dim_y_.RawData()); +} + +// Small-leaf direct body (hybrid growth, non-quantized only): adds each row's +// gradient/hessian pair straight to the leaf's global histogram with plain +// device-scope atomicAdd, skipping the shared-memory accumulation entirely. +// The shared-memory body pays a fixed per-block cost (zero + merge of up to +// 2 * num_bins_per_partition shared entries) that dwarfs the row work when a +// pair's leaves are tiny; at <= SmallLeafRowThreshold() rows per leaf global +// atomic contention is negligible. Rows are covered by a grid-stride loop, so +// any launch grid works: blocks whose row range lies beyond the leaf exit +// after one comparison (no shared zero, no merge), which is what makes the +// over-provisioned batched grids (sized for the level's LARGEST pair) cheap +// for the tiny pairs of the same level. +template +__device__ __forceinline__ void ConstructHistogramDenseDirectInner( + const CUDALeafSplitsStruct* smaller_leaf_splits, + const score_t* cuda_gradients, + const score_t* cuda_hessians, + const float2* cuda_gh, + const BIN_TYPE* data, + const uint32_t* column_hist_offsets, + const uint32_t* column_hist_offsets_full, + const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, + const int8_t* is_feature_used_bytree, + const data_size_t num_data) { + const int partition_column_start = feature_partition_column_index_offsets[blockIdx.x]; + const int partition_column_end = feature_partition_column_index_offsets[blockIdx.x + 1]; + const int num_columns_in_partition = partition_column_end - partition_column_start; + const int column_index = static_cast(threadIdx.x) + partition_column_start; + const bool feat_used = (threadIdx.x < static_cast(num_columns_in_partition)) && + (is_feature_used_bytree == nullptr || is_feature_used_bytree[column_index]); + if (!feat_used) { + return; + } + const data_size_t num_data_in_smaller_leaf = smaller_leaf_splits->num_data_in_leaf; + const int data_row_stride = IS_4BIT ? + packed_partition_byte_offsets[blockIdx.x + 1] - packed_partition_byte_offsets[blockIdx.x] : + num_columns_in_partition; + const BIN_TYPE* data_ptr = data + static_cast( + IS_4BIT ? packed_partition_byte_offsets[blockIdx.x] : partition_column_start) * num_data; + const data_size_t* data_indices_ref = smaller_leaf_splits->data_indices_in_leaf; + // column_hist_offsets is PARTITION-RELATIVE (it indexes the per-partition + // shared histogram in the shared-memory body); the global histogram position + // additionally needs the partition's own start offset + const uint32_t partition_hist_start = column_hist_offsets_full[blockIdx.x]; + hist_t* hist_ptr = smaller_leaf_splits->hist_in_leaf + + ((partition_hist_start + column_hist_offsets[column_index]) << 1); + const data_size_t row_stride = static_cast(gridDim.y) * static_cast(blockDim.y); + for (data_size_t row = static_cast(blockIdx.y) * blockDim.y + threadIdx.y; + row < num_data_in_smaller_leaf; row += row_stride) { + const data_size_t data_index = data_indices_ref[row]; + score_t grad, hess; + if (USE_GH2) { + const float2 gh = cuda_gh[data_index]; + grad = gh.x; + hess = gh.y; + } else { + grad = cuda_gradients[data_index]; + hess = cuda_hessians[data_index]; + } + const uint32_t pos = ReadDenseBin( + data_ptr + static_cast(data_index) * data_row_stride, threadIdx.x) << 1; + atomicAdd(hist_ptr + pos, static_cast(grad)); + atomicAdd(hist_ptr + pos + 1, static_cast(hess)); + } +} + +// Batched per-level variant (hybrid growth): one launch covers all sibling pairs +// of a level; blockIdx.z selects the pair. The x/y grid is sized for the pair with +// the most data; blocks beyond a pair's own data exit early inside the helper. +// The construct gating (host-mirrored min_data/min_hessian early return) is +// evaluated on-device from the pair structs, so the speculative single-sync flow +// can enqueue the level before the child statistics are read back; in the classic +// flow desc->construct_valid carries the identical host decision and the device +// check is a no-op. When level_dim_y is non-null (speculative flow), the launch +// grid is only an upper bound and the row-grouping extent comes from the scalar +// precomputed by ComputeBatchedConstructDimYKernel (bit-identical to the classic +// host sizing). +template +__global__ void CUDAConstructHistogramDenseBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const score_t* cuda_gradients, + const score_t* cuda_hessians, + const float2* cuda_gh, + const BIN_TYPE* data, + const uint32_t* column_hist_offsets, + const uint32_t* column_hist_offsets_full, + const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, + const int8_t* is_feature_used_bytree, + const data_size_t num_data, + const data_size_t min_data_in_leaf, + const double min_sum_hessian_in_leaf, + const int* level_dim_y, + const data_size_t* level_smaller_num_data, + const int min_grid_dim_y, + const int min_rows_per_thread, + const int saturation_floor_total, + const data_size_t small_leaf_threshold, + const uint8_t* bin_used) { + __shared__ HIST_TYPE shared_hist[SHARED_HIST_SIZE]; + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.z; + if (!desc->construct_valid) { + return; + } + // device mirror of ConstructHistogramForLeaf's min_data/min_hessian early + // return (block-uniform, so the early return is divergence-free) + const CUDALeafSplitsStruct* smaller_struct = desc->smaller_struct; + const CUDALeafSplitsStruct* larger_struct = desc->larger_struct; + const data_size_t num_data_smaller = smaller_struct->num_data_in_leaf; + const double sum_hessians_smaller = smaller_struct->sum_of_hessians; + const bool has_larger = larger_struct->leaf_index >= 0; + const data_size_t num_data_larger = has_larger ? larger_struct->num_data_in_leaf : 0; + const double sum_hessians_larger = has_larger ? larger_struct->sum_of_hessians : 0.0; + if ((num_data_smaller <= min_data_in_leaf || sum_hessians_smaller <= min_sum_hessian_in_leaf) && + (num_data_larger <= min_data_in_leaf || sum_hessians_larger <= min_sum_hessian_in_leaf)) { + return; + } + // small-leaf pairs (decided on-device from the ACTUAL smaller-leaf size, so + // the speculative flow's host-side upper bounds never mask a tiny pair) skip + // the shared-memory accumulation entirely; see ConstructHistogramDenseDirectInner + if (num_data_smaller <= small_leaf_threshold) { + ConstructHistogramDenseDirectInner( + smaller_struct, cuda_gradients, cuda_hessians, cuda_gh, data, + column_hist_offsets, column_hist_offsets_full, + feature_partition_column_index_offsets, + packed_partition_byte_offsets, + is_feature_used_bytree, num_data); + return; + } + // effective row-grouping extent: the launch grid in the classic flow; the + // precomputed scalar (ComputeBatchedConstructDimYKernel) for many-pair + // speculative levels; or -- for few-pair speculative levels -- the identical + // formula evaluated right here from the level's actual smaller-child sizes, + // saving that kernel launch (block-uniform, <= 32 loads) + int dim_y; + if (level_dim_y != nullptr) { + dim_y = level_dim_y[0]; + } else if (level_smaller_num_data == nullptr) { + dim_y = static_cast(gridDim.y * blockDim.y); + } else { + data_size_t max_num_data = 0; + const int num_pairs = static_cast(gridDim.z); + for (int i = 0; i < num_pairs; ++i) { + const data_size_t n = level_smaller_num_data[i]; + if (n > max_num_data) { + max_num_data = n; + } + } + dim_y = HybridBatchedConstructGridDimY( + max_num_data, num_pairs, static_cast(blockDim.y), + min_grid_dim_y, min_rows_per_thread, saturation_floor_total) * static_cast(blockDim.y); + } + if (USE_REG_BINS) { + ConstructHistogramDenseInner( + smaller_struct, shared_hist, cuda_gradients, cuda_hessians, cuda_gh, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + is_feature_used_bytree, num_data, dim_y, bin_used); + } else { + ConstructHistogramDenseInner( + smaller_struct, shared_hist, cuda_gradients, cuda_hessians, cuda_gh, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + is_feature_used_bytree, num_data, dim_y, bin_used); } } @@ -528,37 +996,56 @@ __global__ void CUDAConstructHistogramSparseKernel_GlobalMemory( } } -template -__global__ void CUDAConstructDiscretizedHistogramDenseKernel( +// Shared body of the discretized dense histogram kernel (see +// ConstructHistogramDenseInner for the shared-memory-passing and early-exit +// rationale). +template +__device__ __forceinline__ void ConstructDiscretizedHistogramDenseInner( const CUDALeafSplitsStruct* smaller_leaf_splits, + int16_t* shared_hist, const int32_t* cuda_gradients_and_hessians, const BIN_TYPE* data, const uint32_t* column_hist_offsets, const uint32_t* column_hist_offsets_full, const int* feature_partition_column_index_offsets, - const data_size_t num_data) { + const int* packed_partition_byte_offsets, + const data_size_t num_data, + const int8_t* is_feature_used_bytree = nullptr, + const uint8_t* bin_used = nullptr) { const int dim_y = static_cast(gridDim.y * blockDim.y); const data_size_t num_data_in_smaller_leaf = smaller_leaf_splits->num_data_in_leaf; const data_size_t num_data_per_thread = (num_data_in_smaller_leaf + dim_y - 1) / dim_y; + const unsigned int blockIdx_y = blockIdx.y; + const data_size_t block_start = (static_cast(blockIdx_y) * blockDim.y) * num_data_per_thread; + if (block_start >= num_data_in_smaller_leaf) { + return; + } const data_size_t* data_indices_ref = smaller_leaf_splits->data_indices_in_leaf; - __shared__ int16_t shared_hist[SHARED_HIST_SIZE]; int32_t* shared_hist_packed = reinterpret_cast(shared_hist); const unsigned int num_threads_per_block = blockDim.x * blockDim.y; const int partition_column_start = feature_partition_column_index_offsets[blockIdx.x]; const int partition_column_end = feature_partition_column_index_offsets[blockIdx.x + 1]; - const BIN_TYPE* data_ptr = data + static_cast(partition_column_start) * num_data; const int num_columns_in_partition = partition_column_end - partition_column_start; + const int row_stride = IS_4BIT ? + packed_partition_byte_offsets[blockIdx.x + 1] - packed_partition_byte_offsets[blockIdx.x] : + num_columns_in_partition; + const BIN_TYPE* data_ptr = data + static_cast( + IS_4BIT ? packed_partition_byte_offsets[blockIdx.x] : partition_column_start) * num_data; const uint32_t partition_hist_start = column_hist_offsets_full[blockIdx.x]; const uint32_t partition_hist_end = column_hist_offsets_full[blockIdx.x + 1]; const uint32_t num_items_in_partition = (partition_hist_end - partition_hist_start); const unsigned int thread_idx = threadIdx.x + threadIdx.y * blockDim.x; + // bin_used / is_feature_used_bytree (per-tree feature_fraction masks, null in + // the per-pair path and without sampling): unused features' bins are dead + // storage this tree, so their zero/accumulate/merge work is skipped. Used + // bins see the identical arithmetic (integer atomics are order-invariant). for (unsigned int i = thread_idx; i < num_items_in_partition; i += num_threads_per_block) { - shared_hist_packed[i] = 0; + if (bin_used == nullptr || bin_used[partition_hist_start + i]) { + shared_hist_packed[i] = 0; + } } __syncthreads(); const unsigned int threadIdx_y = threadIdx.y; - const unsigned int blockIdx_y = blockIdx.y; - const data_size_t block_start = (blockIdx_y * blockDim.y) * num_data_per_thread; const data_size_t* data_indices_ref_this_block = data_indices_ref + block_start; data_size_t block_num_data = max(0, min(num_data_in_smaller_leaf - block_start, num_data_per_thread * static_cast(blockDim.y))); const data_size_t num_iteration_total = (block_num_data + blockDim.y - 1) / blockDim.y; @@ -566,12 +1053,13 @@ __global__ void CUDAConstructDiscretizedHistogramDenseKernel( const data_size_t num_iteration_this = remainder == 0 ? num_iteration_total : num_iteration_total - static_cast(threadIdx_y >= remainder); data_size_t inner_data_index = static_cast(threadIdx_y); const int column_index = static_cast(threadIdx.x) + partition_column_start; - if (threadIdx.x < static_cast(num_columns_in_partition)) { + if (threadIdx.x < static_cast(num_columns_in_partition) && + (is_feature_used_bytree == nullptr || is_feature_used_bytree[column_index])) { int32_t* shared_hist_ptr = shared_hist_packed + (column_hist_offsets[column_index]); for (data_size_t i = 0; i < num_iteration_this; ++i) { const data_size_t data_index = data_indices_ref_this_block[inner_data_index]; const int32_t grad_and_hess = cuda_gradients_and_hessians[data_index]; - const uint32_t bin = static_cast(data_ptr[static_cast(data_index) * num_columns_in_partition + threadIdx.x]); + const uint32_t bin = ReadDenseBin(data_ptr + static_cast(data_index) * row_stride, threadIdx.x); int32_t* pos_ptr = shared_hist_ptr + bin; atomicAdd_block(pos_ptr, grad_and_hess); inner_data_index += blockDim.y; @@ -581,12 +1069,18 @@ __global__ void CUDAConstructDiscretizedHistogramDenseKernel( if (USE_16BIT_HIST) { int32_t* feature_histogram_ptr = reinterpret_cast(smaller_leaf_splits->hist_in_leaf) + partition_hist_start; for (unsigned int i = thread_idx; i < num_items_in_partition; i += num_threads_per_block) { + if (bin_used != nullptr && !bin_used[partition_hist_start + i]) { + continue; + } const int32_t packed_grad_hess = shared_hist_packed[i]; atomicAdd_system(feature_histogram_ptr + i, packed_grad_hess); } } else { atomic_add_long_t* feature_histogram_ptr = reinterpret_cast(smaller_leaf_splits->hist_in_leaf) + partition_hist_start; for (unsigned int i = thread_idx; i < num_items_in_partition; i += num_threads_per_block) { + if (bin_used != nullptr && !bin_used[partition_hist_start + i]) { + continue; + } const int32_t packed_grad_hess = shared_hist_packed[i]; const int64_t packed_grad_hess_int64 = (static_cast(static_cast(packed_grad_hess >> 16)) << 32) | (static_cast(packed_grad_hess & 0x0000ffff)); atomicAdd_system(feature_histogram_ptr + i, (atomic_add_long_t)(packed_grad_hess_int64)); @@ -594,6 +1088,59 @@ __global__ void CUDAConstructDiscretizedHistogramDenseKernel( } } +template +__global__ void CUDAConstructDiscretizedHistogramDenseKernel( + const CUDALeafSplitsStruct* smaller_leaf_splits, + const int32_t* cuda_gradients_and_hessians, + const BIN_TYPE* data, + const uint32_t* column_hist_offsets, + const uint32_t* column_hist_offsets_full, + const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, + const data_size_t num_data) { + __shared__ int16_t shared_hist[SHARED_HIST_SIZE]; + ConstructDiscretizedHistogramDenseInner( + smaller_leaf_splits, shared_hist, cuda_gradients_and_hessians, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + num_data); +} + +// Batched per-level variant (hybrid growth): blockIdx.z selects the pair. The +// per-pair histogram bit width is a runtime (block-uniform) branch so pairs with +// 16-bit and 32-bit histograms share a single launch. +template +__global__ void CUDAConstructDiscretizedHistogramDenseBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int32_t* cuda_gradients_and_hessians, + const BIN_TYPE* data, + const uint32_t* column_hist_offsets, + const uint32_t* column_hist_offsets_full, + const int* feature_partition_column_index_offsets, + const int* packed_partition_byte_offsets, + const data_size_t num_data, + const int8_t* is_feature_used_bytree, + const uint8_t* bin_used) { + __shared__ int16_t shared_hist[SHARED_HIST_SIZE]; + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.z; + if (!desc->construct_valid) { + return; + } + if (desc->smaller_num_bits <= 16) { + ConstructDiscretizedHistogramDenseInner( + desc->smaller_struct, shared_hist, cuda_gradients_and_hessians, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + num_data, is_feature_used_bytree, bin_used); + } else { + ConstructDiscretizedHistogramDenseInner( + desc->smaller_struct, shared_hist, cuda_gradients_and_hessians, data, + column_hist_offsets, column_hist_offsets_full, feature_partition_column_index_offsets, + packed_partition_byte_offsets, + num_data, is_feature_used_bytree, bin_used); + } +} + template __global__ void CUDAConstructDiscretizedHistogramSparseKernel( const CUDALeafSplitsStruct* smaller_leaf_splits, @@ -869,7 +1416,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( if (USE_GLOBAL_MEM_BUFFER) { if (cuda_row_data_->is_sparse()) { if (num_bits_in_histogram_bins <= 16) { - CUDAConstructDiscretizedHistogramSparseKernel_GlobalMemory<<>>( + CUDAConstructDiscretizedHistogramSparseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -879,7 +1426,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( num_data_, reinterpret_cast(cuda_hist_buffer_.RawData())); } else { - CUDAConstructDiscretizedHistogramSparseKernel_GlobalMemory<<>>( + CUDAConstructDiscretizedHistogramSparseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -891,7 +1438,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( } } else { if (num_bits_in_histogram_bins <= 16) { - CUDAConstructDiscretizedHistogramDenseKernel_GlobalMemory<<>>( + CUDAConstructDiscretizedHistogramDenseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -901,7 +1448,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( num_data_, reinterpret_cast(cuda_hist_buffer_.RawData())); } else { - CUDAConstructDiscretizedHistogramDenseKernel_GlobalMemory<<>>( + CUDAConstructDiscretizedHistogramDenseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -915,7 +1462,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( } else { if (cuda_row_data_->is_sparse()) { if (num_bits_in_histogram_bins <= 16) { - CUDAConstructDiscretizedHistogramSparseKernel<<>>( + CUDAConstructDiscretizedHistogramSparseKernel<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -924,7 +1471,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( cuda_row_data_->cuda_partition_hist_offsets(), num_data_); } else { - CUDAConstructDiscretizedHistogramSparseKernel<<>>( + CUDAConstructDiscretizedHistogramSparseKernel<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), @@ -934,23 +1481,47 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( num_data_); } } else { - if (num_bits_in_histogram_bins <= 16) { - CUDAConstructDiscretizedHistogramDenseKernel<<>>( + if (cuda_row_data_->is_4bit_packed()) { + if (num_bits_in_histogram_bins <= 16) { + CUDAConstructDiscretizedHistogramDenseKernel<<>>( + cuda_smaller_leaf_splits, + reinterpret_cast(cuda_gradients_), + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + cuda_row_data_->cuda_packed_partition_byte_offsets(), + num_data_); + } else { + CUDAConstructDiscretizedHistogramDenseKernel<<>>( + cuda_smaller_leaf_splits, + reinterpret_cast(cuda_gradients_), + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + cuda_row_data_->cuda_packed_partition_byte_offsets(), + num_data_); + } + } else if (num_bits_in_histogram_bins <= 16) { + CUDAConstructDiscretizedHistogramDenseKernel<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), cuda_row_data_->cuda_column_hist_offsets(), cuda_row_data_->cuda_partition_hist_offsets(), cuda_row_data_->cuda_feature_partition_column_index_offsets(), + nullptr, num_data_); } else { - CUDAConstructDiscretizedHistogramDenseKernel<<>>( + CUDAConstructDiscretizedHistogramDenseKernel<<>>( cuda_smaller_leaf_splits, reinterpret_cast(cuda_gradients_), cuda_row_data_->GetBin(), cuda_row_data_->cuda_column_hist_offsets(), cuda_row_data_->cuda_partition_hist_offsets(), cuda_row_data_->cuda_feature_partition_column_index_offsets(), + nullptr, num_data_); } } @@ -958,7 +1529,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( } else { if (!USE_GLOBAL_MEM_BUFFER) { if (cuda_row_data_->is_sparse()) { - CUDAConstructHistogramSparseKernel<<>>( + CUDAConstructHistogramSparseKernel<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, cuda_row_data_->GetBin(), @@ -983,7 +1554,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( // and then flips active_buffer_is_alt_.) uint8_t* active_data = compact_data_uint8_t_.RawData(); if (compact_is_col_major_) { - CUDAConstructHistogramDenseColMajorKernel<<>>( + CUDAConstructHistogramDenseColMajorKernel<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, reinterpret_cast(active_data), @@ -991,8 +1562,19 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( cuda_row_data_->cuda_partition_hist_offsets(), compact_feature_partition_column_index_offsets_.RawData(), num_data_); + } else if (compact_is_4bit_) { + CUDAConstructHistogramDenseKernel<<>>( + cuda_smaller_leaf_splits, + cuda_gradients_, cuda_hessians_, + reinterpret_cast(active_data), + compact_column_hist_offsets_.RawData(), + cuda_row_data_->cuda_partition_hist_offsets(), + compact_feature_partition_column_index_offsets_.RawData(), + compact_packed_partition_byte_offsets_.RawData(), + nullptr, + num_data_); } else { - CUDAConstructHistogramDenseKernel<<>>( + CUDAConstructHistogramDenseKernel<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, reinterpret_cast(active_data), @@ -1000,23 +1582,36 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( cuda_row_data_->cuda_partition_hist_offsets(), compact_feature_partition_column_index_offsets_.RawData(), nullptr, + nullptr, num_data_); } + } else if (cuda_row_data_->is_4bit_packed()) { + CUDAConstructHistogramDenseKernel<<>>( + cuda_smaller_leaf_splits, + cuda_gradients_, cuda_hessians_, + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + cuda_row_data_->cuda_packed_partition_byte_offsets(), + cuda_is_feature_used_bytree_.Size() > 0 ? cuda_is_feature_used_bytree_.RawData() : nullptr, + num_data_); } else { - CUDAConstructHistogramDenseKernel<<>>( + CUDAConstructHistogramDenseKernel<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, cuda_row_data_->GetBin(), cuda_row_data_->cuda_column_hist_offsets(), cuda_row_data_->cuda_partition_hist_offsets(), cuda_row_data_->cuda_feature_partition_column_index_offsets(), + nullptr, cuda_is_feature_used_bytree_.Size() > 0 ? cuda_is_feature_used_bytree_.RawData() : nullptr, num_data_); } } } else { if (cuda_row_data_->is_sparse()) { - CUDAConstructHistogramSparseKernel_GlobalMemory<<>>( + CUDAConstructHistogramSparseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, cuda_row_data_->GetBin(), @@ -1026,7 +1621,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( num_data_, reinterpret_cast(cuda_hist_buffer_.RawData())); } else { - CUDAConstructHistogramDenseKernel_GlobalMemory<<>>( + CUDAConstructHistogramDenseKernel_GlobalMemory<<>>( cuda_smaller_leaf_splits, cuda_gradients_, cuda_hessians_, cuda_row_data_->GetBin(), @@ -1040,7 +1635,7 @@ void CUDAHistogramConstructor::LaunchConstructHistogramKernelInner2( } } -__global__ void SubtractHistogramKernel( +__device__ __forceinline__ void SubtractHistogramInner( const int num_total_bin, const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, const CUDALeafSplitsStruct* cuda_larger_leaf_splits) { @@ -1055,15 +1650,50 @@ __global__ void SubtractHistogramKernel( } } -__global__ void FixHistogramKernel( +__global__ void SubtractHistogramKernel( + const int num_total_bin, + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, + const CUDALeafSplitsStruct* cuda_larger_leaf_splits) { + SubtractHistogramInner(num_total_bin, cuda_smaller_leaf_splits, cuda_larger_leaf_splits); +} + +// Batched per-level variant (hybrid growth): blockIdx.y selects the pair. +// bin_used (per-tree feature_fraction bin mask, may be null) skips histogram +// entries of features outside this tree's sample: nothing of this tree reads +// them (the find kernels are feature-masked), so they are dead storage. +__global__ void SubtractHistogramBatchedKernel( + const int num_total_bin, + const CUDAHybridPairDescriptor* pair_descs, + const uint8_t* bin_used) { + const unsigned int global_thread_index = threadIdx.x + blockIdx.x * blockDim.x; + if (bin_used != nullptr && + global_thread_index < static_cast(2 * num_total_bin) && + !bin_used[global_thread_index >> 1]) { + return; + } + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + SubtractHistogramInner(num_total_bin, desc->smaller_struct, desc->larger_struct); +} + +// When fix_feature_index (the index into the need-fix feature list) is -1 it +// comes from blockIdx.x (the standalone fix kernels); the fused small-leaf +// fix+subtract kernel passes it explicitly because its fix blocks start after +// the subtract blocks. When larger_for_subtract is non-null, thread 0 also +// applies the histogram subtraction at the fixed most-frequent-bin entries +// (larger = parent - fixed smaller, the exact arithmetic the standalone +// subtract kernel would perform after the fix kernel). +__device__ __forceinline__ void FixHistogramInner( const uint32_t* cuda_feature_num_bins, const uint32_t* cuda_feature_hist_offsets, const uint32_t* cuda_feature_most_freq_bins, const int* cuda_need_fix_histogram_features, const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, - const CUDALeafSplitsStruct* cuda_smaller_leaf_splits) { - __shared__ hist_t shared_mem_buffer[WARPSIZE]; - const unsigned int blockIdx_x = blockIdx.x; + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, + hist_t* shared_mem_buffer, + const int fix_feature_index = -1, + const CUDALeafSplitsStruct* larger_for_subtract = nullptr) { + const unsigned int blockIdx_x = fix_feature_index >= 0 ? + static_cast(fix_feature_index) : blockIdx.x; const int feature_index = cuda_need_fix_histogram_features[blockIdx_x]; const uint32_t num_bin_aligned = cuda_need_fix_histogram_features_num_bin_aligned[blockIdx_x]; const uint32_t feature_hist_offset = cuda_feature_hist_offsets[feature_index]; @@ -1079,17 +1709,111 @@ __global__ void FixHistogramKernel( const hist_t sum_gradient = ShuffleReduceSum(bin_gradient, shared_mem_buffer, num_bin_aligned); const hist_t sum_hessian = ShuffleReduceSum(bin_hessian, shared_mem_buffer, num_bin_aligned); if (threadIdx_x == 0) { - feature_hist[most_freq_bin << 1] = leaf_sum_gradients - sum_gradient; - feature_hist[(most_freq_bin << 1) + 1] = leaf_sum_hessians - sum_hessian; + const hist_t fixed_gradient = leaf_sum_gradients - sum_gradient; + const hist_t fixed_hessian = leaf_sum_hessians - sum_hessian; + feature_hist[most_freq_bin << 1] = fixed_gradient; + feature_hist[(most_freq_bin << 1) + 1] = fixed_hessian; + if (larger_for_subtract != nullptr && larger_for_subtract->leaf_index >= 0) { + hist_t* larger_feature_hist = larger_for_subtract->hist_in_leaf + feature_hist_offset * 2; + larger_feature_hist[most_freq_bin << 1] -= fixed_gradient; + larger_feature_hist[(most_freq_bin << 1) + 1] -= fixed_hessian; + } + } +} + +__global__ void FixHistogramKernel( + const uint32_t* cuda_feature_num_bins, + const uint32_t* cuda_feature_hist_offsets, + const uint32_t* cuda_feature_most_freq_bins, + const int* cuda_need_fix_histogram_features, + const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits) { + __shared__ hist_t shared_mem_buffer[WARPSIZE]; + FixHistogramInner(cuda_feature_num_bins, cuda_feature_hist_offsets, + cuda_feature_most_freq_bins, cuda_need_fix_histogram_features, + cuda_need_fix_histogram_features_num_bin_aligned, cuda_smaller_leaf_splits, + shared_mem_buffer); +} + +// Batched per-level variant (hybrid growth): blockIdx.y selects the pair. +// feature_used (per-tree feature_fraction mask, may be null) skips need-fix +// features outside this tree's sample (their bins are dead storage this tree). +__global__ void FixHistogramBatchedKernel( + const uint32_t* cuda_feature_num_bins, + const uint32_t* cuda_feature_hist_offsets, + const uint32_t* cuda_feature_most_freq_bins, + const int* cuda_need_fix_histogram_features, + const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, + const CUDAHybridPairDescriptor* pair_descs, + const int8_t* feature_used) { + if (feature_used != nullptr && + !feature_used[cuda_need_fix_histogram_features[blockIdx.x]]) { + return; + } + __shared__ hist_t shared_mem_buffer[WARPSIZE]; + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + FixHistogramInner(cuda_feature_num_bins, cuda_feature_hist_offsets, + cuda_feature_most_freq_bins, cuda_need_fix_histogram_features, + cuda_need_fix_histogram_features_num_bin_aligned, desc->smaller_struct, + shared_mem_buffer); +} + +// Fused fix + subtract of the small-leaf level path (hybrid growth, +// non-quantized only): one launch replaces the sequential FixHistogramBatched + +// SubtractHistogramBatched pair. blockIdx.y selects the pair. Blocks with +// blockIdx.x < num_subtract_blocks perform the elementwise larger -= smaller +// subtraction but SKIP the entries flagged in fix_mfb_mask (the most-frequent- +// bin gradient/hessian slots of the need-fix features); the remaining blocks +// (one per need-fix feature) run the most-frequent-bin fix of the smaller leaf +// and apply the subtraction at exactly those skipped entries from the fixed +// values. Every histogram entry is therefore written by exactly one block with +// the identical arithmetic of the sequential launches (bit-identical result); +// the subtraction reads no entry the fix writes and vice versa, so no +// cross-block ordering is needed. +__global__ void FixSubtractHistogramSmallLeafBatchedKernel( + const int num_total_bin, + const int num_subtract_blocks, + const uint32_t* cuda_feature_num_bins, + const uint32_t* cuda_feature_hist_offsets, + const uint32_t* cuda_feature_most_freq_bins, + const int* cuda_need_fix_histogram_features, + const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, + const uint8_t* fix_mfb_mask, + const CUDAHybridPairDescriptor* pair_descs, + const uint8_t* bin_used, + const int8_t* feature_used) { + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + if (static_cast(blockIdx.x) < num_subtract_blocks) { + const CUDALeafSplitsStruct* larger_leaf = desc->larger_struct; + if (larger_leaf->leaf_index >= 0) { + const unsigned int i = blockIdx.x * blockDim.x + threadIdx.x; + // bins of features outside this tree's sample are dead storage: skip + if (i < static_cast(2 * num_total_bin) && !fix_mfb_mask[i] && + (bin_used == nullptr || bin_used[i >> 1])) { + larger_leaf->hist_in_leaf[i] -= desc->smaller_struct->hist_in_leaf[i]; + } + } + } else { + if (feature_used != nullptr && + !feature_used[cuda_need_fix_histogram_features[ + static_cast(blockIdx.x) - num_subtract_blocks]]) { + return; // most-frequent bin of an unused feature: dead storage this tree + } + __shared__ hist_t shared_mem_buffer[WARPSIZE]; + FixHistogramInner(cuda_feature_num_bins, cuda_feature_hist_offsets, + cuda_feature_most_freq_bins, cuda_need_fix_histogram_features, + cuda_need_fix_histogram_features_num_bin_aligned, desc->smaller_struct, + shared_mem_buffer, static_cast(blockIdx.x) - num_subtract_blocks, + desc->larger_struct); } } template -__global__ void SubtractHistogramDiscretizedKernel( +__device__ __forceinline__ void SubtractHistogramDiscretizedInner( const int num_total_bin, const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, const CUDALeafSplitsStruct* cuda_larger_leaf_splits, - hist_t* num_bit_change_buffer) { + int32_t* num_bit_change_buffer) { const unsigned int global_thread_index = threadIdx.x + blockIdx.x * blockDim.x; const int cuda_larger_leaf_index_ref = cuda_larger_leaf_splits->leaf_index; if (cuda_larger_leaf_index_ref >= 0) { @@ -1100,7 +1824,7 @@ __global__ void SubtractHistogramDiscretizedKernel( larger_leaf_hist[global_thread_index] -= smaller_leaf_hist[global_thread_index]; } } else if (LARGER_USE_16BIT_HIST) { - int32_t* buffer = reinterpret_cast(num_bit_change_buffer); + int32_t* buffer = num_bit_change_buffer; const int32_t* smaller_leaf_hist = reinterpret_cast(cuda_smaller_leaf_splits->hist_in_leaf); int64_t* larger_leaf_hist = reinterpret_cast(cuda_larger_leaf_splits->hist_in_leaf); if (global_thread_index < num_total_bin) { @@ -1133,6 +1857,52 @@ __global__ void SubtractHistogramDiscretizedKernel( } } +template +__global__ void SubtractHistogramDiscretizedKernel( + const int num_total_bin, + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, + const CUDALeafSplitsStruct* cuda_larger_leaf_splits, + hist_t* num_bit_change_buffer) { + SubtractHistogramDiscretizedInner( + num_total_bin, cuda_smaller_leaf_splits, cuda_larger_leaf_splits, + reinterpret_cast(num_bit_change_buffer)); +} + +// Batched per-level variant (hybrid growth): blockIdx.y selects the pair. The +// per-pair histogram bit widths choose the arithmetic at runtime (block-uniform +// branches), and each pair that needs the 64->32-bit compaction writes its own +// region of the change buffer (stride num_total_bin int32 entries per pair). +__global__ void SubtractHistogramDiscretizedBatchedKernel( + const int num_total_bin, + const CUDAHybridPairDescriptor* pair_descs, + hist_t* num_bit_change_buffer, + const uint8_t* bin_used) { + // bins of features outside this tree's sample are dead storage: skip (the + // change-buffer copy kernel skips the same bins, so no stale data is read) + const unsigned int global_thread_index_gate = threadIdx.x + blockIdx.x * blockDim.x; + if (bin_used != nullptr && + global_thread_index_gate < static_cast(num_total_bin) && + !bin_used[global_thread_index_gate]) { + return; + } + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + int32_t* buffer = reinterpret_cast(num_bit_change_buffer) + + static_cast(blockIdx.y) * static_cast(num_total_bin); + if (desc->parent_num_bits <= 16) { + SubtractHistogramDiscretizedInner( + num_total_bin, desc->smaller_struct, desc->larger_struct, buffer); + } else if (desc->larger_num_bits <= 16) { + SubtractHistogramDiscretizedInner( + num_total_bin, desc->smaller_struct, desc->larger_struct, buffer); + } else if (desc->smaller_num_bits <= 16) { + SubtractHistogramDiscretizedInner( + num_total_bin, desc->smaller_struct, desc->larger_struct, buffer); + } else { + SubtractHistogramDiscretizedInner( + num_total_bin, desc->smaller_struct, desc->larger_struct, buffer); + } +} + __global__ void CopyChangedNumBitHistogram( const int num_total_bin, const CUDALeafSplitsStruct* cuda_larger_leaf_splits, @@ -1145,15 +1915,38 @@ __global__ void CopyChangedNumBitHistogram( } } +// Batched per-level variant: copies each mixed-bit-width pair's change-buffer +// region back into the larger leaf's histogram. Pairs whose bit widths do not +// need the compaction (or whose larger leaf does not exist) exit immediately. +__global__ void CopyChangedNumBitHistogramBatchedKernel( + const int num_total_bin, + const CUDAHybridPairDescriptor* pair_descs, + hist_t* num_bit_change_buffer, + const uint8_t* bin_used) { + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + if (desc->larger_leaf_index < 0 || + !(desc->parent_num_bits > 16 && desc->larger_num_bits <= 16)) { + return; + } + int32_t* hist_dst = reinterpret_cast(desc->larger_struct->hist_in_leaf); + const int32_t* hist_src = reinterpret_cast(num_bit_change_buffer) + + static_cast(blockIdx.y) * static_cast(num_total_bin); + const unsigned int global_thread_index = threadIdx.x + blockIdx.x * blockDim.x; + if (global_thread_index < static_cast(num_total_bin) && + (bin_used == nullptr || bin_used[global_thread_index])) { + hist_dst[global_thread_index] = hist_src[global_thread_index]; + } +} + template -__global__ void FixHistogramDiscretizedKernel( +__device__ __forceinline__ void FixHistogramDiscretizedInner( const uint32_t* cuda_feature_num_bins, const uint32_t* cuda_feature_hist_offsets, const uint32_t* cuda_feature_most_freq_bins, const int* cuda_need_fix_histogram_features, const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, - const CUDALeafSplitsStruct* cuda_smaller_leaf_splits) { - __shared__ int64_t shared_mem_buffer[WARPSIZE]; + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, + int64_t* shared_mem_buffer) { const unsigned int blockIdx_x = blockIdx.x; const int feature_index = cuda_need_fix_histogram_features[blockIdx_x]; const uint32_t num_bin_aligned = cuda_need_fix_histogram_features_num_bin_aligned[blockIdx_x]; @@ -1187,6 +1980,50 @@ __global__ void FixHistogramDiscretizedKernel( } } +template +__global__ void FixHistogramDiscretizedKernel( + const uint32_t* cuda_feature_num_bins, + const uint32_t* cuda_feature_hist_offsets, + const uint32_t* cuda_feature_most_freq_bins, + const int* cuda_need_fix_histogram_features, + const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, + const CUDALeafSplitsStruct* cuda_smaller_leaf_splits) { + __shared__ int64_t shared_mem_buffer[WARPSIZE]; + FixHistogramDiscretizedInner( + cuda_feature_num_bins, cuda_feature_hist_offsets, cuda_feature_most_freq_bins, + cuda_need_fix_histogram_features, cuda_need_fix_histogram_features_num_bin_aligned, + cuda_smaller_leaf_splits, shared_mem_buffer); +} + +// Batched per-level variant (hybrid growth): blockIdx.y selects the pair; the +// per-pair histogram bit width is a runtime (block-uniform) branch. +__global__ void FixHistogramDiscretizedBatchedKernel( + const uint32_t* cuda_feature_num_bins, + const uint32_t* cuda_feature_hist_offsets, + const uint32_t* cuda_feature_most_freq_bins, + const int* cuda_need_fix_histogram_features, + const uint32_t* cuda_need_fix_histogram_features_num_bin_aligned, + const CUDAHybridPairDescriptor* pair_descs, + const int8_t* feature_used) { + if (feature_used != nullptr && + !feature_used[cuda_need_fix_histogram_features[blockIdx.x]]) { + return; // most-frequent bin of an unused feature: dead storage this tree + } + __shared__ int64_t shared_mem_buffer[WARPSIZE]; + const CUDAHybridPairDescriptor* desc = pair_descs + blockIdx.y; + if (desc->smaller_num_bits <= 16) { + FixHistogramDiscretizedInner( + cuda_feature_num_bins, cuda_feature_hist_offsets, cuda_feature_most_freq_bins, + cuda_need_fix_histogram_features, cuda_need_fix_histogram_features_num_bin_aligned, + desc->smaller_struct, shared_mem_buffer); + } else { + FixHistogramDiscretizedInner( + cuda_feature_num_bins, cuda_feature_hist_offsets, cuda_feature_most_freq_bins, + cuda_need_fix_histogram_features, cuda_need_fix_histogram_features_num_bin_aligned, + desc->smaller_struct, shared_mem_buffer); + } +} + void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, const CUDALeafSplitsStruct* cuda_larger_leaf_splits, @@ -1199,7 +2036,7 @@ void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( const int num_subtract_blocks = (num_subtract_threads + SUBTRACT_BLOCK_SIZE - 1) / SUBTRACT_BLOCK_SIZE; global_timer.Start("CUDAHistogramConstructor::FixHistogramKernel"); if (need_fix_histogram_features_.size() > 0) { - FixHistogramKernel<<>>( + FixHistogramKernel<<>>( cuda_feature_num_bins_.RawData(), cuda_feature_hist_offsets_.RawData(), cuda_feature_most_freq_bins_.RawData(), @@ -1209,7 +2046,7 @@ void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( } global_timer.Stop("CUDAHistogramConstructor::FixHistogramKernel"); global_timer.Start("CUDAHistogramConstructor::SubtractHistogramKernel"); - SubtractHistogramKernel<<>>( + SubtractHistogramKernel<<>>( num_total_bin_, cuda_smaller_leaf_splits, cuda_larger_leaf_splits); @@ -1220,7 +2057,7 @@ void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( global_timer.Start("CUDAHistogramConstructor::FixHistogramDiscretizedKernel"); if (need_fix_histogram_features_.size() > 0) { if (smaller_num_bits_in_histogram_bins <= 16) { - FixHistogramDiscretizedKernel<<>>( + FixHistogramDiscretizedKernel<<>>( cuda_feature_num_bins_.RawData(), cuda_feature_hist_offsets_.RawData(), cuda_feature_most_freq_bins_.RawData(), @@ -1228,7 +2065,7 @@ void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( cuda_need_fix_histogram_features_num_bin_aligned_.RawData(), cuda_smaller_leaf_splits); } else { - FixHistogramDiscretizedKernel<<>>( + FixHistogramDiscretizedKernel<<>>( cuda_feature_num_bins_.RawData(), cuda_feature_hist_offsets_.RawData(), cuda_feature_most_freq_bins_.RawData(), @@ -1239,42 +2076,382 @@ void CUDAHistogramConstructor::LaunchSubtractHistogramKernel( } global_timer.Stop("CUDAHistogramConstructor::FixHistogramDiscretizedKernel"); global_timer.Start("CUDAHistogramConstructor::SubtractHistogramDiscretizedKernel"); + // Per-pipeline region of the bit-change buffer: sibling pairs of one level run + // on different pipeline streams concurrently, so the 64->32-bit compaction of + // two pairs must not share scratch space (the shared buffer was a data race + // whenever two concurrent pairs both had a >16-bit parent with a <=16-bit + // larger child). Regions are num_total_bin_ int32 entries per pipeline; the + // buffer is only ever accessed as int32, so the 4-byte alignment is fine. + hist_t* change_buffer = reinterpret_cast( + reinterpret_cast(hist_buffer_for_num_bit_change_.RawData()) + + static_cast(active_pipeline_) * static_cast(num_total_bin_)); if (parent_num_bits_in_histogram_bins <= 16) { CHECK_LE(smaller_num_bits_in_histogram_bins, 16); CHECK_LE(larger_num_bits_in_histogram_bins, 16); - SubtractHistogramDiscretizedKernel<<>>( + SubtractHistogramDiscretizedKernel<<>>( num_total_bin_, cuda_smaller_leaf_splits, cuda_larger_leaf_splits, - hist_buffer_for_num_bit_change_.RawData()); + change_buffer); } else if (larger_num_bits_in_histogram_bins <= 16) { CHECK_LE(smaller_num_bits_in_histogram_bins, 16); - SubtractHistogramDiscretizedKernel<<>>( + SubtractHistogramDiscretizedKernel<<>>( num_total_bin_, cuda_smaller_leaf_splits, cuda_larger_leaf_splits, - hist_buffer_for_num_bit_change_.RawData()); - CopyChangedNumBitHistogram<<>>( + change_buffer); + CopyChangedNumBitHistogram<<>>( num_total_bin_, cuda_larger_leaf_splits, - hist_buffer_for_num_bit_change_.RawData()); + change_buffer); } else if (smaller_num_bits_in_histogram_bins <= 16) { - SubtractHistogramDiscretizedKernel<<>>( + SubtractHistogramDiscretizedKernel<<>>( num_total_bin_, cuda_smaller_leaf_splits, cuda_larger_leaf_splits, - hist_buffer_for_num_bit_change_.RawData()); + change_buffer); } else { - SubtractHistogramDiscretizedKernel<<>>( + SubtractHistogramDiscretizedKernel<<>>( num_total_bin_, cuda_smaller_leaf_splits, cuda_larger_leaf_splits, - hist_buffer_for_num_bit_change_.RawData()); + change_buffer); } global_timer.Stop("CUDAHistogramConstructor::SubtractHistogramDiscretizedKernel"); } } +// ---- batched per-level launchers (hybrid growth) ---------------------------------- +// All launches go to pipeline_streams_[0] (cuda_stream_) so construct -> fix -> +// subtract are ordered by the stream; the caller records subtract_done_events_[0] +// afterwards for the best split finder to wait on. + +void CUDAHistogramConstructor::LaunchConstructHistogramBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data) { + if (cuda_row_data_->shared_hist_size() == DP_SHARED_HIST_SIZE && gpu_use_dp_) { + LaunchConstructHistogramBatchedKernelInner(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } else if (cuda_row_data_->shared_hist_size() == SP_SHARED_HIST_SIZE && !gpu_use_dp_) { + LaunchConstructHistogramBatchedKernelInner(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } else { + Log::Fatal("Unknown shared histogram size %d", cuda_row_data_->shared_hist_size()); + } +} + +template +void CUDAHistogramConstructor::LaunchConstructHistogramBatchedKernelInner( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data) { + if (cuda_row_data_->bit_type() == 8) { + if (gh_interleave_valid_) { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } else { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } + } else if (cuda_row_data_->bit_type() == 16) { + if (gh_interleave_valid_) { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } else { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } + } else if (cuda_row_data_->bit_type() == 32) { + if (gh_interleave_valid_) { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } else { + LaunchConstructHistogramBatchedKernelInner0(pair_descs, num_pairs, max_num_data_in_smaller_leaf, level_smaller_num_data); + } + } else { + Log::Fatal("Unknown bit_type = %d", cuda_row_data_->bit_type()); + } +} + +template +void CUDAHistogramConstructor::LaunchConstructHistogramBatchedKernelInner0( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data) { + // dense shared-memory path only (SupportsBatchedLevel() gates the rest) + int grid_dim_x = 0; + int grid_dim_y = 0; + int block_dim_x = 0; + int block_dim_y = 0; + CalcConstructHistogramBatchedKernelDim(&grid_dim_x, &grid_dim_y, &block_dim_x, &block_dim_y, max_num_data_in_smaller_leaf, num_pairs); + if (use_compact_view_) { + // compact column view: same batched kernel, fed with the per-tree compact + // data/metadata (mirrors the per-pair compact launch). Blocks span the + // USED columns of a partition; the y sizing formula is the batched one. + block_dim_x = std::max(1, max_num_compact_cols_per_partition_); + block_dim_y = std::max(1, NUM_THREADS_PER_BLOCK / block_dim_x); + grid_dim_y = HybridBatchedConstructGridDimY( + max_num_data_in_smaller_leaf, num_pairs, block_dim_y, min_grid_dim_y_, + BatchConstructMinRowsPerThread(), BatchConstructSaturationFloor()); + } + dim3 grid_dim(grid_dim_x, grid_dim_y, num_pairs); + dim3 block_dim(block_dim_x, block_dim_y); + const int* level_dim_y = nullptr; + const data_size_t* level_sizes_for_kernel = nullptr; + if (level_smaller_num_data != nullptr) { + // speculative flow: the grid above was sized from an upper BOUND; the exact + // row-grouping extent comes from the level's actual sizes. Few-pair levels + // evaluate the formula inside the construct kernel itself (saves a launch + // on the per-level critical path); many-pair levels keep the single-block + // reduction kernel so construct blocks read one precomputed scalar. + if (num_pairs <= 32) { + level_sizes_for_kernel = level_smaller_num_data; + } else { + LaunchComputeBatchedConstructDimYKernel(level_smaller_num_data, num_pairs, block_dim_y); + level_dim_y = cuda_hybrid_construct_dim_y_.RawDataReadOnly(); + } + } + if (use_quantized_grad_) { + // quantized training always uses the classic (two-sync) flow, so the exact + // level sizes are host-known and no device grouping override is needed + if (cuda_row_data_->is_4bit_packed()) { + CUDAConstructDiscretizedHistogramDenseBatchedKernel<<>>( + pair_descs, + reinterpret_cast(cuda_gradients_), + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + cuda_row_data_->cuda_packed_partition_byte_offsets(), + num_data_, + any_feature_unused_bytree_ ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } else { + CUDAConstructDiscretizedHistogramDenseBatchedKernel<<>>( + pair_descs, + reinterpret_cast(cuda_gradients_), + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + nullptr, + num_data_, + any_feature_unused_bytree_ ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } + } else if (use_compact_view_) { + // compact data holds only the tree's sampled columns, so no per-column + // feature mask is needed (mirrors the per-pair compact launch). Few-bin + // datasets take the register-accumulation body (see USE_REG_BINS). + if (construct_reg_bins_) { + if (compact_is_4bit_) { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + reinterpret_cast(compact_data_uint8_t_.RawData()), + compact_column_hist_offsets_.RawData(), + cuda_row_data_->cuda_partition_hist_offsets(), + compact_feature_partition_column_index_offsets_.RawData(), + compact_packed_partition_byte_offsets_.RawData(), + nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } else { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + reinterpret_cast(compact_data_uint8_t_.RawData()), + compact_column_hist_offsets_.RawData(), + cuda_row_data_->cuda_partition_hist_offsets(), + compact_feature_partition_column_index_offsets_.RawData(), + nullptr, + nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } + } else { + if (compact_is_4bit_) { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + reinterpret_cast(compact_data_uint8_t_.RawData()), + compact_column_hist_offsets_.RawData(), + cuda_row_data_->cuda_partition_hist_offsets(), + compact_feature_partition_column_index_offsets_.RawData(), + compact_packed_partition_byte_offsets_.RawData(), + nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } else { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + reinterpret_cast(compact_data_uint8_t_.RawData()), + compact_column_hist_offsets_.RawData(), + cuda_row_data_->cuda_partition_hist_offsets(), + compact_feature_partition_column_index_offsets_.RawData(), + nullptr, + nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } + } + } else if (cuda_row_data_->is_4bit_packed()) { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + cuda_row_data_->cuda_packed_partition_byte_offsets(), + cuda_is_feature_used_bytree_.Size() > 0 ? cuda_is_feature_used_bytree_.RawData() : nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } else { + CUDAConstructHistogramDenseBatchedKernel<<>>( + pair_descs, + cuda_gradients_, cuda_hessians_, + USE_GH2 ? cuda_gradients_hessians_.RawDataReadOnly() : nullptr, + cuda_row_data_->GetBin(), + cuda_row_data_->cuda_column_hist_offsets(), + cuda_row_data_->cuda_partition_hist_offsets(), + cuda_row_data_->cuda_feature_partition_column_index_offsets(), + nullptr, + cuda_is_feature_used_bytree_.Size() > 0 ? cuda_is_feature_used_bytree_.RawData() : nullptr, + num_data_, + static_cast(min_data_in_leaf_), + min_sum_hessian_in_leaf_, + level_dim_y, + level_sizes_for_kernel, + min_grid_dim_y_, + BatchConstructMinRowsPerThread(), + BatchConstructSaturationFloor(), + SmallLeafConstructEnabled() ? SmallLeafRowThreshold() : 0, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } +} + +void CUDAHistogramConstructor::LaunchFixSubtractHistogramSmallLeafBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs) { + // block size FIX_HISTOGRAM_BLOCK_SIZE so the fix blocks reduce exactly like + // the standalone fix kernel (bit-identical); the subtract role is elementwise + // and block-size invariant + const int num_subtract_threads = 2 * num_total_bin_; + const int num_subtract_blocks = + (num_subtract_threads + FIX_HISTOGRAM_BLOCK_SIZE - 1) / FIX_HISTOGRAM_BLOCK_SIZE; + const int num_fix_blocks = static_cast(need_fix_histogram_features_.size()); + dim3 grid_dim(num_subtract_blocks + num_fix_blocks, num_pairs); + FixSubtractHistogramSmallLeafBatchedKernel<<>>( + num_total_bin_, + num_subtract_blocks, + cuda_feature_num_bins_.RawData(), + cuda_feature_hist_offsets_.RawData(), + cuda_feature_most_freq_bins_.RawData(), + cuda_need_fix_histogram_features_.RawData(), + cuda_need_fix_histogram_features_num_bin_aligned_.RawData(), + cuda_fix_mfb_mask_.RawDataReadOnly(), + pair_descs, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr, + any_feature_unused_bytree_ ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr); +} + +void CUDAHistogramConstructor::LaunchSubtractHistogramBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const bool any_pair_needs_bit_change_copy) { + if (!use_quantized_grad_) { + const int num_subtract_threads = 2 * num_total_bin_; + const int num_subtract_blocks = (num_subtract_threads + SUBTRACT_BLOCK_SIZE - 1) / SUBTRACT_BLOCK_SIZE; + if (need_fix_histogram_features_.size() > 0) { + dim3 fix_grid(static_cast(need_fix_histogram_features_.size()), num_pairs); + FixHistogramBatchedKernel<<>>( + cuda_feature_num_bins_.RawData(), + cuda_feature_hist_offsets_.RawData(), + cuda_feature_most_freq_bins_.RawData(), + cuda_need_fix_histogram_features_.RawData(), + cuda_need_fix_histogram_features_num_bin_aligned_.RawData(), + pair_descs, + any_feature_unused_bytree_ ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr); + } + dim3 subtract_grid(num_subtract_blocks, num_pairs); + SubtractHistogramBatchedKernel<<>>( + num_total_bin_, + pair_descs, + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } else { + const int num_subtract_threads = num_total_bin_; + const int num_subtract_blocks = (num_subtract_threads + SUBTRACT_BLOCK_SIZE - 1) / SUBTRACT_BLOCK_SIZE; + if (need_fix_histogram_features_.size() > 0) { + dim3 fix_grid(static_cast(need_fix_histogram_features_.size()), num_pairs); + FixHistogramDiscretizedBatchedKernel<<>>( + cuda_feature_num_bins_.RawData(), + cuda_feature_hist_offsets_.RawData(), + cuda_feature_most_freq_bins_.RawData(), + cuda_need_fix_histogram_features_.RawData(), + cuda_need_fix_histogram_features_num_bin_aligned_.RawData(), + pair_descs, + any_feature_unused_bytree_ ? cuda_is_feature_used_bytree_.RawDataReadOnly() : nullptr); + } + dim3 subtract_grid(num_subtract_blocks, num_pairs); + SubtractHistogramDiscretizedBatchedKernel<<>>( + num_total_bin_, + pair_descs, + hist_buffer_for_num_bit_change_.RawData(), + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + if (any_pair_needs_bit_change_copy) { + CopyChangedNumBitHistogramBatchedKernel<<>>( + num_total_bin_, + pair_descs, + hist_buffer_for_num_bit_change_.RawData(), + any_feature_unused_bytree_ ? cuda_bin_used_bytree_.RawDataReadOnly() : nullptr); + } + } +} + } // namespace LightGBM #endif // USE_CUDA diff --git a/src/treelearner/cuda/cuda_histogram_constructor.hpp b/src/treelearner/cuda/cuda_histogram_constructor.hpp index 9b70c7a21c87..abc97fae289c 100644 --- a/src/treelearner/cuda/cuda_histogram_constructor.hpp +++ b/src/treelearner/cuda/cuda_histogram_constructor.hpp @@ -14,7 +14,9 @@ #include #include +#include #include +#include #include #include "cuda_leaf_splits.hpp" @@ -29,6 +31,36 @@ namespace LightGBM { +/*! \brief y-grid sizing formula of the batched per-level construct kernel, + * shared verbatim by the host launch sizing (CalcConstructHistogramBatchedKernelDim) + * and the device row-grouping replica used by the speculative single-sync flow + * (which launches an upper-bound grid but must group rows exactly as the classic + * host sizing would, so histograms stay bit-identical). See the .cpp comment for + * the rationale of the cap/floor structure. */ +__host__ __device__ inline int HybridBatchedConstructGridDimY( + const data_size_t max_num_data_in_smaller_leaf, + const int num_pairs, + const int block_dim_y, + const int min_grid_dim_y, + const int min_rows_per_thread, + const int saturation_floor_total) { + // rows-per-thread target of the per-leaf sizing + const int y_full_rate = ((max_num_data_in_smaller_leaf + NUM_DATA_PER_THREAD - 1) / NUM_DATA_PER_THREAD + + block_dim_y - 1) / block_dim_y; + if (min_rows_per_thread <= 0) { + // disabled: fall back to the per-leaf sizing + return min_grid_dim_y > y_full_rate ? min_grid_dim_y : y_full_rate; + } + const int y_min_rate = ((max_num_data_in_smaller_leaf + min_rows_per_thread - 1) / min_rows_per_thread + + block_dim_y - 1) / block_dim_y; + const int saturation_floor = num_pairs > 0 ? + (saturation_floor_total + num_pairs - 1) / num_pairs : saturation_floor_total; + int inner = y_min_rate > 1 ? y_min_rate : 1; + inner = inner > saturation_floor ? inner : saturation_floor; + inner = inner < min_grid_dim_y ? inner : min_grid_dim_y; + return y_full_rate > inner ? y_full_rate : inner; +} + class CUDAHistogramConstructor { public: CUDAHistogramConstructor( @@ -50,14 +82,30 @@ class CUDAHistogramConstructor { /*! \brief Event recorded on cuda_stream_ after the smaller-leaf histogram is * constructed. The best split finder waits on it (cudaStreamWaitEvent) before * reading the smaller-leaf histogram, replacing a per-split device sync. */ - cudaEvent_t construct_done_event() const { return construct_done_event_; } + cudaEvent_t construct_done_event() const { return construct_done_events_[0]; } + + /*! \brief number of independent (stream, event-pair) pipelines for hybrid + * level-batched growth; pairs of one level round-robin across them so their + * small histogram kernels can execute concurrently */ + static constexpr int kNumHistPipelines = 4; + void SetActivePipeline(const int pipeline) { active_pipeline_ = pipeline; } + cudaStream_t current_stream() const { return pipeline_streams_[active_pipeline_]; } + const cudaEvent_t* construct_done_events() const { return construct_done_events_; } + const cudaEvent_t* subtract_done_events() const { return subtract_done_events_; } /*! \brief Event recorded on cuda_stream_ after the larger-leaf histogram is * produced by subtraction. The best split finder waits on it before reading * the larger-leaf histogram. */ - cudaEvent_t subtract_done_event() const { return subtract_done_event_; } + cudaEvent_t subtract_done_event() const { return subtract_done_events_[0]; } void ZeroHistForLeaf(int leaf_index); + /*! \brief Selective grow-then-prune: zero the histogram slots freed by + * collapsed subtrees so they can be reallocated to later right children + * (the construct kernels accumulate into assumed-zero slots). Async memsets + * on the legacy default stream: ordered before any subsequently enqueued + * construct kernel on the (blocking) histogram streams. */ + void ZeroHistSlots(const std::vector& slots); + void ConstructHistogramForLeaf( const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, const CUDALeafSplitsStruct* cuda_larger_leaf_splits, @@ -77,12 +125,155 @@ class CUDAHistogramConstructor { const uint8_t smaller_num_bits_in_histogram_bins, const uint8_t larger_num_bits_in_histogram_bins); + /*! \brief stream the batched per-level histogram kernels run on; the hybrid + * learner uploads the per-level pair descriptors asynchronously on this + * stream so the copy is ordered before the construct kernel that reads them */ + cudaStream_t hist_stream() const { return cuda_stream_; } + + /*! \brief whether the batched per-level construct/fix/subtract path supports the + * current data layout (dense, shared-memory histograms). The compact column + * view (feature_fraction sampling gathered into a dense per-tree copy) is + * supported through the same batched dense kernel fed with the compact + * pointers/dims; EXABOOST_BATCH_WIDE=0 restores the per-pair fallback for it. */ + bool SupportsBatchedLevel() const { + return cuda_row_data_ != nullptr && + !cuda_row_data_->is_sparse() && + cuda_row_data_->NumLargeBinPartition() == 0 && + (!use_compact_view_ || + (ExaboostBatchWideEnabled() && !compact_is_col_major_ && + cuda_row_data_->bit_type() == 8)); + } + + /*! \brief whether the tree learner's per-tree column-view build can source its + * gather from this constructor's compact matrix (row-major-in-partition over + * the tree's sampled columns) instead of the full bin matrix */ + bool CompactViewSourceAvailable() const { + return use_compact_view_ && !compact_is_col_major_; + } + const std::vector& compact_src_cols() const { return compact_src_cols_host_; } + const std::vector& compact_slot_bytes() const { return compact_slot_byte_host_; } + const std::vector& compact_slot_strides() const { return compact_slot_stride_host_; } + const std::vector& compact_slot_cols() const { return compact_slot_col_host_; } + /*! \brief whether the compact matrix is 4-bit packed (two columns per byte, + * per-partition even-column padding; mirrors CUDARowData::is_4bit_packed) */ + bool compact_src_is_4bit() const { return compact_is_4bit_; } + const uint8_t* compact_data_device() const { return compact_data_uint8_t_.RawDataReadOnly(); } + /*! \brief whether this tree's compact build also produced the column-major + * view (compact_col_major_device()[slot * num_data + row]); the tree learner + * then uses it directly instead of gathering its own copy */ + bool CompactColMajorFilled() const { return compact_col_major_filled_; } + const uint8_t* compact_col_major_device() const { return compact_staging_col_major_.RawDataReadOnly(); } + + /*! \brief one-line gate dump for EXABOOST_HYBRID_DIAG */ + std::string BatchedLevelGateDiag() const { + char buf[256]; + snprintf(buf, sizeof(buf), + "hist: row_data=%d sparse=%d large_bin_parts=%d compact_view=%d " + "col_major=%d bit_type=%d batch_wide=%d", + static_cast(cuda_row_data_ != nullptr), + cuda_row_data_ != nullptr ? static_cast(cuda_row_data_->is_sparse()) : -1, + cuda_row_data_ != nullptr ? cuda_row_data_->NumLargeBinPartition() : -1, + static_cast(use_compact_view_), + static_cast(compact_is_col_major_), + cuda_row_data_ != nullptr ? cuda_row_data_->bit_type() : -1, + static_cast(ExaboostBatchWideEnabled())); + return std::string(buf); + } + + /*! \brief Batched per-level histogram phase for hybrid growth: one construct, one + * fix and one subtract launch cover every sibling pair of a level (pairs indexed + * by a grid dimension). All kernels run on pipeline_streams_[0] (cuda_stream_) in + * order; subtract_done_events_[0] is recorded at the end so the best split finder + * can order its batched find kernel after the whole histogram phase. + * level_smaller_num_data is null in the classic (two-sync) flow, where + * max_num_data_in_smaller_leaf is the level's exact maximum smaller-leaf size. + * In the speculative (single-sync) flow it points at the device array of this + * level's actual smaller-child sizes (written by the batched apply kernels); + * max_num_data_in_smaller_leaf is then only an upper BOUND used to size the + * launch grid, and the construct kernel derives the exact per-thread row + * grouping on-device from the array so histograms are bit-identical to the + * classic sizing (non-quantized dense path only). */ + void ConstructHistogramsForLevel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const bool any_pair_needs_bit_change_copy, + const data_size_t* level_smaller_num_data = nullptr); + + /*! \brief minimum rows-per-thread cap of the batched construct grid sizing + * (EXABOOST_BATCH_CONSTRUCT_MINROWS; 0 = per-leaf sizing). Shared by the host + * grid sizing and the device row-grouping replica. */ + static int BatchConstructMinRowsPerThread() { + static const int min_rows_per_thread = []() { + const char* env = std::getenv("EXABOOST_BATCH_CONSTRUCT_MINROWS"); + return env != nullptr ? std::atoi(env) : 64; + }(); + return min_rows_per_thread; + } + + /*! \brief device-saturation floor (total y-blocks shared across the level's + * pairs) of the batched construct grid sizing + * (EXABOOST_BATCH_CONSTRUCT_FLOOR; default 160 = historical behavior) */ + static int BatchConstructSaturationFloor() { + static const int floor_total = []() { + const char* env = std::getenv("EXABOOST_BATCH_CONSTRUCT_FLOOR"); + return env != nullptr ? std::atoi(env) : 160; + }(); + return floor_total; + } + + /*! \brief interleaved float2 gradient/hessian copy for the non-quantized + * dense construct kernels: their scattered per-row reads then touch ONE + * 32B sector per row instead of two (gradients and hessians live in two + * separate arrays). Values are bit-identical to the separate reads. + * EXABOOST_GH_INTERLEAVE=0 disables; float score_t only. */ + static bool GHInterleaveEnabled() { + static const bool enabled = []() { + const char* env = std::getenv("EXABOOST_GH_INTERLEAVE"); + return env == nullptr || std::string(env) != "0"; + }(); + return enabled && sizeof(score_t) == sizeof(float); + } + + /*! \brief kill-switch of the small-leaf construct path + * (EXABOOST_SMALL_LEAF_CONSTRUCT=0 restores the shared-memory batched + * kernels for every level) */ + static bool SmallLeafConstructEnabled() { + static const bool enabled = []() { + const char* env = std::getenv("EXABOOST_SMALL_LEAF_CONSTRUCT"); + return env == nullptr || std::string(env) != "0"; + }(); + return enabled; + } + + /*! \brief leaf-size threshold of the DIRECT small-leaf construct body: a pair + * whose actual smaller-leaf size (device-side check) is at most this many + * rows skips the shared-memory histogram accumulation and adds straight to + * the global histogram (non-quantized training only). Default 0 = disabled: + * the direct body changes the float accumulation (per-row double adds vs + * per-block float partial sums), which trades the fraud 63/6 exact-quality + * reproduction for a measured ~1-2% fraud-deep gain -- not worth it by + * default. Set EXABOOST_SMALL_LEAF_ROWS=8192 to enable. */ + static data_size_t SmallLeafRowThreshold() { + static const data_size_t threshold = []() { + const char* env = std::getenv("EXABOOST_SMALL_LEAF_ROWS"); + return env != nullptr ? static_cast(std::atoi(env)) : 0; + }(); + return threshold; + } + void ResetTrainingData(const Dataset* train_data, TrainingShareStates* share_states); void ResetConfig(const Config* config); void BeforeTrain(const score_t* gradients, const score_t* hessians); + /*! \brief number of leaf histogram slots the finished tree dirtied (== its + * final leaf count; leaf k's histogram is slot k). BeforeTrain only zeroes + * that prefix of cuda_hist_ instead of all num_leaves slots. -1 (initial / + * after resets, and kept on the NCCL path) zeroes everything. */ + void SetNumDirtyLeaves(const int num_dirty_leaves) { num_dirty_leaves_ = num_dirty_leaves; } + // Per-tree feature sampling mask (host-side vector, length == num_features_). // Copied to cuda_is_feature_used_bytree_ so the histogram kernel can skip // features not selected by feature_fraction sampling. @@ -107,6 +298,9 @@ class CUDAHistogramConstructor { private: void InitFeatureMetaInfo(const Dataset* train_data, const std::vector& feature_hist_offsets); + /*! \brief (re)build cuda_fix_mfb_mask_ from the need-fix feature metadata */ + void InitFixMFBMask(); + void CalcConstructHistogramKernelDim( int* grid_dim_x, int* grid_dim_y, @@ -114,6 +308,20 @@ class CUDAHistogramConstructor { int* block_dim_y, const data_size_t num_data_in_smaller_leaf); + /*! \brief Grid sizing for the batched per-level construct kernel. Shares the + * min_grid_dim_y_ device-saturation floor across the level's pairs and caps + * the y-grid at a minimum rows-per-thread: every active block pays a fixed + * shared-histogram zero + global-merge cost, which dominates small leaves. + * Identical to the per-leaf sizing for single-pair levels and for leaves + * large enough that the cap is inactive. */ + void CalcConstructHistogramBatchedKernelDim( + int* grid_dim_x, + int* grid_dim_y, + int* block_dim_x, + int* block_dim_y, + const data_size_t max_num_data_in_smaller_leaf, + const int num_pairs); + template void LaunchConstructHistogramKernelInner( const CUDALeafSplitsStruct* cuda_smaller_leaf_splits, @@ -151,6 +359,56 @@ class CUDAHistogramConstructor { const uint8_t smaller_num_bits_in_histogram_bins, const uint8_t larger_num_bits_in_histogram_bins); + // ---- batched per-level launchers (hybrid growth) ---- + + template + void LaunchConstructHistogramBatchedKernelInner( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data); + + /*! \brief USE_GH2: feed the construct kernel the interleaved float2 + * gradient/hessian copy (see GHInterleaveEnabled); compile-time so the + * non-interleaved instantiation stays identical to the historical kernel */ + template + void LaunchConstructHistogramBatchedKernelInner0( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data); + + void LaunchConstructHistogramBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const data_size_t max_num_data_in_smaller_leaf, + const data_size_t* level_smaller_num_data); + + void LaunchSubtractHistogramBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs, + const bool any_pair_needs_bit_change_copy); + + // ---- small-leaf level launchers (hybrid growth, non-quantized only) ---- + + /*! \brief fused fix + subtract of the small-leaf path: one launch does both + * the most-frequent-bin fix of the smaller leaf and the parent-minus-smaller + * subtraction of the larger leaf. Subtract blocks skip the fix features' + * most-frequent-bin entries (cuda_fix_mfb_mask_); the fix blocks write those + * for both leaves with the identical arithmetic, so the result is + * bit-identical to the sequential fix -> subtract launches. */ + void LaunchFixSubtractHistogramSmallLeafBatchedKernel( + const CUDAHybridPairDescriptor* pair_descs, + const int num_pairs); + + /*! \brief one tiny launch per speculative level: reduce the level's actual + * smaller-child sizes and apply the exact host grid-sizing formula, writing + * the effective row-grouping extent to cuda_hybrid_construct_dim_y_ */ + void LaunchComputeBatchedConstructDimYKernel( + const data_size_t* level_smaller_num_data, + const int num_pairs, + const int block_dim_y); + // Host memory /*! \brief size of training data */ @@ -175,6 +433,10 @@ class CUDAHistogramConstructor { double min_sum_hessian_in_leaf_; /*! \brief cuda stream for histogram construction */ cudaStream_t cuda_stream_; + cudaStream_t pipeline_streams_[kNumHistPipelines]; + cudaEvent_t construct_done_events_[kNumHistPipelines]; + cudaEvent_t subtract_done_events_[kNumHistPipelines]; + int active_pipeline_ = 0; /*! \brief recorded on cuda_stream_ after smaller-leaf histogram construction */ cudaEvent_t construct_done_event_ = nullptr; /*! \brief recorded on cuda_stream_ after larger-leaf histogram subtraction */ @@ -185,6 +447,8 @@ class CUDAHistogramConstructor { std::vector need_fix_histogram_features_num_bin_aligend_; /*! \brief minimum number of blocks allowed in the y dimension */ const int min_grid_dim_y_ = 160; + /*! \brief leaf histogram slots dirtied by the previous tree (-1 = all) */ + int num_dirty_leaves_ = -1; // CUDA memory, held by this object @@ -204,6 +468,26 @@ class CUDAHistogramConstructor { /*! \brief Per-tree feature mask (1 = feature in this tree's sample, 0 = skip). * Indexed by column_index (== inner_feature_index for dense single-feature groups). */ CUDAVector cuda_is_feature_used_bytree_; + /*! \brief effective row-grouping extent of the speculative batched construct + * launch (see LaunchComputeBatchedConstructDimYKernel) */ + CUDAVector cuda_hybrid_construct_dim_y_; + /*! \brief mask over the 2 * num_total_bin_ histogram entries marking the + * most-frequent-bin gradient/hessian slots of the need-fix features; the + * fused small-leaf fix+subtract kernel's subtract blocks skip these (the fix + * blocks own them) */ + CUDAVector cuda_fix_mfb_mask_; + /*! \brief per-tree bin-level used mask (1 = bin belongs to a feature in this + * tree's feature_fraction sample). The batched fix/subtract kernels and the + * batched construct's shared-histogram zero/merge skip unused bins: nothing + * of this tree ever reads them (the find kernels are feature-masked), so + * their content is dead storage. Only active when sampling is on. */ + std::vector host_bin_used_bytree_; + CUDAVector cuda_bin_used_bytree_; + bool any_feature_unused_bytree_ = false; + /*! \brief every feature fits the register-accumulation bin cap (<= 8 bins); + * selects the contention-free construct body on the batched compact path + * (non-quantized only; EXABOOST_BATCH_REGHIST=0 disables) */ + bool construct_reg_bins_ = false; // ======================================================================== // Compact-view buffers: when feature_fraction < 1.0, build a contiguous @@ -228,6 +512,29 @@ class CUDAHistogramConstructor { bool use_compact_view_; /*! \brief if true, compact_data_uint8_t_ is column-major-in-partition (used when source is host) */ bool compact_is_col_major_ = false; + /*! \brief host copies of the compact layout (source column per slot and the + * row-major-in-partition placement), kept so the tree learner's per-tree + * column-view build can gather from the ~10x smaller compact matrix instead + * of re-reading the full bin matrix (see CompactViewSource*) */ + std::vector compact_src_cols_host_; + std::vector compact_slot_byte_host_; + std::vector compact_slot_stride_host_; + std::vector compact_slot_col_host_; + /*! \brief compact matrix is 4-bit packed (set iff the row data is packed) */ + bool compact_is_4bit_ = false; + /*! \brief prefix over partitions of packed compact per-row byte widths (4-bit mode) */ + CUDAVector compact_packed_partition_byte_offsets_; + /*! \brief per destination-byte-slot metadata of the 4-bit compact fill kernel + * (source nibble bases of the byte's two columns, per-row nibble stride, + * destination byte base/stride) */ + CUDAVector cuda_bs_src_nib0_; + CUDAVector cuda_bs_src_nib1_; + CUDAVector cuda_bs_src_stride_nib_; + CUDAVector cuda_bs_dst_byte_; + CUDAVector cuda_bs_dst_stride_; + /*! \brief whether compact_staging_col_major_ holds this tree's column-major + * compact view (fused second output of the compact fill) */ + bool compact_col_major_filled_ = false; /*! \brief column_hist_offsets for compact view (length num_compact_columns+1) */ CUDAVector compact_column_hist_offsets_; /*! \brief partition_hist_offsets for compact view: [0, total_compact_bins] */ @@ -260,6 +567,11 @@ class CUDAHistogramConstructor { CUDAVector cuda_need_fix_histogram_features_num_bin_aligned_; /*! \brief histogram buffer used in histogram subtraction with different number of bits for histogram bins */ CUDAVector hist_buffer_for_num_bit_change_; + /*! \brief per-tree interleaved (gradient, hessian) pairs (see GHInterleaveEnabled); + * filled by BeforeTrain, consumed by the non-quantized dense construct kernels */ + CUDAVector cuda_gradients_hessians_; + /*! \brief whether cuda_gradients_hessians_ holds this tree's gradients */ + bool gh_interleave_valid_ = false; // CUDA memory, held by other object diff --git a/src/treelearner/cuda/cuda_leaf_splits.cpp b/src/treelearner/cuda/cuda_leaf_splits.cpp index 5b5dec54799d..4a96e015a5fc 100644 --- a/src/treelearner/cuda/cuda_leaf_splits.cpp +++ b/src/treelearner/cuda/cuda_leaf_splits.cpp @@ -31,8 +31,9 @@ void CUDALeafSplits::Init(const bool use_quantized_grad) { } void CUDALeafSplits::InitValues() { + // legacy default stream: later consumers on blocking streams implicitly + // order after this kernel, so no device sync is needed LaunchInitValuesEmptyKernel(); - SynchronizeCUDADevice(__FILE__, __LINE__); } void CUDALeafSplits::InitValues( @@ -40,15 +41,32 @@ void CUDALeafSplits::InitValues( const score_t* cuda_gradients, const score_t* cuda_hessians, const data_size_t* cuda_bagging_data_indices, const data_size_t* cuda_data_indices_in_leaf, const data_size_t num_used_indices, hist_t* cuda_hist_in_leaf, - double* root_sum_gradients, double* root_sum_hessians) { + double* root_sum_gradients, double* root_sum_hessians, + const bool defer_root_sum_readback) { cuda_gradients_ = cuda_gradients; cuda_hessians_ = cuda_hessians; - cuda_sum_of_gradients_buffer_.SetValue(0); - cuda_sum_of_hessians_buffer_.SetValue(0); + // async memsets on the legacy default stream: the init kernels follow on the + // same stream and the synchronous D2H copies below block until completion + // (SetValue would pay one full device sync each, plus one at the end) + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_sum_of_gradients_buffer_.RawData()), 0, + cuda_sum_of_gradients_buffer_.Size() * sizeof(double))); + CUDASUCCESS_OR_FATAL(cudaMemset(reinterpret_cast(cuda_sum_of_hessians_buffer_.RawData()), 0, + cuda_sum_of_hessians_buffer_.Size() * sizeof(double))); LaunchInitValuesKernel(lambda_l1, lambda_l2, cuda_bagging_data_indices, cuda_data_indices_in_leaf, num_used_indices, cuda_hist_in_leaf); - CopyFromCUDADeviceToHost(root_sum_gradients, cuda_sum_of_gradients_buffer_.RawData(), 1, __FILE__, __LINE__); - CopyFromCUDADeviceToHost(root_sum_hessians, cuda_sum_of_hessians_buffer_.RawData(), 1, __FILE__, __LINE__); - SynchronizeCUDADevice(__FILE__, __LINE__); + if (!defer_root_sum_readback) { + // synchronous D2H: blocks on everything enqueued so far, including the + // (large) per-tree histogram-buffer zeroing on the legacy stream. Flows that + // do not need the host root sums up front (single-sync hybrid growth derives + // all root gating on-device) defer this to CopyRootSumsToHost, letting the + // level-0 search enqueue overlap the zeroing. + CopyFromCUDADeviceToHost(root_sum_gradients, cuda_sum_of_gradients_buffer_.RawData(), 1, __FILE__, __LINE__); + CopyFromCUDADeviceToHost(root_sum_hessians, cuda_sum_of_hessians_buffer_.RawData(), 1, __FILE__, __LINE__); + } +} + +void CUDALeafSplits::CopyRootSumsToHost(double* root_sum_gradients, double* root_sum_hessians) const { + CopyFromCUDADeviceToHost(root_sum_gradients, cuda_sum_of_gradients_buffer_.RawDataReadOnly(), 1, __FILE__, __LINE__); + CopyFromCUDADeviceToHost(root_sum_hessians, cuda_sum_of_hessians_buffer_.RawDataReadOnly(), 1, __FILE__, __LINE__); } void CUDALeafSplits::InitValues( @@ -61,9 +79,10 @@ void CUDALeafSplits::InitValues( cuda_gradients_ = reinterpret_cast(cuda_gradients_and_hessians); cuda_hessians_ = nullptr; LaunchInitValuesKernel(lambda_l1, lambda_l2, cuda_bagging_data_indices, cuda_data_indices_in_leaf, num_used_indices, cuda_hist_in_leaf, grad_scale, hess_scale); + // the synchronous D2H copies block until the kernels above complete; no + // extra device sync is needed CopyFromCUDADeviceToHost(root_sum_gradients, cuda_sum_of_gradients_buffer_.RawData(), 1, __FILE__, __LINE__); CopyFromCUDADeviceToHost(root_sum_hessians, cuda_sum_of_hessians_buffer_.RawData(), 1, __FILE__, __LINE__); - SynchronizeCUDADevice(__FILE__, __LINE__); } void CUDALeafSplits::Resize(const data_size_t num_data) { diff --git a/src/treelearner/cuda/cuda_leaf_splits.cu b/src/treelearner/cuda/cuda_leaf_splits.cu index d26302841723..7968fed9b3f4 100644 --- a/src/treelearner/cuda/cuda_leaf_splits.cu +++ b/src/treelearner/cuda/cuda_leaf_splits.cu @@ -328,9 +328,9 @@ void CUDALeafSplits::LaunchInitValuesKernel( cuda_gradients_, cuda_hessians_, num_used_indices, cuda_bagging_data_indices, cuda_sum_of_gradients_buffer_.RawData(), cuda_sum_of_hessians_buffer_.RawData()); } - SynchronizeCUDADevice(__FILE__, __LINE__); if (nccl_communicator_ != nullptr) { + SynchronizeCUDADevice(__FILE__, __LINE__); ReduceGradKernel<<<1, NUM_THREADS_PER_BLOCK_LEAF_SPLITS>>>(num_blocks_init_from_gradients_, cuda_sum_of_gradients_buffer_.RawData(), cuda_sum_of_hessians_buffer_.RawData(), num_used_indices); SynchronizeCUDADevice(__FILE__, __LINE__); @@ -355,7 +355,8 @@ void CUDALeafSplits::LaunchInitValuesKernel( cuda_hist_in_leaf, cuda_struct_.RawData()); } - SynchronizeCUDADevice(__FILE__, __LINE__); + // callers read the root sums back with a synchronous D2H copy that blocks + // until these legacy-default-stream kernels complete; no device sync needed } void CUDALeafSplits::LaunchInitValuesKernel( @@ -376,9 +377,8 @@ void CUDALeafSplits::LaunchInitValuesKernel( cuda_sum_of_hessians_buffer_.RawData(), cuda_sum_of_gradients_hessians_buffer_.RawData(), grad_scale, hess_scale); } - SynchronizeCUDADevice(__FILE__, __LINE__); - if (nccl_communicator_ != nullptr) { + SynchronizeCUDADevice(__FILE__, __LINE__); ReduceGradKernel<<<1, NUM_THREADS_PER_BLOCK_LEAF_SPLITS>>>(num_blocks_init_from_gradients_, cuda_sum_of_gradients_buffer_.RawData(), cuda_sum_of_hessians_buffer_.RawData(), cuda_sum_of_gradients_hessians_buffer_.RawData(), num_used_indices); @@ -407,7 +407,8 @@ void CUDALeafSplits::LaunchInitValuesKernel( cuda_hist_in_leaf, cuda_struct_.RawData()); } - SynchronizeCUDADevice(__FILE__, __LINE__); + // callers read the root sums back with a synchronous D2H copy that blocks + // until these legacy-default-stream kernels complete; no device sync needed } } // namespace LightGBM diff --git a/src/treelearner/cuda/cuda_leaf_splits.hpp b/src/treelearner/cuda/cuda_leaf_splits.hpp index 3dd87a05060a..2f8b368bed0b 100644 --- a/src/treelearner/cuda/cuda_leaf_splits.hpp +++ b/src/treelearner/cuda/cuda_leaf_splits.hpp @@ -15,11 +15,25 @@ #include #include +#include +#include + #define NUM_THREADS_PER_BLOCK_LEAF_SPLITS (1024) #define NUM_DATA_THREAD_ADD_LEAF_SPLITS (6) namespace LightGBM { +/*! \brief kill switch for the wide-shape batched level support (many split-find + * tasks and/or compact-column-view histogram data): EXABOOST_BATCH_WIDE=0 + * restores the previous fallback to the per-pair kernels for those shapes */ +inline bool ExaboostBatchWideEnabled() { + static const bool enabled = []() { + const char* env = std::getenv("EXABOOST_BATCH_WIDE"); + return env == nullptr || std::string(env) != "0"; + }(); + return enabled; +} + struct CUDALeafSplitsStruct { public: int leaf_index; @@ -33,6 +47,31 @@ struct CUDALeafSplitsStruct { hist_t* hist_in_leaf; }; +/*! \brief Per-sibling-pair metadata for the hybrid level-batched growth phase. + * One entry per pair of a level; uploaded to the device in a single H2D copy so + * the batched construct/fix/subtract/find/sync kernels can cover all pairs of a + * level with one launch each (indexed by a grid dimension). All fields are + * host-known at level start (single-GPU only, so global == local leaf counts). */ +struct CUDAHybridPairDescriptor { + const CUDALeafSplitsStruct* smaller_struct; + const CUDALeafSplitsStruct* larger_struct; + int smaller_leaf_index; + int larger_leaf_index; + data_size_t num_data_in_smaller_leaf; + data_size_t num_data_in_larger_leaf; + /*! \brief histogram construction needed (mirror of ConstructHistogramForLeaf's + * min_data/min_hessian early-return) */ + uint8_t construct_valid; + /*! \brief smaller/larger leaf pass the best-split-search validity checks + * (min_data_in_leaf, min_sum_hessian_in_leaf, below max_depth) */ + uint8_t smaller_valid; + uint8_t larger_valid; + /*! \brief per-pair histogram bit widths for quantized training (0 when unused) */ + uint8_t parent_num_bits; + uint8_t smaller_num_bits; + uint8_t larger_num_bits; +}; + class CUDALeafSplits: public NCCLInfo { public: explicit CUDALeafSplits(const data_size_t num_data); @@ -46,7 +85,11 @@ class CUDALeafSplits: public NCCLInfo { const score_t* cuda_gradients, const score_t* cuda_hessians, const data_size_t* cuda_bagging_data_indices, const data_size_t* cuda_data_indices_in_leaf, const data_size_t num_used_indices, - hist_t* cuda_hist_in_leaf, double* root_sum_gradients, double* root_sum_hessians); + hist_t* cuda_hist_in_leaf, double* root_sum_gradients, double* root_sum_hessians, + const bool defer_root_sum_readback = false); + + /*! \brief deferred counterpart of InitValues' root-sum readback (see there) */ + void CopyRootSumsToHost(double* root_sum_gradients, double* root_sum_hessians) const; void InitValues( const double lambda_l1, const double lambda_l2, diff --git a/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp b/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp index 85cc72408256..11e7d7144a2b 100644 --- a/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp +++ b/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp @@ -16,7 +16,9 @@ #include #include +#include #include +#include #include #include #include @@ -66,6 +68,29 @@ void CUDASingleGPUTreeLearner::Init(const Dataset* train_data, bool is_constant_ cuda_data_partition_->Init(); select_features_by_node_ = !config_->interaction_constraints_vector.empty() || config_->feature_fraction_bynode < 1.0; + // hybrid level-batched growth is on by default; EXABOOST_HYBRID_GROWTH=0 disables + // it (falls back to the classic one-split-at-a-time leaf-wise loop everywhere) + const char* hybrid_env = std::getenv("EXABOOST_HYBRID_GROWTH"); + use_hybrid_growth_ = (hybrid_env == nullptr || std::string(hybrid_env) != std::string("0")); + // batched per-level kernels for the hybrid prefix (one construct/fix/subtract/ + // find/sync launch per level instead of per pair); "0" keeps the per-pair path + const char* batch_env = std::getenv("EXABOOST_HYBRID_BATCH_KERNELS"); + use_hybrid_batch_kernels_ = (batch_env == nullptr || std::string(batch_env) != std::string("0")); + // batched per-level APPLY phase for the hybrid prefix (one launch per kernel + // family per level and zero per-split host syncs, instead of ~7 launches and + // 2 device syncs per split); "0" keeps the per-split deferred loop + const char* batch_apply_env = std::getenv("EXABOOST_HYBRID_BATCH_APPLY"); + use_hybrid_batch_apply_ = (batch_apply_env == nullptr || std::string(batch_apply_env) != std::string("0")); + // single-sync (speculative) level pipeline for the hybrid prefix: each level's + // apply and the children's search are enqueued back to back and read back with + // ONE device synchronization per level; "0" keeps the classic two-sync flow + const char* one_sync_env = std::getenv("EXABOOST_HYBRID_ONE_SYNC"); + use_hybrid_one_sync_ = (one_sync_env == nullptr || std::string(one_sync_env) != std::string("0")); + // selective (grow-then-prune) growth for budget-limited configs + // (num_leaves << 2^max_depth): exactly leaf-wise-equivalent level batching + // with end-of-selection pruning; "0" falls back to the classic loop there + const char* selective_env = std::getenv("EXABOOST_HYBRID_SELECTIVE"); + use_hybrid_selective_ = (selective_env == nullptr || std::string(selective_env) != std::string("0")); cuda_best_split_finder_.reset(new CUDABestSplitFinder(cuda_histogram_constructor_->cuda_hist(), train_data_, this->share_state_->feature_hist_offsets(), select_features_by_node_, config_)); cuda_best_split_finder_->Init(); @@ -73,8 +98,8 @@ void CUDASingleGPUTreeLearner::Init(const Dataset* train_data, bool is_constant_ // per-leaf FindBestSplits kernels are ordered after histogram construction/subtraction // via cudaStreamWaitEvent instead of per-split device syncs. cuda_best_split_finder_->SetHistogramEvents( - cuda_histogram_constructor_->construct_done_event(), - cuda_histogram_constructor_->subtract_done_event()); + cuda_histogram_constructor_->construct_done_events(), + cuda_histogram_constructor_->subtract_done_events()); leaf_best_split_feature_.resize(config_->num_leaves, -1); leaf_best_split_threshold_.resize(config_->num_leaves, 0); @@ -155,6 +180,13 @@ void CUDASingleGPUTreeLearner::BeforeTrain() { } } else { cuda_histogram_constructor_->BeforeTrain(gradients_, hessians_); + // The single-sync hybrid prefix derives every root gate on-device, so the + // synchronous root-sum readback (which would block on the histogram-buffer + // zeroing above) is deferred; EnsureRootSumsReadBack performs it lazily on + // the paths that do need host root sums. Optimistic: SupportsBatchedLevel + // may flip once BuildCompactView below runs, which the classic prefix + // handles by reading the sums back at its start. + root_sums_deferred_ = HybridGrowthUsable() && UseOneSyncPrefix(); cuda_smaller_leaf_splits_->InitValues( config_->lambda_l1, config_->lambda_l2, @@ -165,7 +197,8 @@ void CUDASingleGPUTreeLearner::BeforeTrain() { root_num_data, cuda_histogram_constructor_->cuda_hist_pointer(), &leaf_sum_gradients_[0], - &leaf_sum_hessians_[0]); + &leaf_sum_hessians_[0], + root_sums_deferred_); } leaf_num_data_[0] = root_num_data; cuda_larger_leaf_splits_->InitValues(); @@ -186,6 +219,23 @@ void CUDASingleGPUTreeLearner::BeforeTrain() { } } +namespace { + +// Split kernels read the 4-bit packed compact matrix directly (bit_type 4 +// descriptors) instead of materializing the per-tree column-major buffer +// (~1.5GB write per tree on numerai-shaped data); the scattered packed reads +// cost one 32B sector per row but only the split columns are ever touched. +// EXABOOST_SPLIT_PACKED_READ=0 restores the column-major gather. +bool SplitPackedReadEnabled() { + static const bool enabled = []() { + const char* env = std::getenv("EXABOOST_SPLIT_PACKED_READ"); + return env == nullptr || std::string(env) != "0"; + }(); + return enabled; +} + +} // anonymous namespace + // Forward decl: defined in cuda_histogram_constructor.cu. extern void LaunchRowToColCompactKernel( cudaStream_t stream, @@ -195,7 +245,8 @@ extern void LaunchRowToColCompactKernel( const int* slot_p_stride, const int* slot_col_in_p, int num_compact_cols, - data_size_t num_data); + data_size_t num_data, + bool src_is_4bit); void CUDASingleGPUTreeLearner::BuildCompactColumnView() { const auto* row_data = cuda_histogram_constructor_->cuda_row_data_internal(); @@ -241,20 +292,55 @@ void CUDASingleGPUTreeLearner::BuildCompactColumnView() { } // Build per-slot source-frame metadata host-side (one entry per compact slot). + // When the histogram constructor already gathered this tree's sampled columns + // into its compact matrix (same column set, row-major-in-partition), gather + // from THAT instead of the full bin matrix: the transpose then reads + // ~fraction of the bytes (the full-matrix gather effectively streams the + // whole dataset through L2 once per tree). const auto& src_part_col_offsets_h = row_data->host_feature_partition_column_index_offsets(); + const bool compact_src = cuda_histogram_constructor_->CompactViewSourceAvailable() && + cuda_histogram_constructor_->compact_src_cols() == compact_column_to_orig_; + if (compact_src && cuda_histogram_constructor_->CompactColMajorFilled()) { + // the histogram constructor's compact fill already produced the column-major + // view as a fused second output (same slot order, verified above): point the + // column data at it directly, no gather kernel at all + col_data->SetCompactColumnView(orig_column_to_compact_slot_, + const_cast(cuda_histogram_constructor_->compact_col_major_device()), + static_cast(num_data)); + compact_col_signature_ = sig; + return; + } + const uint8_t* gather_src = compact_src ? cuda_histogram_constructor_->compact_data_device() : src; + const bool gather_src_is_4bit = compact_src ? + cuda_histogram_constructor_->compact_src_is_4bit() : row_data->is_4bit_packed(); std::vector slot_p_byte_h(num_compact_cols); std::vector slot_p_stride_h(num_compact_cols); std::vector slot_col_in_p_h(num_compact_cols); - for (int s = 0; s < num_compact_cols; ++s) { - const int orig_col = compact_column_to_orig_[s]; - int p = 0; - while (p + 1 < static_cast(src_part_col_offsets_h.size()) && - orig_col >= src_part_col_offsets_h[p + 1]) ++p; - const int p_start = src_part_col_offsets_h[p]; - const int p_end = src_part_col_offsets_h[p + 1]; - slot_p_byte_h[s] = static_cast(p_start) * static_cast(num_data); - slot_p_stride_h[s] = p_end - p_start; - slot_col_in_p_h[s] = orig_col - p_start; + if (compact_src) { + // 8-bit: slot byte offsets already include the column-in-partition offset + // (col entries are 0); 4-bit: byte offsets are the packed partition bases + // and the logical column-in-partition rides in compact_slot_cols() + slot_p_byte_h = cuda_histogram_constructor_->compact_slot_bytes(); + slot_p_stride_h = cuda_histogram_constructor_->compact_slot_strides(); + slot_col_in_p_h = cuda_histogram_constructor_->compact_slot_cols(); + } else { + const std::vector& packed_offsets_h = row_data->host_packed_partition_byte_offsets(); + for (int s = 0; s < num_compact_cols; ++s) { + const int orig_col = compact_column_to_orig_[s]; + int p = 0; + while (p + 1 < static_cast(src_part_col_offsets_h.size()) && + orig_col >= src_part_col_offsets_h[p + 1]) ++p; + const int p_start = src_part_col_offsets_h[p]; + const int p_end = src_part_col_offsets_h[p + 1]; + if (gather_src_is_4bit) { + slot_p_byte_h[s] = static_cast(packed_offsets_h[p]) * static_cast(num_data); + slot_p_stride_h[s] = packed_offsets_h[p + 1] - packed_offsets_h[p]; + } else { + slot_p_byte_h[s] = static_cast(p_start) * static_cast(num_data); + slot_p_stride_h[s] = p_end - p_start; + } + slot_col_in_p_h[s] = orig_col - p_start; + } } if (cuda_slot_p_byte_.Size() < static_cast(num_compact_cols)) { cuda_slot_p_byte_.Resize(num_compact_cols); @@ -268,15 +354,38 @@ void CUDASingleGPUTreeLearner::BuildCompactColumnView() { CopyFromHostToCUDADevice(cuda_slot_col_in_p_.RawData(), slot_col_in_p_h.data(), num_compact_cols, __FILE__, __LINE__); + compact_packed_view_active_ = false; + if (SplitPackedReadEnabled() && gather_src_is_4bit && HybridGrowthUsable()) { + // packed split read: skip the column-major gather entirely; the batched + // apply descriptors address the packed source per column (byte of row 0, + // per-row byte stride, nibble shift). The slot metadata just uploaded stays + // valid for the lazy classic fallback (EnsureClassicColumnView), which any + // classic per-split Split() triggers via ApplySplit. + std::vector col_base_byte_h(num_compact_cols); + std::vector col_shift_h(num_compact_cols); + for (int s = 0; s < num_compact_cols; ++s) { + col_base_byte_h[s] = slot_p_byte_h[s] + static_cast(slot_col_in_p_h[s] >> 1); + col_shift_h[s] = static_cast((slot_col_in_p_h[s] & 1) << 2); + } + col_data->SetCompactPackedColumnView(orig_column_to_compact_slot_, gather_src, + col_base_byte_h, slot_p_stride_h, col_shift_h); + compact_packed_view_active_ = true; + compact_gather_src_ = gather_src; + compact_gather_src_is_4bit_ = gather_src_is_4bit; + compact_col_signature_ = sig; + return; + } + LaunchRowToColCompactKernel( 0, - src, + gather_src, compact_column_buffer_.RawData(), cuda_slot_p_byte_.RawData(), cuda_slot_p_stride_.RawData(), cuda_slot_col_in_p_.RawData(), num_compact_cols, - num_data); + num_data, + gather_src_is_4bit); CUDASUCCESS_OR_FATAL(cudaDeviceSynchronize()); // Update CUDAColumnData's data_by_column_ pointers to slots in compact buffer. @@ -286,6 +395,36 @@ void CUDASingleGPUTreeLearner::BuildCompactColumnView() { compact_col_signature_ = sig; } +void CUDASingleGPUTreeLearner::EnsureClassicColumnView() { + if (!compact_packed_view_active_) { + return; + } + // run the gather that BuildCompactColumnView skipped for the packed split + // read; once per tree at most (the classic view stays registered after) + const data_size_t num_data = cuda_histogram_constructor_->cuda_row_data_internal()->num_data(); + const int num_compact_cols = static_cast(compact_column_to_orig_.size()); + const size_t needed_bytes = static_cast(num_compact_cols) * static_cast(num_data); + if (compact_column_buffer_.Size() < needed_bytes) { + compact_column_buffer_.Resize(needed_bytes); + } + LaunchRowToColCompactKernel( + 0, + compact_gather_src_, + compact_column_buffer_.RawData(), + cuda_slot_p_byte_.RawData(), + cuda_slot_p_stride_.RawData(), + cuda_slot_col_in_p_.RawData(), + num_compact_cols, + num_data, + compact_gather_src_is_4bit_); + CUDASUCCESS_OR_FATAL(cudaDeviceSynchronize()); + CUDAColumnData* col_data = const_cast(train_data_->cuda_column_data()); + col_data->SetCompactColumnView(orig_column_to_compact_slot_, + compact_column_buffer_.RawData(), + static_cast(num_data)); + compact_packed_view_active_ = false; +} + void CUDASingleGPUTreeLearner::AddPredictionToScore(const Tree* tree, double* out_score) const { if (config_->linear_tree && last_tree_is_linear_ && tree->is_linear()) { // Linear tree: score += leaf_const + sum(coeff * raw_feat), NaN -> leaf_output. @@ -306,135 +445,1274 @@ void CUDASingleGPUTreeLearner::AddPredictionToScore(const Tree* tree, double* ou cuda_data_partition_->UpdateTrainScore(tree, out_score); } -Tree* CUDASingleGPUTreeLearner::Train(const score_t* gradients, - const score_t* hessians, bool is_first_tree) { - gradients_ = gradients; - hessians_ = hessians; - global_timer.Start("CUDASingleGPUTreeLearner::BeforeTrain"); - BeforeTrain(); - global_timer.Stop("CUDASingleGPUTreeLearner::BeforeTrain"); - // linear trees need per-leaf branch features to know which features each leaf's - // linear model uses (mirrors the CPU LinearTreeLearner using new Tree(.., true, true)). - const bool track_branch_features = !(config_->interaction_constraints_vector.empty()) || config_->linear_tree; - std::unique_ptr tree(new CUDATree(config_->num_leaves, track_branch_features, - config_->linear_tree, gpu_device_id_, has_categorical_feature_)); - // set the root value by hand, as it is not handled by splits +namespace { + +// EXABOOST_HYBRID_AGGRESSIVE=1 opts into plain level batching in budget-limited +// configs, accepting the approximate (breadth-biased) growth policy in exchange +// for hybrid speed. Off by default; exact for depth-limited configs either way. +bool HybridAggressiveEnv() { + static const bool aggressive = + std::getenv("EXABOOST_HYBRID_AGGRESSIVE") != nullptr && + std::string(std::getenv("EXABOOST_HYBRID_AGGRESSIVE")) == std::string("1"); + return aggressive; +} + +} // anonymous namespace + +bool CUDASingleGPUTreeLearner::HybridGrowthUsable() const { + // Plain level batching is exactly leaf-wise-equivalent only in the + // depth-limited regime (2^max_depth <= num_leaves + 1): there the budget can + // only bind at the final (max-depth) level, where the batched top-K-by-gain + // selection is optimal because no deeper candidates can exist. In + // budget-limited configs (num_leaves << 2^max_depth) the SELECTIVE + // grow-then-prune mode (UseSelectiveGrowth) provides exact leaf-wise + // equivalence; where it does not apply, EXABOOST_HYBRID_AGGRESSIVE=1 opts + // into the approximate (breadth-biased) plain batching, and otherwise the + // classic one-split-at-a-time loop runs. + const bool depth_limited = config_->max_depth > 0 && config_->max_depth < 31 && + (1LL << config_->max_depth) <= static_cast(config_->num_leaves) + 1; + const bool base = use_hybrid_growth_ && + nccl_communicator_ == nullptr && + !has_categorical_feature_ && + !select_features_by_node_ && + (forced_split_json_ == nullptr || forced_split_json_->is_null()) && + config_->num_leaves > 2; + if (!base) { + return false; + } + return depth_limited || HybridAggressiveEnv() || UseSelectiveGrowth(); +} + +bool CUDASingleGPUTreeLearner::UseSelectiveGrowth() const { + const bool depth_limited = config_->max_depth > 0 && config_->max_depth < 31 && + (1LL << config_->max_depth) <= static_cast(config_->num_leaves) + 1; + if (depth_limited || HybridAggressiveEnv()) { + // depth-limited configs keep the (bit-exact-locked) exact-fit prefix; + // AGGRESSIVE explicitly selects the approximate plain batching + return false; + } + // Selective growth needs the batched apply phase (host-assigned, recyclable + // right-child leaf indices) and is only the default when both batching env + // switches are untouched: any explicit fallback env reproduces the previous + // behavior (classic loop for budget-limited configs). Linear trees renumber + // leaves through their own device paths, so they keep the classic loop. + return use_hybrid_selective_ && + use_hybrid_batch_apply_ && + use_hybrid_batch_kernels_ && + !config_->linear_tree && + nccl_communicator_ == nullptr && + !has_categorical_feature_ && + !select_features_by_node_ && + (forced_split_json_ == nullptr || forced_split_json_->is_null()) && + config_->num_leaves > 2; +} + +bool CUDASingleGPUTreeLearner::UseOneSyncPrefix() const { + return use_hybrid_one_sync_ && + !config_->use_quantized_grad && + use_hybrid_batch_kernels_ && + use_hybrid_batch_apply_ && + cuda_histogram_constructor_->SupportsBatchedLevel() && + cuda_best_split_finder_->SupportsBatchedLevel(); +} + +void CUDASingleGPUTreeLearner::EnsureRootSumsReadBack(CUDATree* tree) { + if (!root_sums_deferred_) { + return; + } + root_sums_deferred_ = false; + cuda_smaller_leaf_splits_->CopyRootSumsToHost(&leaf_sum_gradients_[0], &leaf_sum_hessians_[0]); + // the deferred half of Train()'s root initialization (see there) tree->SetLeafOutput(0, CUDALeafSplits::CalculateSplittedLeafOutput( - leaf_sum_gradients_[smaller_leaf_index_], leaf_sum_hessians_[smaller_leaf_index_], - config_->lambda_l1, config_->lambda_l2, config_->path_smooth, + leaf_sum_gradients_[0], leaf_sum_hessians_[0], + config_->lambda_l1, config_->lambda_l2, config_->path_smooth, static_cast(num_data_), 0)); tree->SyncLeafOutputFromHostToCUDA(); - int num_splits_done = 0; - ForceSplitsCUDA(tree.get(), &num_splits_done); - for (int i = num_splits_done; i < config_->num_leaves - 1; ++i) { - global_timer.Start("CUDASingleGPUTreeLearner::ConstructHistogramForLeaf"); - const data_size_t global_num_data_in_smaller_leaf = nccl_communicator_ != nullptr ? - global_num_data_in_leaf_[smaller_leaf_index_] : - leaf_num_data_[smaller_leaf_index_]; - const data_size_t global_num_data_in_larger_leaf = nccl_communicator_ != nullptr ? - (larger_leaf_index_ < 0 ? 0 : global_num_data_in_leaf_[larger_leaf_index_]) : - (larger_leaf_index_ < 0 ? 0 : leaf_num_data_[larger_leaf_index_]); - const data_size_t num_data_in_smaller_leaf = leaf_num_data_[smaller_leaf_index_]; - const data_size_t num_data_in_larger_leaf = larger_leaf_index_ < 0 ? 0 : leaf_num_data_[larger_leaf_index_]; - const double sum_hessians_in_smaller_leaf = leaf_sum_hessians_[smaller_leaf_index_]; - const double sum_hessians_in_larger_leaf = larger_leaf_index_ < 0 ? 0 : leaf_sum_hessians_[larger_leaf_index_]; - const uint8_t num_bits_in_histogram_bins = config_->use_quantized_grad ? (nccl_communicator_ != nullptr ? - cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_) : - cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_)) : 0; - cuda_histogram_constructor_->ConstructHistogramForLeaf( - cuda_smaller_leaf_splits_->GetCUDAStruct(), - cuda_larger_leaf_splits_->GetCUDAStruct(), - global_num_data_in_smaller_leaf, - global_num_data_in_larger_leaf, - num_data_in_smaller_leaf, - num_data_in_larger_leaf, - sum_hessians_in_smaller_leaf, - sum_hessians_in_larger_leaf, - num_bits_in_histogram_bins); - global_timer.Stop("CUDASingleGPUTreeLearner::ConstructHistogramForLeaf"); +} - global_timer.Start("CUDASingleGPUTreeLearner::NCCLReduceHistogram"); - if (nccl_communicator_ != nullptr) { - NCCLReduceHistogram(); +void CUDASingleGPUTreeLearner::EnqueuePairBestSplitSearch(const CUDATree* tree, + const CUDALeafSplitsStruct* smaller_struct, const CUDALeafSplitsStruct* larger_struct, + const int smaller_leaf_index, const int larger_leaf_index, const bool synchronize, + const int pipeline, const int parent_leaf_index_arg) { + cuda_histogram_constructor_->SetActivePipeline(pipeline); + cuda_best_split_finder_->SetActiveHistPipeline(pipeline); + global_timer.Start("CUDASingleGPUTreeLearner::ConstructHistogramForLeaf"); + const data_size_t global_num_data_in_smaller_leaf = nccl_communicator_ != nullptr ? + global_num_data_in_leaf_[smaller_leaf_index] : + leaf_num_data_[smaller_leaf_index]; + const data_size_t global_num_data_in_larger_leaf = nccl_communicator_ != nullptr ? + (larger_leaf_index < 0 ? 0 : global_num_data_in_leaf_[larger_leaf_index]) : + (larger_leaf_index < 0 ? 0 : leaf_num_data_[larger_leaf_index]); + const data_size_t num_data_in_smaller_leaf = leaf_num_data_[smaller_leaf_index]; + const data_size_t num_data_in_larger_leaf = larger_leaf_index < 0 ? 0 : leaf_num_data_[larger_leaf_index]; + const double sum_hessians_in_smaller_leaf = leaf_sum_hessians_[smaller_leaf_index]; + const double sum_hessians_in_larger_leaf = larger_leaf_index < 0 ? 0 : leaf_sum_hessians_[larger_leaf_index]; + const uint8_t num_bits_in_histogram_bins = config_->use_quantized_grad ? (nccl_communicator_ != nullptr ? + cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index) : + cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index)) : 0; + cuda_histogram_constructor_->ConstructHistogramForLeaf( + smaller_struct, + larger_struct, + global_num_data_in_smaller_leaf, + global_num_data_in_larger_leaf, + num_data_in_smaller_leaf, + num_data_in_larger_leaf, + sum_hessians_in_smaller_leaf, + sum_hessians_in_larger_leaf, + num_bits_in_histogram_bins); + global_timer.Stop("CUDASingleGPUTreeLearner::ConstructHistogramForLeaf"); + + global_timer.Start("CUDASingleGPUTreeLearner::NCCLReduceHistogram"); + if (nccl_communicator_ != nullptr) { + NCCLReduceHistogram(); + } + global_timer.Stop("CUDASingleGPUTreeLearner::NCCLReduceHistogram"); + + global_timer.Start("CUDASingleGPUTreeLearner::FindBestSplitsForLeaf"); + uint8_t parent_num_bits_bin = 0; + uint8_t smaller_num_bits_bin = 0; + uint8_t larger_num_bits_bin = 0; + if (config_->use_quantized_grad) { + if (larger_leaf_index != -1) { + const int parent_leaf_index = parent_leaf_index_arg >= 0 ? + parent_leaf_index_arg : std::min(smaller_leaf_index, larger_leaf_index); + if (nccl_communicator_ != nullptr) { + parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInNode(parent_leaf_index); + smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index); + larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index); + } else { + parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInNode(parent_leaf_index); + smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index); + larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index); + } + } else { + if (nccl_communicator_ != nullptr) { + parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + } else { + parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + } } - global_timer.Stop("CUDASingleGPUTreeLearner::NCCLReduceHistogram"); + } + cuda_histogram_constructor_->SubtractHistogramForLeaf( + smaller_struct, + larger_struct, + config_->use_quantized_grad, + parent_num_bits_bin, + smaller_num_bits_bin, + larger_num_bits_bin); + + SelectFeatureByNode(tree); + + if (config_->use_quantized_grad) { + const uint8_t smaller_leaf_num_bits_bin = nccl_communicator_ == nullptr ? + cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index) : + cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index); + const uint8_t larger_leaf_num_bits_bin = larger_leaf_index < 0 ? 32 : (nccl_communicator_ == nullptr ? + cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index) : + cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index)); + cuda_best_split_finder_->FindBestSplitsForLeaf( + smaller_struct, + larger_struct, + smaller_leaf_index, larger_leaf_index, + global_num_data_in_smaller_leaf, global_num_data_in_larger_leaf, + sum_hessians_in_smaller_leaf, sum_hessians_in_larger_leaf, + cuda_gradient_discretizer_->grad_scale_ptr(), + cuda_gradient_discretizer_->hess_scale_ptr(), + smaller_leaf_num_bits_bin, + larger_leaf_num_bits_bin, + config_->max_depth <= 0 || GrowthLeafDepth(tree, smaller_leaf_index) < config_->max_depth, + larger_leaf_index < 0 || config_->max_depth <= 0 || + GrowthLeafDepth(tree, larger_leaf_index) < config_->max_depth, + synchronize); + } else { + cuda_best_split_finder_->FindBestSplitsForLeaf( + smaller_struct, + larger_struct, + smaller_leaf_index, larger_leaf_index, + global_num_data_in_smaller_leaf, global_num_data_in_larger_leaf, + sum_hessians_in_smaller_leaf, sum_hessians_in_larger_leaf, + nullptr, nullptr, 0, 0, + config_->max_depth <= 0 || GrowthLeafDepth(tree, smaller_leaf_index) < config_->max_depth, + larger_leaf_index < 0 || config_->max_depth <= 0 || + GrowthLeafDepth(tree, larger_leaf_index) < config_->max_depth, + synchronize); + } + global_timer.Stop("CUDASingleGPUTreeLearner::FindBestSplitsForLeaf"); + cuda_histogram_constructor_->SetActivePipeline(0); + cuda_best_split_finder_->SetActiveHistPipeline(0); +} - global_timer.Start("CUDASingleGPUTreeLearner::FindBestSplitsForLeaf"); - uint8_t parent_num_bits_bin = 0; - uint8_t smaller_num_bits_bin = 0; - uint8_t larger_num_bits_bin = 0; +void CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearch(const CUDATree* tree, + const std::vector& pairs) { + const int num_pairs = static_cast(pairs.size()); + if (num_pairs <= 0) { + return; + } + global_timer.Start("CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearch"); + host_hybrid_pair_descs_.resize(static_cast(num_pairs)); + data_size_t max_num_data_in_smaller_leaf = 0; + bool any_bit_change_copy = false; + for (int i = 0; i < num_pairs; ++i) { + const HybridPendingPair& pair = pairs[i]; + CUDAHybridPairDescriptor& desc = host_hybrid_pair_descs_[i]; + desc.smaller_struct = pair.smaller_struct; + desc.larger_struct = pair.larger_struct; + desc.smaller_leaf_index = pair.smaller; + desc.larger_leaf_index = pair.larger; + const data_size_t num_data_in_smaller_leaf = leaf_num_data_[pair.smaller]; + const data_size_t num_data_in_larger_leaf = pair.larger < 0 ? 0 : leaf_num_data_[pair.larger]; + const double sum_hessians_in_smaller_leaf = leaf_sum_hessians_[pair.smaller]; + const double sum_hessians_in_larger_leaf = pair.larger < 0 ? 0.0 : leaf_sum_hessians_[pair.larger]; + desc.num_data_in_smaller_leaf = num_data_in_smaller_leaf; + desc.num_data_in_larger_leaf = num_data_in_larger_leaf; + // mirror of ConstructHistogramForLeaf's min_data/min_hessian early return + desc.construct_valid = + ((num_data_in_smaller_leaf <= config_->min_data_in_leaf || + sum_hessians_in_smaller_leaf <= config_->min_sum_hessian_in_leaf) && + (num_data_in_larger_leaf <= config_->min_data_in_leaf || + sum_hessians_in_larger_leaf <= config_->min_sum_hessian_in_leaf)) ? 0 : 1; + // mirror of CUDABestSplitFinder::FindBestSplitsForLeaf's leaf validity checks + const bool smaller_below_max_depth = + config_->max_depth <= 0 || GrowthLeafDepth(tree, pair.smaller) < config_->max_depth; + const bool larger_below_max_depth = pair.larger < 0 || + config_->max_depth <= 0 || GrowthLeafDepth(tree, pair.larger) < config_->max_depth; + desc.smaller_valid = (num_data_in_smaller_leaf > config_->min_data_in_leaf && + sum_hessians_in_smaller_leaf > config_->min_sum_hessian_in_leaf && + smaller_below_max_depth) ? 1 : 0; + desc.larger_valid = (pair.larger >= 0 && + num_data_in_larger_leaf > config_->min_data_in_leaf && + sum_hessians_in_larger_leaf > config_->min_sum_hessian_in_leaf && + larger_below_max_depth) ? 1 : 0; if (config_->use_quantized_grad) { - if (larger_leaf_index_ != -1) { - const int parent_leaf_index = std::min(smaller_leaf_index_, larger_leaf_index_); - if (nccl_communicator_ != nullptr) { - parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInNode(parent_leaf_index); - smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_); - larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index_); - } else { - parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInNode(parent_leaf_index); - smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_); - larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index_); + if (pair.larger >= 0) { + // pair.parent == the split's left leaf == min(smaller, larger) unless + // selective growth recycled the right-child index + const int parent_leaf_index = pair.parent >= 0 ? + pair.parent : std::min(pair.smaller, pair.larger); + desc.parent_num_bits = cuda_gradient_discretizer_->GetHistBitsInNode(parent_leaf_index); + desc.smaller_num_bits = cuda_gradient_discretizer_->GetHistBitsInLeaf(pair.smaller); + desc.larger_num_bits = cuda_gradient_discretizer_->GetHistBitsInLeaf(pair.larger); + if (desc.parent_num_bits > 16 && desc.larger_num_bits <= 16) { + any_bit_change_copy = true; } } else { - if (nccl_communicator_ != nullptr) { - parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); - smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); - larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + desc.parent_num_bits = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + desc.smaller_num_bits = desc.parent_num_bits; + desc.larger_num_bits = desc.parent_num_bits; + } + } else { + desc.parent_num_bits = 0; + desc.smaller_num_bits = 0; + desc.larger_num_bits = 0; + } + if (num_data_in_smaller_leaf > max_num_data_in_smaller_leaf) { + max_num_data_in_smaller_leaf = num_data_in_smaller_leaf; + } + } + if (cuda_hybrid_pair_descs_.Size() < static_cast(num_pairs)) { + // preallocate for the deepest possible level so this resizes at most once + cuda_hybrid_pair_descs_.Resize(static_cast( + std::max(num_pairs, config_->num_leaves / 2 + 2))); + } + // single async H2D per level on the histogram stream: ordered before the + // construct kernel on the same stream, and before the find/sync kernels via + // subtract_done_events_[0]. The host staging buffer is only rewritten after + // the next FinishSplitBatch/SyncAllLeafBestSplitsToHost full sync, and a + // pageable async H2D returns only once the data is staged, so reuse is safe. + CopyFromHostToCUDADeviceAsync( + cuda_hybrid_pair_descs_.RawData(), host_hybrid_pair_descs_.data(), + static_cast(num_pairs), cuda_histogram_constructor_->hist_stream(), + __FILE__, __LINE__); + cuda_histogram_constructor_->ConstructHistogramsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), num_pairs, + max_num_data_in_smaller_leaf, any_bit_change_copy); + cuda_best_split_finder_->FindBestSplitsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), num_pairs, + config_->use_quantized_grad ? cuda_gradient_discretizer_->grad_scale_ptr() : nullptr, + config_->use_quantized_grad ? cuda_gradient_discretizer_->hess_scale_ptr() : nullptr); + global_timer.Stop("CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearch"); +} + +void CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearchSpeculative(const CUDATree* tree, + const std::vector& applied) { + const int num_pairs = static_cast(applied.size()); + if (num_pairs <= 0) { + return; + } + global_timer.Start("CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearch"); + host_hybrid_pair_descs_.resize(static_cast(num_pairs)); + // the level's exact smaller-child sizes are not host-known yet (the apply's + // readback is deferred); bound the construct grid by the parents' sizes + // (smaller child <= floor(parent / 2)) -- the kernel restores the exact row + // grouping from the device-side actual sizes written by the apply kernels + data_size_t max_smaller_leaf_bound = 0; + for (int i = 0; i < num_pairs; ++i) { + const HybridAppliedSplit& a = applied[i]; + CUDAHybridPairDescriptor& desc = host_hybrid_pair_descs_[i]; + desc.smaller_struct = a.smaller_slot; + desc.larger_struct = a.larger_slot; + // informational only: the batched kernels read the leaf indices from the + // structs (the smaller/larger designation is decided on-device) + desc.smaller_leaf_index = a.left; + desc.larger_leaf_index = a.right; + // derived on-device from the child structs + desc.num_data_in_smaller_leaf = 0; + desc.num_data_in_larger_leaf = 0; + desc.construct_valid = 1; + // both children of a split are siblings at the same (host-known) depth; the + // min_data/min_hessian gates are evaluated on-device + const uint8_t below_max_depth = (config_->max_depth <= 0 || + GrowthLeafDepth(tree, a.left) < config_->max_depth) ? 1 : 0; + desc.smaller_valid = below_max_depth; + desc.larger_valid = below_max_depth; + // single-sync flow is non-quantized only + desc.parent_num_bits = 0; + desc.smaller_num_bits = 0; + desc.larger_num_bits = 0; + // leaf_num_data_[a.left] still holds the PARENT's size (its readback-based + // update is deferred to the next level's combined readback) + const data_size_t smaller_child_bound = leaf_num_data_[a.left] / 2; + if (smaller_child_bound > max_smaller_leaf_bound) { + max_smaller_leaf_bound = smaller_child_bound; + } + } + if (cuda_hybrid_pair_descs_.Size() < static_cast(num_pairs)) { + cuda_hybrid_pair_descs_.Resize(static_cast( + std::max(num_pairs, config_->num_leaves / 2 + 2))); + } + CopyFromHostToCUDADeviceAsync( + cuda_hybrid_pair_descs_.RawData(), host_hybrid_pair_descs_.data(), + static_cast(num_pairs), cuda_histogram_constructor_->hist_stream(), + __FILE__, __LINE__); + // order the whole speculative search phase after this level's batched apply + // kernels: they write the child structs (and the smaller-count array) the + // construct/fix/subtract/find/sync kernels read. Everything enqueued before + // the previous level's combined readback has already completed. + CUDASUCCESS_OR_FATAL(cudaStreamWaitEvent( + cuda_histogram_constructor_->hist_stream(), + cuda_data_partition_->apply_done_event(), 0)); + cuda_histogram_constructor_->ConstructHistogramsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), num_pairs, + max_smaller_leaf_bound, /*any_pair_needs_bit_change_copy=*/false, + cuda_data_partition_->level_smaller_leaf_counts()); + cuda_best_split_finder_->FindBestSplitsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), num_pairs, nullptr, nullptr); + global_timer.Stop("CUDASingleGPUTreeLearner::EnqueueLevelBestSplitSearch"); +} + +void CUDASingleGPUTreeLearner::CollectSplittableLeaves(const CUDATree* tree, + std::vector* splittable) { + splittable->clear(); + for (int leaf = 0; leaf < tree->num_leaves(); ++leaf) { + const CUDASplitInfo& info = host_leaf_best_splits_[leaf]; + if (info.is_valid) { + // keep the host-side split arrays consistent with the device cache for + // every candidate leaf: the leaf-wise tail may apply any of them later + leaf_best_split_feature_[leaf] = info.inner_feature_index; + leaf_best_split_threshold_[leaf] = info.threshold; + leaf_best_split_default_left_[leaf] = static_cast(info.default_left); + splittable->push_back(leaf); + } + } + static const bool hybrid_debug = std::getenv("EXABOOST_HYBRID_DEBUG") != nullptr; + if (hybrid_debug) { + double max_gain = -1e300, min_gain = 1e300; + for (const int leaf : *splittable) { + max_gain = std::max(max_gain, host_leaf_best_splits_[leaf].gain); + min_gain = std::min(min_gain, host_leaf_best_splits_[leaf].gain); + } + Log::Warning("[hybrid] leaves=%d splittable=%d gain=[%g, %g]", + tree->num_leaves(), static_cast(splittable->size()), min_gain, max_gain); + } +} + +bool CUDASingleGPUTreeLearner::ArbitrateLevelBudget(const CUDATree* tree, + std::vector* splittable, bool* final_partial_level) { + // batch the level whenever it fits in the leaf budget. If the finished tree + // never exhausts num_leaves the result is exactly the leaf-wise tree (split + // decisions are node-local); only when the budget binds does the final level + // defer to the leaf-wise tail, which selects among the cached candidates in + // exact best-gain order. + *final_partial_level = false; + if (tree->num_leaves() + static_cast(splittable->size()) > config_->num_leaves) { + // The budget binds. In general the leaf-wise tail must arbitrate between + // this level's candidates and their future children, so defer to it. But + // when every child this level could create would sit at max_depth, no + // child can ever become a candidate itself (FindBestSplitsForLeaf marks + // leaves at max_depth invalid) and splitting one frontier leaf leaves the + // cached best splits of the other candidates untouched: the tail would + // simply apply the top-(remaining budget) candidates in descending gain + // order, one full search-partition-sync round trip per split. Reproduce + // exactly that as one final batched partial level. + const int remaining_budget = config_->num_leaves - tree->num_leaves(); + bool children_depth_capped = config_->max_depth > 0; + if (children_depth_capped) { + for (const int leaf : *splittable) { + if (tree->leaf_depth(leaf) + 1 < config_->max_depth) { + children_depth_capped = false; + break; + } + } + } + if (!children_depth_capped || remaining_budget <= 0) { + return false; + } + // descending gain; ties go to the lower leaf index (splittable is leaf- + // ascending and the sort is stable), matching FindBestFromAllSplitsKernel's + // strict-greater selection. Applying in this order reassigns right-child + // leaf indices exactly as the tail's sequential splits would. + std::stable_sort(splittable->begin(), splittable->end(), + [this](const int a, const int b) { + return host_leaf_best_splits_[a].gain > host_leaf_best_splits_[b].gain; + }); + splittable->resize(static_cast(remaining_budget)); + *final_partial_level = true; + } + // debug: cap splits per level to isolate multi-pair interactions + static const char* max_splits_env = std::getenv("EXABOOST_HYBRID_MAXSPLITS"); + if (max_splits_env != nullptr) { + const size_t cap = static_cast(std::atoi(max_splits_env)); + if (splittable->size() > cap) splittable->resize(cap); + } + return true; +} + +void CUDASingleGPUTreeLearner::ApplyLevelBatched(CUDATree* tree, + const std::vector& splittable, std::vector* applied) { + // one launch per kernel family for the whole level: batched tree-structure + // update (CUDATree::SplitBatch) + batched partition (SplitLevelBatched); + // right children take consecutive leaf indices exactly as the per-split + // loop would assign them + applied->clear(); + applied->reserve(splittable.size()); + const int base_num_leaves = tree->num_leaves(); + host_tree_batch_splits_.clear(); + host_apply_split_inputs_.clear(); + size_t slot_id = 0; + for (const int leaf : splittable) { + CUDALeafSplitsStruct* smaller_slot = hybrid_pair_slots_.RawData() + slot_id; + CUDALeafSplitsStruct* larger_slot = hybrid_pair_slots_.RawData() + slot_id + 1; + const int right_leaf_index = base_num_leaves + static_cast(applied->size()); + const CUDASplitInfo* best_split_info = cuda_best_split_finder_->leaf_best_split_info_ptr(leaf); + const int inner_feature_index = leaf_best_split_feature_[leaf]; + host_tree_batch_splits_.push_back({ + leaf, + right_leaf_index, // == tree num_leaves at the time of this split + train_data_->RealFeatureIndex(inner_feature_index), + train_data_->RealThreshold(inner_feature_index, leaf_best_split_threshold_[leaf]), + static_cast(train_data_->FeatureBinMapper(inner_feature_index)->missing_type()), + best_split_info}); + host_apply_split_inputs_.push_back({ + best_split_info, + leaf, + right_leaf_index, + inner_feature_index, + leaf_best_split_threshold_[leaf], + leaf_best_split_default_left_[leaf], + leaf_num_data_[leaf], + leaf_data_start_[leaf], + smaller_slot, + larger_slot}); + applied->push_back({leaf, right_leaf_index, smaller_slot, larger_slot}); + slot_id += 2; + } + // ZeroHistForLeaf is a no-op (BeforeTrain zeroes the full histogram + // buffer), so the batched path skips the per-split calls entirely + tree->SplitBatch(host_tree_batch_splits_); + cuda_data_partition_->SplitLevelBatched(host_apply_split_inputs_); +} + +void CUDASingleGPUTreeLearner::FinishLevelBookkeeping( + const std::vector& applied, + const std::vector& batch_info, + std::vector* next_pairs, + int* num_splits) { + for (size_t k = 0; k < applied.size(); ++k) { + const HybridAppliedSplit& a = applied[k]; + const int* info = batch_info.data() + 18 * k; + const double* dinfo = reinterpret_cast(info + 8); + leaf_num_data_[a.left] = info[1]; + leaf_data_start_[a.left] = info[2]; + leaf_num_data_[a.right] = info[4]; + leaf_data_start_[a.right] = info[5]; + leaf_sum_hessians_[a.left] = dinfo[0]; + leaf_sum_hessians_[a.right] = dinfo[1]; + leaf_sum_gradients_[a.left] = dinfo[2]; + leaf_sum_gradients_[a.right] = dinfo[3]; + const int smaller = leaf_num_data_[a.left] < leaf_num_data_[a.right] ? a.left : a.right; + const int larger = smaller == a.left ? a.right : a.left; + if (config_->use_quantized_grad) { + cuda_gradient_discretizer_->SetNumBitsInHistogramBin( + a.left, a.right, leaf_num_data_[a.left], leaf_num_data_[a.right]); + } + if (next_pairs != nullptr) { + next_pairs->push_back({smaller, larger, a.left, a.smaller_slot, a.larger_slot}); + } + smaller_leaf_index_ = smaller; + larger_leaf_index_ = larger; + ++(*num_splits); + } +} + +int CUDASingleGPUTreeLearner::TrainLevelWisePrefix(CUDATree* tree) { + // two struct slots per split of a level, reused across levels; one device + // slab, sized once (the kernels initialize every slot they use before any + // read, exactly as with the previous individually-allocated structs) + const size_t max_slots = static_cast(config_->num_leaves) + 2; + if (hybrid_pair_slots_.Size() < max_slots) { + hybrid_pair_slots_.Resize(max_slots); + } + // batched per-level kernels: one construct/fix/subtract/find/sync launch covers + // all pairs of a level. Falls back to the per-pair loop when the data layout or + // config is outside the batched kernels' support, or when disabled via env. + const bool use_batched_level_kernels = use_hybrid_batch_kernels_ && + cuda_histogram_constructor_->SupportsBatchedLevel() && + cuda_best_split_finder_->SupportsBatchedLevel(); + static const bool hybrid_diag = std::getenv("EXABOOST_HYBRID_DIAG") != nullptr; + if (hybrid_diag) { + static bool diag_logged = false; + if (!diag_logged) { + diag_logged = true; + fprintf(stderr, "[hybrid-diag] prefix: batched_level_kernels=%d batch_kernels_env=%d one_sync=%d\n", + static_cast(use_batched_level_kernels), + static_cast(use_hybrid_batch_kernels_), + static_cast(UseOneSyncPrefix())); + fprintf(stderr, "[hybrid-diag] %s\n", cuda_histogram_constructor_->BatchedLevelGateDiag().c_str()); + fprintf(stderr, "[hybrid-diag] %s\n", cuda_best_split_finder_->BatchedLevelGateDiag().c_str()); + } + } + // batched apply: all tree-structure + partition updates of a level in one + // launch per kernel family with zero per-split host syncs. The hybrid prefix + // guarantees the preconditions (single GPU, no categorical splits; bagging + // shares the same index array), so this only falls back when disabled by env. + const bool use_batched_level_apply = use_hybrid_batch_apply_; + // single-sync (speculative) level pipeline: requires both batched phases and + // non-quantized training (the quantized path selects histogram kernels host- + // side from per-leaf bit widths, which needs the classic readback ordering). + if (UseOneSyncPrefix()) { + return TrainLevelWisePrefixOneSync(tree); + } + // the classic flow needs the host root sums up front (root descriptor + // validity); a no-op unless BeforeTrain optimistically deferred the readback + EnsureRootSumsReadBack(tree); + // the root "pair" lives in the fixed pair structs, initialized by BeforeTrain + std::vector pairs; + pairs.push_back({smaller_leaf_index_, larger_leaf_index_, /*parent=*/-1, + cuda_smaller_leaf_splits_->GetCUDAStruct(), + cuda_larger_leaf_splits_->GetCUDAStruct()}); + int num_splits = 0; + while (true) { + // enqueue histogram + best-split search for every pair of this level; device + // work only, ordered per pair by the histogram-completion events + if (use_batched_level_kernels) { + EnqueueLevelBestSplitSearch(tree, pairs); + } else { + int pair_counter = 0; + for (const HybridPendingPair& pair : pairs) { + static const bool sync_pairs = std::getenv("EXABOOST_HYBRID_SYNCPAIRS") != nullptr; + EnqueuePairBestSplitSearch(tree, pair.smaller_struct, pair.larger_struct, + pair.smaller, pair.larger, /*synchronize=*/sync_pairs, + pair_counter % CUDAHistogramConstructor::kNumHistPipelines); + ++pair_counter; + } + } + // one device synchronization + one transfer for the whole level + cuda_best_split_finder_->SyncAllLeafBestSplitsToHost(tree->num_leaves(), &host_leaf_best_splits_); + std::vector splittable; + CollectSplittableLeaves(tree, &splittable); + if (splittable.empty()) { + break; + } + bool final_partial_level = false; + if (!ArbitrateLevelBudget(tree, &splittable, &final_partial_level)) { + break; + } + pairs.clear(); + // apply the whole level without any device synchronization: every launch + // parameter is host-known from the previous level, and split info is read + // back once for the whole level below (FinishSplitBatch synchronizes) + std::vector applied; + if (use_batched_level_apply) { + ApplyLevelBatched(tree, splittable, &applied); + } else { + applied.reserve(splittable.size()); + size_t slot_id = 0; + for (const int leaf : splittable) { + CUDALeafSplitsStruct* smaller_slot = hybrid_pair_slots_.RawData() + slot_id; + CUDALeafSplitsStruct* larger_slot = hybrid_pair_slots_.RawData() + slot_id + 1; + const int right_leaf_index = ApplySplit( + tree, cuda_best_split_finder_->leaf_best_split_info_ptr(leaf), leaf, + smaller_slot, larger_slot, static_cast(applied.size())); + applied.push_back({leaf, right_leaf_index, smaller_slot, larger_slot}); + slot_id += 2; + } + } + std::vector batch_info; + cuda_data_partition_->FinishSplitBatch(static_cast(applied.size()), &batch_info); + FinishLevelBookkeeping(applied, batch_info, &pairs, &num_splits); + if (final_partial_level) { + // the tree is full and every child sits at max_depth: nothing is left + // for the leaf-wise tail to search or split + break; + } + } + return num_splits; +} + +void CUDASingleGPUTreeLearner::EnqueueRootLevelSearchOneSync() { + // Root level of the single-sync flow: device-derived gating like every later + // level, so the root-sum readback of InitValues stays deferred and the + // level-0 search enqueue overlaps the per-tree histogram-buffer zeroing. The + // construct grid is exact (the root size is host-known), matching the + // classic sizing bit-for-bit. + host_hybrid_pair_descs_.resize(1); + CUDAHybridPairDescriptor& desc = host_hybrid_pair_descs_[0]; + desc.smaller_struct = cuda_smaller_leaf_splits_->GetCUDAStruct(); + desc.larger_struct = cuda_larger_leaf_splits_->GetCUDAStruct(); + desc.smaller_leaf_index = 0; + desc.larger_leaf_index = -1; + desc.num_data_in_smaller_leaf = leaf_num_data_[0]; + desc.num_data_in_larger_leaf = 0; + desc.construct_valid = 1; // min_data/min_hessian evaluated on-device + // the root sits at depth 0 + desc.smaller_valid = (config_->max_depth <= 0 || 0 < config_->max_depth) ? 1 : 0; + desc.larger_valid = 0; + desc.parent_num_bits = 0; + desc.smaller_num_bits = 0; + desc.larger_num_bits = 0; + if (cuda_hybrid_pair_descs_.Size() < 1) { + cuda_hybrid_pair_descs_.Resize(static_cast(config_->num_leaves / 2 + 2)); + } + CopyFromHostToCUDADeviceAsync( + cuda_hybrid_pair_descs_.RawData(), host_hybrid_pair_descs_.data(), 1, + cuda_histogram_constructor_->hist_stream(), __FILE__, __LINE__); + cuda_histogram_constructor_->ConstructHistogramsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), 1, leaf_num_data_[0], + /*any_pair_needs_bit_change_copy=*/false); + cuda_best_split_finder_->FindBestSplitsForLevel( + cuda_hybrid_pair_descs_.RawDataReadOnly(), 1, nullptr, nullptr); +} + +int CUDASingleGPUTreeLearner::TrainLevelWisePrefixOneSync(CUDATree* tree) { + // Single-sync level pipeline: the level's APPLY and the children's SEARCH are + // enqueued back to back; the search kernels derive per-child validity, sizes + // and hessian sums from the child structs the apply kernels write, so no host + // value is needed in between. Each iteration then reads back the level's + // deferred split info AND the children's best splits with ONE device + // synchronization (the second, deferred-info copy finds the device idle). + EnqueueRootLevelSearchOneSync(); + int num_splits = 0; + std::vector pending; // applied splits awaiting readback + std::vector applied; + std::vector batch_info; + std::vector splittable; + while (true) { + // the ONE device synchronization of the level: the synchronous best-split + // readback drains the previous apply and its children's search + cuda_best_split_finder_->SyncAllLeafBestSplitsToHost(tree->num_leaves(), &host_leaf_best_splits_); + if (!pending.empty()) { + cuda_data_partition_->FinishSplitBatch(static_cast(pending.size()), &batch_info); + FinishLevelBookkeeping(pending, batch_info, nullptr, &num_splits); + pending.clear(); + } + CollectSplittableLeaves(tree, &splittable); + if (splittable.empty()) { + break; + } + bool final_partial_level = false; + if (!ArbitrateLevelBudget(tree, &splittable, &final_partial_level)) { + // leave the budget-bound level to the leaf-wise tail; every current leaf + // already has a valid cached best-split candidate + break; + } + ApplyLevelBatched(tree, splittable, &applied); + if (final_partial_level) { + // the tree is full and every child sits at max_depth: nothing is left to + // search, so read the apply info back immediately and stop + cuda_data_partition_->FinishSplitBatch(static_cast(applied.size()), &batch_info); + FinishLevelBookkeeping(applied, batch_info, nullptr, &num_splits); + break; + } + // speculative: the children's search goes out before their statistics are + // host-known; the next iteration's readback collects both + EnqueueLevelBestSplitSearchSpeculative(tree, applied); + pending.swap(applied); + } + if (num_splits == 0) { + // the root has no valid split: the leaf-wise tail (and a possible stump) + // needs the host root sums after all + EnsureRootSumsReadBack(tree); + } + return num_splits; +} + +// ---- selective (grow-then-prune) hybrid growth ---- + +void CUDASingleGPUTreeLearner::SelectiveReset() { + // hybrid leaf indices are recycled on collapse, so live leaves never exceed + // num_leaves (the greedy selection holds <= num_leaves - 1 splits and fresh + // indices are only allocated when the free list is empty, i.e. when every + // allocated index is live); +2 is head room for the CHECK below + const size_t cap = static_cast(config_->num_leaves) + 2; + sel_applied_.clear(); + sel_frontier_.clear(); + sel_leaf_parent_.assign(cap, -1); + sel_leaf_depth_.assign(cap, 0); + sel_leaf_classic_.assign(cap, 0); + sel_leaf_frontier_.assign(cap, -1); + sel_leaf_live_.assign(cap, 0); + sel_free_leaves_.clear(); + sel_order_.clear(); + sel_order_classic_leaf_.clear(); + sel_level_apply_.clear(); + sel_num_allocated_ = 1; + sel_num_splits_ = 0; + sel_leaf_live_[0] = 1; +} + +void CUDASingleGPUTreeLearner::SelectiveDiscoverFrontier(const std::vector& child_leaves) { + for (const SelectiveFrontier& f : sel_frontier_) { + sel_leaf_frontier_[f.leaf] = -1; + } + sel_frontier_.clear(); + for (const int leaf : child_leaves) { + const CUDASplitInfo& info = host_leaf_best_splits_[leaf]; + if (!info.is_valid) { + continue; // permanently dormant leaf (no positive gain / gates failed) + } + sel_leaf_frontier_[leaf] = static_cast(sel_frontier_.size()); + sel_frontier_.push_back({sel_leaf_parent_[leaf], leaf, info.gain}); + } +} + +int CUDASingleGPUTreeLearner::SelectiveTieBreak(const std::vector& tied_classic_leaves) const { + // FindBestFromAllSplitsKernel runs with 256 threads: thread t scans leaves + // t, t + 256, ... with a strict-greater update (the SMALLEST tied leaf wins + // within a thread), then an 8-warp shuffle-down reduction and a final warp + // reduction combine the threads, each with a strict-greater comparison that + // KEEPS the receiving lane's value on ties. Candidates with smaller gains + // can never displace a max-gain candidate and are always displaced by one, + // so only the tied candidates' lane positions matter: simulate the exact + // reduction over occupied/empty lanes. + constexpr int kThreads = 256; // NUM_THREADS_FIND_BEST_LEAF + std::array thread_winner; + thread_winner.fill(-1); + for (const int leaf : tied_classic_leaves) { + const int t = leaf % kThreads; + if (thread_winner[t] < 0 || leaf < thread_winner[t]) { + thread_winner[t] = leaf; + } + } + // shuffle-down reduction over 32 lanes: lane i takes lane i + offset's value + // only when lane i holds no tied candidate (out-of-warp reads return the + // caller's own value); all lanes update simultaneously from the pre-step state + const auto reduce32 = [](std::array lanes) { + for (int offset = 16; offset > 0; offset >>= 1) { + std::array next = lanes; + for (int i = 0; i + offset < 32; ++i) { + if (lanes[i] < 0 && lanes[i + offset] >= 0) { + next[i] = lanes[i + offset]; + } + } + lanes = next; + } + return lanes[0]; + }; + std::array lanes; + std::array warp_winner_lanes; + warp_winner_lanes.fill(-1); + for (int w = 0; w < kThreads / 32; ++w) { + for (int l = 0; l < 32; ++l) { + lanes[l] = thread_winner[w * 32 + l]; + } + warp_winner_lanes[w] = reduce32(lanes); + } + const int winner = reduce32(warp_winner_lanes); + CHECK_GE(winner, 0); + return winner; +} + +int CUDASingleGPUTreeLearner::RunSelectiveLevel() { + const int budget = config_->num_leaves - 1; + // ---- 1. greedy simulation over live applied splits + the current frontier. + // Exactly the classic leaf-wise selection: repeatedly take the max-gain + // candidate whose parent split is already selected, until the budget binds or + // no candidate remains. Ties go to the smaller CLASSIC leaf index, matching + // FindBestFromAllSplitsKernel's reduction (exact for <= 256 concurrent + // leaves; equal double gains across leaves are otherwise vanishingly rare). + // Classic leaf indices are assigned along the way: the t-th selected split + // keeps its leaf index for the left child and hands index t + 1 to the right + // child, exactly as the classic loop numbers them. + for (SelectiveApplied& node : sel_applied_) { + node.selected = false; + } + sel_order_.clear(); + sel_order_classic_leaf_.clear(); + struct HeapEntry { + double gain; + int classic_leaf; + int id; // >= 0: applied record id; < 0: ~frontier index + }; + const auto worse = [](const HeapEntry& a, const HeapEntry& b) { + return a.gain < b.gain || (a.gain == b.gain && a.classic_leaf > b.classic_leaf); + }; + std::priority_queue, decltype(worse)> heap(worse); + sel_leaf_classic_[0] = 0; + if (!sel_applied_.empty()) { + // record 0 is the root split (first pop of the first simulation); the root + // is always available first, so it can never be displaced + heap.push({sel_applied_[0].gain, 0, 0}); + } else if (!sel_frontier_.empty()) { + heap.push({sel_frontier_[0].gain, 0, ~0}); // the root candidate (leaf 0) + } + const auto push_child = [this, &heap](const SelectiveApplied& node, const bool left, + const int classic_leaf) { + const int kid = left ? node.kid_left : node.kid_right; + if (kid >= 0) { + heap.push({sel_applied_[kid].gain, classic_leaf, kid}); + return; + } + const int leaf = left ? node.left_leaf : node.right_leaf; + const int fidx = sel_leaf_frontier_[leaf]; + if (fidx >= 0) { + heap.push({sel_frontier_[fidx].gain, classic_leaf, ~fidx}); + } + }; + std::vector tied; + while (static_cast(sel_order_.size()) < budget && !heap.empty()) { + HeapEntry e = heap.top(); + heap.pop(); + if (!heap.empty() && heap.top().gain == e.gain) { + // equal-gain candidates: resolve with the exact device tie-break + tied.clear(); + tied.push_back(e); + while (!heap.empty() && heap.top().gain == e.gain) { + tied.push_back(heap.top()); + heap.pop(); + } + std::vector tied_leaves; + tied_leaves.reserve(tied.size()); + for (const HeapEntry& c : tied) { + tied_leaves.push_back(c.classic_leaf); + } + const int winner_leaf = SelectiveTieBreak(tied_leaves); + for (const HeapEntry& c : tied) { + if (c.classic_leaf == winner_leaf) { + e = c; } else { - parent_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); - smaller_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); - larger_num_bits_bin = cuda_gradient_discretizer_->GetHistBitsInLeaf(0); + heap.push(c); } } + } + const int t = static_cast(sel_order_.size()); + sel_order_.push_back(e.id); + sel_order_classic_leaf_.push_back(e.classic_leaf); + if (e.id >= 0) { + SelectiveApplied& node = sel_applied_[e.id]; + node.selected = true; + sel_leaf_classic_[node.left_leaf] = e.classic_leaf; + sel_leaf_classic_[node.right_leaf] = t + 1; + push_child(node, true, e.classic_leaf); + push_child(node, false, t + 1); + } + // selected frontier candidates have no discovered children yet + } + + // ---- 2. eager collapse of displaced applied splits. By monotonicity a + // displaced split can never re-enter the selection, and by availability + // closure its whole subtree is displaced with it: revert the topmost + // displaced node to a leaf (its left-child index), rewrite the window's + // data-index-to-leaf-index entries to it, and recycle every right-child leaf + // index (and histogram slot) of the subtree. Children ids are always larger + // than their parent's, so the ascending scan visits topmost nodes first and + // the dead flag skips their (already collapsed) descendants. + sel_collapse_windows_.clear(); + sel_freed_hist_slots_.clear(); + std::vector stack; + int num_displaced = 0; + for (int id = 0; id < static_cast(sel_applied_.size()); ++id) { + SelectiveApplied& node = sel_applied_[id]; + if (node.dead || node.selected) { + continue; + } + // detach from the (still selected) parent's child slot + if (node.parent >= 0) { + SelectiveApplied& p = sel_applied_[node.parent]; + if (p.kid_left == id) { + p.kid_left = -1; + } else if (p.kid_right == id) { + p.kid_right = -1; + } + } + stack.clear(); + stack.push_back(id); + while (!stack.empty()) { + const int m_id = stack.back(); + stack.pop_back(); + SelectiveApplied& m = sel_applied_[m_id]; + m.dead = true; + ++num_displaced; + // the right-child leaf index (and its histogram slot, which is always + // cuda_hist + 2 * right_leaf * num_total_bin) becomes reusable; the slot + // must be re-zeroed (construct kernels accumulate into assumed-zero slots) + sel_leaf_live_[m.right_leaf] = 0; + sel_leaf_parent_[m.right_leaf] = -1; + sel_free_leaves_.push_back(m.right_leaf); + sel_freed_hist_slots_.push_back(m.right_leaf); + if (m.kid_left >= 0) { + stack.push_back(m.kid_left); + } + if (m.kid_right >= 0) { + stack.push_back(m.kid_right); + } + } + // the collapsed node reverts to a (permanently dormant) leaf at its + // left-child index, with its pre-split window + sel_leaf_parent_[node.left_leaf] = node.parent; + leaf_num_data_[node.left_leaf] = node.num_data; + leaf_data_start_[node.left_leaf] = node.data_start; + sel_collapse_windows_.push_back({node.data_start, node.num_data, node.left_leaf}); + } + if (!sel_collapse_windows_.empty()) { + // device work is idle here (post-readback); the rewrite goes on the apply + // stream ahead of this level's batched apply, the slot zeroing on the + // legacy stream ahead of the next construct + cuda_data_partition_->CollapseLeafWindows(sel_collapse_windows_); + cuda_histogram_constructor_->ZeroHistSlots(sel_freed_hist_slots_); + } + + // ---- 3. create the applied records for the selected frontier candidates (in + // greedy order; right-child indices from the free list, else fresh) ---- + sel_level_apply_.clear(); + for (size_t t = 0; t < sel_order_.size(); ++t) { + const int id = sel_order_[t]; + if (id >= 0) { + continue; + } + const SelectiveFrontier& f = sel_frontier_[~id]; + int right = -1; + if (!sel_free_leaves_.empty()) { + right = sel_free_leaves_.back(); + sel_free_leaves_.pop_back(); } else { - parent_num_bits_bin = 0; - smaller_num_bits_bin = 0; - larger_num_bits_bin = 0; + right = sel_num_allocated_++; } - cuda_histogram_constructor_->SubtractHistogramForLeaf( - cuda_smaller_leaf_splits_->GetCUDAStruct(), - cuda_larger_leaf_splits_->GetCUDAStruct(), - config_->use_quantized_grad, - parent_num_bits_bin, - smaller_num_bits_bin, - larger_num_bits_bin); + // invariant: live leaves <= num_leaves, and fresh indices are allocated + // only when every allocated index is live, so this can never fire (the + // host leaf arrays are sized config_->num_leaves) + CHECK_LT(right, config_->num_leaves); + const int new_id = static_cast(sel_applied_.size()); + SelectiveApplied node; + node.parent = f.parent; + node.left_leaf = f.leaf; + node.right_leaf = right; + node.kid_left = -1; + node.kid_right = -1; + node.dead = false; + node.selected = true; + node.gain = f.gain; + node.data_start = leaf_data_start_[f.leaf]; + node.num_data = leaf_num_data_[f.leaf]; + node.info = host_leaf_best_splits_[f.leaf]; + sel_applied_.push_back(node); + if (f.parent >= 0) { + SelectiveApplied& p = sel_applied_[f.parent]; + if (p.left_leaf == f.leaf) { + p.kid_left = new_id; + } else { + p.kid_right = new_id; + } + } + sel_leaf_parent_[f.leaf] = new_id; + sel_leaf_parent_[right] = new_id; + sel_leaf_live_[right] = 1; + sel_leaf_depth_[right] = sel_leaf_depth_[f.leaf] + 1; + ++sel_leaf_depth_[f.leaf]; + sel_level_apply_.push_back(new_id); + } + sel_stat_applied_ += static_cast(sel_level_apply_.size()); + sel_stat_displaced_ += num_displaced; + ++sel_stat_levels_; + return static_cast(sel_level_apply_.size()); +} - SelectFeatureByNode(tree.get()); +void CUDASingleGPUTreeLearner::ApplyLevelBatchedSelective(std::vector* applied) { + applied->clear(); + applied->reserve(sel_level_apply_.size()); + host_apply_split_inputs_.clear(); + size_t slot_id = 0; + for (const int id : sel_level_apply_) { + const SelectiveApplied& node = sel_applied_[id]; + CUDALeafSplitsStruct* smaller_slot = hybrid_pair_slots_.RawData() + slot_id; + CUDALeafSplitsStruct* larger_slot = hybrid_pair_slots_.RawData() + slot_id + 1; + host_apply_split_inputs_.push_back({ + cuda_best_split_finder_->leaf_best_split_info_ptr(node.left_leaf), + node.left_leaf, + node.right_leaf, + node.info.inner_feature_index, + node.info.threshold, + static_cast(node.info.default_left), + node.num_data, + node.data_start, + smaller_slot, + larger_slot}); + applied->push_back({node.left_leaf, node.right_leaf, smaller_slot, larger_slot}); + slot_id += 2; + } + cuda_data_partition_->SplitLevelBatched(host_apply_split_inputs_); +} - if (config_->use_quantized_grad) { - const uint8_t smaller_leaf_num_bits_bin = nccl_communicator_ == nullptr ? - cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_) : - cuda_gradient_discretizer_->GetHistBitsInLeaf(smaller_leaf_index_); - const uint8_t larger_leaf_num_bits_bin = larger_leaf_index_ < 0 ? 32 : (nccl_communicator_ == nullptr ? - cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index_) : - cuda_gradient_discretizer_->GetHistBitsInLeaf(larger_leaf_index_)); - cuda_best_split_finder_->FindBestSplitsForLeaf( - cuda_smaller_leaf_splits_->GetCUDAStruct(), - cuda_larger_leaf_splits_->GetCUDAStruct(), - smaller_leaf_index_, larger_leaf_index_, - global_num_data_in_smaller_leaf, global_num_data_in_larger_leaf, - sum_hessians_in_smaller_leaf, sum_hessians_in_larger_leaf, - cuda_gradient_discretizer_->grad_scale_ptr(), - cuda_gradient_discretizer_->hess_scale_ptr(), - smaller_leaf_num_bits_bin, - larger_leaf_num_bits_bin, - config_->max_depth <= 0 || tree->leaf_depth(smaller_leaf_index_) < config_->max_depth, - larger_leaf_index_ < 0 || config_->max_depth <= 0 || - tree->leaf_depth(larger_leaf_index_) < config_->max_depth); +void CUDASingleGPUTreeLearner::TrainSelectiveOneSync(CUDATree* tree) { + EnqueueRootLevelSearchOneSync(); + std::vector pending; // applied splits awaiting readback + std::vector applied; + std::vector batch_info; + std::vector child_leaves; + while (true) { + // the ONE device synchronization of the level + cuda_best_split_finder_->SyncAllLeafBestSplitsToHost(sel_num_allocated_, &host_leaf_best_splits_); + child_leaves.clear(); + if (!pending.empty()) { + cuda_data_partition_->FinishSplitBatch(static_cast(pending.size()), &batch_info); + FinishLevelBookkeeping(pending, batch_info, nullptr, &sel_num_splits_); + for (const HybridAppliedSplit& a : pending) { + child_leaves.push_back(a.left); + child_leaves.push_back(a.right); + } + pending.clear(); + } else if (sel_applied_.empty()) { + child_leaves.push_back(0); // the root candidate + } + SelectiveDiscoverFrontier(child_leaves); + if (RunSelectiveLevel() == 0) { + // no frontier candidate enters the greedy selection: by monotonicity the + // live applied splits ARE the final leaf-wise selection + break; + } + ApplyLevelBatchedSelective(&applied); + // speculative: the children's search goes out before their statistics are + // host-known; the next iteration's readback collects both + EnqueueLevelBestSplitSearchSpeculative(tree, applied); + pending.swap(applied); + } +} + +void CUDASingleGPUTreeLearner::TrainSelectiveTwoSync(CUDATree* tree) { + // the two-sync flow builds host-known pair descriptors, which need the host + // root sums (a no-op unless BeforeTrain optimistically deferred the readback) + EnsureRootSumsReadBack(tree); + std::vector pairs; + pairs.push_back({smaller_leaf_index_, larger_leaf_index_, /*parent=*/-1, + cuda_smaller_leaf_splits_->GetCUDAStruct(), + cuda_larger_leaf_splits_->GetCUDAStruct()}); + const bool use_batched_level_kernels = use_hybrid_batch_kernels_ && + cuda_histogram_constructor_->SupportsBatchedLevel() && + cuda_best_split_finder_->SupportsBatchedLevel(); + static const bool hybrid_diag = std::getenv("EXABOOST_HYBRID_DIAG") != nullptr; + if (hybrid_diag) { + static bool diag_logged = false; + if (!diag_logged) { + diag_logged = true; + fprintf(stderr, "[hybrid-diag] selective-two-sync: batched_level_kernels=%d batch_kernels_env=%d\n", + static_cast(use_batched_level_kernels), + static_cast(use_hybrid_batch_kernels_)); + fprintf(stderr, "[hybrid-diag] %s\n", cuda_histogram_constructor_->BatchedLevelGateDiag().c_str()); + fprintf(stderr, "[hybrid-diag] %s\n", cuda_best_split_finder_->BatchedLevelGateDiag().c_str()); + } + } + std::vector applied; + std::vector batch_info; + std::vector child_leaves; + while (true) { + if (use_batched_level_kernels) { + EnqueueLevelBestSplitSearch(tree, pairs); + } else { + int pair_counter = 0; + for (const HybridPendingPair& pair : pairs) { + EnqueuePairBestSplitSearch(tree, pair.smaller_struct, pair.larger_struct, + pair.smaller, pair.larger, /*synchronize=*/false, + pair_counter % CUDAHistogramConstructor::kNumHistPipelines, + pair.parent); + ++pair_counter; + } + } + cuda_best_split_finder_->SyncAllLeafBestSplitsToHost(sel_num_allocated_, &host_leaf_best_splits_); + child_leaves.clear(); + for (const HybridPendingPair& pair : pairs) { + child_leaves.push_back(pair.smaller); + if (pair.larger >= 0) { + child_leaves.push_back(pair.larger); + } + } + SelectiveDiscoverFrontier(child_leaves); + if (RunSelectiveLevel() == 0) { + break; + } + ApplyLevelBatchedSelective(&applied); + pairs.clear(); + cuda_data_partition_->FinishSplitBatch(static_cast(applied.size()), &batch_info); + FinishLevelBookkeeping(applied, batch_info, &pairs, &sel_num_splits_); + } +} + +void CUDASingleGPUTreeLearner::SelectiveFinalize(CUDATree* tree) { + // the root output seeds the internal_value_ chain of the rebuilt tree (and is + // the stump value); a no-op unless the one-sync flow deferred the readback + EnsureRootSumsReadBack(tree); + // the final greedy order is the last simulation's order: it selected no + // frontier candidate, hence (by the eager collapse invariant) exactly the + // live applied splits, in classic leaf-wise apply order with classic + // node/leaf numbering + std::vector seq; + seq.reserve(sel_order_.size()); + for (size_t t = 0; t < sel_order_.size(); ++t) { + const int id = sel_order_[t]; + CHECK_GE(id, 0); + const SelectiveApplied& node = sel_applied_[id]; + const int inner_feature = node.info.inner_feature_index; + CUDATreeHostSplit s; + s.leaf_index = sel_order_classic_leaf_[t]; + s.real_feature_index = train_data_->RealFeatureIndex(inner_feature); + s.real_threshold = train_data_->RealThreshold(inner_feature, node.info.threshold); + s.missing_type = static_cast(train_data_->FeatureBinMapper(inner_feature)->missing_type()); + s.inner_feature_index = inner_feature; + s.threshold_in_bin = node.info.threshold; + s.gain = node.info.gain; + s.default_left = node.info.default_left ? 1 : 0; + s.left_sum_hessians = node.info.left_sum_hessians; + s.right_sum_hessians = node.info.right_sum_hessians; + s.left_count = node.info.left_count; + s.right_count = node.info.right_count; + s.left_value = node.info.left_value; + s.right_value = node.info.right_value; + seq.push_back(s); + } + tree->RebuildFromHostSplits(seq); + const int final_num_leaves = tree->num_leaves(); + if (final_num_leaves < config_->num_leaves) { + Log::Warning("No further splits with positive gain, training stopped with %d leaves.", + final_num_leaves); + } + // hybrid -> final leaf index map (pruned subtrees' rows were already rewritten + // to their reverted dormant leaf at collapse time, so every row's entry names + // a live hybrid leaf) + final per-leaf data layout, host and device + std::vector remap(sel_num_allocated_, 0); + std::vector final_num(final_num_leaves, 0); + std::vector final_start(final_num_leaves, 0); + int live_count = 0; + for (int leaf = 0; leaf < sel_num_allocated_; ++leaf) { + if (!sel_leaf_live_[leaf]) { + continue; + } + const int classic = sel_leaf_classic_[leaf]; + remap[leaf] = classic; + final_num[classic] = leaf_num_data_[leaf]; + final_start[classic] = leaf_data_start_[leaf]; + ++live_count; + } + CHECK_EQ(live_count, final_num_leaves); + cuda_data_partition_->RemapDataIndexToLeafIndex(remap); + cuda_data_partition_->SetLeafDataLayout(final_num, final_start, final_num_leaves); + for (int i = 0; i < final_num_leaves; ++i) { + leaf_num_data_[i] = final_num[i]; + leaf_data_start_[i] = final_start[i]; + } + sel_last_peak_ = sel_num_allocated_; + ++sel_stat_trees_; + static const bool hybrid_debug = std::getenv("EXABOOST_HYBRID_DEBUG") != nullptr; + if (hybrid_debug) { + // stderr (not the logger): churn statistics must be visible under verbose=-1 + fprintf(stderr, "[selective] trees=%lld leaves=%d cumulative: applied=%lld displaced=%lld levels=%lld\n", + static_cast(sel_stat_trees_), final_num_leaves, + static_cast(sel_stat_applied_), + static_cast(sel_stat_displaced_), + static_cast(sel_stat_levels_)); + } +} + +void CUDASingleGPUTreeLearner::TrainSelective(CUDATree* tree) { + global_timer.Start("CUDASingleGPUTreeLearner::TrainSelective"); + selective_active_ = true; + SelectiveReset(); + // two struct slots per split of a level (levels hold at most (num_leaves+1)/2 + // splits: a level's splits plus their ancestor chains all fit the selection) + const size_t max_slots = static_cast(config_->num_leaves) + 2; + if (hybrid_pair_slots_.Size() < max_slots) { + hybrid_pair_slots_.Resize(max_slots); + } + if (UseOneSyncPrefix()) { + TrainSelectiveOneSync(tree); + } else { + TrainSelectiveTwoSync(tree); + } + SelectiveFinalize(tree); + selective_active_ = false; + global_timer.Stop("CUDASingleGPUTreeLearner::TrainSelective"); +} + +Tree* CUDASingleGPUTreeLearner::Train(const score_t* gradients, + const score_t* hessians, bool is_first_tree) { + gradients_ = gradients; + hessians_ = hessians; + global_timer.Start("CUDASingleGPUTreeLearner::BeforeTrain"); + BeforeTrain(); + global_timer.Stop("CUDASingleGPUTreeLearner::BeforeTrain"); + // linear trees need per-leaf branch features to know which features each leaf's + // linear model uses (mirrors the CPU LinearTreeLearner using new Tree(.., true, true)). + const bool track_branch_features = !(config_->interaction_constraints_vector.empty()) || config_->linear_tree; + // reusable slab for the per-tree device arrays: one allocation for the whole + // training run instead of ~17 cudaMallocs + ~16 cudaFrees per tree (trees are + // trained one at a time and release the slab in ToHost()); lazily (re)sized so + // ResetConfig num_leaves changes are honored + const size_t tree_pool_bytes = CUDATree::PooledDeviceBufferSize(config_->num_leaves); + if (cuda_tree_pool_buffer_.Size() < tree_pool_bytes) { + cuda_tree_pool_buffer_.Resize(tree_pool_bytes); + } + std::unique_ptr tree(new CUDATree(config_->num_leaves, track_branch_features, + config_->linear_tree, gpu_device_id_, has_categorical_feature_, + cuda_tree_pool_buffer_.RawData())); + // set the root value by hand, as it is not handled by splits. When the root + // sums were deferred (single-sync hybrid growth), this happens lazily in + // EnsureRootSumsReadBack -- the root value only survives when the tree ends + // up a stump (any split overwrites leaf 0's output on device). + if (!root_sums_deferred_) { + tree->SetLeafOutput(0, CUDALeafSplits::CalculateSplittedLeafOutput( + leaf_sum_gradients_[smaller_leaf_index_], leaf_sum_hessians_[smaller_leaf_index_], + config_->lambda_l1, config_->lambda_l2, config_->path_smooth, + static_cast(num_data_), 0)); + tree->SyncLeafOutputFromHostToCUDA(); + } + int num_splits_done = 0; + ForceSplitsCUDA(tree.get(), &num_splits_done); + bool pair_search_cached = false; + bool selective_handled = false; + if (num_splits_done == 0 && HybridGrowthUsable()) { + if (UseSelectiveGrowth()) { + // budget-limited exact grow-then-prune: builds the COMPLETE tree + // (including the final pruned structure); no leaf-wise tail runs + TrainSelective(tree.get()); + selective_handled = true; } else { - cuda_best_split_finder_->FindBestSplitsForLeaf( + global_timer.Start("CUDASingleGPUTreeLearner::TrainLevelWisePrefix"); + num_splits_done = TrainLevelWisePrefix(tree.get()); + global_timer.Stop("CUDASingleGPUTreeLearner::TrainLevelWisePrefix"); + // every current leaf already has a cached best-split candidate; the first + // tail iteration must not redo the pair search + pair_search_cached = num_splits_done > 0; + } + } + for (int i = num_splits_done; !selective_handled && i < config_->num_leaves - 1; ++i) { + if (!pair_search_cached) { + EnqueuePairBestSplitSearch(tree.get(), cuda_smaller_leaf_splits_->GetCUDAStruct(), cuda_larger_leaf_splits_->GetCUDAStruct(), - smaller_leaf_index_, larger_leaf_index_, - global_num_data_in_smaller_leaf, global_num_data_in_larger_leaf, - sum_hessians_in_smaller_leaf, sum_hessians_in_larger_leaf, - nullptr, nullptr, 0, 0, - config_->max_depth <= 0 || tree->leaf_depth(smaller_leaf_index_) < config_->max_depth, - larger_leaf_index_ < 0 || config_->max_depth <= 0 || - tree->leaf_depth(larger_leaf_index_) < config_->max_depth); + smaller_leaf_index_, larger_leaf_index_, /*synchronize=*/true); } - - global_timer.Stop("CUDASingleGPUTreeLearner::FindBestSplitsForLeaf"); + pair_search_cached = false; global_timer.Start("CUDASingleGPUTreeLearner::FindBestFromAllSplits"); const CUDASplitInfo* best_split_info = nullptr; if (larger_leaf_index_ >= 0) { @@ -503,9 +1781,27 @@ Tree* CUDASingleGPUTreeLearner::Train(const score_t* gradients, if (config_->use_quantized_grad && config_->quant_train_renew_leaf) { global_timer.Start("CUDASingleGPUTreeLearner::RenewDiscretizedTreeLeaves"); RenewDiscretizedTreeLeaves(tree.get()); + if (selective_handled) { + // the selective path already finalized the host tree (no ToHost below): + // pull the renewed device leaf values back explicitly + tree->SyncLeafOutputFromCUDAToHost(); + } global_timer.Stop("CUDASingleGPUTreeLearner::RenewDiscretizedTreeLeaves"); } - tree->ToHost(); + if (!selective_handled) { + // the selective path rebuilt the host tree from captured split info and + // released the device arrays in RebuildFromHostSplits already + tree->ToHost(); + } + // only the leaf histogram slots this tree used can be dirty; the next + // BeforeTrain zeroes just that prefix (single-GPU only: the NCCL path keeps + // the conservative full zeroing). Selective growth dirties every hybrid slot + // it ever allocated (recycled slots are re-zeroed on the fly, but the last + // occupants remain), so it reports its allocation peak instead. + if (nccl_communicator_ == nullptr) { + cuda_histogram_constructor_->SetNumDirtyLeaves( + selective_handled ? sel_last_peak_ : tree->num_leaves()); + } last_tree_is_linear_ = false; if (config_->linear_tree) { // gradients_/hessians_ are device pointers after BeforeTrain (the Train args stay on host) @@ -515,7 +1811,12 @@ Tree* CUDASingleGPUTreeLearner::Train(const score_t* gradients, return tree.release(); } -int CUDASingleGPUTreeLearner::ApplySplit(CUDATree* tree, const CUDASplitInfo* best_split_info, const int leaf_index) { +int CUDASingleGPUTreeLearner::ApplySplit(CUDATree* tree, const CUDASplitInfo* best_split_info, const int leaf_index, + CUDALeafSplitsStruct* smaller_slot, CUDALeafSplitsStruct* larger_slot, + int deferred_slot) { + // classic per-split path: needs the plain per-column view (only the batched + // apply kernels understand the packed compact source) + EnsureClassicColumnView(); int right_leaf_index = 0; if (train_data_->FeatureBinMapper(leaf_best_split_feature_[leaf_index])->bin_type() == BinType::CategoricalBin) { right_leaf_index = tree->SplitCategorical(leaf_index, @@ -550,8 +1851,8 @@ int CUDASingleGPUTreeLearner::ApplySplit(CUDATree* tree, const CUDASplitInfo* be leaf_best_split_default_left_[leaf_index], leaf_num_data_[leaf_index], leaf_data_start_[leaf_index], - cuda_smaller_leaf_splits_->GetCUDAStructRef(), - cuda_larger_leaf_splits_->GetCUDAStructRef(), + smaller_slot != nullptr ? smaller_slot : cuda_smaller_leaf_splits_->GetCUDAStructRef(), + larger_slot != nullptr ? larger_slot : cuda_larger_leaf_splits_->GetCUDAStructRef(), &leaf_num_data_[leaf_index], &leaf_num_data_[right_leaf_index], &leaf_data_start_[leaf_index], @@ -561,7 +1862,9 @@ int CUDASingleGPUTreeLearner::ApplySplit(CUDATree* tree, const CUDASplitInfo* be &leaf_sum_gradients_[leaf_index], &leaf_sum_gradients_[right_leaf_index], global_num_data_in_leaf_.data() + leaf_index, - global_num_data_in_leaf_.data() + right_leaf_index); + global_num_data_in_leaf_.data() + right_leaf_index, + /*point_structs_at_main=*/smaller_slot != nullptr, + deferred_slot); #ifdef DEBUG CheckSplitValid(leaf_index, right_leaf_index); #endif // DEBUG diff --git a/src/treelearner/cuda/cuda_single_gpu_tree_learner.hpp b/src/treelearner/cuda/cuda_single_gpu_tree_learner.hpp index 09028c89b065..459af004ff47 100644 --- a/src/treelearner/cuda/cuda_single_gpu_tree_learner.hpp +++ b/src/treelearner/cuda/cuda_single_gpu_tree_learner.hpp @@ -61,6 +61,8 @@ class CUDASingleGPUTreeLearner: public SerialTreeLearner, public NCCLInfo { data_size_t global_num_data) override; protected: + bool IsCUDATreeLearner() const override { return true; } + void BeforeTrain() override; // ---- linear tree support ---- @@ -103,8 +105,194 @@ class CUDASingleGPUTreeLearner: public SerialTreeLearner, public NCCLInfo { // Apply a split (device CUDASplitInfo + host-known feature/threshold) to the tree // and the data partition. Shared by the main loop and the forced-split pre-pass. - // Returns the new (right) leaf index. - int ApplySplit(CUDATree* tree, const CUDASplitInfo* best_split_info, const int leaf_index); + // Returns the new (right) leaf index. When smaller_slot/larger_slot are given the + // data partition writes the children's leaf-splits structs into them instead of the + // fixed pair structs (used by the hybrid level-batched growth phase). + int ApplySplit(CUDATree* tree, const CUDASplitInfo* best_split_info, const int leaf_index, + CUDALeafSplitsStruct* smaller_slot = nullptr, + CUDALeafSplitsStruct* larger_slot = nullptr, + int deferred_slot = -1); + + // ---- hybrid growth: depth-wise (level-batched) prefix + leaf-wise tail ---- + // While the leaf budget cannot bind, every positive-gain frontier leaf will be + // split by leaf-wise growth eventually, so whole levels are grown with one + // device synchronization instead of one full pipeline sync per split. Split + // decisions are node-local, hence the resulting tree equals the leaf-wise tree + // whenever the budget does not bind; the tail preserves exact leaf-wise + // semantics once it might. + bool HybridGrowthUsable() const; + // Whether the hybrid prefix will run the single-sync (speculative) level + // pipeline (see TrainLevelWisePrefixOneSync); requires both batched phases + // and non-quantized training. + bool UseOneSyncPrefix() const; + // Lazily perform the root-sum readback (and the root leaf-output init) that + // BeforeTrain deferred for the single-sync flow. No-op when not deferred. + void EnsureRootSumsReadBack(CUDATree* tree); + // Enqueue histogram construction + best-split search for one sibling pair + // (device work only; no host synchronization). + void EnqueuePairBestSplitSearch(const CUDATree* tree, + const CUDALeafSplitsStruct* smaller_struct, const CUDALeafSplitsStruct* larger_struct, + int smaller_leaf_index, int larger_leaf_index, bool synchronize = true, + int pipeline = 0, int parent_leaf_index = -1); + // One sibling pair of a level awaiting its best-split search. parent is the + // pair's parent leaf index (== the split's left leaf; -1 for the root pair): + // the quantized path stores the parent's histogram bit width at that index. + // Without selective growth right > left always holds, so parent == + // min(smaller, larger); with selective growth right-child indices are + // recycled and the explicit field is required. + struct HybridPendingPair { + int smaller; + int larger; + int parent; + const CUDALeafSplitsStruct* smaller_struct; + const CUDALeafSplitsStruct* larger_struct; + }; + // Batched replacement for the per-pair EnqueuePairBestSplitSearch loop of one + // level: builds per-pair descriptors (validity flags, leaf sizes, quantized + // histogram bit widths), uploads them in a single H2D copy, then runs ONE + // batched construct/fix/subtract launch (histogram constructor) and ONE batched + // find + sync launch (best split finder) covering every pair. Device work only; + // the caller synchronizes via SyncAllLeafBestSplitsToHost. + void EnqueueLevelBestSplitSearch(const CUDATree* tree, + const std::vector& pairs); + // One applied split of a level: host-assigned child leaf indices plus the pair + // struct slots the batched apply kernels fill (smaller/larger designation is + // decided on-device; the host learns it from the deferred readback). + struct HybridAppliedSplit { + int left; + int right; + CUDALeafSplitsStruct* smaller_slot; + CUDALeafSplitsStruct* larger_slot; + }; + // Speculative variant of EnqueueLevelBestSplitSearch used by the single-sync + // flow: enqueued right after the level's batched apply, BEFORE the child + // statistics are read back. Descriptors carry only host-known data (struct slot + // pointers, the max_depth gate); the batched kernels derive leaf indices, + // sizes, hessian sums and the min_data/min_hessian gates from the child structs + // the apply kernels write (ordered via the partition's apply-done event). The + // construct grid is sized by the parents' sizes (smaller child <= parent / 2) + // and the kernel restores the exact row grouping from the device-side actual + // sizes, so results are bit-identical to the classic two-sync flow. + void EnqueueLevelBestSplitSearchSpeculative(const CUDATree* tree, + const std::vector& applied); + // Collect all leaves with a valid cached best-split candidate, mirroring the + // device cache into the host split arrays (the leaf-wise tail may apply any of + // them later). + void CollectSplittableLeaves(const CUDATree* tree, std::vector* splittable); + // Leaf-budget arbitration of one level: returns false when the budget binds + // and the level must defer to the leaf-wise tail; truncates splittable to a + // final partial level (*final_partial_level = true) when every child would sit + // at max_depth. Also applies the EXABOOST_HYBRID_MAXSPLITS debug cap. + bool ArbitrateLevelBudget(const CUDATree* tree, std::vector* splittable, + bool* final_partial_level); + // Batched apply of one level's splits: tree-structure update (SplitBatch) + + // partition (SplitLevelBatched), zero host synchronization; fills *applied. + void ApplyLevelBatched(CUDATree* tree, const std::vector& splittable, + std::vector* applied); + // Shared per-level readback bookkeeping: consume the deferred split info of one + // level's applied splits (child counts/starts/sums), update the host leaf + // arrays, quantized histogram bit widths and smaller/larger designation, count + // the splits, and (when next_pairs is non-null) emit the next level's pairs. + void FinishLevelBookkeeping(const std::vector& applied, + const std::vector& batch_info, + std::vector* next_pairs, + int* num_splits); + // Grow full levels until the next level could exceed num_leaves. Returns the + // number of splits applied; afterwards every current leaf has a valid cached + // best-split candidate and smaller_/larger_leaf_index_ name the last applied pair. + int TrainLevelWisePrefix(CUDATree* tree); + // Single-sync variant of TrainLevelWisePrefix: enqueues each level's APPLY and + // the children's SEARCH back to back, then reads best splits + deferred split + // info back with ONE device synchronization per level (the classic flow pays + // two). Non-quantized batched-kernels + batched-apply configurations only. + int TrainLevelWisePrefixOneSync(CUDATree* tree); + + // ---- selective (grow-then-prune) hybrid growth: exact leaf-wise equivalence + // in budget-limited configs (num_leaves << 2^max_depth) ---- + // + // Leaf-wise growth is greedy selection: repeatedly apply the max-gain + // candidate among AVAILABLE candidates (available = parent split already + // applied) until num_leaves - 1 splits or no positive gain. Key monotonicity + // property: newly discovered (deeper) candidates can only DISPLACE currently + // selected candidates from the greedy selection, never promote unselected + // ones (a new candidate inserts into the greedy sequence right after its + // parent's selection and consumes budget earlier, shifting the tail out). + // Hence, per level: run the greedy selection over all live applied splits + + // the newly discovered frontier candidates, apply ONLY the selected frontier + // candidates (unselected ones are permanently dormant), and eagerly collapse + // applied splits displaced from the selection (their whole subtree is + // displaced by availability closure). At termination the live applied splits + // equal the classic leaf-wise selection, in the classic greedy order given by + // the final simulation; the tree is then rebuilt host-side in that order + // (classic node/leaf numbering => bit-identical model files in deterministic + // configs). Eager collapse recycles hybrid leaf indices (and their histogram + // slots), so live leaves never exceed num_leaves and no device buffer grows. + + // One applied split record of the selective growth phase. + struct SelectiveApplied { + int parent; // id of the parent applied record (-1 for the root split) + int left_leaf; // hybrid leaf that was split (left child keeps it) + int right_leaf; // hybrid right child leaf (recycled index possible) + int kid_left; // applied child record at left_leaf (-1 none) + int kid_right; // applied child record at right_leaf (-1 none) + bool dead; // collapsed (pruned) -- never selectable again + bool selected; // in the current greedy selection (scratch) + double gain; + data_size_t data_start; // the split node's window in the main index array + data_size_t num_data; + // captured host copy of the device CUDASplitInfo at apply time (categorical + // pointers scrubbed by SyncAllLeafBestSplitsToHost) + CUDASplitInfo info; + }; + // One frontier candidate (discovered this level, not yet applied). + struct SelectiveFrontier { + int parent; // applied record that created this leaf (-1 root) + int leaf; // hybrid leaf index + double gain; + }; + + // Whether the budget-limited selective mode governs this tree's growth. + bool UseSelectiveGrowth() const; + // Exact host replica of FindBestFromAllSplitsKernel's equal-gain tie-break: + // the winner among equal-gain candidates is decided by the kernel's + // strided per-thread scan + shuffle-down reductions (keep-current-on-tie), + // NOT by the smallest leaf index. Returns the winning classic leaf index. + int SelectiveTieBreak(const std::vector& tied_classic_leaves) const; + // Enqueue the root level's batched search of the single-sync flow (descriptor + // with device-derived gating, exact host-known construct grid). Shared by the + // exact-fit one-sync prefix and the selective one-sync loop. + void EnqueueRootLevelSearchOneSync(); + // Reset the per-tree selective growth state. + void SelectiveReset(); + // Collect the new frontier candidates from the given child leaves (children of + // last level's applied splits; the root leaf on the first level). + void SelectiveDiscoverFrontier(const std::vector& child_leaves); + // One level's host-side selection step: greedy simulation over live applied + // splits + the current frontier (budget num_leaves - 1, gains descending, + // ties to the smaller classic leaf index, matching FindBestFromAllSplits), + // then eager collapse of displaced applied splits (device window rewrite + + // histogram slot recycling) and creation of the applied records for the + // selected frontier candidates (right-child indices from the free list). + // Returns the number of frontier candidates to apply this level. + int RunSelectiveLevel(); + // Batched partition-only apply of the selected frontier splits (the CUDA tree + // object is not touched during selective growth; the final tree is rebuilt + // host-side after pruning). Fills *applied for the readback bookkeeping. + void ApplyLevelBatchedSelective(std::vector* applied); + // The two growth loops (same level pipelines as the exact-fit prefix). + void TrainSelectiveOneSync(CUDATree* tree); + void TrainSelectiveTwoSync(CUDATree* tree); + // Rebuild the final (pruned) tree host-side in classic greedy order, remap the + // partition's data-index-to-leaf-index to final numbering and upload the final + // per-leaf data layout. + void SelectiveFinalize(CUDATree* tree); + // Full selective training of one tree (grow + prune + finalize). + void TrainSelective(CUDATree* tree); + // Leaf depth during hybrid growth: the CUDA tree's host mirror on the exact-fit + // paths, the selective host array when selective growth is active. + int GrowthLeafDepth(const CUDATree* tree, const int leaf) const { + return selective_active_ ? sel_leaf_depth_[leaf] : tree->leaf_depth(leaf); + } // Apply forcedsplits_filename before the main split loop, mirroring // SerialTreeLearner::ForceSplits (numerical features, single-GPU, non-quantized). @@ -126,6 +314,13 @@ class CUDASingleGPUTreeLearner: public SerialTreeLearner, public NCCLInfo { // a column-major buffer for partition kernels, avoiding the 17 GB per-column // allocation in CUDAColumnData. void BuildCompactColumnView(); + // Lazy fallback of the packed split read (EXABOOST_SPLIT_PACKED_READ): the + // batched apply path reads the packed compact matrix directly, so the + // column-major gather is skipped in BuildCompactColumnView; any classic + // per-split Split() (leaf-wise tail, forced splits, batched-apply fallback + // env) still needs the plain per-column buffer, so ApplySplit calls this + // first to run the deferred gather once for the tree. + void EnsureClassicColumnView(); CUDAVector compact_column_buffer_; std::vector compact_column_to_orig_; // [slot] -> original column index std::vector orig_column_to_compact_slot_; // [col] -> slot, or -1 @@ -137,6 +332,12 @@ class CUDASingleGPUTreeLearner: public SerialTreeLearner, public NCCLInfo { CUDAVector cuda_slot_p_stride_; CUDAVector cuda_slot_col_in_p_; uint64_t compact_col_signature_ = 0; + // packed split read: this tree's column view is the packed compact matrix + // (no column-major gather ran yet; see EnsureClassicColumnView) + bool compact_packed_view_active_ = false; + // gather inputs stashed for the lazy fallback (device pointer + 4-bit flag) + const uint8_t* compact_gather_src_ = nullptr; + bool compact_gather_src_is_4bit_ = false; // number of threads on CPU int num_threads_; @@ -146,6 +347,60 @@ class CUDASingleGPUTreeLearner: public SerialTreeLearner, public NCCLInfo { // leaf splits information for smaller and larger leaves std::unique_ptr cuda_smaller_leaf_splits_; std::unique_ptr cuda_larger_leaf_splits_; + // hybrid growth phase-1 state: per-split (smaller, larger) struct slots for the + // pairs of one level (2 slots per applied split, reused across levels), and the + // host copy of the device per-leaf best-split cache. The slots are one device + // slab of bare CUDALeafSplitsStructs (they are only ever written/read by the + // split kernels through their struct pointers); allocating num_leaves + 2 + // CUDALeafSplits objects instead costs ~3 cudaMallocs each (~3000 calls, + // ~3.7 ms) inside the first Train() call. + bool use_hybrid_growth_ = false; + CUDAVector hybrid_pair_slots_; + std::vector host_leaf_best_splits_; + // hybrid growth: batched per-level kernels (EXABOOST_HYBRID_BATCH_KERNELS, + // default on; "0" falls back to the per-pair kernel launches) + bool use_hybrid_batch_kernels_ = false; + std::vector host_hybrid_pair_descs_; + CUDAVector cuda_hybrid_pair_descs_; + // hybrid growth: batched per-level APPLY phase (EXABOOST_HYBRID_BATCH_APPLY, + // default on; "0" falls back to the per-split deferred ApplySplit loop) + bool use_hybrid_batch_apply_ = false; + std::vector host_tree_batch_splits_; + std::vector host_apply_split_inputs_; + // hybrid growth: single-sync (speculative) level pipeline + // (EXABOOST_HYBRID_ONE_SYNC, default on; "0" keeps the classic two-sync flow) + bool use_hybrid_one_sync_ = false; + // selective (grow-then-prune) hybrid growth for budget-limited configs + // (EXABOOST_HYBRID_SELECTIVE, default on; "0" falls back to the classic loop) + bool use_hybrid_selective_ = false; + bool selective_active_ = false; + std::vector sel_applied_; + std::vector sel_frontier_; + std::vector sel_leaf_parent_; // hybrid leaf -> applied record id (-1 root) + std::vector sel_leaf_depth_; + std::vector sel_leaf_classic_; // classic leaf index (last simulation) + std::vector sel_leaf_frontier_; // hybrid leaf -> frontier idx (per level) + std::vector sel_leaf_live_; + std::vector sel_free_leaves_; // recycled hybrid leaf indices (LIFO) + std::vector sel_order_; // last simulation: greedy order (applied ids, ~frontier idx) + std::vector sel_order_classic_leaf_; + std::vector sel_level_apply_; // applied record ids to apply this level + std::vector sel_collapse_windows_; + std::vector sel_freed_hist_slots_; + int sel_num_allocated_ = 0; // peak hybrid leaf count (dirty hist slots) + int sel_num_splits_ = 0; // applied splits surviving readback bookkeeping + int sel_last_peak_ = 0; // sel_num_allocated_ of the finished tree + // churn statistics (EXABOOST_HYBRID_DEBUG): applied / displaced split totals + int64_t sel_stat_applied_ = 0; + int64_t sel_stat_displaced_ = 0; + int64_t sel_stat_levels_ = 0; + int64_t sel_stat_trees_ = 0; + // reusable device slab for CUDATree's per-tree arrays (see CUDATree ctor) + CUDAVector cuda_tree_pool_buffer_; + // whether BeforeTrain deferred the root-sum readback (single-sync flow); the + // readback (plus the root leaf-output init) then happens lazily in + // EnsureRootSumsReadBack on the paths that need host root sums + bool root_sums_deferred_ = false; // data partition that partitions data indices into different leaves std::unique_ptr cuda_data_partition_; // for histogram construction diff --git a/src/treelearner/serial_tree_learner.cpp b/src/treelearner/serial_tree_learner.cpp index 23d7525de74b..b2834866c149 100644 --- a/src/treelearner/serial_tree_learner.cpp +++ b/src/treelearner/serial_tree_learner.cpp @@ -90,12 +90,14 @@ void SerialTreeLearner::GetShareStates(const Dataset* dataset, share_state_.reset(dataset->GetShareStates( reinterpret_cast(gradient_discretizer_->ordered_int_gradients_and_hessians()), nullptr, col_sampler_.is_feature_used_bytree(), is_constant_hessian, - config_->force_col_wise, config_->force_row_wise, config_->num_grad_quant_bins)); + config_->force_col_wise, config_->force_row_wise, config_->num_grad_quant_bins, + IsCUDATreeLearner())); } else { share_state_.reset(dataset->GetShareStates( ordered_gradients_.data(), ordered_hessians_.data(), col_sampler_.is_feature_used_bytree(), is_constant_hessian, - config_->force_col_wise, config_->force_row_wise, config_->num_grad_quant_bins)); + config_->force_col_wise, config_->force_row_wise, config_->num_grad_quant_bins, + IsCUDATreeLearner())); } } else { CHECK_NOTNULL(share_state_); @@ -104,12 +106,14 @@ void SerialTreeLearner::GetShareStates(const Dataset* dataset, share_state_.reset(dataset->GetShareStates( reinterpret_cast(gradient_discretizer_->ordered_int_gradients_and_hessians()), nullptr, col_sampler_.is_feature_used_bytree(), is_constant_hessian, - share_state_->is_col_wise, !share_state_->is_col_wise, config_->num_grad_quant_bins)); + share_state_->is_col_wise, !share_state_->is_col_wise, config_->num_grad_quant_bins, + IsCUDATreeLearner())); } else { share_state_.reset(dataset->GetShareStates( ordered_gradients_.data(), ordered_hessians_.data(), col_sampler_.is_feature_used_bytree(), is_constant_hessian, share_state_->is_col_wise, - !share_state_->is_col_wise, config_->num_grad_quant_bins)); + !share_state_->is_col_wise, config_->num_grad_quant_bins, + IsCUDATreeLearner())); } } CHECK_NOTNULL(share_state_); diff --git a/src/treelearner/serial_tree_learner.h b/src/treelearner/serial_tree_learner.h index b753827b1bd7..81e160149ffa 100644 --- a/src/treelearner/serial_tree_learner.h +++ b/src/treelearner/serial_tree_learner.h @@ -131,6 +131,10 @@ class SerialTreeLearner: public TreeLearner { void GetShareStates(const Dataset* dataset, bool is_constant_hessian, bool is_first_time); + /*! \brief Whether this learner constructs histograms on the CUDA device + * (lets Dataset::GetShareStates skip the host multi-val bin build). */ + virtual bool IsCUDATreeLearner() const { return false; } + void RecomputeBestSplitForLeaf(Tree* tree, int leaf, SplitInfo* split); /*!