[bench] reproducible GPU benchmark suite (ExaBoost vs LightGBM/XGBoost/CatBoost)#33
Merged
Conversation
…Boost/CatBoost) Adds benchmarks/: a self-contained, resumable harness comparing ExaBoost (built from the checkout) against upstream LightGBM (CUDA), XGBoost and CatBoost on GPU, across the classic gbm-bench datasets (fraud, covtype, year, higgs, epsilon, airline) plus optionally Numerai v5 all-data with official per-era corr/Sharpe metrics. Measures construction/train time, peak GPU/host memory, quality metrics and time-to-quality curves; writes a markdown report with charts. See benchmarks/README.md for reproduction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A bench.py subprocess that segfaults (observed: upstream LightGBM with use_quantized_grad on CUDA at HIGGS scale) leaves no line in runs.jsonl, so a resumed orchestrator would retry it forever and the warmup-failed skip never triggers. Record the failure from the orchestrator side when the child exits non-zero without having written its own record. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gimes Engine defaults differ (xgboost lambda=1, lightgbm lambda_l2=0, catboost l2_leaf_reg=3). At lr 0.1 the unregularized LightGBM-family models degenerate on imbalanced data (creditcard fraud: 0.56 AUC vs 0.97 with lambda_l2=1, xgboost 0.98 with its default lambda=1), so unaligned runs measure default regularization choices rather than the engines. The numerai regime intentionally keeps each engine's own defaults, mirroring how the official example config is used in practice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Building a full GPU DMatrix of a wide test set (numerai: 1.3M x 2748) requested 26GB with the trained booster still resident on a 32GB card. Predict through inplace_predict in 200K-row chunks instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l timings An absent bar (e.g. upstream lightgbm-quant segfaulting on higgs) read as 'not measured'; a fast bar for a sane=false run (upstream quant garbage models on fraud/covtype/year) read as a legitimate win. Crashes now get an explicit marker and insane runs are hatched/dimmed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e regressions The crash marker was annotated in data coordinates before the log scale and limits existed, so it floated outside the axes; use a blended transform (data x, axes y) drawn after the bars. Also hatch regression runs whose RMSE exceeds the xgboost reference by >15% even when they squeak past the std-based sanity flag (upstream quant on year: RMSE 10.85 vs std 10.9). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One scatter panel per dataset/regime (time log-x, quality oriented so up is better) makes degenerate-but-fast runs visually obvious; crash notes live in panel titles to avoid point collisions. The numerai regime now gets its own bar chart (it is neither shallow nor deep and appeared in no chart before). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…te from the matrix) The cross-library matrix stays float32-fed for fairness; ingest_bench.py measures ExaBoost's native int8 path on numerai in its own results file. Reference on RTX 5090 (medians of 3): construct 38.9s -> 15.8s, peak host RSS 86.4GB -> 43.9GB, models md5-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYUXNtCAXDd3zYT71m8YSJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproducible GPU benchmark harness under benchmarks/: gbm-bench dataset suite + optional Numerai v5, six library configs, aligned hyperparameter regimes (incl. aligned L2 — engine defaults differ and unaligned runs measure regularization defaults rather than engines), resumable run matrix, speed/quality/memory measurement, time-to-quality curves, markdown report generation. Companion to #32 (its results were produced by this suite). See benchmarks/README.md for one-command reproduction.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EYUXNtCAXDd3zYT71m8YSJ