diff --git a/okf/bundles/ml_experiment_trail/experiments/baseline_logistic_regression.md b/okf/bundles/ml_experiment_trail/experiments/baseline_logistic_regression.md new file mode 100644 index 00000000..6159f335 --- /dev/null +++ b/okf/bundles/ml_experiment_trail/experiments/baseline_logistic_regression.md @@ -0,0 +1,39 @@ +--- +type: Experiment +title: Baseline logistic regression +description: First end-to-end baseline establishing the validation protocol and a reference score. +tags: +- experiment +- baseline +- logistic-regression +timestamp: '2026-01-14T09:00:00+00:00' +--- + +A minimal end-to-end run to lock the validation protocol before investing in +feature engineering. Establishes the reference score every later experiment is +compared against. + +# Setup + +| Field | Value | +| --- | --- | +| Algorithm | Logistic regression (L2, `C=1.0`) | +| Features | 11 raw numeric columns, standardized | +| Validation | 5-fold stratified cross-validation, seed 42 | +| Metric | ROC AUC | + +# Result + +| Split | ROC AUC | +| --- | --- | +| CV mean | 0.782 | +| CV std | 0.006 | + +# Notes + +An early version of this experiment scored a suspiciously high 0.991; that run +is written up in [target leakage via row id](/lessons/target_leakage_via_row_id.md). +Superseded as current best by [gradient boosting v2](/experiments/gradient_boosting_v2.md). + +# Citations +- https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression diff --git a/okf/bundles/ml_experiment_trail/experiments/gradient_boosting_v2.md b/okf/bundles/ml_experiment_trail/experiments/gradient_boosting_v2.md new file mode 100644 index 00000000..53ee2ee7 --- /dev/null +++ b/okf/bundles/ml_experiment_trail/experiments/gradient_boosting_v2.md @@ -0,0 +1,38 @@ +--- +type: Experiment +title: Gradient boosting v2 +description: Gradient-boosted trees with engineered features; current best cross-validation score. +tags: +- experiment +- gradient-boosting +- feature-engineering +timestamp: '2026-02-11T16:20:00+00:00' +--- + +Builds on the baseline by adding ratio and count-encoding features, trained with +gradient-boosted decision trees. Applies the leakage fix from the baseline +post-mortem. + +# Setup + +| Field | Value | +| --- | --- | +| Algorithm | Gradient-boosted trees (400 rounds, depth 6, lr 0.03) | +| Features | 11 raw + 9 engineered (ratios, count encodings) | +| Validation | 5-fold stratified cross-validation, seed 42 | +| Metric | ROC AUC | + +# Result + +| Split | ROC AUC | +| --- | --- | +| CV mean | 0.834 | +| CV std | 0.005 | + +# Notes + +Explicitly drops `row_id` per [target leakage via row id](/lessons/target_leakage_via_row_id.md). +The tuned artifact exported from this experiment is [gbdt tuned](/models/gbdt_tuned.md). + +# Citations +- https://xgboost.readthedocs.io/en/stable/ diff --git a/okf/bundles/ml_experiment_trail/experiments/index.md b/okf/bundles/ml_experiment_trail/experiments/index.md new file mode 100644 index 00000000..f158dede --- /dev/null +++ b/okf/bundles/ml_experiment_trail/experiments/index.md @@ -0,0 +1,4 @@ +# Experiment + +* [Baseline logistic regression](baseline_logistic_regression.md) - First end-to-end baseline establishing the validation protocol and a reference score. +* [Gradient boosting v2](gradient_boosting_v2.md) - Gradient-boosted trees with engineered features; current best cross-validation score. diff --git a/okf/bundles/ml_experiment_trail/index.md b/okf/bundles/ml_experiment_trail/index.md new file mode 100644 index 00000000..3b9b8a5d --- /dev/null +++ b/okf/bundles/ml_experiment_trail/index.md @@ -0,0 +1,6 @@ +# Subdirectories + +* [experiments](experiments/index.md) - Model-training experiments, each recording its setup, data, and validation result. +* [models](models/index.md) - Trained models promoted from experiments. +* [submissions](submissions/index.md) - Scored submissions against the competition leaderboard. +* [lessons](lessons/index.md) - Durable lessons carried across the experiment trail. diff --git a/okf/bundles/ml_experiment_trail/lessons/index.md b/okf/bundles/ml_experiment_trail/lessons/index.md new file mode 100644 index 00000000..7c151e12 --- /dev/null +++ b/okf/bundles/ml_experiment_trail/lessons/index.md @@ -0,0 +1,3 @@ +# Lesson + +* [Target leakage via row id](target_leakage_via_row_id.md) - A near-perfect early score traced to an ordered row identifier leaking the label. diff --git a/okf/bundles/ml_experiment_trail/lessons/target_leakage_via_row_id.md b/okf/bundles/ml_experiment_trail/lessons/target_leakage_via_row_id.md new file mode 100644 index 00000000..bbde78d7 --- /dev/null +++ b/okf/bundles/ml_experiment_trail/lessons/target_leakage_via_row_id.md @@ -0,0 +1,32 @@ +--- +type: Lesson +title: Target leakage via row id +description: A near-perfect early score traced to an ordered row identifier leaking the label. +tags: +- lesson +- data-leakage +- validation +timestamp: '2026-01-15T08:15:00+00:00' +--- + +# Symptom + +An early version of [baseline logistic regression](/experiments/baseline_logistic_regression.md) +scored 0.991 ROC AUC in cross-validation — far above anything the available +features plausibly support. + +# Cause + +The exported training table was sorted by label before a monotonic `row_id` was +assigned, so `row_id` was almost perfectly rank-correlated with the target. The +model learned the identifier, not the signal. + +# Fix + +Drop `row_id` (and any other index-like column) before training, and shuffle +rows at export time. Applied in [gradient boosting v2](/experiments/gradient_boosting_v2.md). + +# Takeaway + +Treat any single feature that produces a near-perfect score as a leak until +proven otherwise; audit identifier and timestamp columns first. diff --git a/okf/bundles/ml_experiment_trail/models/gbdt_tuned.md b/okf/bundles/ml_experiment_trail/models/gbdt_tuned.md new file mode 100644 index 00000000..fc9fccff --- /dev/null +++ b/okf/bundles/ml_experiment_trail/models/gbdt_tuned.md @@ -0,0 +1,29 @@ +--- +type: Model +title: GBDT tuned +description: Tuned gradient-boosted tree model exported from experiment gradient-boosting v2. +resource: https://example.com/artifacts/gbdt_tuned_2026-02-11.pkl +tags: +- model +- gradient-boosting +timestamp: '2026-02-11T17:05:00+00:00' +--- + +The serialized model artifact selected for submission. Produced by +[gradient boosting v2](/experiments/gradient_boosting_v2.md). + +# Interface + +| Property | Value | +| --- | --- | +| Format | pickled scikit-learn pipeline | +| Input | 20 features (see the source experiment) | +| Output | Probability in `[0, 1]` | + +# Notes + +Scored on the public leaderboard by +[leaderboard submission 2026-02-18](/submissions/leaderboard_2026_02_18.md). + +# Citations +- https://scikit-learn.org/stable/model_persistence.html diff --git a/okf/bundles/ml_experiment_trail/models/index.md b/okf/bundles/ml_experiment_trail/models/index.md new file mode 100644 index 00000000..468c854b --- /dev/null +++ b/okf/bundles/ml_experiment_trail/models/index.md @@ -0,0 +1,3 @@ +# Model + +* [GBDT tuned](gbdt_tuned.md) - Tuned gradient-boosted tree model exported from experiment gradient-boosting v2. diff --git a/okf/bundles/ml_experiment_trail/submissions/index.md b/okf/bundles/ml_experiment_trail/submissions/index.md new file mode 100644 index 00000000..0954a2da --- /dev/null +++ b/okf/bundles/ml_experiment_trail/submissions/index.md @@ -0,0 +1,3 @@ +# Submission + +* [Leaderboard submission 2026-02-18](leaderboard_2026_02_18.md) - Public-leaderboard submission generated from the GBDT tuned model. diff --git a/okf/bundles/ml_experiment_trail/submissions/leaderboard_2026_02_18.md b/okf/bundles/ml_experiment_trail/submissions/leaderboard_2026_02_18.md new file mode 100644 index 00000000..ef3a2dd5 --- /dev/null +++ b/okf/bundles/ml_experiment_trail/submissions/leaderboard_2026_02_18.md @@ -0,0 +1,26 @@ +--- +type: Submission +title: Leaderboard submission 2026-02-18 +description: Public-leaderboard submission generated from the GBDT tuned model. +tags: +- submission +- leaderboard +timestamp: '2026-02-18T11:30:00+00:00' +--- + +The prediction file submitted to the competition, generated from +[gbdt tuned](/models/gbdt_tuned.md). + +# Result + +| Field | Value | +| --- | --- | +| Public score | 0.828 | +| CV score | 0.834 | +| Public rank | 142 / 1,930 | + +# Notes + +The public score tracks cross-validation within 0.006, which suggests the +validation protocol from the [baseline](/experiments/baseline_logistic_regression.md) +generalizes and there is no public-set overfitting yet.