CE888 Agentic Data Scientist

Module: CE888 Data Science and Decision Making Assignment: Offline Agentic AI for Data Science Academic year: 2025/2026 Deadline: 21 April 2026

Overview

This repository contains an offline agentic ML pipeline that profiles a CSV dataset, plans an end-to-end ML workflow, trains candidate models, evaluates the results, reflects on performance, and (if it is not happy) replans. No LLM and no external API are involved; every decision about what to try is driven by the dataset profile and a handful of statistical rules. The outer loop runs at most max_replans + 1 iterations before it terminates.

This project was originally based on a course scaffold provided by Professor Haider and has since been substantially extended and modified.

Quick start

pip install -r requirements.txt

# one-off: download the spaCy model used by target-column inference (~33 MB)
python -m spacy download en_core_web_md

# run on a CSV, letting the agent infer the target column
python run_agent.py --data data/example_dataset.csv --target auto

# run with explicit target and non-default settings
python run_agent.py \
    --data data/your_dataset.csv \
    --target label \
    --seed 42 \
    --test_size 0.15 \
    --val_size 0.15 \
    --max_replans 3

Each run writes its outputs to outputs/<YYYYMMDD_HHMMSS_uuid>/:

File	Contents
report.md	Human-readable run report
eda_summary.json	Dataset profile
plan.json	Generated execution plan
metrics.json	Evaluation payload
reflection.json	Diagnosis and replan decision
*.png	EDA plots, confusion matrix, ROC curves (binary), etc.

CLI arguments

Argument	Default	Description
--data	(required)	Path to CSV dataset
--target	(required)	Target column name, or auto to infer
--output_root	outputs	Root directory for run outputs
--seed	3006	Random seed
--test_size	0.15	Fraction held out as final test set
--val_size	0.15	Fraction of train-val used for validation
--max_replans	5	Maximum replan iterations
--quiet	False	Suppress verbose logging

Project structure

ce888-agentic-data-scientist/
├── run_agent.py                    # CLI entry point
├── agentic_data_scientist.py       # orchestrator + replan loop
├── requirements.txt
├── pyproject.toml                  # ruff / mypy / pytest config
│
├── agents/
│   ├── planner.py                  # create_plan(): 8 decision sub-stages
│   ├── reflector.py                # reflect() + apply_replan_strategy()
│   ├── memory.py                   # JSONMemory: similarity search, write guard
│   └── logger.py                   # ReactLogger: Thought/Action/Observation trace
│
├── tools/
│   ├── data_profiler.py            # profile_dataset()
│   ├── _profiler_stats.py          # normality, LDA, multicollinearity tests
│   ├── _profiler_features.py       # feature / task / time-series detection
│   ├── _profiler_plots.py          # EDA plots
│   ├── modelling.py                # build_preprocessor, select_models, train_models
│   ├── _modelling_preprocessor.py  # FrequencyEncoder, IQROutlierClipper
│   ├── evaluation.py               # evaluate_best, write_markdown_report
│   └── _modelling_extended.py      # SARIMA + collaborative-filtering pipelines
│
├── tests/                          # pytest suite (see Testing below)
├── data/
│   ├── README.md                   # one-line description per dataset
│   ├── demo.csv                    # tiny smoke-test dataset
│   ├── example_dataset.csv         # same content as demo.csv (kept for back-compat)
│   └── *.csv                       # Kaggle datasets (see data/README.md)
├── outputs/                        # generated per run (gitignored)
└── report/
    ├── REPORT.md                   # written technical report
    └── system_architecture.png     # referenced from REPORT.md

Pipeline stages

The end-to-end run is driven by AgenticDataScientist.run() and is broken into 14 numbered stages. Stages 6-13 repeat inside the replan loop; stages 1-5 run once up front, and Stage 14 covers both the in-loop replan-strategy application (when a replan is recommended) and the end-of-run memory + report write.

Stage 1   load_data()                 CSV -> DataFrame
Stage 2   profile_dataset()           DataFrame -> profile dict
                                      (target detection, feature types, stats,
                                      time-series / recommendation checks)
Stage 3   memory.get_dataset_record() fingerprint -> prior record | None
Stage 4   create_plan()               profile -> plan dict (8 sub-stages 4a-4h)
Stage 5   train_test_split (3-way)    X, y -> train / val / test (ONCE)

--- replan loop (up to max_replans + 1 iterations) -----------------------
Stage 14  apply_replan_strategy()     plan, reflection -> new plan
                                      [in-loop; skipped on iter 0]
Stage  6  build_preprocessor()        plan -> sklearn ColumnTransformer
Stage  7  select_models()             plan -> [(name, estimator), ...]
Stage  8  apply_smote()               resampled train set if requested
Stage  9  train_models()              candidates -> list of result dicts
Stage 10  evaluate_best()             results -> eval_payload
Stage 11  reflect()                   eval_payload -> reflection
Stage 12  save_json()                 writes metrics.json, reflection.json
Stage 13  session_memory update       snapshot-revert if delta < 0
--------------------------------------------------------------------------

Stage 14  memory.upsert_dataset_record() + write_markdown_report()
                                      -> agent_memory.json (with write guard)
                                      -> report.md

Key design decisions

The system is adaptive rather than fixed: most of the decisions below are conditional branches on properties the profiler measures, not defaults applied uniformly.

Auto target inference. When --target auto is passed, infer_target_column scores each column using keyword match (target, label, class, y, outcome), positional bonus, and spaCy semantic similarity between the column name tokens and the CSV filename. Scores below 1.5 route the run into the unsupervised pipeline instead of guessing.

Validation strategy. Nested CV (outer 5-fold, inner 4-fold GridSearchCV) is used when n_rows < max(500, 10 × n_features); otherwise a three-way split. The absolute floor of 500 keeps the held-out folds statistically meaningful; the samples-per-variable condition reflects the heuristic that reliable fitting needs ~10 observations per input dimension. For time series, KFold is swapped for TimeSeriesSplit at both levels.

Primary metric. Chosen before training. balanced_accuracy for balanced classification (imbalance_ratio < 3), macro_f1 for imbalanced. The reason is that balanced_accuracy assumes equal-class performance is the goal, which is inappropriate when one class is rare and a model that ignores the minority can still look acceptable.

Per-column preprocessing. Scaling and encoding are chosen per column from the profiler's measurements:

Column type	Trigger	Choice
Numeric	is_normal=True (D'Agostino-Pearson)	StandardScaler
Numeric	|skew| > 0.5	log1p then StandardScaler
Numeric	otherwise	MinMaxScaler
Categorical	cardinality = 2	binary
Categorical	cardinality 3-5	one-hot
Categorical	cardinality > 5, supervised	target encoding
Categorical	otherwise	ordinal / frequency

Multicollinearity response. With > 20 features, PCA at 0.70 variance is applied and the LogisticRegression grid is restricted to L2 (L1's feature selection fights a PCA basis). With ≤ 20 features, the profiler's correlated columns are dropped directly and LR is allowed both L1 and L2.

Held-out test set. The three-way split runs once before the replan loop and is not touched again. Validation data drives model selection inside the loop; test data is only used for the final numbers in report.md.

Threshold tuning. For binary classification with imbalance_ratio ≥ 3, the default 0.5 decision threshold is replaced by Youden's J (argmax of TPR - FPR) computed on the validation set, since 0.5 is only optimal at equal class priors. Skipped for multi-class (no single J-equivalent).

Diagnosis priority order. The reflector checks diagnoses in strict priority and stops at the first hit:

1. imbalance_overlap: class imbalance degrading minority recall
2. overfitting: train-test gap > 0.15
3. underfitting: best model barely above the dummy baseline
4. non_convergence: any model hit its max_iter limit
5. plateau: relative margin ≤ 0.10 over the dummy
6. no_signal: plateau at the final replan, agent exits

Replan trigger. Any non-ok / non-no_signal diagnosis requests another loop (subject to max_replans and the not-already-tried guard). The replan is diagnosis-driven, not margin-driven: a model that beats the dummy by a wide margin while overfitting still triggers a replan, because the diagnosis identifies signal worth recovering even when headline performance is fine. Snapshot-revert below makes this safe.

Snapshot-revert. After each replan iteration, if the new strategy hurts performance (delta < 0), the plan is reverted to its pre-replan state and the strategy is marked _REVERTED in session memory so it is not retried inside the same run.

Occam's razor. When two models are within 2% on the primary metric, the simpler one wins. The complexity ordering used for ties, from tools/evaluation.py:_COMPLEXITY_RANK, is:

Dummy < GaussianNB < LogisticRegression < LDA < KNN < LinearSVC
      < DecisionTree < RandomForest < ExtraTrees < GradientBoosting
      < XGBoost < LightGBM < VotingEnsemble

Model pool

Model selection is driven by size_tier (small / normal / large):

Model	Small	Normal	Large
DummyMostFrequent	✓	✓	✓
LogisticRegression	✓	✓	✓
GaussianNB	✓	✓	-
LDA	✓	✓	-
KNN	✓*	✓*	-
DecisionTree	✓	✓	-
LinearSVC	-	✓	-
SVC_RBF	-	✓**	-
RandomForest	-	✓	✓
ExtraTrees	-	✓	✓
GradientBoosting	-	✓	-
XGBoost	-	-	✓
LightGBM	-	-	✓
VotingEnsemble	plateau replan only

* KNN excluded if rows < 50 (small) or < 500 (normal). ** SVC_RBF excluded if rows > 10,000.

Imbalance handling

Imbalance ratio	Strategy
< 3	none
3-9	class_weight
≥ 10	smote (adaptive k)

If replanning is triggered for imbalance_overlap, the strategy escalates smote -> smoteenn -> adasyn.

Extended pipelines

Three single-pass pipelines replace the standard replan loop when the profiler detects a task type that doesn't fit classification / regression.

Unsupervised (tools/modelling.py:train_unsupervised): triggered when no target column can be inferred. Sweeps K-Means across k = 2..min(8, n/10) and keeps the best silhouette. If compare_with_without_pca is set, the sweep is repeated on a PCA-reduced copy and the better of the two wins. When the best silhouette is still below 0.40, falls back to DBSCAN. Optionally runs IsolationForest for anomaly flags. Saves pca_clusters.png, pca_variance.png, cluster_labels.csv, and (if enabled) anomaly_detection.png.

SARIMA (tools/_modelling_extended.py:train_sarima): regression task with a non-stationary time series. Steps: log1p transform, ADF-based auto-differencing, pmdarima.auto_arima, forecast. Reports MAE, RMSE, sMAPE, R^2, and the fitted model order.

Collaborative filtering (tools/_modelling_extended.py:train_recommendation): triggered when user/item/rating columns are detected. Strategies are nmf_cf (NMF, k=20) or user_based_cf (cosine-similarity KNN). Reports Recall@10, Precision@10, and sparsity.

Memory system

agents/memory.py implements JSONMemory, which persists learned knowledge in agent_memory.json. Two lookup paths, used for different purposes:

Exact match (primary warm-start). At Stage 3, the orchestrator computes a dataset fingerprint from shape + target column + column names and calls get_dataset_record. If a record exists, the past winner is placed first in the model pool and previously-failing models are excluded before the first training iteration.

Similarity match (per-model grid narrowing). Inside planner.py:_build_hyperparameter_grids, each candidate model queries find_similar_record(profile, model_type=<model>). This uses a 6-dimensional feature vector and weighted Euclidean distance:

features:  [log_n_samples, log_n_features, n_classes, log_imbalance_ratio, missing_pct_avg, cat_feature_ratio]
weights:   [2.0,           2.0,            1.5,       1.0,                 1.0,             1.0]
threshold: distance < 0.30

When a similar record is found for that model, its best_params are expanded by build_centred_grid into a narrow grid centred on the known winner. If a parameter previously hit a grid boundary, the grid extends in that direction rather than bracketing symmetrically.

Write guard. upsert_dataset_record drops records where best_score <= dummy_score + 0.10. Weak results are not written, so future similarity lookups are not polluted by runs that barely beat the baseline.

Testing

# full suite
pytest tests/ -v

# with coverage report
pytest --cov=agents --cov=tools --cov-report=html tests/

# single file
pytest tests/test_planner.py -v

# smoke test: full end-to-end run on a small CSV
pytest tests/test_smoke_run.py -v

Code quality

black .                    # format (line length 100)
ruff check . --fix         # lint with auto-fix
mypy agents tools          # type checking

Configured in pyproject.toml: line length 100, Google-style docstrings on public functions, type annotations required on function signatures in agents/ and tools/.