Extract ML service feature pipeline into standalone ETL module

[Smartdevs17]

Context

ML feature engineering logic is embedded in the FastAPI inference service, mixing data transformation with model serving and making both hard to maintain.

Current Limitation/Problem

Feature computation runs synchronously during inference requests, adding 200-500ms latency. Feature transformation code is untested and versioned poorly.

Expected Outcome

Standalone ETL pipeline that pre-computes features into a feature store, with versioned transformations, backfill capability, and feature drift monitoring.

Acceptance Criteria

• Feature pipeline extracted to services/feature-pipeline/ as a separate Python service
• Feature store: Redis-compatible store with TTL-based expiration for real-time features
• Versioned transformations: each feature set has a hash, cache invalidation on change
• Batch computation: hourly cron for real-time features, daily cron for historical aggregates
• Backfill command: recompute features for a date range with progress tracking
• Inference API reads from feature store; on cache miss, computes online and writes to store
• Feature drift detection: Kolmogorov-Smirnov test comparing current vs. reference distribution
• Edge case: feature store unavailable (all features computed online with store update)

Technical Scope

• services/feature-pipeline/ - new ETL service (Python/FastAPI)
• services/feature-pipeline/features/ - transformation modules (one per feature group)
• ml-service/ - remove embedded feature computation, replace with feature store reads
• docker-compose.yml - feature store (Redis) service
• services/feature-pipeline/jobs/ - cron-based refresh with Airflow integration

[udoyechinenyevictoria]

@udoyechinenyevictoria has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Hello! Please can I work on this issue?

ℹ️ Repo Maintainers: To accept this application, review their application or assign @udoyechinenyevictoria to this issue.

[edwarddavid929-png]

@edwarddavid929-png has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

please i would like to work on this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @edwarddavid929-png to this issue.

[drips-wave]

Congratulations, @udoyechinenyevictoria! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 30, 2026.

🧑‍💻 @udoyechinenyevictoria: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊