[FSTORE-2055] incremental training datasets#612
Conversation
There was a problem hiding this comment.
Pull request overview
Adds documentation for incremental (“append-only”) materialized training datasets in the Feature Store, clarifying how large parquet training datasets can be extended over time without full rewrites.
Changes:
- Documented
feature_view.insert_training_data(...)for appending new batches to an existing training dataset version. - Added a conceptual overview of incremental training data to the Offline API concepts page with a pointer to the user guide.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/user_guides/fs/feature_view/training-data.md | Adds a new “Appending to a Training Dataset” section with example usage and behavioral notes. |
| docs/concepts/fs/feature_view/offline_api.md | Introduces a new “Incremental Training Data” concept section and links to the detailed guide. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| !!! note "Requirements and behavior" | ||
| - Appending is only supported for the `parquet` data format. | ||
| - Statistics are not recomputed on append by default, since that reads the whole dataset back every time. Pass `compute_statistics=True` to `insert_training_data` to refresh the descriptive statistics over all increments after the batch is written, or call `feature_view.compute_training_dataset_statistics(training_dataset_version)` explicitly when fresh statistics are needed, for example periodically or right before retraining. | ||
| - Model-dependent transformation functions transform each appended batch with the statistics computed when the training dataset version was created, so all increments and serving stay consistent with each other. To refit those statistics, rebuild the version with `overwrite=True` or create a new training dataset version. |
There was a problem hiding this comment.
If we recompute the training dataset statistics I think the transformation functions should use the new statistics of the training data.
There was a problem hiding this comment.
That's intentional. compute_training_dataset_statistics only refreshes the descriptive statistics the transformation statistics stay pinned, because the materialized data was already transformed with them.
Refitting transformation statistics without rewriting the data would make new appends and serving inconsistent with what's on disk. To refit currently, rebuild the version with insert_training_data(overwrite=True) or create a new TD version. I've added a runtime warning to make this explicit at the call site.
Maybe we should look into improving stats further in the future, but i think it should probably be a new PR and maybe someone with more XP in this should do it :/
There was a problem hiding this comment.
I agree with what you have mentioned, I did not think about the data that is already transformed. I am also not really sure what is the best way to to handle this case. We will leave it for now and handle it later.
No description provided.