[HWORKS-2807] Native Partitioning, Liquid Clustering, Z-Ordering#617
Draft
jimdowling wants to merge 4 commits into
Draft
[HWORKS-2807] Native Partitioning, Liquid Clustering, Z-Ordering#617jimdowling wants to merge 4 commits into
jimdowling wants to merge 4 commits into
Conversation
… hidden partitioning, Delta liquid clustering, z-ordering https://hopsworks.atlassian.net/browse/HWORKS-2807 Reinterpret partitioned_by as a list of native partition transform expressions instead of the materialized time-grain columns shipped under HWORKS-2802, with each layout axis its own parameter: partitioned_by (Iceberg partition spec; Hudi grains), clustered_by (Delta liquid clustering), bucket_index (Hudi), zorder_by, and sort_order (Iceberg). The docs gain a "Partitioning and Clustering" user guide covering the transform grammar with partition-field aliases (Iceberg-only, unique field names), the explicit identity form for grain-named columns, and per-format availability, hidden partitioning on Iceberg with a day + bucket example, the persistent Iceberg sort_order (mutually exclusive with zorder_by), liquid clustering on Delta with the Spark-only writer warning and the auto-widened delta.dataSkippingNumIndexedCols note, a catalog-capability matrix for the Iceberg operations, the rewrite_all=False optimize default with the explicit initial full z-order and the Iceberg where filter, the Hudi grain columns and bucket index (engine "simple" only; hoodie.bucket.index.* options and a partition-level bucket index settable directly through write_options), zorder_by with an hour + z-order example, the per-format optimize() behavior including the Delta full/zorder combination rules, layout evolution (update_partition_spec with actual-spec persistence, the partition_key restriction, and the no-arg metadata re-sync; update/disable_clustering), layout introspection (the format-gated get_* methods and describe_layout, with the user-catalog caveat), the migration note for the removed bare-grain form including the explicit write rejection, and the stream/online restrictions. The create guide's table-format list gains ICEBERG and its partition-key section links to the new guide. Merge after the hopsworks-api branch: the API cross-references (FeatureGroup.optimize) resolve against hopsworks-api@main in the docs build. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMpkURDybXS7BnP4C6eMsW
…io into HWORKS-2807-partition-transforms # Conflicts: # docs/user_guides/fs/feature_group/create.md
…data https://hopsworks.atlassian.net/browse/HWORKS-2807 Adds a "Layout for point-in-time training data" subsection to the partitioning guide: day(event_ts) pruning engages through the PIT join when a lookback (or event-time filter) bounds the history, bucket(N, pk) plus a key sort order serve the key-equality join, appends preserve the clustering that upserts discard, and partition sizing guidance (day x bucket combinations become files, so match the granularity to volume). Backed by an in-cluster verification of the generated PIT plans showing the bound pushed into the Iceberg BatchScan for both mechanisms. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMpkURDybXS7BnP4C6eMsW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: https://hopsworks.atlassian.net/browse/HWORKS-2807
Redesigns
partitioned_byfrom materialized time-grain columns (HWORKS-2802) into native partition transform expressions, and splits each layout axis into its own creation parameter so no format accepts and erases a mechanism it cannot express:partitioned_by: native partition spec from transforms (identity,bucket,truncate,year/month/day/hour,weekHudi-only,voidIceberg-only), optionally aliased. Iceberg hidden partitioning; Hudi identity + grains; rejected on Delta.clustered_by: Delta liquid clustering (Spark-only writes; statistics widened when a clustering column sits past the indexed range).bucket_index: Hudi bucket index (engine="simple").zorder_by: z-order columns (Iceberg optimize rewrite; Hudi inline clustering).sort_order: persistent Iceberg write sort order (mutually exclusive withzorder_by).Also adds layout evolution (
update_partition_spec/update_clustering/disable_clustering), actual-table introspection (get_partition_spec(s)/get_sort_order/get_clustering_columns/describe_layout), and richeroptimize()(strategy, columns, rewrite_all, target_file_size_mb, where, full) returning rewrite metrics. Every layout column reference is canonicalized to the sanitized feature name. Breaking the bare-grain form is accepted: it is rejected with a migration hint, and writes to pre-transform feature groups fail explicitly.Coordinated change across hopsworks-api, hopsworks-ee, loadtest, and logicalclocks.github.io. Cluster-verified on jim-partitioned-by: stage A REST matrix 46/46 and stage B in-cluster Spark job 38/38.
In this repository (logicalclocks.github.io): a new "Partitioning and Clustering" user guide, plus
create.mdandmkdocs.ymlupdates. Merge after the hopsworks-api PR so theFeatureGroup.optimizeAPI cross-references resolve againsthopsworks-api@main.🤖 Generated with Claude Code