Implement Parquet GH-583 INT96 timestamp ColumnOrder - #10106
Merged
Conversation
etseidl
changed the title
ColumnOrderColumnOrder
etseidl
marked this pull request as ready for review
August 3, 2026 17:48
Contributor
|
🚀 please let me know when this is ready for a review @etseidl |
Contributor
Author
alamb
approved these changes
Aug 4, 2026
| TOTAL_ORDER, | ||
| /// Use INT96 timestamp order (see [parquet-format/#584]). | ||
| /// | ||
| /// [parquet-format/#584]: https://github.com/apache/parquet-format/pull/584 |
Contributor
There was a problem hiding this comment.
Maybe we should also link to the current parquet.thrift spec?
| let exp_min = stats.min_bytes_opt().expect("min stats missing"); | ||
| let exp_max = stats.max_bytes_opt().expect("max stats missing"); | ||
|
|
||
| let col_idx = reader |
Contributor
There was a problem hiding this comment.
a minor nit is that the mechanics to extract the INT96 stats from a column is somewhat verbose and repeated twice -- it might help readability to make a helper function or something that extracts the min/max values so there is less code duplication and thus the actual intent of the test is clearer
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
parquet::basic::ColumnOrder::sort_order_for_type#10104 (which depends on Implement PARQUET-2249: Introduce IEEE 754 total order #9619)Rationale for this change
Spark continues to use INT96 timestamps, despite INT96 being marked as deprecated in 2018. Query engines want valid statistics to allow reliably pruning on INT96 columns. apache/parquet-format#584 adds a new
ColumnOrdervariant which can be used to signal compliance with the only known use of INT96 (4-byte julian day from epoch, 8-byte nanosecond).What changes are included in this PR?
Adds support for the new enum variant, and writes the appropriate value in the
FileMetaData.column_ordersfield.This builds on changes introduced in #7687.
Are these changes tested?
Yes
Are there any user-facing changes?
Yes, this adds a new variant to public enums (
ColumnOrder::INT96_TIMESTAMP_ORDER,SortOrder::INT96_TIMESTAMP).