Add Variant Logical Type for Parquet Conversions #804
Description
Feature Request / Improvement
A Variant is a binary encoding designed to store nested, schema-less data (JSON-like) while allowing for shredding—the ability to pull common fields out into their own columns for lightning-fast access.
Parquet users had to choose between two extremes:
- Map/Struct types: High performance, but requires a fixed schema.
- JSON Strings: Flexible, but requires scanning the whole string to find one value (very slow).
The Variant type is the modern "middle ground" solution, recently standardized by the Apache Parquet and Apache Iceberg communities.
This type is not yet supported for Syncing Parquet files into either Iceberg, Delta or Hudi.
Original PR supporting the current type into Parquet Format is:
Added Variant to Parquet Format
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct