Core: Read and write geometry and geography values in Avro#17119
Core: Read and write geometry and geography values in Avro#17119huan233usc wants to merge 1 commit into
Conversation
| } | ||
|
|
||
| @Override | ||
| protected boolean supportsGeospatial() { |
There was a problem hiding this comment.
TestAvroEncoderUtil extends DataTestBase and round-trips generic records through this same TypeToSchema + RandomAvroData path via AvroEncoderUtil.encode/decode, and on main it mirrors this test's type flags exactly (supportsUnknown, supportsTimestampNanos, supportsVariant). Enabling supportsGeospatial() there as well covers geo through the encoder entry point for free and keeps the two generic-model Avro tests in parity. szehon-ho raised the same coverage-parity point on the sibling Parquet PR #16982.
There was a problem hiding this comment.
Done — enabled supportsGeospatial() on TestAvroEncoderUtil in 67619a7, so geo now round-trips through the AvroEncoderUtil.encode/decode entry point too, keeping the two generic-model Avro tests in parity (good call on the #16982 coverage-parity precedent).
I also added explicit WKB round-trip tests while here:
- TestAvroDataWriter.testGeospatialWkbRoundTrip — geometry/geography through the Avro DataWriter → PlannedDataReader path, including nulls.
- TestParquet.testGeospatialWkbRoundTrip — through ParquetAvroWriter → ParquetAvroValueReaders,
500ea8d to
7b2cc6d
Compare
TypeToSchema mapped Iceberg types to Avro schema for every primitive except geometry and geography, so converting a schema with a geo column threw UnsupportedOperationException before any value could be read or written. Map both to an Avro bytes field carrying WKB, per the Avro type mapping in the spec (binary, geometry, and geography all use bytes). The value read/write paths dispatch on the Avro physical type, so once the schema is bytes the existing byteBuffers reader/writer handle geo unchanged in both the generic and internal object models (geo values are WKB ByteBuffers, like binary). Enable the shared DataTest geospatial coverage for the Avro object models (TestGenericAvro, TestInternalAvro, generic data TestGenericData) and add the GEOMETRY/GEOGRAPHY -> ByteBuffer handling to RandomAvroData and AvroTestHelpers that the generic Avro path needs, mirroring RandomInternalData/InternalTestHelpers.
7b2cc6d to
67619a7
Compare
Summary
TypeToSchemamapped every Iceberg primitive to an Avro schema exceptgeometryandgeography,so converting a schema with a geo column threw
UnsupportedOperationException: Unsupported type ID: GEOMETRYbefore any value could be read or written through the Avro object model. This wires up thevalue path.
Per the Avro type mapping in the spec,
geometryandgeographyare stored as an Avrobytesfieldcarrying WKB — the same representation as
binary:binarybytesgeometrybytes(WKB)geographybytes(WKB)So
TypeToSchema.primitive()now maps both geo types to the existingBINARY_SCHEMA. The valueread/write paths dispatch on the Avro physical type, so once the schema is
bytesthe existingbyteBuffersreader/writer handle geo unchanged in both the generic and internal object models (geovalues are WKB
ByteBuffers, exactly likebinary) — no value-path code changes are needed.This is a follow-up split out of #16982 (which added the Parquet WKB value path and deliberately left
the Avro object model as a follow-up).
Note on the reverse direction
SchemaToTypeintentionally still maps Avrobytes → binary. Plain Avrobytescarries no marker todistinguish geometry/geography from binary, and Iceberg resolves the real column type from the
expected/table schema, never by reverse-inferring from the file's Avro schema — the same way
binaryand other
bytes-backed types already behave.Test plan
DataTestgeospatial coverage (supportsGeospatial()) for the Avro object models:TestGenericAvro,TestInternalAvro, and the generic-dataTestGenericData. This round-tripsgeometry and geography across multiple CRS and edge algorithms with randomly generated WKB values.
GEOMETRY/GEOGRAPHY→ByteBufferhandling that the generic Avro path needs inRandomAvroDataandAvroTestHelpers, mirroring the existingRandomInternalData/InternalTestHelpershandling../gradlew :iceberg-core:test --tests org.apache.iceberg.avro.TestGenericAvro --tests org.apache.iceberg.avro.TestInternalAvro./gradlew :iceberg-data:test --tests org.apache.iceberg.data.avro.TestGenericData