Skip to content

Bug Fix: Cast UINT32 to INT32 to ensure compatibility with other engines #3529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JeroenSchmidt wants to merge 2 commits into
base: main
Choose a base branch
from
+19 −4
Open

Bug Fix: Cast UINT32 to INT32 to ensure compatibility with other engines #3529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0b3a04c
fix and tests
JeroenSchmidt Jun 18, 2026
bcf5779
add test to validate exception is raised when uint32 values > INT32_MAX
JeroenSchmidt Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions pyiceberg/io/pyarrow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1970,13 +1970,14 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array:
return values.cast(target_type)
raise ValueError(f"Unsupported schema projection from {values.type} to {target_type}")
elif isinstance(field.field_type, (IntegerType, LongType)):
# Cast smaller integer types to target type for cross-platform compatibility
# Only allow widening conversions (smaller bit width to larger)
# Narrowing conversions fall through to promote() handling below
# Cast integer types for cross-platform compatibility (e.g. Spark reads):
# widening (smaller bit width to larger) and unsigned-to-signed at same width
if pa.types.is_integer(values.type):
source_width = values.type.bit_width
target_width = target_type.bit_width
if source_width < target_width:
if source_width < target_width or (
pa.types.is_unsigned_integer(values.type) and source_width <= target_width

@sungwy sungwy Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for digging into this @JeroenSchmidt.

I thought about this a bit more, and taking a step back: the root issue is that we're writing non-conformant Parquet.

Table Spec: Appendix A, which should be our source of truth, maps int/long to signed int32/int64 with no unsigned annotation, so the UINT_32 we emit today as a result of Arrow→Iceberg conversion is already out of spec. This PR fixes that symptom, but I think the cleaner fix is one step earlier.

There are really two steps on the write path: (1) the Arrow→Iceberg mapping picks the Iceberg type for a uint32 column, then (2) the write casts the data to match it. Today step 1 (_ConvertToIceberg.primitive) keys only on bit width, so uint32 maps to int (int32). IMHO that's where the loss is locked in, since int32 can't hold uint32 values ≥ 2^31. This PR's same-width cast is then left narrowing into that a type that's already too small.

If step 1 instead mapped uint32 → long, step 2 becomes a plain lossless widening (uint32 → int64), it's spec-conformant, and the same-width cast isn't needed at all. Same idea for uint8/uint16 → int (already lossless).

The only cases with no lossless target are uint64 (no Iceberg type wider than int64) and a uint32 appended to an already-declared int column. There I'd rather reject the write with a clear error than silently cast it to its signed counterpart. A lossy conversion should be the caller's explicit choice, not something we do quietly.

So I'm hesitant to land the same-width cast as-is. Could we instead (1) map uint32 → long and (2) reject unsigned writes with no lossless target?

):
return values.cast(target_type)

if field.field_type != file_field.field_type:
Expand Down
14 changes: 14 additions & 0 deletions tests/io/test_pyarrow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3084,6 +3084,7 @@ def test__to_requested_schema_timestamps_without_downcast_raises_exception(
(pa.int8(), IntegerType(), pa.int32()),
(pa.int16(), IntegerType(), pa.int32()),
(pa.uint16(), IntegerType(), pa.int32()),
(pa.uint32(), IntegerType(), pa.int32()),
(pa.uint32(), LongType(), pa.int64()),
(pa.int32(), LongType(), pa.int64()),
],
Expand All @@ -3109,6 +3110,19 @@ def test__to_requested_schema_integer_promotion(
assert result.column(0).to_pylist() == [1, 2, 3, None]


def test__to_requested_schema_uint32_overflow_raises() -> None:
"""Test that uint32 values exceeding INT32_MAX raise an error rather than wrapping."""
requested_schema = Schema(NestedField(1, "col", IntegerType(), required=False))
file_schema = requested_schema

arrow_schema = pa.schema([pa.field("col", pa.uint32())])
data = pa.array([2**31], type=pa.uint32())
batch = pa.RecordBatch.from_arrays([data], schema=arrow_schema)

with pytest.raises(pa.lib.ArrowInvalid, match="Integer value .* not in range"):
_to_requested_schema(requested_schema, file_schema, batch, downcast_ns_timestamp_to_us=False, include_field_ids=False)


def test_pyarrow_file_io_fs_by_scheme_cache() -> None:
# It's better to set up multi-region minio servers for an integration test once `endpoint_url` argument
# becomes available for `resolve_s3_region`
Expand Down