Skip to content

postgres_scan: reject out-of-range timestamps - #484

Open
marknefedov wants to merge 3 commits into
Snowflake-Labs:mainfrom
marknefedov:fix/postgres-scan-timestamp-overflow
Open

postgres_scan: reject out-of-range timestamps#484
marknefedov wants to merge 3 commits into
Snowflake-Labs:mainfrom
marknefedov:fix/postgres-scan-timestamp-overflow

Conversation

@marknefedov

@marknefedov marknefedov commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

postgres_scan silently corrupts out-of-range timestamps instead of rejecting them. PostgreSQL's timestamp/timestamptz range is wider than DuckDB's. When PostgresBinaryReader::ReadTimestamp() adds the PostgreSQL→DuckDB epoch offset to a value near either end of PostgreSQL's range, the addition overflows DuckDB's internal int64 microseconds representation and silently wraps — e.g. a valid PostgreSQL value of 294276-12-31 23:59:59 is returned as 290279-12-14 (BC) 15:58:09.448384. The same unchecked arithmetic exists in reverse in postgres_binary_writer.hpp, so a DuckDB value near its own bounds can silently wrap into a bogus PostgreSQL timestamp on write.

Fix

This backports duckdb/duckdb-postgres#530 onto the older vendored duckdb-postgres revision pg_lake currently pins, via a new patch in duckdb_pglake/patches/duckdb-postgres/timestamp-range.patch (the upstream PR targets a later API — PostgresBinaryParser vs. this revision's PostgresBinaryReader — so the checks are ported rather than cherry-picked verbatim):

  • Reader: use TryAddOperator for the epoch-offset addition and reject the result with ConversionException("timestamp out of range") on overflow or if it isn't Timestamp::IsFinite.
  • Writer: use TrySubtractOperator and validate against PostgreSQL's exact MIN_TIMESTAMP/END_TIMESTAMP bounds (from datatype/timestamp.h), raising InvalidInputException instead of wrapping.
  • Infinity/-infinity sentinels are preserved and still round-trip correctly on both sides.

Tests

Added pgduck_server/tests/pytests/test_postgres_scanner_timestamp_range.py, covering the reproduction from #439:

  • both timestamp and timestamptz scanning reject the out-of-range value with an error instead of returning wrapped data
  • the same rejection propagates through COPY ... TO PARQUET
  • supported boundary/control values (including a BC date) still round-trip correctly — kept in a separate source table since postgres_scan converts a whole batch before DuckDB's outer filter applies, so a single mixed table can't isolate the failing row
  • the writer path is exercised via the existing postgres_binary.test fixture from duckdb-postgres, adjusted to use a cross-database-representable BC boundary instead of DuckDB's own (more extreme) minimum, which previously round-tripped silently through matching reader/writer overflows

Before the patch: 4 failed, 1 passed. After: 5 passed. Focused scanner suite (test_postgres_scanner.py + test_postgres_scanner_timestamp_range.py): 18 passed.

Fixes #439.
Upstream fix: duckdb/duckdb-postgres#530

Signed-off-by: Mark Nefedov <mvnefedov@avito.ru>
Signed-off-by: Mark Nefedov <mvnefedov@avito.ru>
Comment thread duckdb_pglake/patches/duckdb-postgres/timestamp-range.patch
@marknefedov
marknefedov marked this pull request as draft July 28, 2026 10:51
Signed-off-by: Mark Nefedov <mvnefedov@avito.ru>
@marknefedov
marknefedov marked this pull request as ready for review July 28, 2026 12:04
@marknefedov
marknefedov requested a review from sfc-gh-mslot July 28, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

postgres_scan out-of-range timestamps wrap instead of erroring

2 participants