Skip to content

Flink: Fix watermark extractor error message showing field id instead of file#17124

Open
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:fix/flink-watermark-error-message-file
Open

Flink: Fix watermark extractor error message showing field id instead of file#17124
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:fix/flink-watermark-error-message-file

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 7, 2026

Copy link
Copy Markdown
### What

`ColumnStatsWatermarkExtractor.extractWatermark` builds its "missing statistics"
error with a `Preconditions.checkArgument` format string that has exactly two
`%s` placeholders:

"Missing statistics for column name = %s in file = %s"


but passes three arguments to it: `eventTimeFieldName`, `eventTimeFieldId`, and
`scanTask.file()`. Guava's `lenientFormat` fills the two `%s` slots with the
first two arguments and appends any extras in trailing brackets. As a result the
message renders the numeric field id where the file is supposed to be, and dumps
the actual file in an overflow bracket, for example:

Missing statistics for column name = ts in file = 5 [GenericDataFile{...}]


The `file =` slot shows `5` (the field id) instead of the file, which is
misleading when diagnosing a table that is missing lower bounds for the
watermark column.

### Why

The `file =` label should point at the file that lacks statistics. Printing the
field id there (and hiding the file in brackets) makes the diagnostic harder to
act on. The bug is present identically in the v1.20, v2.0, and v2.1 Flink
modules.

### How

Remove the redundant `eventTimeFieldId` argument so the two remaining arguments
(column name and file) line up with the two placeholders. The message text is
unchanged; `eventTimeFieldId` is still used by the surrounding null-check guard
and the `Conversions.fromByteBuffer` lookup, so nothing else is affected.

The existing `testEmptyStatistics` assertion is strengthened from a substring
check to an exact `hasMessage(...)` that computes the expected file from the
split, so it now fails if the file slot is ever wrong again. Verified this is a
genuine red/green guard: with the extra argument re-added the test fails with
`... in file = 10 [<file>]`; with the fix it passes.

### Tests

- `TestColumnStatsWatermarkExtractor` passes for all three Flink modules
  (v1.20, v2.0, v2.1) run with `-DflinkVersions=1.20,2.0,2.1`.
- `spotlessCheck` is clean for the three modules.

One concern per PR: only the argument fix and its regression assertion are
touched, applied identically across the three module copies.

---
**AI Disclosure**
- Model: Claude Opus 4.8
- Platform/Tool: opencode
- Human Oversight: partially reviewed
- Prompt Summary: Locate and fix a Flink `ColumnStatsWatermarkExtractor` error
  message that printed the watermark column's field id in the "file =" slot
  instead of the file, remove the stray format argument, and add a regression
  assertion. Diff and tests were independently reviewed and run before opening.

Files changed (6 files, +15/-9)

  • flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java (-1)
  • flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java (+5/-2)
  • flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java (-1)
  • flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java (+5/-2)
  • flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/source/reader/ColumnStatsWatermarkExtractor.java (-1)
  • flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/source/reader/TestColumnStatsWatermarkExtractor.java (+5/-2)

… of file

ColumnStatsWatermarkExtractor.extractWatermark passed three arguments to a
Preconditions.checkArgument format string that has only two %s placeholders
("Missing statistics for column name = %s in file = %s"). The extra
eventTimeFieldId argument shifted the file into an overflow position, so
Guava's lenientFormat rendered the field id in the "file =" slot and appended
the actual file in trailing brackets, e.g. "... in file = 5 [<file>]".

Remove the redundant eventTimeFieldId argument so the two remaining arguments
(column name and file) line up with the two placeholders. Applied identically
to the v1.20, v2.0, and v2.1 Flink modules, and strengthen the existing
testEmptyStatistics assertion to pin the rendered message to the file.

Generated-by: opencode
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@github-actions github-actions Bot added the flink label Jul 7, 2026
@@ -99,7 +99,6 @@ public long extractWatermark(IcebergSourceSplit split) {
&& scanTask.file().lowerBounds().get(eventTimeFieldId) != null,
"Missing statistics for column name = %s in file = %s",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we instead add the id to the message too?

Suggested change
"Missing statistics for column name = %s in file = %s",
"Missing statistics for column name = %s with fieldId = %d in file = %s",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants