Skip to content

[FLINK-30514] Fix HybridSourceReader's method calling order for underlying subsources during source switch#28069

Open
shanzi wants to merge 2 commits intoapache:masterfrom
shanzi:FLINK-30514
Open

[FLINK-30514] Fix HybridSourceReader's method calling order for underlying subsources during source switch#28069
shanzi wants to merge 2 commits intoapache:masterfrom
shanzi:FLINK-30514

Conversation

@shanzi
Copy link
Copy Markdown

@shanzi shanzi commented Apr 29, 2026

What is the purpose of the change

This MR is intended to fix the method calling order of the HybridSourceReader described in FLINK-30514. We found it not only affects the File connector but also the Iceberg connector.

In our usage, we'd like to use Iceberg Connector as the first source of the HybridSource, however, we found due to the HybridSource is calling start before addSplits(which will be called during recovery), it breaks IcebergSourceSplitReader's assumption: the recovered splits will be added before start.

In the start method, the IcebergSourceSplitReader with check if there is recovered splits on start, and decide if to request new splits. The wrong order in the HybridSource will lead to a undesired requests of new splits, which breaks watermark, slows down the reading and endangers the stability of a job.

Brief change log

  • Fixed the order of method call: calling start() after the recovered splits were added to the reader first.
  • Updated unit tests to test the behavior.

Verifying this change

Unit tests was updated to verify the behavior.

Does this pull request potentially affect one of the following parts:

  • The behavior of HybridSource on switching sources should have been fixed to match the correct order.

Documentation

  • Does this pull request introduce a new feature? (no)

@flinkbot
Copy link
Copy Markdown
Collaborator

flinkbot commented Apr 29, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@shanzi
Copy link
Copy Markdown
Author

shanzi commented Apr 29, 2026

@flinkbot run connector-base

@shanzi
Copy link
Copy Markdown
Author

shanzi commented May 1, 2026

cc @tweise @WencongLiu @xccui Can you have a look given you contributed to this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants