Skip to content

Flink Sink V2: Add CommitGate plugin interface for deferred commits#15771

Open
herbherbherb wants to merge 2 commits intoapache:mainfrom
herbherbherb:flink-sink-commit-gate
Open

Flink Sink V2: Add CommitGate plugin interface for deferred commits#15771
herbherbherb wants to merge 2 commits intoapache:mainfrom
herbherbherb:flink-sink-commit-gate

Conversation

@herbherbherb
Copy link

Summary

Adds a CommitGate plugin interface to IcebergSink that controls whether committables are emitted downstream or buffered in Flink ListState. This enables use cases like pausing commits during catalog maintenance operations while keeping the Flink job running.

Resolves #15770.

Changes

  • New: CommitGate.java -- @FunctionalInterface with a single method: boolean isCommitAllowed(long checkpointId)
  • Modified: IcebergWriteAggregator -- accepts optional gate, adds ListState for buffering, gate check + buffer/flush logic in prepareSnapshotPreBarrier(), state initialization in initializeState()
  • Modified: IcebergSink.Builder -- new commitGate() method, passed through to aggregator

Compatibility

  • No behavioral change when the gate is not set (null default)
  • Buffered committables are checkpointed in ListState, so recovery works correctly
  • No changes to public API signatures of existing methods
  • Fully backward compatible

@github-actions github-actions bot added the flink label Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flink Sink V2: Add CommitGate plugin interface for deferred commits

1 participant