[VL][DELTA] Support Delta CDF scan offload

[malinjawi]

What changes are proposed in this pull request?

Addresses #12195.

Delta CDF reads enter Spark as CDCReader.DeltaCDFRelation, so they do not initially have the normal FileSourceScanExec + DeltaParquetFileFormat shape that Gluten's existing Delta scan offload rule recognizes.

This PR adds a Gluten Delta planner strategy, wired from the Velox Delta component, that recognizes DeltaCDFRelation, expands it through Delta's own CDF batch planning path, and rewrites the original projection/filter attributes onto the expanded logical plan. After that, the existing Delta scan offload path can plan the underlying CDF file scans as DeltaScanTransformer.

The change is intentionally scoped to Gluten's Delta/Spark planning layer rather than Velox C++:

• Add DeltaCDFScanStrategy for table_changes(...) and DataFrame readChangeFeed scans.
• Add Delta-version helper shims for Delta 2.3, 2.4, 3.3, and 4.x API differences.
• Register the planner strategy from VeloxDeltaComponent.
• Add Delta regression coverage for insert/update/delete CDF rows, filter/projection handling, bounded version reads, DataFrame readChangeFeed, column mapping, and a startingVersion = 0 case.

One planner/test nuance: Delta CDF expansion can keep a Spark-side ExistingRDD branch for synthesized change rows, including on the tested update/delete CDF paths. The regression suite therefore compares results against vanilla Spark and asserts that the expanded CDF file scans are transformed to DeltaScanTransformer, rather than requiring the entire expanded CDF union to be globally fallback-free.

How was this patch tested?

Local checks used JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home.

• git diff --check
• ./dev/format-scala-code.sh check
• ./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile

Earlier cross-version compile checks also passed before the test-expectation-only update:

• ./build/mvn -pl gluten-delta -am -Pspark-3.3 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-3.4 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
• ./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile

Full native Velox runtime and benchmarking are left to CI / a native Gluten-optimized environment; this local checkout does not have cpp/build/releases/libgluten.so and the Velox external project build available.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[felipepessoto]

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:

  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);

compared against vanilla Spark (checkAnswer)

[felipepessoto]

Do you know what happens in Delta 4.1/4.2/4.3? Would it take the code from 4.0 folder?

[malinjawi]

Yes @felipepessoto I checked the current Maven profile/source wiring:

• spark-4.0 sets delta.version=4.0.1 and delta.binary.version=40
• spark-4.1 sets delta.version=4.1.0 and delta.binary.version=40
• the Delta profile adds sources from src-delta${delta.binary.version}/main/scala

So with the current profile setup, both Spark 4.0 / Delta 4.0.x and Spark 4.1 / Delta 4.1.x use gluten-delta/src-delta40/....

For future Delta 4.2 / 4.3 support, it depends on how those profiles are introduced. If they keep delta.binary.version=40, they will continue to use the same src-delta40 helper. If Delta changes the relevant CDF APIs and Gluten introduces a new binary bucket, then we should add a new version-specific helper.

This also relates to the earlier folder-activation discussion in #11924: supporting family-level folders like src-spark4 / src-delta4 would make this cleaner and avoid copying code across Spark/Delta 4.x profiles when the APIs stay compatible. I think that belongs in a separate Maven/source-layout refactor rather than in this CDF PR.

[malinjawi]

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:

  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);

compared against vanilla Spark (checkAnswer)

Thanks @felipepessoto good point! I agree this is worth covering explicitly.

I’ll add a focused DV-enabled CDF regression with delta.enableChangeDataFeed=true and delta.enableDeletionVectors=true, comparing table_changes(...) against vanilla Spark via the existing runAndCompare / checkAnswer path.

One important nuance: this PR is not intended to claim full native DV support. DV-backed Delta scan execution is still a separate active area in Gluten/Velox and may continue to fall back where native DV handling is not available. The goal of the regression here is to make sure enabling DV does not break the CDF planning path or result correctness.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86