Delta column mapping mode caused wrong result with partition column filters

[sezruby]

Backend

VL (Velox)

Bug description

 spark.sql(s"""
                    |create table delta_cm2 (id int, name string) using delta
                    |partitioned by (id)
                    |tblproperties ("delta.columnMapping.mode"= "name")
                    |""".stripMargin)
spark.sql(s"""
                    |insert into delta_cm2 values (1, "v1"), (2, "v2"), (3, "v3")
                    |""".stripMargin)
spark.sql("select name from delta_cm2 where id > 2")

[Expected behavior]
returns ["v3"]

[actual behavior]
returns ["v1", "v2", "v3"]

  == Physical Plan ==
  VeloxColumnarToRow
  +- ^(32) ProjectExecTransformer [col-4aba4d55-65eb-461f-9a50-154cb6b1c6ec#6087 AS name#6087]
     +- ^(32) FileScanTransformer parquet spark_catalog.default.delta_cm2[col-4aba4d55-65eb-461f-9a50-154cb6b1c6ec#6087,col-9a0726cb-9edd-4f44-b7f7-a1ccfc5474c3#6086] Batched: true, DataFilters: [], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[file:/root/gluten/backends-velox/spark-warehouse/org.apache.glute..., PartitionFilters: [isnotnull(col-9a0726cb-9edd-4f44-b7f7-a1ccfc5474c3#6086), (col-9a0726cb-9edd-4f44-b7f7-a1ccfc547..., PushedFilters: [], ReadSchema: struct<col-4aba4d55-65eb-461f-9a50-154cb6b1c6ec:string> NativeFilters: []
  
  == Results ==
  
  == Results ==
  !== Correct Answer - 1 ==   == Gluten Answer - 3 ==
   struct<>                   struct<>
  ![v3]                       [v1]
  !                           [v2]
  !                           [v3] (GlutenQueryTest.scala:437)

This is because Delta filtering expects logical column name in filters:
https://github.com/delta-io/delta/blob/44c619e51846f3e98aa6605d13fa4517de049281/spark/src/main/scala/org/apache/spark/sql/delta/stats/PrepareDeltaScan.scala#L356

Though Gluten changed logical column names to physical column names for parquet reader, filter pushdown.
In JVM Spark, Delta changes the name to physical column names when creating parquet reader.
https://github.com/delta-io/delta/blob/44c619e51846f3e98aa6605d13fa4517de049281/spark/src/main/scala/org/apache/spark/sql/delta/DeltaParquetFileFormat.scala#L179

We need to add a fallback for column mapping mode, both id and name.

Gluten version

Gluten-1.3

Spark version

Spark-3.5.x

Spark configurations

Spark3.5 and Delta 3.2

System information

No response

Relevant logs

[FelixYBW]

@zhztheplayer is this fixed in main?

[sezruby]

Also id mapping is not fully supported - parquet.field.id is not used in Gluten/Velox

[zhztheplayer]

@sezruby May I know if you are using Gluten only for Delta Scan or for all the Gluten features?

[sezruby]

@zhztheplayer I work at MS Fabric Spark Runtime, providing Gluten/Velox version for customers.
We internally validated the feature before and add column mapping fallback for NEE scan as we found unit tests failures.
Recently tried to enable it and found this design fault.

The right fix would be keeping both logical and physical name in Gluten side.

[FelixYBW]

@sezruby @zhztheplayer is the issue fixed?

[zhztheplayer]

I ran a similar test with latest Gluten + Spark 3.5 / Delta 3.3 and it passed:

withTable("delta_cm2") {
  spark.sql(s"""
               |create table delta_cm2 (id int, name string) using delta
               |tblproperties ("delta.columnMapping.mode"= "name")
               |""".stripMargin)
  spark.sql(s"""
               |insert into delta_cm2 values (1, "v1"), (2, "v2"), (3, "v3")
               |""".stripMargin)

  val df2 = runQueryAndCompare("select name from delta_cm2 where id > 2") { _ => }
  checkLengthAndPlan(df2, 1)
  checkAnswer(df2, Row("v3") :: Nil)
}

[sezruby]

Opened #12240 with a Gluten-only fix.

Root cause: DeltaPostTransformRules rewrote partition filters and partitionSchema to physical names, but Delta's PreparedDeltaFileIndex.matchingFiles / DeltaLog.rewritePartitionFilters resolve them against the logical metadata.partitionSchema, so partition pruning silently no-op'd (and file-level stats skipping was disabled for the same reason).

Fix: keep filters and partition schema logical on the scan node; only output / dataSchema / data fields of requiredSchema go physical. Native side gets a physical-translated copy via DeltaScanTransformer.scanFilters (filter binding to native is by exprId, so logical-named filter attributes still resolve correctly against the physical-named output). Tests added in DeltaSuite for both name and id modes covering partition pruning, multi-partition, partition+data filter combos, IS [NOT] NULL, and column rename.

@zhztheplayer @FelixYBW Could you review the PR?
FYI I didn't run the tests in local as I don't have velox native build setup on my local. I'll fix any test failure after CI done.

I believe id mapping issue still persists, let me check further and open a new issue.