[GLUTEN-11622][VL] Add basic TIMESTAMP_NTZ type support (#11939) by rui-mo · Pull Request #12229 · apache/gluten

rui-mo · 2026-06-04T01:32:55Z

What changes are proposed in this pull request?

Enables TimestampNTZ scan by default. Supports Arrow conversion and fallback of unsupported function.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Related issue: #11622

github-actions · 2026-06-04T01:33:24Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-04T01:36:40Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-04T02:07:12Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-04T03:46:11Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-09T01:31:49Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Adds initial Velox-backend support for Spark TIMESTAMP_NTZ by enabling native scan paths, wiring Substrait/Arrow type conversions, and adjusting validator/fallback behavior and test coverage across supported Spark versions.

Changes:

Enable native Parquet scan for TIMESTAMP_NTZ and add validation/translation plumbing (Substrait + C++ parser/expr handling).
Extend Spark↔Arrow and Spark↔Substrait type mappings for timestamp_ntz, plus related iterator/columnar-to-row handling.
Add/adjust unit tests and backend test settings/exclusions for TIMESTAMP_NTZ behaviors.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala	Adds cache/uncache coverage for `TIMESTAMP_NTZ`.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Updates Velox exclusions related to `TIMESTAMP_NTZ` and associated behaviors.
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala	Adds cache/uncache coverage for `TIMESTAMP_NTZ`.
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Updates Velox exclusions related to `TIMESTAMP_NTZ` and associated behaviors.
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala	Adds cache/uncache coverage for `TIMESTAMP_NTZ`.
gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Updates Velox exclusions related to `TIMESTAMP_NTZ` and associated behaviors.
gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala	Adjusts supported Spark SQL query-test input lists (notably timestamp-related inputs).
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala	Adds cache/uncache coverage for `TIMESTAMP_NTZ`.
gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Updates Velox exclusions related to `TIMESTAMP_NTZ` and associated behaviors.
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala	Adds cache/uncache coverage for `TIMESTAMP_NTZ`.
gluten-ut/spark33/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Updates Velox exclusions related to `TIMESTAMP_NTZ` and associated behaviors.
gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenSQLTestsBaseTrait.scala	Adjusts Spark test conf to bind driver host/address to localhost.
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala	Refines validator/fallback rules around `TIMESTAMP_NTZ` (scan vs non-scan).
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala	Adds `TIMESTAMP_NTZ` Spark/Substrait conversions and signatures.
gluten-substrait/src/main/java/org/apache/gluten/substrait/type/TypeBuilder.java	Adds TypeBuilder entry point for `TimestampNTZTypeNode`.
gluten-substrait/src/main/java/org/apache/gluten/substrait/type/TimestampNTZTypeNode.java	Introduces Substrait type node for `TIMESTAMP_NTZ` (mapped via Substrait TIMESTAMP).
gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/TimestampNTZLiteralNode.java	Introduces literal node for `TIMESTAMP_NTZ`.
gluten-delta/src/test/scala/org/apache/gluten/execution/DeltaSuite.scala	Updates Delta test expectation for `TIMESTAMP_NTZ` table behavior.
gluten-arrow/src/main/scala/org/apache/spark/sql/utils/SparkArrowUtil.scala	Adds Arrow mapping for `timestamp_ntz` (no timezone).
cpp/velox/substrait/SubstraitToVeloxPlan.cc	Updates includes for Arrow/Velox utilities during plan conversion.
cpp/velox/substrait/SubstraitToVeloxExpr.cc	Extends literal type mapping for Substrait `timestamp` literals.
cpp/velox/substrait/SubstraitParser.cc	Extends Substrait type parsing and literal extraction for `timestamp` (no-tz).
backends-velox/src/test/scala/org/apache/gluten/functions/DateFunctionsValidateSuite.scala	Adds coverage for reading `timestamp_ntz` and ensuring unsupported functions fall back.
backends-velox/src/test/scala/org/apache/gluten/execution/VeloxParquetDataTypeValidationSuite.scala	Updates validation test to assert native scan for `TIMESTAMP_NTZ`.
backends-velox/src/test/scala/org/apache/gluten/execution/FallbackSuite.scala	Adjusts fallback behavior test to Spark 3.4+ and new expectations.
backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToRowExec.scala	Allows `timestamp_ntz` through columnar-to-row validation.
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala	Changes default for `enableTimestampNtzValidation`.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxValidatorApi.scala	Updates schema validation allowance for `timestamp_ntz` when validation disabled.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala	Adds partition value formatting support for `timestamp_ntz`.
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaInsertIntoTableSuite.scala	Ignores a Delta `timestamp_ntz` round-trip test due to unsupported cast behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-09T06:19:39Z

Run Gluten Clickhouse CI on x86

zhouyuan

👍

zhouyuan · 2026-06-09T13:55:47Z

Run Gluten Clickhouse CI on x86

…alue

github-actions · 2026-06-10T03:11:25Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

+  private class FallbackByTimestampNTZ(enableValidation: Boolean) extends Validator {
    override def validate(plan: SparkPlan): Validator.OutCome = {
      if (!enableValidation) {
-        // Validation is disabled, allow TimestampNTZ
-        return pass()
-      }
-
-      def containsNTZ(dataType: DataType): Boolean = dataType match {
-        case dt if dt.catalogString == "timestamp_ntz" => true
-        case st: StructType => st.exists(f => containsNTZ(f.dataType))
-        case at: ArrayType => containsNTZ(at.elementType)
-        case mt: MapType => containsNTZ(mt.keyType) || containsNTZ(mt.valueType)
-        case _ => false
-      }
-      val hasNTZ = plan.output.exists(a => containsNTZ(a.dataType)) ||
-        plan.children.exists(_.output.exists(a => containsNTZ(a.dataType)))
-      if (hasNTZ) {
-        fail(s"${plan.nodeName} has TimestampNTZType in input/output schema")
-      } else {
-        pass()
+        // Validation is disabled, allow supported operators.
+        def containsNTZ(dataType: DataType): Boolean = dataType match {
+          case dt if dt.typeName == "timestamp_ntz" => true
+          case st: StructType => st.exists(f => containsNTZ(f.dataType))
+          case at: ArrayType => containsNTZ(at.elementType)
+          case mt: MapType => containsNTZ(mt.keyType) || containsNTZ(mt.valueType)
+          case _ => false
+        }
+        val isScan = plan match {
+          case _: BatchScanExec => true
+          case _: FileSourceScanExec => true
+          case p if HiveTableScanExecTransformer.isHiveTableScan(p) => true
+          case _ => false
+        }
+        val hasNTZ = plan.output.exists(a => containsNTZ(a.dataType)) ||
+          plan.children.exists(_.output.exists(a => containsNTZ(a.dataType)))
+        if (isScan || !hasNTZ) {
+          return pass()
+        }
      }
+      fail(s"${plan.nodeName} has TimestampNTZType in input/output schema")
    }


github-actions · 2026-06-10T04:27:15Z

Run Gluten Clickhouse CI on x86

github-actions Bot added CORE works for Gluten Core VELOX DATA_LAKE labels Jun 4, 2026

rui-mo force-pushed the ts_ntz_dev branch from d5bbbf7 to 63fc54b Compare June 4, 2026 01:36

rui-mo force-pushed the ts_ntz_dev branch from 63fc54b to 45e2bac Compare June 4, 2026 02:06

Copilot AI review requested due to automatic review settings June 9, 2026 01:31

rui-mo force-pushed the ts_ntz_dev branch from 65182e0 to df10cf0 Compare June 9, 2026 01:31

Copilot started reviewing on behalf of rui-mo June 9, 2026 01:31 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

rui-mo force-pushed the ts_ntz_dev branch from df10cf0 to 5afbafd Compare June 9, 2026 06:19

rui-mo mentioned this pull request Jun 9, 2026

[VL] Enable hour(timestamp_ntz) native execution in Velox backend #12064

Open

zhouyuan approved these changes Jun 9, 2026

View reviewed changes

rui-mo added 2 commits June 9, 2026 19:30

[GLUTEN-11622][VL] Add basic TIMESTAMP_NTZ type support (#11939)

59485e6

[GLUTEN-11622][VL] Use formatted string for TIMESTAMP_NTZ partition v…

d36e9b0

…alue

Copilot AI review requested due to automatic review settings June 10, 2026 03:10

rui-mo force-pushed the ts_ntz_dev branch from 5afbafd to 3a14212 Compare June 10, 2026 03:10

Copilot started reviewing on behalf of rui-mo June 10, 2026 03:10 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Fallback TimestampNTZ for CH backend

22a7b49

rui-mo force-pushed the ts_ntz_dev branch from 3a14212 to 22a7b49 Compare June 10, 2026 04:26

Conversation

rui-mo commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

zhouyuan left a comment

Choose a reason for hiding this comment

Uh oh!

zhouyuan commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rui-mo commented Jun 4, 2026 •

edited

Loading