[#11590] fix(doris): upgrade type system for Doris 3.0+/4.0.x compatibility by jiangxt2 · Pull Request #11763 · apache/gravitino

jiangxt2 · 2026-06-22T11:13:58Z

What changes were proposed in this pull request?

Type Converter Enhancement (`DorisTypeConverter.java`)

toGravitino() direction (Doris → Gravitino):

Parameterized type parsing: Extract base type name from parameterized types
("int(11)" → "int", "decimal(10,2)" → "decimal") before switch matching,
preventing them from falling into ExternalType fallback
datetime(N) precision: Parse precision directly from type string when JDBC
datetimePrecision is null, bypassing inaccurate COLUMN_SIZE from older MySQL drivers
New type mappings: binary/varbinary → BinaryType, json/variant/ipv4/ipv6
/largeint/bitmap/hll/bigint unsigned → ExternalType, datev2 → DateType
Error handling: parseTypeParamsOrExternal() catches NumberFormatException on
malformed type strings and returns ExternalType instead of crashing
Malformed input guard: endsWith(")") check prevents StringIndexOutOfBoundsException
for inputs like "datetime(" or "varchar("

fromGravitino() direction (Gravitino → Doris):

datev2: DateType → "datev2" (Doris 1.2+ accepts datev2; 4.0.x requires it
due to disable_datev1=true)
ExternalType round-trip: ExternalType → catalogString() (e.g.
ExternalType("json") → "json")
Complex types: array/map/struct are mapped to ExternalType in toGravitino
direction. fromGravitino does not handle ListType/MapType/StructType (throws
IllegalArgumentException). Full complex type support requires SHOW CREATE TABLE parsing,
which will be added in a follow-up PR.

DATETIME Precision Fix (`DorisTableOperations.java`)

DATETIME(N) first: Parse precision directly from type string (e.g. "DATETIME(3)"
→ 3), no dependency on JDBC COLUMN_SIZE or driver version
Plain DATETIME fallback: Uses COLUMN_SIZE with MySQL driver version guard
(requires >= 8.0.16 for accurate precision)

Does this PR introduce any user-facing change?

No. Type mapping improvements are transparent to users. Complex types (array/map/struct)
are recognized as ExternalType rather than being completely unknown.

How was this patch tested?

Unit tests (TestDorisTypeConverter):

All scalar types, datev2, datetime(N) precision (0/3/6)
Parameterized types: int(11), varchar(100), decimal(10,2), datetime(3)
External type round-trip: json, variant, ipv4, ipv6, largeint, bitmap, hll,
bigint unsigned
Malformed type string fallback: varchar(abc), char(xyz), decimal(a,b)

DATETIME precision tests (TestDorisTableOperations):

DATETIME(0/3/6) precision parsing, DATETIME(x) invalid input
DATETIME(3) with unsupported driver version (returns 3, not null)

Cross-version E2E verification (local Doris clusters):

Scalar type create/load round-trip on Doris 1.2.2, 3.0.6.2, 4.0.6
datev2 column creation verified on all three versions
DATETIME(N) precision verified on all three versions

Related to #11590, Fixes #11762

Note to maintainer: Could you please add - [ ] #11762 to the task list
in #11590 to link this sub-issue? I don't have write access to do it directly.
Thank you!

Copilot

Pull request overview

Improves Apache Doris JDBC catalog type handling to better interoperate with Doris 3.0+/4.0.x, focusing on parameterized type parsing and DATETIME precision extraction, with accompanying unit test updates.

Changes:

Enhance DorisTypeConverter to normalize/parse parameterized scalar types (e.g., decimal(10,2), datetime(3)) and add mappings for additional Doris scalar/external types.
Update DorisTableOperations.calculateDatetimePrecision() to parse DATETIME(N) directly from the type string before falling back to JDBC metadata/driver-version gating.
Extend/adjust unit tests around DATETIME precision and type conversion.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/converter/DorisTypeConverter.java	Adds base-type stripping and parameter parsing; expands scalar/external type mappings; changes DateType output to `datev2`.
catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java	Parses `DATETIME(N)` precision from type string first; keeps driver-version guard for plain `DATETIME`.
catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/converter/TestDorisTypeConverter.java	Adds coverage for new mappings, parameterized type parsing, and malformed parameter handling.
catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperations.java	Adds tests for `DATETIME(N)` parsing and behavior under unsupported MySQL driver versions; adjusts unsupported-type test set.

@@ -475,11 +475,7 @@ public void testCreateNotSupportTypeTable() {
            Types.IntervalDayType.get(),
            Types.IntervalYearType.get(),
            Types.UUIDType.get(),
-            Types.ListType.of(Types.DateType.get(), true),
-            Types.MapType.of(Types.StringType.get(), Types.IntegerType.get(), true),
-            Types.UnionType.of(Types.IntegerType.get()),
-            Types.StructType.of(
-                Types.StructType.Field.notNullField("col_1", Types.IntegerType.get())));
+            Types.UnionType.of(Types.IntegerType.get()));


+      if (parenIndex > 0) {
+        try {
+          String precisionStr = typeName.substring(parenIndex + 1, typeName.length() - 1);
+          int precision = Integer.parseInt(precisionStr);
+          return Types.TimestampType.withoutTimeZone(precision);
+        } catch (NumberFormatException e) {
+          // Fall through to default datetime handling
+        }
+      }


+    if (p1 == 0 && parenIndex > 0) {
+      try {
+        String[] parts = typeName.substring(parenIndex + 1, typeName.length() - 1).split(",");
+        p1 = Integer.parseInt(parts[0].trim());
+        p2 = parts.length >= 2 ? Integer.parseInt(parts[1].trim()) : 0;
+      } catch (NumberFormatException e) {
+        return Types.ExternalType.of(typeName);
+      }
+    }


github-actions · 2026-06-24T02:11:12Z

Code Coverage Report

Overall Project	67.17% `+0.05%`	🟢
Files changed	71.52%	🟢

Module	Coverage
aliyun	1.72%	🔴
api	46.82%	🟢
authorization-common	85.96%	🟢
aws	3.66% `-0.28%`	🔴
azure	2.47%	🔴
catalog-common	10.4%	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.91%	🟢
catalog-hive	79.42%	🟢
catalog-jdbc-clickhouse	80.02%	🟢
catalog-jdbc-common	44.22%	🟢
catalog-jdbc-doris	80.75% `+3.15%`	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	80.72% `+7.24%`	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	58.53%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.86%	🟢
catalog-lakehouse-paimon	82.14%	🟢
catalog-model	77.72%	🟢
cli	44.51%	🟢
client-java	78.01%	🟢
common	50.17%	🟢
core	82.58%	🟢
filesystem-hadoop3	77.27%	🟢
flink	0.0%	🔴
flink-common	47.12%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-common	10.88%	🔴
hive-metastore-common	53.77%	🟢
iceberg-common	58.15%	🟢
iceberg-rest-server	73.9%	🟢
idp-basic	85.71%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.83%	🔴
lance-rest-server	60.13%	🟢
lineage	53.02%	🟢
optimizer	82.95%	🟢
optimizer-api	21.95%	🔴
server	85.96%	🟢
server-common	74.18%	🟢
spark	28.57%	🔴
spark-common	41.66%	🟢
trino-connector	40.25%	🟢

Files

Module	File	Coverage
aws	AwsIrsaCredentialGenerator.java	3.25%	🔴
catalog-jdbc-doris	DorisTypeConverter.java	90.43%	🟢
	DorisTableOperations.java	81.82%	🟢
catalog-jdbc-oceanbase	OceanBaseTableOperations.java	83.33%	🟢

…ompatibility - datev2: fromGravitino(DateType) returns "datev2" for Doris 4.0.x compatibility - DATETIME precision: parse from "datetime(N)" type string when JDBC metadata is null - Parameter type parsing: extract base type from "int(11)" to prevent ExternalType fallback - Type mapping: binary/json/variant/ip/largeint/bitmap/hll to BinaryType/ExternalType - ExternalType round-trip: fromGravitino(ExternalType) returns catalogString() - Error handling: parseTypeParamsOrExternal catches NumberFormatException, returns ExternalType Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com> Co-Authored-By: Chang-Tong <zdcheerful@hotmail.com> Co-Authored-By: ArtificialIdoit <bill.sea@hotmail.com> Co-Authored-By: cwq222 <15503804976@163.com>

jiangxt2 · 2026-06-24T11:25:43Z

Updated the PR to address review feedback and code review findings:

Narrowed scope to scalar types only — complex types (array/map/struct) now mapped to ExternalType in toGravitino; fromGravitino does not handle ListType/MapType/StructType. Full complex type support requires SHOW CREATE TABLE parsing (follow-up PR).
Added endsWith(")") guards for malformed input like "datetime(" / "varchar(" to prevent StringIndexOutOfBoundsException.
Fixed test coverage: added bigint unsigned to round-trip tests, restored ListType/MapType/StructType to unsupported-type test list.
Corrected misleading comments and aligned code style (constant references, default value explanations).
Cross-version E2E verified on Doris 1.2.2, 3.0.6.2, and 4.0.6.

CI checks triggered — ready for review.

jiangxt2 force-pushed the feat/doris-type-system branch from 6fe4bf5 to 6d5e661 Compare June 23, 2026 07:16

FANNG1 requested a review from Copilot June 24, 2026 00:52

Copilot started reviewing on behalf of FANNG1 June 24, 2026 00:52 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

jiangxt2 force-pushed the feat/doris-type-system branch from be05223 to d8fc157 Compare June 24, 2026 11:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[#11590] fix(doris): upgrade type system for Doris 3.0+/4.0.x compatibility#11763

[#11590] fix(doris): upgrade type system for Doris 3.0+/4.0.x compatibility#11763
jiangxt2 wants to merge 1 commit into
apache:mainfrom
jiangxt2:feat/doris-type-system

jiangxt2 commented Jun 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

jiangxt2 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jiangxt2 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Type Converter Enhancement (DorisTypeConverter.java)

DATETIME Precision Fix (DorisTableOperations.java)

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jun 24, 2026

Code Coverage Report

Uh oh!

jiangxt2 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jiangxt2 commented Jun 22, 2026 •

edited

Loading

Type Converter Enhancement (`DorisTypeConverter.java`)

DATETIME Precision Fix (`DorisTableOperations.java`)