feat: Delta to Iceberg v3 conversion support tracking issue

[vladislav-sidorovich]

Feature Request / Improvement

Tracking issue for updating delta-lake module to support conversion from Delta Lake format to Iceberg v3 format.

Initial update PR: #15407

Millestones:

• Converts all data types
• Converts table schema and partitions spec
• Support UPDATES and DELETS (Delta Lake Remove action)
• Support Delta VACUUM scenario
• Support DVs
• Handle Schema evolution
• Handle Edge cases for partitions and Generated columns
• Support All Delta Lake actions
• Support All Delta Lake features (column mapping, generated columns and so on)
• Incremental Conversion (from/to a specific Delta Version)

Detailed plan

SCHEMA & METADATA

• Map Delta (Spark) primitive types (IntegerType, StringType, TimestampNTZType) to Iceberg primitive types.
• Map complex types (StructType, ArrayType, MapType) recursively.
• Implement ID assignment strategy (map Delta's physical IDs if columnMapping is enabled, otherwise generate sequential IDs).
• Translate column nullability and documentation/comments.

TABLE LAYOUT & PARTITIONING

• Parse Delta's partitionColumns from the metadata action.
• Analyze Delta's generatedColumns expressions (e.g., CAST(ts AS DATE), YEAR(ts)).
• Map Delta generated expressions to Iceberg Transforms (day(ts), year(ts)).
• Map explicit (non-generated) partition columns to Iceberg 'identity' transforms.
• Read Delta's latest OPTIMIZE commit info to extract zOrderBy or Liquid Clustering configurations.
• Translate Delta OPTIMIZE into an Iceberg SortOrder attached to the table metadata.

DATA FILES & STATISTICS TRANSLATION

• Handle Add/Remove Delta Actions
• Extract minValues, maxValues, nullCount, and numRecords from Delta metadata (add action)
• Convert Delta string stats to Iceberg's truncated byte buffers (handle 16-byte truncation and upper-bound increments)
• Handle the NaN gap (leave Iceberg nan_value_counts empty).
• Handle VACUUM operation for Delta Lake tables

HANDLE ALL DELTA LAKE PROTOCOL ACTION

• Metadata (Schema definition/evolution) action
• Add action
• Remove action
• CDC action
• Txn action
• Protocol action
• Commit info action
• Domain metadata action
• Sidecar action
• Check point action
• Reconciliation action

ADVANCED FEATURES & MUTABILITY

• Deletion Vectors Support
• Table Features & Properties

CONTINUOUS SYNC

• Store the last synced Delta version in the Iceberg table properties (e.g., delta_sync_version=15).
• Build a poller that reads Delta commits starting from delta_sync_version + 1.
• Handle on-the-fly schema evolution (apply Iceberg UpdateSchema API if a new Delta commit contains altered Metadata).

CLEAN-UP

• Remove usage of Delta Kernel internal classes

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time