Skip to content

[iceberg] TestSparkDataWrite show mismatch results #2118

@hsiang-c

Description

@hsiang-c

Describe the bug

Take from GitHub CI of #1987

TestSparkDataWrite > testPartitionedCreateWithTargetFileSizeViaOption() > format = PARQUET, branch = null FAILED
    org.opentest4j.AssertionFailedError: [Result rows should match] 
TestSparkDataWrite > testPartitionedFanoutCreateWithTargetFileSizeViaOption2() > format = PARQUET, branch = main FAILED
    org.opentest4j.AssertionFailedError: [Result rows should match] 

Steps to reproduce

SparkSession configs used:

            .config("spark.plugins", "org.apache.spark.CometPlugin")
            .config("spark.shuffle.manager", "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
            .config("spark.comet.explainFallback.enabled", "true")
            .config("spark.sql.iceberg.parquet.reader-type", "COMET")
            .config("spark.memory.offHeap.enabled", "true")
            .config("spark.memory.offHeap.size", "10g")
            .config("spark.comet.use.lazyMaterialization", "false")
            .config("spark.comet.schemaEvolution.enabled", "true")

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions