Dropping a schema field that was part of an old spec corrupts the table

[gabeiglio]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

Steps to reproduce:

run_spark_commands(spark, [
    "DROP TABLE IF EXISTS ns.table",
    "CREATE TABLE ns.table (a int, b int) USING ICEBERG PARTITIONED BY (a)",
    "INSERT INTO ns.table VALUES (1, 2), (1, 2), (2, 1)",
    "ALTER TABLE ns.table DROP PARTITION FIELD a",
    "ALTER TABLE ns.table DROP COLUMN a"
])

table = catalog.load_table("ns.table")

# ERROR
print(len(table.scan().plan_files()))

Here is the PR with the fix

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

Thanks @gabeiglio, closed by #2367