Filtered scans fail entirely for tables where a historical partition spec's source column has been dropped

[anoopj]

Apache Iceberg Rust version

0.6.0 (latest version)

Describe the bug

plan_files() returns an error for any scan with a predicate, even one that touches no partition column if the snapshot contains a manifest written under a historical partition spec whose source column is no longer in the current schema. The unfiltered scan of the same table succeeds, so the data is readable; adding any filter makes the table's historical data unqueryable.

This is a legal table state in format v2 and later (including v3): a column may be dropped once it is no longer used by the default spec, but historical specs referencing it are retained in table metadata because existing manifests still use them.

Related: #2842 (the same table state panics the manifests metadata table; fix in #2843). This issue is the scan-planning counterpart and is independent of that fix.

To Reproduce

1. Create a v2 table (v3 behaves identically) with schema (id int, is_test boolean) and spec identity(is_test) (spec 0), and write a snapshot so a manifest with spec 0 exists.
2. Evolve the spec to remove the is_test partition field (spec 1 becomes default).
3. Drop the is_test column from the schema.
4. Scan with any predicate on a live column, e.g. id > 0, and call plan_files().

Step 4 returns:

Unexpected => No column with source column id 2 in schema [...]

An unfiltered scan of the same table succeeds and returns the expected FileScanTasks (with the correct partition values, since manifest reading resolves types from the manifest's own embedded metadata).

Expected behavior

The scan should succeed. Partition pruning is an optimization: a manifest whose spec can't be fully resolved should at worst not be pruned, never cause planning to fail. Two levels of fix:

1. Stopgap: when a spec's partition type can't be resolved against the current schema, skip partition pruning for that spec's manifests (treat the partition filter as always-true). This should be strictly safe since a predicate cannot reference a dropped column, so the skipped filter could never have pruned based on the missing field anyway; remaining row-level filtering is unaffected. I'd implement this as a resolvability pre-check (all of the spec's source IDs present in the schema) rather than by matching the error, so it composes with the longer-term fix.
2. Longer term: make partition_type succeed for this state: derive the output type from the transform where it doesn't depend on the source (bucket, year/month/day/hour, void) and degrade to the v3 unknown type only for identity/truncate. This is the approach iceberg-java is taking in API, Core: Only fall back to unknown partition type for source-dependent transforms iceberg#17262 (context: Core: Add v4 manifest reader iceberg#16958); for rust it also depends on adding the v3 unknown type, which I'll file separately. With that in place, pruning keeps working on the still-resolvable fields and the stopgap becomes unnecessary.

I can contribute the stopgap fix.

Willingness to contribute

I can contribute a fix for this bug independently