Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found"

[wjixiang]

Apache Iceberg Rust version

iceberg-rust: git rev 713e201
(crate metadata 0.10.0, 2026-07-01)

Describe the bug

Summary

A table with a user data column named pos writes fine and appears correctly in the catalog schema, but any scan that projects it fails with External(Unexpected => "field not found"). Renaming the column to anything else (e.g. position) makes it readable again. The same applies to file_path.

The bare names pos and file_path (no leading underscore) are reserved as metadata-column names (RESERVED_COL_NAME_DELETE_FILE_POS, RESERVED_COL_NAME_DELETE_FILE_P ATH), and the data-table scan resolves column names with metadata-first precedence and no fallback to a same-named data column, so a real data column named pos/f ile_path is shadowed.

Minimal reproduction (pure iceberg crate, no catalog/storage)

Uses only the public metadata_columns API — runs in milliseconds with no infrastructure:

use iceberg::metadata_columns::{
    get_metadata_field_id, is_metadata_column_name,
    RESERVED_COL_NAME_DELETE_FILE_POS, RESERVED_FIELD_ID_DELETE_FILE_POS,
};

fn main() {
    // The reserved name is the bare `pos` (not the spec's `_pos`).
    assert_eq!(RESERVED_COL_NAME_DELETE_FILE_POS, "pos");

    // A user data column named `pos` is misclassified as a metadata column.
    // Per the Iceberg spec only `_`-prefixed names are metadata columns,
    // so this should be `false`.
    assert!(is_metadata_column_name("pos"));
    assert!(is_metadata_column_name("file_path"));

    // So a data column `pos` resolves to the reserved delete-file field id
    // (i32::MAX - 102) instead of its real schema field id.
    assert_eq!(
        get_metadata_field_id("pos").unwrap(),
        RESERVED_FIELD_ID_DELETE_FILE_POS,
    );
}

Expected behavior

A user data column named pos should be readable. The Iceberg spec only reserves _-prefixed names for metad
ata columns (_file, _pos, _partition, _spec_id, _deleted). The bare name pos is a valid data column name — the Java reference implementation does not treat it as a metadata column and reads it normally.

Actual behavior (end-to-end)

With a real table whose schema is chrom: String (id=1), pos: Int (id=2):

SELECT chrom FROM t   -> OK
SELECT pos    FROM t   -> External error: Unexpected => "field not found"
SELECT *      FROM t   -> External error: Unexpected => "field not found"   (pos is in the projection)

The catalog schema is correct (id=2 name=pos type=Int), so this is purely a read-path problem. Workaround: rename the column before writing (pos -> position, chro m_pos, etc.).

Root cause

crates/iceberg/src/metadata_columns.rs defines the bare reserved names — these describe the columns inside a position-delete file, not metadata columns of a data t
able:

pub const RESERVED_COL_NAME_DELETE_FILE_POS: &str  = "pos";        // no underscore
pub const RESERVED_COL_NAME_DELETE_FILE_PATH: &str = "file_path";  // no underscore

is_metadata_column_name() (via get_metadata_field_id) returns Ok for both.

crates/iceberg/src/scan/mod.rs applies that check to every projected column of a data-table scan, with metadata-first precedence and continue (no fallback to the
data schema):

for column_name in column_names.iter() {
    if is_metadata_column_name(column_name) {
        field_ids.push(get_metadata_field_id(column_name)?); // pos -> RESERVED_FIELD_ID_DELETE_FILE_POS
        continue;                                            // never reaches field_id_by_name
    }
    let field_id = schema.field_id_by_name(column_name)?;    // real data column never consulted
    ...
}

So even though the table has pos as field id 2, the name pos is intercepted by the metadata branch before the data-schema lookup, and the scanner goes looking for
RESERVED_FIELD_ID_DELETE_FILE_POS in the Parquet file, which has no such column → field not found.

Why metadata-first precedence is usually fine — and why it breaks here

The data-table metadata columns (_file, _pos, _partition, _spec_id, _deleted) are virtual: they don't exist in the data schema or the Parquet file, they ar
e injected by the scanner via a reserved field id. The "check metadata first, then continue" rule is how those virtual columns get materialized. The spec makes this sa
fe by restricting metadata names to a _ prefix, which cannot collide with user data columns. The bug is that pos/file_path (delete-file internal schema columns
) were added to the same name set consulted by data-table scans, but without the _ prefix — so they can and do collide with legitimate data column names, and the m
etadata-first precedence silently shadows them.

name	spec role	_-prefixed?	metadata-first safe?
_file _pos _partition	virtual metadata columns	yes	yes (can't collide)
pos file_path	delete-file internal schema columns	no	no (collides with user data)

This is a real-world problem for bioinformatics: VCF POS is one of the most common column names (oxbow emits it as pos), so any VCF → Iceberg pipeline hits this.

Suggested fix

Separate the two concepts so data-table scans only match the spec's _-prefixed metadata names:

• Keep RESERVED_COL_NAME_POS = "_pos" (and _file, _partition, _spec_id, _deleted) in the set consulted by the scan path's is_metadata_column_name check.
• Do not let RESERVED_COL_NAME_DELETE_FILE_POS = "pos" / RESERVED_COL_NAME_DELETE_FILE_PATH = "file_path" satisfy the scan-path metadata check — they should only
  be used when reading a delete file's own schema.

(Alternative: in the metadata branch, if the name also exists as a real data column in the current table schema, prefer the data column's field id. But restricting the s
et to _-prefixed names is cleaner and restores the spec's collision-safety guarantee.)

Environment

• iceberg-rust: git rev 713e201 (workspace crates/iceberg)
• observed via the iceberg-datafusion table provider + REST catalog, but the bug is in core scan + metadata_columns

Happy to open a PR for the suggested fix if the approach sounds right.

To Reproduce

No response

Expected behavior

No response

Willingness to contribute

None