[EPIC] Deletion vector read support (`deletion-vector-v1` Puffin blobs)

[mbutrovich]

Is your feature request related to a problem or challenge?

Iceberg format version 3 stores positional deletes as deletion vectors instead of position delete files. A deletion vector is a roaring bitmap of deleted row positions, written as a deletion-vector-v1 Puffin blob and referenced from a delete manifest entry by content_offset and content_size_in_bytes.

iceberg-rust does not apply them on read. CachingDeleteFileLoader::load_file_for_task branches only on DataContentType (PositionDeletes / EqualityDeletes / Data), and the PositionDeletes arm unconditionally calls parquet_to_batch_stream(...) on the file path (crates/iceberg/src/arrow/caching_delete_file_loader.rs). A V3 deletion vector arrives as PositionDeletes content but points at a Puffin file, so the read fails when the loader parses that file as Parquet. The DV branch is stubbed: DeleteFileContext carries // TODO: Delete Vector loader from Puffin files (caching_delete_file_loader.rs:54). Any V3 table with positional deletes is therefore unreadable.

This issue tracks the read path under the V3 epic #2411, scoped to land independently of the DV write path.

Current state and prior art

The primitives are already on main:

• DataFile parses the V3 manifest fields referenced_data_file, content_offset, content_size_in_bytes (crates/iceberg/src/spec/manifest/data_file.rs).
• DeleteVector (a RoaringTreemap wrapper) and the ParsedDeleteFileContext::DelVecs loader variant exist (crates/iceberg/src/delete_vector.rs, caching_delete_file_loader.rs).
• Positional deletes already flow DeleteVector -> DeleteFilter -> ArrowReader, so DV application reuses that path unchanged.
• PuffinReader, PuffinWriter, and the DELETION_VECTOR_V1 blob-type constant exist (crates/iceberg/src/puffin/).

Open PRs cover parts of the read path, but none is landable read-first:

• feat(delete-vector): parse deletion-vector-v1 puffin blob - read support #2414 implements the decode codec (deletion-vector-v1 blob into a RoaringTreemap) with Java-produced golden fixtures. Decode only.
• feat(scan): apply V3 deletion vectors on read (stacked on #2678) #2681 implements the scan-task coordinates and the loader/apply branch (a DeleteFileIndex that supersedes position deletes, load-by-offset), validated against DuckDB-written DVs. It is stacked on feat(transaction): RowDelta merge-on-read deletion vectors (carries forward #2203) #2678 (DV write / RowDelta merge-on-read), which is stacked on feat(transaction): add RowDelta transaction action for row-level modi… #2203 (copy-on-write RowDelta). Both are conflicting and unreviewed.
• feat(transaction): RowDelta merge-on-read deletion vectors (carries forward #2203) #2678 and feat(scan): apply V3 deletion vectors on read (stacked on #2678) #2681 also carry their own copy of the codec, duplicating feat(delete-vector): parse deletion-vector-v1 puffin blob - read support #2414.

These are useful references but not dependencies. This issue implements the read path independently and decoupled from the write stack, reusing the Java golden fixtures as the test oracle and #2681's apply logic as a design reference.

Describe the solution you'd like

Implement DV read as a series of small PRs that depend only on each other, not on the write path (#2678 / #2203). DV writing is out of scope here; it belongs with #2678 under #2411.

Task list

• Task 1: DV blob codec (decode). Parse a deletion-vector-v1 blob into deleted positions, tested against the Java-produced golden fixtures (as in feat(delete-vector): parse deletion-vector-v1 puffin blob - read support #2414). Byte layout below. No scan or reader API changes, so it merges first.
• Task 2: Carry DV coordinates on the scan task. Add content_offset, content_size_in_bytes, and referenced_data_file to FileScanTaskDeleteFile (crates/iceberg/src/scan/task.rs), populated from DataFile during scan planning, depending only on Task 1. This matches Iceberg-Java, where a DV is a DeleteFile (content=POSITION_DELETES, format=PUFFIN) attached to the task via FileScanTask.deletes().
• Task 3: Loader branch, apply rules, end-to-end read. In load_file_for_task, route entries with content_offset.is_some() to a DV path: read the blob at content_offset / content_size_in_bytes via PuffinReader, decode with Task 1, and return ParsedDeleteFileContext::DelVecs. Apply the spec rules below (DeleteFileIndex supersede logic as in feat(scan): apply V3 deletion vectors on read (stacked on #2678) #2681, decoupled from write). Validate that the decoded cardinality equals record_count and the blob length equals content_size_in_bytes. Add an end-to-end test reading a V3 table whose deletes are DVs.

Two read-path rules from the table spec must hold. A DV applies to a data file when the file path equals the DV's referenced_data_file, the file's data sequence number is <= the DV's, and the partitions match. When a DV applies to a data file, readers ignore any position delete files that would otherwise match it, because the DV subsumes them. At most one DV exists per data file per snapshot.

Task 1 byte layout

From the specs, confirmed against Iceberg-Java BitmapPositionDeleteIndex and RoaringPositionBitmap (reference implementation in #2414):

• Blob: [length: u32 BE][magic: D1 D3 39 64][vector][crc: u32 BE], where length = 4 + vector_size and the CRC-32 covers the magic and vector, not the length prefix. Java writes the magic as the little-endian int 1681511377 (0x6439D3D1); writing the four bytes D1 D3 39 64 directly is equivalent.
• Vector (portable 64-bit layout): an 8-byte little-endian bitmap count, then per ascending 32-bit key a 4-byte little-endian key followed by a standard 32-bit Roaring bitmap. Build the framing on roaring::RoaringBitmap (32-bit) serialize_into / deserialize_from, which implement the 32-bit RoaringFormatSpec, not RoaringTreemap::serialize_into, which uses a crate-native format. Add crc32fast for the CRC-32.

References

• Iceberg table spec format/spec.md, "Deletion Vectors": semantics, at-most-one-per-data-file, DV-subsumes-position-deletes, apply-scope rules, and the referenced_data_file / content_offset / content_size_in_bytes fields (required for DVs, must match the Puffin footer).
• Iceberg Puffin spec format/puffin-spec.md: the deletion-vector-v1 blob byte layout.
• Iceberg-Java: core/.../deletes/BitmapPositionDeleteIndex.java (framing, length, magic, CRC) and core/.../deletes/RoaringPositionBitmap.java (portable 64-bit layout).

Willingness to contribute

I can contribute to this feature independently.

[mbutrovich]

I have it working end-to-end in a draft PR in Comet pointing to my feature branch (with all DV tasks complete) apache/datafusion-comet#4887. The tasks will be submitted as separate, stacked PRs.

[raghav-reglobe]

Nice — that was fast. Since you have it end-to-end already, the most useful thing we can offer is validation coverage: we run V3 tables whose DVs are written by DuckDB's iceberg extension, which today omits referenced_data_file from the manifest entry while writing it correctly in the Puffin blob. Binding DVs purely from manifest metadata silently misses those — binding via the scan task's delete entries handles both. Happy to run your stacked PRs against our DuckDB-DV fixtures once they're up, and we have CRC/framing test vectors for the blob layout if useful.

On overlap: once your read stack lands we'll rebase #2678 (DV write / RowDelta) onto it and retire the read half of #2681 — the write path staying decoupled from read should make both easier to review.