Skip to content

Pass manifests to icechunk as pyarrow arrays#861

Draft
TomNicholas wants to merge 24 commits intozarr-developers:mainfrom
TomNicholas:pyarrow-to-icechunk
Draft

Pass manifests to icechunk as pyarrow arrays#861
TomNicholas wants to merge 24 commits intozarr-developers:mainfrom
TomNicholas:pyarrow-to-icechunk

Conversation

@TomNicholas
Copy link
Copy Markdown
Member

@TomNicholas TomNicholas commented Jan 26, 2026

Requires earth-mover/icechunk#1576

Does not yet give the benefits I hoped for

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 26, 2026

Codecov Report

❌ Patch coverage is 30.00000% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.78%. Comparing base (333070f) to head (dd01697).

Files with missing lines Patch % Lines
virtualizarr/writers/arrow.py 34.48% 19 Missing ⚠️
virtualizarr/writers/icechunk.py 23.80% 16 Missing ⚠️

❌ Your patch check has failed because the patch coverage (30.00%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (71.78%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (333070f) and HEAD (dd01697). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (333070f) HEAD (dd01697)
unittests 3 1
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #861       +/-   ##
===========================================
- Coverage   89.18%   71.78%   -17.40%     
===========================================
  Files          33       34        +1     
  Lines        2052     2070       +18     
===========================================
- Hits         1830     1486      -344     
- Misses        222      584      +362     
Files with missing lines Coverage Δ
virtualizarr/writers/icechunk.py 15.55% <23.80%> (-76.70%) ⬇️
virtualizarr/writers/arrow.py 34.48% <34.48%> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

TomNicholas and others added 8 commits January 26, 2026 14:59
- Add virtualizarr/writers/arrow.py with ArrowChunkManifest class and
  validation functions extracted from icechunk.py
- Refactor manifest_array fixture to use ChunkManifest.from_arrays for
  scalar arrays, avoiding ambiguity around chunk keys ("c" vs "0")
- Update test_broadcast_scalar to expect the new scalar manifest format

The dict-based ChunkManifest constructor infers shape from chunk keys,
but scalar chunk key conventions differ between zarr versions. Using
from_arrays with shape=() arrays is unambiguous, matching what the HDF
parser does.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid memory copies during serialization

1 participant