Regression in support for indexing into structured arrays within a Zarr store (v2 -> v3)

[rachitk]

Zarr version

v3.2.1

Numcodecs version

0.16.5

Python Version

3.12.7

Operating System

Linux

Installation

Pip into virtual environment

Description

In Zarr-Python 2, saving a structured array into a Zarr store enabled one to load the store and index into the structured array to load a specific field of the structured array. This same functionality does not appear to work in Zarr-Python 3, instead returning an error similar to that of:

TypeError: Cannot cast array data from dtype([('A', '<U8'), ('B', '<i4')]) to dtype('<i4') according to the rule 'unsafe'

This appears to be a regression from something that was supported in Zarr-Python 2. I know that structured array support was pending for a while in Zarr-Python 3, but I was under the impression that #2874 had fixed this (or at least achieved feature parity with Zarr-Python 2 in this regard) per the discussion in #2134.

(As a note, I'm not as involved in the Zarr ecosystem but am hoping to use it more heavily in my future projects. Apologies if any of my terminology is not exact when it comes to describing Zarr stores, groups, and arrays.)

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
#   "numpy",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr
import numpy as np

n_samples = 1000

# Structured array example from https://numpy.org/doc/stable/user/basics.rec.html
x = np.array([('Rex', 9, 81.0), ('Fido', 3, 27.0)],
             dtype=[('name', 'U10'), ('age', 'i4'), ('weight', 'f4')])

# Save into Zarr (format v2)
zarr_f = zarr.open('test.zarr', mode='w', zarr_format=2)
zarr_struct = zarr_f.create_array('test', data=x)

# errors out with " TypeError: Cannot cast array data from dtype([('name', '<U10'), ('age', '<i4'), ('weight', '<f4')]) to dtype('<U10') according to the rule 'unsafe' "
name_array = zarr_struct['name']

# zarr.print_debug_info()

Additional output

zarr.print_debug_info() output:

platform: Linux-5.14.0-427.110.1.el9_4.x86_64-x86_64-with-glibc2.34
python: 3.12.7
zarr: 3.2.1

**Required dependencies:**
packaging: 26.2
numpy: 2.1.2
numcodecs: 0.16.5
typing_extensions: 4.15.0
donfig: 0.8.1.post1

**Optional dependencies:**

fsspec: 2026.4.0
numcodecs: 0.16.5

[d-v-b]

thanks for raising this issue! I'm not sure we want to add this type of indexing behavior through __getitem__. If we want Zarr to comply with the array api standard, the proscribed signature of __getitem__ doesn't take a string parameter. My read is that this type of indexing is something zarr copied from numpy before the array api standard existed, and also that there was no performance advantage to indexing this way -- zarr_array["field"] was not any different from zarr_array[:]["field"]. But I could be wrong here.

@rachitk could you describe your use case a bit more? I'd like to find a solution that doesn't require us deviating from the indexing convention that's now formalized in the array API

[rachitk]

Thanks so much for the reply!

I was under the impression that there was a performance benefit in terms of indexing this way, though I haven't had the chance to reinstall Zarr 2.x.x to compare the speed of indexing zarr_array["field"] vs zarr_array[:]["field"]. My impression was that the former loaded only the field in question into memory from the Zarr store while the latter loads every field in the array into memory and then indexes into the field. From my recollection, the speed of indexing in the latter manner in 3.0 is slower than the speed of indexing in the former manner in 2.0. I'll see if I can try to benchmark the two in Zarr-Python 2.x.x to confirm.

I'd actually also be fine with having this be an explicit function like .get_orthogonal_selection (which is what I believe using a field in this way calls regardless, and trying to use that directly results in the same error currently) instead of needing to directly index using __getitem__ - I don't really have an attachment to the current API, so long as I can index at all based on fields (ideally without loading the entire array into memory).

My use case is mostly genomics-related. In the structured array, I have something like 20,000,000 elements per field with each element representing a property of a genetic variant, with all fields being of the same length. One of those fields is the variant allele frequency, which I'd sometimes like to filter on to only include variants above a certain frequency (and there are many other fields where filtering can be done, as well). It was a bit cumbersome to index into multiple arrays at the same time to try to get the other fields that corresponded to the filtered variants and so I thought of using a structured array to, after creating a boolean mask, index into all of the fields simultaneously within Zarr. This also allowed me to add new fields in a fairly friction-free manner (as I wouldn't need to add the field anywhere except on the instantiation of the array).

I also am working with a memory-sensitive application where squeezing every bit of memory saving (at least in obvious ways) would be ideal for me, so if I could avoid loading the whole array into memory that would be my goal (since I rarely ever need to, as I filter based on a single field to determine which parts of the other fields to actually load).

I use Zarr mostly because of its compression capabilities and due to the fact that another array (my variant genotype array) is far too large to load into memory (usually ~20,000,000 x 60,000 int8) and my use-case involves indexing into this array in discrete chunks that I can align with (or closely align with) the Zarr chunking to speed things up dramatically.

(Incidentally, but completely separate from this issue) I've been trying to figure out how to put variable-length string fields (such as variant IDs) into a structured array and save it in a Zarr store. np.dtypes.StringDType() seemed like the perfect solution but is currently not supported by NumPy in structured arrays, sadly, so I'm simply working around this by using fixed-length strings that sometimes truncate with the hopes that this will be supported by Numpy in the future. I previously used the 'object' dtype with an object codec but know this functionality is not supported in Zarr-Python 3.0 (and I think the reasons seem rational to me).

[d-v-b]

From my recollection, the speed of indexing in the latter manner in 3.0 is slower than the speed of indexing in the former manner in 2.0. I'll see if I can try to benchmark the two in Zarr-Python 2.x.x to confirm.

we have a known performance regression in array access for memory storage and local storage which requires some large-scale changes to fix; that might explain what you see. We have a plan to fix this but it takes a while to get all the required pieces in place.

As for your access pattern, maybe @ilan-gold has some ideas on how to make this more ergonomic?