Improve Cubed support in xarray

[jsignell]

Context

Lazy indexing (pydata/xarray#5081, zarr-developers/zarr-python#3906) is a great start. But people do genuinely need to do work that is larger than the memory available. They need to be able to take advantage of on-disk chunks to divy up work.

Proposal: Invest in Cubed

Cubed has a simplified API that is focused exclusively on arrays. It benefited from being created after dask which means that it came out after the array API was solidified (design docs).

Improve xarray integration

Xarray needs functionality from the array libraries that is not included in the strict array API. That's why the ChunkManager API exists.

Would be useful to get hypothesis tests for duck arrays. There already are strategies that Justus and Tom Nicholas had worked on, but might not actually be used.

Improve local use

• Single machine tracking issue cubed-dev/cubed#514

Questions

• does cubed do anything for parallelization without a cluster? -> yes
• does cubed do any task culling or optimization? -> yes some but perhaps more is available
• does cube write everything to disk? -> I think so. This makes it even more important to optimize the graph

[maxrjones]

Great ideas! In case it's helpful, we discussed a PI objective on cubed previously, but never got to it due to time limitation. We had a discussion with Yuvi, Julius, Tom Nicholas, and Tom White last August as a follow-up to NASA-IMPACT/veda-odd#233, which was summarized by Julius as:

I think we have identified two major usecases for cubed:

1. As a dask replacement for „science notebook“type workflows, where users are expected to use cubed via the xarray API. Some aspects to investigate:

• Do we see performance/reliability/cost improvements using cubed over dask
• Do science workflows work out of the box with cubed or are there parts of the xarray api that do not work or are not supported.

2. As compute layer for dynamic tiling apps (e.g. for timeseries) and ichechunk „updaters“

• This requires a more clear scope before its actionable

Suggested next steps:

• Collect sample workflows from the veda community. E.g. use cases that need to run reliably on small memory resources, and some representative large computations that need scalable compute
• Prototype xarray>cubed>lithops>aws lambda as first MVP
• Prototype how the hub could „hide“ auth/containers etc from the user similarly to dask gateway
• Explore Kubernetes as backend alternative

[jsignell]

Jupyterhub team has also had a similar objective in the past NASA-IMPACT/veda-jupyterhub#92

[yuvipanda]

We were never able to start that objective due to lack of people time. I'd love for this to be picked up :)

[jsignell]

I tried to pull all these ideas into this ODSI Innovation Proposal https://docs.google.com/document/d/10NjY2kfueSpFmvVQhgNA78tundJdz0UFxEnL5nB0fGw/edit?tab=t.0