Results from other studies

[gmaze]

Under https://guide.cloudnativegeo.org/overview.html#

I found this very helpful study on cloud-optimised format: https://ntrs.nasa.gov/citations/20200001178

We shall keep in mind that this work dates back to 2020.

Cloud-Optimized Format Study
The cloud infrastructure provides a number of capabilities that can dramatically improve access and use of Earth Observation data. However, in many cases, data may need to be reorganized and/or reformatted in order to make them tractable to support cloud-native analysis/access patterns. The purpose of this study is to examine the pros and cons of different formats for storing data on the cloud. The evaluation will focus on both enabling high-performance data access and usage as well as to meet the existing scientific data stewardship needs of EOSDIS.

Exert from the conclusion:

it is clear from this study that the choice of format and compression itself is critical for enabling archival integrity, broad tool support, and fast, cost-effective use. We find no one- size-fits-all solution but offer some broad guidelines to inform choices for specific needs.

They looked "at Cloud-Optimized GeoTIFF, HDF in the Cloud, Parquet, and Zarr, Parquet in depth" and provided a weighted matrix of evaluated criteria that I reproduce here:

Based on the criteria we would find the most important (larger weights) this could help us give a first idea of the different format to prioritize.

[gmaze]

They also provide a spreadsheet I copied in order for us to work with, it's here:

https://docs.google.com/spreadsheets/d/1hrlLiAph8_b9K5kWRWSwRd2FlaR7HdV1Q1p-7SqX52o/edit?usp=sharing