A Luigi workflow to download Sentinel 1 and 2 products from CDSE

The workflow uses the Copernicus Data Space Ecosystem (CDSE) STAC API to get a subset of the Sentinel 1 and 2 raw products.

The CDSE data portal

Luigi workflow

Installation and setup

Create virtual env

cd workflow

python3 -m venv .venv

source .venv/bin/activate

pip install -r requirements.txt

Integrating this worklow into another using a python package

pip install git+https://github.com/jncc/cdse_downloader.git

Then import the required tasks into your workflow and use them:

...
from workflow import DownloadProductsFromList

@requires(DownloadProductsFromList)
class GetProducts(luigi.Task):
    ...

Setup and runnning

Prerequisites

You'll need an luigi.cfg file and a .env file containing the CDSE S3 bucket credentials. See the example files in the repo.

Download products by area

This will retrieve S2 products that intersect with the specified geometry within a specified date range. The data will be downloaded to /downloads and the state files generated by each task in the workflow will be saved in /state.

All of the options can be overridden on the command line. Removing an option from the config file mandates that it be supplied on the command line.

Specify the date rane on the command line with the startDate and endDate parameters. Note that these parameters have a time dimension.

LUIGI_CONFIG_PATH=s2-england-luigi.cfg PYTHONPATH='.' luigi --module workflow DownloadProductsByArea --startDate=2019-07-06T0000 --endDate=2019-07-08T1200 --local-scheduler

Luigi Arguments

--collection
    sentinel-1-grd, sentinel-1-slc, or sentinel-2-l1c
--platform
    Specific satellite, e.g. sentinel-1a, sentinel-2a.
--startDate         
    Start date of date range as ISO datetime format YYYY-MM-DDThhmmss
--endDate           
    End date of date range as ISO datetime format YYYY-MM-DDThhmmss
--orbitDirection    
    'ascending' or 'descending' filter (for use with S1 primarily)
--relativeOrbitNumber      
    E.g. 123

Sentinel 1 arguments
--s1Polarisation 
    The polorisation of the data (VV|VH|HH|HV)
--s1InstrumentMode
    The S1 instrument mode (IW|EW|SM)

Sentinel 2 arguments         
--s2CloudCover      
    Maximum cloudcover as a percentage, e.g. 95

Download a list of products

This will retrieve a list of S2 products and store downloads in /downloads and state in /state

Creat a list of the products you want to download like so:

productlist.txt

S2B_MSIL1C_20190930T113319_N0208_R080_T30VVH_20190930T133803
S2B_MSIL1C_20190930T113319_N0208_R080_T30UWG_20190930T133803
S2B_MSIL1C_20190930T113319_N0208_R080_T30UWF_20190930T133803
S2B_MSIL1C_20190930T113319_N0208_R080_T30UVD_20190930T133803

Specifiy the product list file on the command line using productListFile. This option could be moved into the configuration file if you have standard input file names.

LUIGI_CONFIG_PATH=s2-list-luigi.cfg  PYTHONPATH='.' luigi --module workflow DownloadProductsFromList --productListFile=productlist.txt --local-scheduler