HARMONY-2312: Implement steps endpoint.#915
Conversation
flamingbear
force-pushed
the
mhs/HARMONY-2312/MVP-steps-endpoint
branch
2 times, most recently
from
79f3f5d to
30972e0
Compare
flamingbear
force-pushed
the
mhs/HARMONY-2312/MVP-steps-endpoint
branch
from
30972e0 to
8008ac2
Compare
|
|
||
| // Routes and middleware not dealing with service requests | ||
| result.get('/service-results/:bucket/public/:jobId/:workItemId/:remainingPath(*)', asyncHandler(getServiceResult)); | ||
| result.get('/service-results/:bucket/public/:jobId([0-9a-fA-F-]{36})/:workItemId/:remainingPath(*)', asyncHandler(getServiceResult)); |
There was a problem hiding this comment.
Ensures the jobID is in fact a uuid. I need this to sign artifact locations like
http://localhost:3000/service-results/local-artifact-bucket/public/sds/smap-l2-subsetter-net2cog/0b10e62f-db76-4cee-937d-6b1075c663a8/184/SMAP_L2_SM_P_47226_A_20231204T154857_R19240_001_Soil_Moisture_Retrieval_Data_landcover_class_fraction_subsetted_regridded.nc
| import { createPublicPermalink, providerCollectionCache } from '../app/frontends/service-results'; | ||
| import { FileStore } from '../app/util/object-store/file-store'; | ||
|
|
||
| const someJobID = uuid(); |
There was a problem hiding this comment.
when I updated the route to ensure a uuid, I needed to update these fixtures to also be uuids.
| | /docs | These documentation pages | | ||
| | /docs/api | The Swagger documentation for the OGC Coverages API | | ||
| | /jobs | [The jobs API for getting job status, pausing/continuing/canceling jobs](#jobs-details) | | ||
| | /jobs/\<JobID\>/steps | [The introspection service for understanding a jobs steps and intermediate results](#jobs-steps) | |
There was a problem hiding this comment.
Adds this line, but tidies the markdown.
flamingbear
requested review from
chris-durbin,
indiejames and
ygliuvt
as code owners
flamingbear
changed the title
chris-durbin
left a comment
chris-durbin
left a comment
There was a problem hiding this comment.
Tested successfully in sandbox - I was still able to open outputs, verified intermediate artifacts were accessible, and documentation looked correct.
2 participants
Jira Issue ID
HARMONY-2312
Description
Implements MVP for steps endpoint.
stepsendpoint allowing a user to interrogate the intermediate steps of a job.job/:jobIdresponsejob/:jobIdresponseLocal Test Steps
Pull this branch and run against local Harmony-In-A-Box.
here are some sample requests to try.
Request that goes into a destination bucket: (to test that S3 refs are revealed to the end user directly)
http://localhost:3000/C1233800302-EEDTEST/ogc-api-coverages/1.0.0/collections/blue_var/coverage/rangeset?subset=lat(20%3A60)&subset=lon(-140%3A-50)&granuleId=G1233800343-EEDTEST&outputCrs=EPSG%3A31975&format=PNG&destinationUrl=s3%3A%2F%2Flocal-upload-bucket&subset=time(%222002-09-15T00%3A00%3A00.000Z%22%3A%222022-09-25T00%3A00%3A00Z%22)&label=HARMONY-2312-destination-bucket
Request that has a HOSS failure.
http://localhost:3000/C1268429762-EEDTEST/ogc-api-coverages/1.0.0/collections/parameter_vars/coverage/rangeset?forceAsync=true&subset=lat(41.73504%3A43.2472)&subset=lon(5.0196%3A7.91089)&subset=time(%222015-03-31T17%3A06%3A00Z%22%3A%222015-03-31T17%3A10%3A00Z%22)&label=HARMONY-2312-errors&format=image%2Ftiff&variable=Soil_Moisture_Retrieval_Data_1km%2Fretrieval_qual_flag_1km&variable=Soil_Moisture_Retrieval_Data_1km%2Fretrieval_qual_flag_apm_1km&variable=Soil_Moisture_Retrieval_Data_1km%2Fsoil_moisture_1km&variable=Soil_Moisture_Retrieval_Data_1km%2Fsoil_moisture_apm_1km
Request to verify net2cog output files.
http://localhost:3000/C1268429309-EEDTEST/ogc-api-coverages/1.0.0/collections/parameter_vars/coverage/rangeset?forceAsync=true&granuleId=G1281797307-EEDTEST&format=image%2Ftiff&variable=Soil_Moisture_Retrieval_Data%2Flandcover_class_fraction&label=HARMONY-2312-net2cog
Request to demonstrate paging:
http://localhost:3000/C1234088182-EEDTEST/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=image%2Ftiff&label=HARMONY-2312-paging
Request with batch-catalog stac files (batchee)
http://localhost:3000/C1254854453-LARC_CLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?subset=lat(-70%3A30)&subset=lon(-157.5%3A-10)&concatenate=true&maxResults=8&extend=true&serviceId=l2-subsetter-batchee-stitchee-concise
Follow the job page to the steps page and verify the outputs look like what we want.
I did also deploy to mhs sandbox and you can pull that information from the sandbox account and see some outputs running there (including the 110000 granule request for C1234724470-POCLOUD [this one is actually slow 2s because of the bunches of query cmrs that are kicked off but each 2000 granule output list is still bounded at 100 catalogs])
PR Acceptance Checklist