503 Service Unavailable due to CMR Payload Too Large

[jhkennedy]

Jira: https://asfdaac.atlassian.net/browse/TOOL-3721

Note: The above link is accessible only to members of ASF.

---

When attempting to submit jobs like:

jobs = sdk.Batch()
for batch in sdk.util.chunk(prepared_jobs):
    jobs += hyp3.submit_prepared_jobs(batch)

I kept getting these errors:

requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://hyp3-jhk-sandbox.asf.alaska.edu/jobs
hyp3_sdk.exceptions.ServiceUnavailableError: <Response [503]> Could not submit jobs due to a CMR error. Please try again later.

However, there was no CMR outage at the time. Looking at the API logs, _get_cmr_metadata is raising this error:

CMR search failed: 413 Client Error: Payload Too Large for url: https://cmr.earthdata.nasa.gov/search/granules.json

Since validate_jobs is gathering all the granules from all the jobs, the payload can exceed the CMR limit.

We should:

• Return a descriptive error in this case
• Batch queries to CMR to stay under the limit

[asjohnston-asf]

Can you provide a specific example of a submission that fails?

[jhkennedy]

I was trying to submit 196 verification jobs for the hyp3-autorift ISCE2 --> ISCE3 transition to my sandbox deployment:
its-live-isce3-verification-jobs.json

Using chunks of 200 (default), I was unable to submit any jobs. Using chunks of 50, I was able to submit the first 150 jobs. To submit the remaining 50, I ended up needing to go down to 5 job chunks.