Duplicate product files sometimes exist for HyP3 jobs

[jhkennedy]

HyP3 product files often include a unique random hash to not clobber previously generated products using the same inputs.

HyP3 also utilizes the AWS SPOT market with retries for failed jobs.

Occasionally^, a job may fail due to being interrupted at the end of a job while uploading its files, and thus will end up with multiple duplicate product files for the same job upon retry.

Those duplicate files are listed in the job response and are viewable in the AWS S3 bucket (for deployments with public buckets).

For example, this job:
https://hyp3-cargill.asf.alaska.edu/jobs/134539d2-173f-4c9e-89cd-8f6cc11c2196

reports multiple product files, browse images, and thumbnail images. And if you list the bucket, you see:

$ aws  s3 ls s3://hyp3-cargill-contentbucket-1l3ubqacgyr1a/134539d2-173f-4c9e-89cd-8f6cc11c2196/
2025-05-07 06:49:07       1056 S1_195695_IW3_20220619_20220725_VV_INT40_870D.txt
2025-05-07 06:49:05   56399313 S1_195695_IW3_20220619_20220725_VV_INT40_870D.zip
2025-05-07 06:49:07      48897 S1_195695_IW3_20220619_20220725_VV_INT40_870D_conncomp.tif
2025-05-07 06:49:07    4289398 S1_195695_IW3_20220619_20220725_VV_INT40_870D_corr.tif
2025-05-07 06:49:07    1102674 S1_195695_IW3_20220619_20220725_VV_INT40_870D_dem.tif
2025-05-07 06:49:08    5347868 S1_195695_IW3_20220619_20220725_VV_INT40_870D_lon_rdr.tif
2025-05-07 06:49:06   10833599 S1_195695_IW3_20220619_20220725_VV_INT40_870D_los_rdr.tif
2025-05-07 06:49:06    2441332 S1_195695_IW3_20220619_20220725_VV_INT40_870D_lv_phi.tif
2025-05-07 06:49:08     448701 S1_195695_IW3_20220619_20220725_VV_INT40_870D_unw_phase.png
2025-05-07 06:49:07       1543 S1_195695_IW3_20220619_20220725_VV_INT40_870D_unw_phase_thumb.png
2025-05-07 06:49:06      27116 S1_195695_IW3_20220619_20220725_VV_INT40_870D_water_mask.tif
2025-05-07 06:49:07    4915198 S1_195695_IW3_20220619_20220725_VV_INT40_870D_wrapped_phase.tif
2025-05-07 07:00:13       1056 S1_195695_IW3_20220619_20220725_VV_INT40_D52A.txt
2025-05-07 07:00:11   56399311 S1_195695_IW3_20220619_20220725_VV_INT40_D52A.zip
2025-05-07 07:00:15      22454 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_README.md.txt
2025-05-07 07:00:13      48897 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_conncomp.tif
2025-05-07 07:00:13    4289398 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_corr.tif
2025-05-07 07:00:14    1102674 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_dem.tif
2025-05-07 07:00:14    4889137 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lat_rdr.tif
2025-05-07 07:00:13    5347868 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lon_rdr.tif
2025-05-07 07:00:15   10833599 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_los_rdr.tif
2025-05-07 07:00:14    2441332 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lv_phi.tif
2025-05-07 07:00:14    3294233 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_lv_theta.tif
2025-05-07 07:00:15     448701 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase.png
2025-05-07 07:00:14    4356826 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase.tif
2025-05-07 07:00:14       1543 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_unw_phase_thumb.png
2025-05-07 07:00:14      27116 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_water_mask.tif
2025-05-07 07:00:15    4915198 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_wrapped_phase.tif
2025-05-07 07:00:13   16945191 S1_195695_IW3_20220619_20220725_VV_INT40_D52A_wrapped_phase_rdr.tif

with the D52A product files being uploaded ~11 minutes after the partial upload of the 870D product files.

---

I have not quantified how often this happens, but I suspect rarely is a more appropriate descriptor.

[jhkennedy]

On the user side of this:

1. The set of files uploaded the latest should be the full set of product files for a succeded job, but:
   • we don't report in HyP3 the file creation (upload) time, so this is only determinable in deployments with a public S3 bucket (or where users are provided with S3 access).
2. If two zip files are uploaded, they will contain the full set of product files, so using either should be fine, but:
   • we do have users that are directly accessing the "sidecar" product files uploaded next to the zip, and the earlier product may have missing files, which brings us back to (1)