diff --git a/pipelines/_ingredients/DECam/DRP.yaml b/pipelines/_ingredients/DECam/DRP.yaml index 717fc202..8a411f68 100644 --- a/pipelines/_ingredients/DECam/DRP.yaml +++ b/pipelines/_ingredients/DECam/DRP.yaml @@ -167,53 +167,73 @@ subsets: Jointcal uses PreSources, generated by consolidatePreSourceTable, for all visits that overlap a tract. It also produces solutions per-tract, per-visit. - step2d: + step2cde: subset: - finalizeCharacterization - - writeRecalibratedSourceTable - - transformSourceTable - - consolidateSourceTable - updateVisitSummary - description: | - Visit-level tasks. - Allowed data query constraints: visit - - writeRecalibratedSourceTable, transformSourceTable run per-detector - consolidateSourceTable produces one data product per visit. - finalizeCharacterization will eventually model full focal plane PSFs. - step2e: - subset: - makeCcdVisitTable - makeVisitTable description: | - Global-level tasks that must not be run with any data query constraints - Can be run anytime after subset step2d. - - Allowed data query constraints: instrument - - Tasks generate one data product per collection. - make[Ccd]VisitTable produces per-collection summary of the Visits - and CcdVisits. + Per-visit and per-collection tasks that can be run together after step2b + with no data query constraints other than instrument. step3: subset: - - selectGoodSeeingVisits - - makeWarp - - templateGen + - mergeMeasurements - deblend + - transformObjectTable + - forcedPhotCoadd + - healSparsePropertyMaps + - templateGen + - consolidateObjectTable - detection + - makeWarp - writeObjectTable + - selectGoodSeeingVisits + - mergeDetections - measure - - mergeMeasurements - assembleCoadd - - forcedPhotCoadd - - transformObjectTable - - consolidateObjectTable - - mergeDetections description: | Tract-level tasks that can be run together, but only after the 'step1' and 'step2' subsets. These should be run with explicit 'tract' constraints essentially all the - time, because otherwise quanta will be created for jobs with only partial visit - coverage. - # Steps 4-7 are not currently implemented in this DECam DRP processing pipeline. + time, because otherwise quanta will be created for jobs with only partial + visit coverage. + step4: + subset: + - writeRecalibratedSourceTable + - transformSourceTable + description: | + Tasks that can be run together, but only after the 'step1', 'step2' and + 'step3' subsets + + These detector-level tasks should not be run with 'tract' or 'patch' as + part of the data ID expression if all reference catalogs or diffIm + templates that cover these detector-level quanta are desired. + # Step 5 is not currently implemented in this DECam DRP processing pipeline. + step6: + subset: + - consolidateSourceTable + description: | + Tasks that can be run together, but only after the 'step1', 'step2', + 'step3', and 'step4' subsets. + + This step includes visit-level aggregation tasks. Running without tract + or patch in the data query is recommended, otherwise the outputs of + consolidateSourceTable will not contain complete visits. + + This subset is separate from step4 to signal to operators to pause to + assess unexpected image differencing failures before these aggregation + steps. Otherwise, if run in the same quantum graph, aggregated data + products (e.g. sourceTable_visit) would not be created if one or more of + the expected inputs is missing. + step7: + subset: + - consolidateHealSparsePropertyMaps + description: | + Tasks that should be run as the final step that require global inputs, + and can be run after the 'step3' subset. + + This step has global aggregation tasks to run over all visits, detectors, + tracts, etc. This step should be run only with the instrument constraint + in the data query.