From 8d6b2c159f762ae6ee69ca658079ab7ccd0d6a08 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 13:42:59 -0700 Subject: [PATCH] add sections --- .../curation/fix-metadata-in-raw.rst | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/usdf-applications/curation/fix-metadata-in-raw.rst b/docs/usdf-applications/curation/fix-metadata-in-raw.rst index 2d10d47..ef1640f 100644 --- a/docs/usdf-applications/curation/fix-metadata-in-raw.rst +++ b/docs/usdf-applications/curation/fix-metadata-in-raw.rst @@ -1,8 +1,10 @@ + .. _LSST: http://lsst.org + ############################# Fixing Metadata in Raw Images ############################# -**Caution:** The fix described here actually happens in butler repos that have those problematic +⚠️ **Caution:** The fix described here actually happens in butler repos that have those problematic raw images in their datastores. The raw images themselves are never altered. From time to time, there may be errors within obs headers, typical issues are: @@ -15,8 +17,10 @@ From time to time, there may be errors within obs headers, typical issues are: * Wrong filter information in the image headers This results in wrong dimension records be ingested to the embargo butler repo. The process below -will allow the dimension records in butler to be fixed. The procedure to report and obtain a fix -for these issues is as follows: +will allow the dimension records in butler to be fixed. It includes several steps: + +Report and obtain a fix +======================= #. An issue is usually identified and reported by someone who notice the problem. Data Curation team should file a JIRA ticket to the OBS team to report the problem, and a JIRA ticket to @@ -35,6 +39,9 @@ for these issues is as follows: `translator code `__. Get this reviewed and merged to the ``main`` branch. +Fix the metadata in a butler repo +================================= + The following uses butler repo ``embargo`` as an example, but the same process applies to any butler repo that has the affected images in its datastore. @@ -100,6 +107,12 @@ butler repo that has the affected images in its datastore. * ``--update-records``: is required to fix headers +Fix downstream butler repos and Rucio +===================================== + +Manual embargo unembargo +------------------------ + If the raw images haven't been unembargoed, the following process can manually unembargo the data after the metadata is fixed. @@ -116,6 +129,9 @@ data after the metadata is fixed. bash ./catchup-raw.sh +Fix downstead repos and Rucio +----------------------------- + If the raw images has been unembargoed, then the downstream butler repos (USDF ``main`` bulter and FrDF and UKDF butlers) will need those updated butler dimension records as well.i @@ -165,7 +181,7 @@ To do so: echo rucio did metadata set --key arcBackup --value SLAC_RAW_DISK_BKUP:need $obsDataset echo rucio did update --close $obsDataset -The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been -updated. If a DF hasn't ingested the raw data yet, it can directly use the new _dimensions.1.yaml to +**The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been +updated**. If a DF hasn't ingested the raw data yet, it can directly use the new _dimensions.1.yaml to ingest the data. If the raw data has been ingested, then the DF will need to update the dimension records in their butler repos using a similar process as described above for the embargo butler repo.