From a14cd00abaee6a8d575122a3fd925fb0d8012b16 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:01:39 -0700 Subject: [PATCH 01/15] describe procedure to update downstream butlers after a dimension record fix --- .../curation/fix-metadata-in-raw.rst | 171 ++++++++++++++++++ .../curation/manual-unembargo.rst | 80 -------- docs/usdf-applications/data-curation.rst | 2 +- 3 files changed, 172 insertions(+), 81 deletions(-) create mode 100644 docs/usdf-applications/curation/fix-metadata-in-raw.rst delete mode 100644 docs/usdf-applications/curation/manual-unembargo.rst diff --git a/docs/usdf-applications/curation/fix-metadata-in-raw.rst b/docs/usdf-applications/curation/fix-metadata-in-raw.rst new file mode 100644 index 0000000..7536a4e --- /dev/null +++ b/docs/usdf-applications/curation/fix-metadata-in-raw.rst @@ -0,0 +1,171 @@ +############################# +Fixing Metadata in Raw Images +############################# + +**Caution:** The fix described here actually happens in butler repos that have those problematic +raw images in their datastores. The raw images themselves are never altered. + +From time to time, there may be errors within obs headers, typical issues are: + +* Faults or bugs that cause images to be taken with ``can_see_sky=True`` or ``can_see_sky=None`` + when in fact it was not possible to see the sky. These images would normally have to wait for + the full embargo period before being copied out of embargo, which can significantly hamper the + Calibrations Team. + +* Wrong filter information in the image headers + +This results in wrong dimension records be ingested to the embargo butler repo. The process below +will allow the dimension records in butler to be fixed. The procedure to report and obtain a fix +for these issues is as follows: + +#. An issue is usually identified and reported by someone who notice the problem. Data Curation + team should file a JIRA ticket to the OBS team to report the problem, and a JIRA ticket to + the DM Data Engineering team to obtain a fix in ``obs_lsst`` (usually in ``lsstCam.py``). + Information in thos ticket usually include observation ids (``day_obs`` and ``seq_num``) of + the images that were taken with incorrect metadata. + +#. The Data Engineering team usually confirms this issue and provides a fix. In some simple cases, + The Data Curation team can also fix those issues by **Checking** what the header should be + changed to (It is best to check with experts). For ``can_see_sky``, + check that the images were in fact taken inside the dome, which may require experts spot-checking + images in RubinTV to ensure that they don't have astronomical objects. + +#. **Skip this step if the Data Engineering team will provide a fix in ``obs_lsst``**: Add the + ``day_obs`` and ``seq_num`` ranges to the ``fix_ranges`` dictionary in the ``obs_lsst`` + `translator code `__. + Get this reviewed and merged to the ``main`` branch. + +The following uses butler repo ``embargo`` as an example, but the same process applies to any +butler repo that has the affected images in its datastore. + +#. Setup the local version of ``obs_lsst`` + + .. code:: bash + + # update environment + source /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib/w_2026_17/loadLSST.sh + setup lsst_distrib + + # Add obs_lsst to eups environment + git clone https://github.com/lsst/obs_lsst.git && cd obs_lsst + git checkout main + + scons + + # add to eups + setup -r . + + # verify local obs_lsst, should say LOCAL:{path} + eups list --setup obs_lsst + +#. Ensure s3 and database credentials are set + + .. code:: bash + + # s3 credentials are located in $HOME/.lsst/aws-credentials.ini + export AWS_PROFILE=embargo_rw + + # postgres credentials + export PGPASSFILE=$HOME/.lsst/postgres-credentials.txt + export PGUSER=rubin + + +#. Update the butler dimension records (``echo`` is for a dry-run) + + .. code:: bash + + # Make sure SHELL is bash + # Remove echo when ready + day_obs=20250715 + REPO=embargo + for seq_num in $(seq -w 000205 001218); do + # Old command + #echo butler ingest-raws $REPO --regex '[W\d]\d.fits$' -t direct -j 10 --output-run LSSTCam/raw/all --update-records s3://embargo@rubin-summit/LSSTCam/${day_obs}/MC_O_${day_obs}_${seq_num}/; + # New command + echo butler update-exposures-from-raws $REPO LSSTCam --where "exposure=${day_obs}${seq_num: -5}" + done + + # Verifying the ingest + # Exposure sequence numbers (XXXXX below) are 5 digits and need to be padded with zeroes + butler query-datasets --collections LSSTCam/raw/all $REPO --where "instrument='LSSTCam' and exposure=YYYYMMDDXXXXX" + + + * ``--regex``: Exclude guiders that cannot be ingested using ``ingest-raws`` + + * ``-t direct``: Mandatory because raws do not live in the butler repo's datastore + + * ``-j 10``: Speed things up a bit + + * ``--output-run``: the default, but it's there for safety + + * ``--update-records``: is required to fix headers + +If the raw images haven't been unembargoed, the following process can manually unembargo the +data after the metadata is fixed. + +#. Login to the ``usdf-embargo-dmz`` vcluster using ``https://k8s.slac.stanford.edu/usdf-embargo-dmz`` and execute the ``catchup-raw.sh`` script from ``slaclab/usdf-embargo-deploy/kubernetes/overlays/transfer`` with the date. This script deploys a Kubernetes Job to the cluster that re-scans the exposure dimension and unembargo the data. + + .. code:: bash + + git clone https://github.com/slaclab/usdf-embargo-deploy + + # script is in this branch + git checkout tickets/DM-51916 + + cd kubernetes/overlays/transfer + + bash ./catchup-raw.sh + +If the raw images has been unembargoed, then the downstream butler repos (USDF ``main`` bulter +and FrDF and UKDF butlers) will need those updated butler dimension records as well.i + +Each exposure has a corresponding Rucio dataset (e.g. ``raw:Dataset/LSSTCam/raw/Obs/20250715/MC_O_20250715_000205``). +This dataset has a .zip file that contains the raw .fits files and json files, and a _dimensions.yaml +file that contains the (incorrect) dimension records for that exposure. We will need to create +a new _dimensions.1.yaml file with the correct dimension records, and add it to the same Rucio dataset. +To do so: + +#. Download and run `create_rawdata_dimensions_yaml.py `_ + + .. code:: bash + + git clone https://github.com/lsst-dm/data-curation-tools.git && cd data-curation-tools/bin.src + + day_obs=20250715 + seq_num=000205 + + python create_rawdata_dimensions_yaml.py ${day_obs}${seq_num: -5} + # This will create a file named ``MC_O_20250715_000205_dimensions.1.yaml`` in the current directory. + +#. Upload the new dimensions yaml file to Rucio + + .. code:: bash + + day_obs=20250715 + seq_num=000205 + + # instrCode="MC" + # controller="O" + # obsId=${instrCode}_${controller} + obsId="MC_O" # !!! sometimes this could be "MC_C". Check what was created above. + + newDimensionsYaml="${obsId}_${day_obs}_${seq_num}_dimensions.1.yaml" + didName="LSSTCam/${day_obs}/${newDimensionsYaml}" + obsDataset="raw:Dataset/LSSTCam/raw/Obs/${day_obs}/${obsId}_${day_obs}_${seq_num}" + + # Note "rucio upload" (below) will not work unless you login to `rubinmgr` and temporarily change + # the permission of /sdf/data/rubin/lsstdata/offline/instrument/20250715 to world-writeable (777). + # Remember to change the permission back after running the rucio upload command in the next step. + + echo rucio upload --rse SLAC_RAW_DISK --scope raw --dataset $didName $newDimensionsYaml + + echo rucio did update --open $obsDataset + echo rucio did content add --to-did $obsDataset $lfn + echo rucio did metadata set --key SafeCopies --value "" $obsDataset + echo rucio did metadata set --key arcBackup --value SLAC_RAW_DISK_BKUP:need $obsDataset + echo rucio did update --close $obsDataset + +The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been +updated. If a DF hasn't ingested the raw data yet, it can directly use the new _dimensions.1.yaml to +ingest the data. If the raw data has been ingested, then the DF will need to update the dimension +records in their butler repos using a similar process as described above for the embargo butler repo. diff --git a/docs/usdf-applications/curation/manual-unembargo.rst b/docs/usdf-applications/curation/manual-unembargo.rst deleted file mode 100644 index 9269f00..0000000 --- a/docs/usdf-applications/curation/manual-unembargo.rst +++ /dev/null @@ -1,80 +0,0 @@ -################ -Manual Unembargo -################ - -From time to time, there may be errors within obs headers, such as faults or bugs that cause images to be taken with ``can_see_sky=True`` or ``can_see_sky=None`` when in fact it was not possible to see the sky. -These images would normally have to wait for the full embargo period before being copied out of embargo, which can significantly hamper the Calibrations Team. -The process below will allow these images to be unembargoed manually. - -#. Ensure that a DM Jira ticket has been filed with the observation ids (``day_obs`` and ``seq_num``) of the images that were taken with incorrect metadata. - -#. **Check** what the header should be changed to. This may involve certification from experts. For ``can_see_sky``, check that the images were in fact taken inside the dome, which may require experts spot-checking images in RubinTV to ensure that they don't have astronomical objects. - -#. Add the ``day_obs`` and ``seq_num`` ranges to the ``fix_ranges`` dictionary in the ``obs_lsst`` `translator code `__. Get this reviewed and merged to ``main``. - -#. Setup the local version of ``obs_lsst`` - - .. code:: bash - - # update environment - source /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib/w_2025_29/loadLSST.sh - setup lsst_distrib - - # Add obs_lsst to eups environment - git clone https://github.com/lsst/obs_lsst.git && cd obs_lsst - git checkout main - - scons - - # add to eups - setup -r . - - # verify local obs_lsst, should say LOCAL:{path} - eups list --setup obs_lsst - -#. Ensure s3 and database credentials are set - - .. code:: bash - - # s3 credentials are located in $HOME/.lsst/aws-credentials.ini - export AWS_PROFILE=embargo_rw - - # postgres credentials - export PGPASSFILE=$HOME/.lsst/postgres-credentials.txt - export PGUSER=rubin - - -#. Reingest the raw images (``echo`` is for a dry-run) - - .. code:: bash - - # Remove echo when ready - day_obs=20250715; for seq_num in $(seq -w 000205 001218); do echo butler ingest-raws embargo --regex '[W\d]\d.fits$' -t direct -j 10 --output-run LSSTCam/raw/all --update-records s3://embargo@rubin-summit/LSSTCam/${day_obs}/MC_O_${day_obs}_${seq_num}/; done - - # Verifying the ingest - # Exposure sequence numbers are 5 digits and need to be padded with zeroes - butler query-datasets --collections LSSTCam/raw/all embargo --where "instrument='LSSTCam' and exposure=YYYYMMDDXXXXX" - - - * ``--regex``: Exclude guiders that cannot be ingested using ``ingest-raws`` - - * ``-t direct``: Mandatory because raws do not live in the butler repo's datastore - - * ``-j 10``: Speed things up a bit - - * ``--output-run``: the default, but it's there for safety - - * ``--update-records``: is required to fix headers - -#. Login to the ``usdf-embargo-dmz`` vcluster using ``https://k8s.slac.stanford.edu/usdf-embargo-dmz`` and execute the ``catchup-raw.sh`` script from ``slaclab/usdf-embargo-deploy/kubernetes/overlays/transfer`` with the date. This script deploys a Kubernetes Job to the cluster that re-scans the exposure dimension and unembargo the data. - - .. code:: bash - - git clone https://github.com/slaclab/usdf-embargo-deploy - - # script is in this branch - git checkout tickets/DM-51916 - - cd kubernetes/overlays/transfer - - bash ./catchup-raw.sh diff --git a/docs/usdf-applications/data-curation.rst b/docs/usdf-applications/data-curation.rst index fdfa217..ac9408d 100644 --- a/docs/usdf-applications/data-curation.rst +++ b/docs/usdf-applications/data-curation.rst @@ -11,6 +11,6 @@ Data Curation curation/embargo-dataflow/index curation/embargo-transfer/index curation/lfa-replication/index - curation/manual-unembargo.rst + curation/fix-metadata-in-raw.rst curation/prompt-output-unembargo/index curation/s3-file-notifications/index From 11e4c98bb415ac28fcd25f37d43c30cc717124f7 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:09:30 -0700 Subject: [PATCH 02/15] trim trailing whitespace --- .../curation/fix-metadata-in-raw.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/usdf-applications/curation/fix-metadata-in-raw.rst b/docs/usdf-applications/curation/fix-metadata-in-raw.rst index 7536a4e..6db9928 100644 --- a/docs/usdf-applications/curation/fix-metadata-in-raw.rst +++ b/docs/usdf-applications/curation/fix-metadata-in-raw.rst @@ -7,7 +7,7 @@ raw images in their datastores. The raw images themselves are never altered. From time to time, there may be errors within obs headers, typical issues are: -* Faults or bugs that cause images to be taken with ``can_see_sky=True`` or ``can_see_sky=None`` +* Faults or bugs that cause images to be taken with ``can_see_sky=True`` or ``can_see_sky=None`` when in fact it was not possible to see the sky. These images would normally have to wait for the full embargo period before being copied out of embargo, which can significantly hamper the Calibrations Team. @@ -20,19 +20,19 @@ for these issues is as follows: #. An issue is usually identified and reported by someone who notice the problem. Data Curation team should file a JIRA ticket to the OBS team to report the problem, and a JIRA ticket to - the DM Data Engineering team to obtain a fix in ``obs_lsst`` (usually in ``lsstCam.py``). + the DM Data Engineering team to obtain a fix in ``obs_lsst`` (usually in ``lsstCam.py``). Information in thos ticket usually include observation ids (``day_obs`` and ``seq_num``) of the images that were taken with incorrect metadata. -#. The Data Engineering team usually confirms this issue and provides a fix. In some simple cases, - The Data Curation team can also fix those issues by **Checking** what the header should be +#. The Data Engineering team usually confirms this issue and provides a fix. In some simple cases, + The Data Curation team can also fix those issues by **Checking** what the header should be changed to (It is best to check with experts). For ``can_see_sky``, check that the images were in fact taken inside the dome, which may require experts spot-checking images in RubinTV to ensure that they don't have astronomical objects. #. **Skip this step if the Data Engineering team will provide a fix in ``obs_lsst``**: Add the ``day_obs`` and ``seq_num`` ranges to the ``fix_ranges`` dictionary in the ``obs_lsst`` - `translator code `__. + `translator code `__. Get this reviewed and merged to the ``main`` branch. The following uses butler repo ``embargo`` as an example, but the same process applies to any @@ -76,9 +76,9 @@ butler repo that has the affected images in its datastore. # Make sure SHELL is bash # Remove echo when ready - day_obs=20250715 + day_obs=20250715 REPO=embargo - for seq_num in $(seq -w 000205 001218); do + for seq_num in $(seq -w 000205 001218); do # Old command #echo butler ingest-raws $REPO --regex '[W\d]\d.fits$' -t direct -j 10 --output-run LSSTCam/raw/all --update-records s3://embargo@rubin-summit/LSSTCam/${day_obs}/MC_O_${day_obs}_${seq_num}/; # New command @@ -121,7 +121,7 @@ and FrDF and UKDF butlers) will need those updated butler dimension records as w Each exposure has a corresponding Rucio dataset (e.g. ``raw:Dataset/LSSTCam/raw/Obs/20250715/MC_O_20250715_000205``). This dataset has a .zip file that contains the raw .fits files and json files, and a _dimensions.yaml -file that contains the (incorrect) dimension records for that exposure. We will need to create +file that contains the (incorrect) dimension records for that exposure. We will need to create a new _dimensions.1.yaml file with the correct dimension records, and add it to the same Rucio dataset. To do so: @@ -154,7 +154,7 @@ To do so: obsDataset="raw:Dataset/LSSTCam/raw/Obs/${day_obs}/${obsId}_${day_obs}_${seq_num}" # Note "rucio upload" (below) will not work unless you login to `rubinmgr` and temporarily change - # the permission of /sdf/data/rubin/lsstdata/offline/instrument/20250715 to world-writeable (777). + # the permission of /sdf/data/rubin/lsstdata/offline/instrument/20250715 to world-writeable (777). # Remember to change the permission back after running the rucio upload command in the next step. echo rucio upload --rse SLAC_RAW_DISK --scope raw --dataset $didName $newDimensionsYaml @@ -165,7 +165,7 @@ To do so: echo rucio did metadata set --key arcBackup --value SLAC_RAW_DISK_BKUP:need $obsDataset echo rucio did update --close $obsDataset -The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been +The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been updated. If a DF hasn't ingested the raw data yet, it can directly use the new _dimensions.1.yaml to -ingest the data. If the raw data has been ingested, then the DF will need to update the dimension +ingest the data. If the raw data has been ingested, then the DF will need to update the dimension records in their butler repos using a similar process as described above for the embargo butler repo. From f7eb850a419ad716dac095e190869c4f89375128 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:16:40 -0700 Subject: [PATCH 03/15] update a URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index e69d796..773ca9e 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -4,7 +4,7 @@ SLAC Kubernetes Overview Operations activities will be carried out at the SLAC US Data Facility (USDF). Where possible, all USDF services will reside on top of a kubernetes platform. -SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. +SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. SLAC Virtual Clusters, aka "Projects" From 6049504b52ae46e7a44f16ac5894b7f4ba2c8723 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:19:24 -0700 Subject: [PATCH 04/15] update a URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index 773ca9e..900e3c3 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -4,7 +4,7 @@ SLAC Kubernetes Overview Operations activities will be carried out at the SLAC US Data Facility (USDF). Where possible, all USDF services will reside on top of a kubernetes platform. -SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. +SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. SLAC Virtual Clusters, aka "Projects" From 7dbdaaa4f343178d649d902a5d8d2293818c0357 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:25:25 -0700 Subject: [PATCH 05/15] remove broken URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index 900e3c3..20689b1 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -4,7 +4,7 @@ SLAC Kubernetes Overview Operations activities will be carried out at the SLAC US Data Facility (USDF). Where possible, all USDF services will reside on top of a kubernetes platform. -SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. +SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to ``openshift's projects`` (https://docs.redhat.com/en/documentation/openshift_container_platform/4.6/html/applications/projects#working-with-projects) or `GKE's projects and folders `__. SLAC Virtual Clusters, aka "Projects" From 50cef1ef20b9a6bfb0a1ee75328fb640ba816e17 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:37:26 -0700 Subject: [PATCH 06/15] ignore links that require human interaction --- conf.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/conf.py b/conf.py index ad9231c..94df47d 100644 --- a/conf.py +++ b/conf.py @@ -4,3 +4,10 @@ # Remove this later after we fix documenteer mermaid_version = "11.9.0" + +# Ignore links + +linkcheck_ignore = [ + r'http://grafana.slac.stanford.edu.+/', + r'http://docs.redhat.com.+', +] From 384e7ee99239547a0a680035b34d56026e09ba2f Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:37:38 -0700 Subject: [PATCH 07/15] minor fix --- .../curation/fix-metadata-in-raw.rst | 50 +++++++++---------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/usdf-applications/curation/fix-metadata-in-raw.rst b/docs/usdf-applications/curation/fix-metadata-in-raw.rst index 6db9928..2d10d47 100644 --- a/docs/usdf-applications/curation/fix-metadata-in-raw.rst +++ b/docs/usdf-applications/curation/fix-metadata-in-raw.rst @@ -127,43 +127,43 @@ To do so: #. Download and run `create_rawdata_dimensions_yaml.py `_ - .. code:: bash + .. code:: bash - git clone https://github.com/lsst-dm/data-curation-tools.git && cd data-curation-tools/bin.src + git clone https://github.com/lsst-dm/data-curation-tools.git && cd data-curation-tools/bin.src - day_obs=20250715 - seq_num=000205 + day_obs=20250715 + seq_num=000205 - python create_rawdata_dimensions_yaml.py ${day_obs}${seq_num: -5} - # This will create a file named ``MC_O_20250715_000205_dimensions.1.yaml`` in the current directory. + python create_rawdata_dimensions_yaml.py ${day_obs}${seq_num: -5} + # This will create a file named MC_O_20250715_000205_dimensions.1.yaml in the current directory. #. Upload the new dimensions yaml file to Rucio - .. code:: bash + .. code:: bash - day_obs=20250715 - seq_num=000205 + day_obs=20250715 + seq_num=000205 - # instrCode="MC" - # controller="O" - # obsId=${instrCode}_${controller} - obsId="MC_O" # !!! sometimes this could be "MC_C". Check what was created above. + # instrCode="MC" + # controller="O" + # obsId=${instrCode}_${controller} + obsId="MC_O" # !!! sometimes this could be "MC_C". Check what was created above. - newDimensionsYaml="${obsId}_${day_obs}_${seq_num}_dimensions.1.yaml" - didName="LSSTCam/${day_obs}/${newDimensionsYaml}" - obsDataset="raw:Dataset/LSSTCam/raw/Obs/${day_obs}/${obsId}_${day_obs}_${seq_num}" + newDimensionsYaml="${obsId}_${day_obs}_${seq_num}_dimensions.1.yaml" + didName="LSSTCam/${day_obs}/${newDimensionsYaml}" + obsDataset="raw:Dataset/LSSTCam/raw/Obs/${day_obs}/${obsId}_${day_obs}_${seq_num}" - # Note "rucio upload" (below) will not work unless you login to `rubinmgr` and temporarily change - # the permission of /sdf/data/rubin/lsstdata/offline/instrument/20250715 to world-writeable (777). - # Remember to change the permission back after running the rucio upload command in the next step. + # Note "rucio upload" (below) will not work unless you login to `rubinmgr` and temporarily change + # the permission of /sdf/data/rubin/lsstdata/offline/instrument/20250715 to world-writeable (777). + # Remember to change the permission back after running the rucio upload command in the next step. - echo rucio upload --rse SLAC_RAW_DISK --scope raw --dataset $didName $newDimensionsYaml + echo rucio upload --rse SLAC_RAW_DISK --scope raw --dataset $didName $newDimensionsYaml - echo rucio did update --open $obsDataset - echo rucio did content add --to-did $obsDataset $lfn - echo rucio did metadata set --key SafeCopies --value "" $obsDataset - echo rucio did metadata set --key arcBackup --value SLAC_RAW_DISK_BKUP:need $obsDataset - echo rucio did update --close $obsDataset + echo rucio did update --open $obsDataset + echo rucio did content add --to-did $obsDataset $lfn + echo rucio did metadata set --key SafeCopies --value "" $obsDataset + echo rucio did metadata set --key arcBackup --value SLAC_RAW_DISK_BKUP:need $obsDataset + echo rucio did update --close $obsDataset The present of this new _dimensions.1.yaml is an indication that the butler dimesion records has been updated. If a DF hasn't ingested the raw data yet, it can directly use the new _dimensions.1.yaml to From 15d9be1d6ea614fbae6a702d6b0301e8f2a33b65 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:40:19 -0700 Subject: [PATCH 08/15] update a URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index 20689b1..900e3c3 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -4,7 +4,7 @@ SLAC Kubernetes Overview Operations activities will be carried out at the SLAC US Data Facility (USDF). Where possible, all USDF services will reside on top of a kubernetes platform. -SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to ``openshift's projects`` (https://docs.redhat.com/en/documentation/openshift_container_platform/4.6/html/applications/projects#working-with-projects) or `GKE's projects and folders `__. +SLAC operates a single large kubernetes cluster. The benefits of this are with increased scale (sharing of resources) and reduced management overhead. We run 'vanilla' kubernetes, deployed via `kubeadm `__. On top of this, to provide segregation and project control we use `loft.sh's vcluster `__. The latter provides a virtual kubernetes cluster from which we can provide a similar experience to `openshift's projects `__ or `GKE's projects and folders `__. SLAC Virtual Clusters, aka "Projects" From 80a255a8ed7deedb18eb871ccb0b37305a0c837e Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:49:24 -0700 Subject: [PATCH 09/15] ignore links that require human interaction --- conf.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/conf.py b/conf.py index 94df47d..14a6747 100644 --- a/conf.py +++ b/conf.py @@ -8,6 +8,7 @@ # Ignore links linkcheck_ignore = [ - r'http://grafana.slac.stanford.edu.+/', - r'http://docs.redhat.com.+', + r'https://grafana.slac.stanford.edu.+', + r'https://github.com/slaclab.+', + r'https://docs.redhat.com.+', ] From ad5369088da9e4638388ca214b51a02f3d8eb844 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Thu, 7 May 2026 23:55:24 -0700 Subject: [PATCH 10/15] fix typo --- docs/usdf-applications/curation/embargo-dataflow/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/usdf-applications/curation/embargo-dataflow/index.rst b/docs/usdf-applications/curation/embargo-dataflow/index.rst index 53917fc..73a2b98 100644 --- a/docs/usdf-applications/curation/embargo-dataflow/index.rst +++ b/docs/usdf-applications/curation/embargo-dataflow/index.rst @@ -56,7 +56,7 @@ image data and how and where the automated process runs. The ``transfer_raw_zip`` Tool ----------------------------- -The ``transfer_raw_zip.py`` in https://github.comr/lsst-dm/transfer_embargo (branch tickets/DM-51619) is +The ``transfer_raw_zip.py`` in https://github.com/lsst-dm/transfer_embargo (branch tickets/DM-51619) is used to unembargo raw image data. This tool will unembargo the data: - Transfer from ``embargo`` S3 storage's ``rubin-summit`` bucket to the destination directory From c77d03d7290402e4ad7cee8ede3920612897f103 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 00:06:39 -0700 Subject: [PATCH 11/15] disable TLS verification --- conf.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/conf.py b/conf.py index 14a6747..ff772d3 100644 --- a/conf.py +++ b/conf.py @@ -9,6 +9,12 @@ linkcheck_ignore = [ r'https://grafana.slac.stanford.edu.+', + r'https://k8s.slac.stanford.edu.+', r'https://github.com/slaclab.+', r'https://docs.redhat.com.+', + r'https://slactraining.skillport.com.+', ] + +# disable TLS verification + +tls_verify = False From 9f27c68c97aaf43a6bb7fb684c62feaf1cded5bf Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 00:19:43 -0700 Subject: [PATCH 12/15] update a URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index 900e3c3..c39b568 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -61,7 +61,7 @@ Connecting and Authenticating Generically: - Determine the 'project' that you wish to access, eg usdf-butler -- Go to https://k8s.slac.stanford.edu/ +- Go to https://k8s.slac.stanford.edu/usdf-butler - Click 'Sign-In' to begin the authentication procedure - Enter your SLAC credentials into the login page, and possibly your Duo 2Factor if requested. This step may automatically skip if you already have valid single sign on credentials in place already. - Click on 'Grant Access' to agree to register From 97421817ac998c119ca1a0318de21dae69b7f2a4 Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 00:26:57 -0700 Subject: [PATCH 13/15] update a URL --- docs/developers/kubernetes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/kubernetes.rst b/docs/developers/kubernetes.rst index c39b568..3a259d5 100644 --- a/docs/developers/kubernetes.rst +++ b/docs/developers/kubernetes.rst @@ -74,6 +74,6 @@ We currently provide kubernetes API access without the need for VPNs etc. i.e. y Miscellaneous ============= -- if you encounter an error like "Unable to connect to the server: No valid id-token, and cannot refresh without refresh-token" when running your kubectl, you will need to log back in via https://k8s.slac.stanford.edu/, re-executing the commands in the second box. This is because our OIDC (dex) implementation does not and cannot generate refresh tokens from our SAML2 (windows ADFS) backend. (Actually, only the ``set-credentials`` command is needed, but it doesn't hurt to execute them all.) +- if you encounter an error like "Unable to connect to the server: No valid id-token, and cannot refresh without refresh-token" when running your kubectl, you will need to log back in via ``https://k8s.slac.stanford.edu/``, re-executing the commands in the second box. This is because our OIDC (dex) implementation does not and cannot generate refresh tokens from our SAML2 (windows ADFS) backend. (Actually, only the ``set-credentials`` command is needed, but it doesn't hurt to execute them all.) Kubernetes secrets are usually held in Vault (vault.slac.stanford.edu). The vault command is available on USDF interactive nodes. You may need to activate it with ``module load vault``. Then login using the commands ``export VAULT_ADDR=https://vault.slac.stanford.edu; vault login -method=ldap`` with your SLAC Windows password. You can then use ``vault kv list -mount=secret rubin[/PATH]`` and ``vault kv get -mount=secret PATH/TO/SECRET`` to access secrets for which you have permission. From 81b887398a1a87990bf9305363ee2e99f22718fc Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 00:28:52 -0700 Subject: [PATCH 14/15] ignore some http links --- conf.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/conf.py b/conf.py index ff772d3..cdddf4a 100644 --- a/conf.py +++ b/conf.py @@ -10,7 +10,8 @@ linkcheck_ignore = [ r'https://grafana.slac.stanford.edu.+', r'https://k8s.slac.stanford.edu.+', - r'https://github.com/slaclab.+', + r'https://confluence.lsstcorp.org.+', + r'https://github.com/.+', r'https://docs.redhat.com.+', r'https://slactraining.skillport.com.+', ] From f54df8be7ea71f5877fa313c882bd38e1ba10bbf Mon Sep 17 00:00:00 2001 From: Wei Yang Date: Fri, 8 May 2026 00:33:35 -0700 Subject: [PATCH 15/15] ignore some http links --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index cdddf4a..c1eb849 100644 --- a/conf.py +++ b/conf.py @@ -9,7 +9,7 @@ linkcheck_ignore = [ r'https://grafana.slac.stanford.edu.+', - r'https://k8s.slac.stanford.edu.+', + r'https://.+.slac.stanford.edu.+', r'https://confluence.lsstcorp.org.+', r'https://github.com/.+', r'https://docs.redhat.com.+',