From 7be9455cbbed3f592a9ac4eb6e9301443ed189f6 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 18:57:39 -0500 Subject: [PATCH 01/19] add scratch docs page --- docs/policies/scratch.md | 119 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/policies/scratch.md diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md new file mode 100644 index 00000000..c87de3c7 --- /dev/null +++ b/docs/policies/scratch.md @@ -0,0 +1,119 @@ +# $SCRATCH Policies + +The `$SCRATCH` (`/scratch/users/`) filesystem is intended to be temporary, +high performance storage for running jobs. +It is not intended for persistent storage. +In order to maintain performance and stability for all users, the `$SCRATCH` has a +**60 day purge policy** where files older than 60 days will be permanently deleted. +Files are not backed up in `$SCRATCH`, so important data or scripts need to be `/projects` (project directories are not backed up) or `$HOME` (home directories have a 7 day rolling snapshot based backup) to be retained. + +## Changes beginning July 1, 2026 + +- The 60 day purge policy on `$SCRATCH` will be automated. +- A quota of 50TB and 10,000,000 files per user will be added to `$SCRATCH`. + +These changes are meant to improve the performance and availability of the +`$SCRATCH` storage space for all users. + +### Planned 60 day purge policy automation rollout and methodology + +In order to reduce stress on the HPC storage systems, the `$SCRATCH` +purge automations will begin by deleting the oldest files stored in +`$SCRATCH` over several weeks until the age of all remaining files +are within the 60-day limit. + +#### Communications + +- All users currently utilizing `$SCRATCH` storage space will receive +personalized emails summarizing their their usage with reminders of +the upcoming changes. +- At least 5 business days before the automated `$SCRATCH` purge begins, +users will be able to access a manifest of all files that will be deleted. + +#### Tentative Schedule + +Note, the schedule may be slowed or accelerated based on the observed load on +the HPC systems. + +##### July 1 + +The `$SCRATCH` purge automation will begin deleting files that are over 2 years old. + +##### July 6 + +The `$SCRATCH` purge automation will begin deleting files that are over 18 months old. + +##### July 9 + +The `$SCRATCH` purge automation will begin deleting files that are over 12 months old. + +##### July 15 + +The `$SCRATCH` purge automation will begin deleting files that are over 6 months old. + +##### July 22 + +The `$SCRATCH` purge automation will begin deleting files that are over 90 days old. + +##### August 3 + +The `$SCRATCH` purge automation will begin normal operation and will regularly delete files 60 days old. +In most cases, a manifest of eligible files will be generated on Sundays and deletion of the files on that +manifest will begin on the following Friday. + +### Quotas + +Quotas will be added on July that with the following limits: + +------------------------------------------------------------------------------------------------------ +| Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | +------------------------------------------------------------------------------------------------------ +| 50 TB | 100 TB | 10,000,000 | 11,000,000 | 10 Days | +------------------------------------------------------------------------------------------------------ + +If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they be able to temporarily exceed this quota, +up to the "Hard Quota" (100TB, 11,000,000 files) for up to the "Grace Period" (10 days). + +When either the "Hard Quota" is reached or usage has been over the "Soft Quota" for more than the "Grace Period," +it will not be possible to create or modify files until the user is under the "Soft Quota." +Once usage is below the "Soft Quota," the "Grace Period" will reset. + +This structure is intended to allow running jobs to complete without being terminated +due to the quota and allow users time to clean up any excess files. +We will attempt to notify users that have exceeded the "Soft Quota" within 48 hours using +their SMU email address, but it is the responsibility of each user to monitor their own usage. + + +## 60 Day Purge Policy + +All files stored in the `$SCRATCH` (`/scratch/users/`) are eligible to +be deleted without warning if they are over 60 days old. + +### Age determination + +Files will be determined to be over 60 days old if the following criteria are met: + +1. The file's `atime` is greater than or equal to 60 days old. +2. The file's `mtime` is greater than or equal to 60 days old. + +Definitions: + +- `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process directly or through commands and scripts) +- `mtime` is "modification time" and it is the last time the file's contents where modified. + +Attempts to circumvent the purge policy by modifying either the `atime` or `ctime` is a violation of policy and will be reported. + +### Transparency + +A manifest of files that are going to be deleted will be available at least 5 days +before the deletion occurs. +These manifests will typically be generated on Sundays. +When the files a deleted, a log of the deleted files will also be available. + +We highly recommend reviewing these lists and moving any files you need to retain. + +### Notifications + +By default, users will be sent a notice when a new manifest of files is available +to their SMU email address. +User may opt out of these emails in ColdFront. From 7de8d26749c806ab16060e73660a9627208e8699 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 18:58:45 -0500 Subject: [PATCH 02/19] update table --- docs/policies/scratch.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index c87de3c7..1f7c4ee3 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -65,11 +65,9 @@ manifest will begin on the following Friday. Quotas will be added on July that with the following limits: ------------------------------------------------------------------------------------------------------- | Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | ------------------------------------------------------------------------------------------------------ | 50 TB | 100 TB | 10,000,000 | 11,000,000 | 10 Days | ------------------------------------------------------------------------------------------------------- If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they be able to temporarily exceed this quota, up to the "Hard Quota" (100TB, 11,000,000 files) for up to the "Grace Period" (10 days). From b9caa049db39f4f8bdc4a45750b9e0b560563acb Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 18:59:34 -0500 Subject: [PATCH 03/19] fix table --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 1f7c4ee3..04f76edd 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -66,7 +66,7 @@ manifest will begin on the following Friday. Quotas will be added on July that with the following limits: | Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | ------------------------------------------------------------------------------------------------------- +|-----------------|-----------------|------------------------|------------------------|--------------| | 50 TB | 100 TB | 10,000,000 | 11,000,000 | 10 Days | If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they be able to temporarily exceed this quota, From eb40b3e45cbebe7d6d0d7fd9fdfceb3946bbb5db Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:10:05 -0500 Subject: [PATCH 04/19] add "will" --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 04f76edd..6557a09a 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -69,7 +69,7 @@ Quotas will be added on July that with the following limits: |-----------------|-----------------|------------------------|------------------------|--------------| | 50 TB | 100 TB | 10,000,000 | 11,000,000 | 10 Days | -If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they be able to temporarily exceed this quota, +If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they will be able to temporarily exceed this quota, up to the "Hard Quota" (100TB, 11,000,000 files) for up to the "Grace Period" (10 days). When either the "Hard Quota" is reached or usage has been over the "Soft Quota" for more than the "Grace Period," From 16c5387fbbff5fe7204726f66fb9f9bf0a79b6b7 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:12:42 -0500 Subject: [PATCH 05/19] remove extra directly --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 6557a09a..6ed98dca 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -96,7 +96,7 @@ Files will be determined to be over 60 days old if the following criteria are me Definitions: -- `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process directly or through commands and scripts) +- `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process or through commands and scripts) - `mtime` is "modification time" and it is the last time the file's contents where modified. Attempts to circumvent the purge policy by modifying either the `atime` or `ctime` is a violation of policy and will be reported. From 2851b45100e39051da1f531a66706d879348a587 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:14:32 -0500 Subject: [PATCH 06/19] change ctime -> mtime --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 6ed98dca..6aaa4c97 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -99,7 +99,7 @@ Definitions: - `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process or through commands and scripts) - `mtime` is "modification time" and it is the last time the file's contents where modified. -Attempts to circumvent the purge policy by modifying either the `atime` or `ctime` is a violation of policy and will be reported. +Attempts to circumvent the purge policy by modifying either the `atime` or `mtime` is a violation of policy and will be reported. ### Transparency From 61e189d18cb27f5fdb60e158e5471f839099b6d1 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:16:42 -0500 Subject: [PATCH 07/19] add move ... --- docs/policies/scratch.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 6aaa4c97..6d580dbe 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -5,7 +5,8 @@ high performance storage for running jobs. It is not intended for persistent storage. In order to maintain performance and stability for all users, the `$SCRATCH` has a **60 day purge policy** where files older than 60 days will be permanently deleted. -Files are not backed up in `$SCRATCH`, so important data or scripts need to be `/projects` (project directories are not backed up) or `$HOME` (home directories have a 7 day rolling snapshot based backup) to be retained. +Files are not backed up in `$SCRATCH`, so important data or scripts need to be moved to `/projects` directories (project directories are not backed up) `$HOME` directories (home directories have a 7 day rolling snapshot based backup) or +off of the HPC systems to be retained. ## Changes beginning July 1, 2026 From d53ba65cb3b237aa5c37529586bf0a0216750004 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:32:08 -0500 Subject: [PATCH 08/19] add quota exception --- docs/policies/scratch.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 6d580dbe..e5d49a17 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -82,6 +82,18 @@ due to the quota and allow users time to clean up any excess files. We will attempt to notify users that have exceeded the "Soft Quota" within 48 hours using their SMU email address, but it is the responsibility of each user to monitor their own usage. +#### Quota Exceptions + +Exceptions to the quotas may be considered on a case-by-case basis. +Approval of any requests require documentation of at least: + +- Your workflow and an explanation of why that workflow is not possible +with the default quota amounts. +- A plan and demonstrable process(es) for using the data within the 60 day purge policy. +Requests that cannot demonstrate that the data will be deleted, used, or moved off of +the HPC systems within 60 days will not be approved. +- Reasonable efforts have been made to reduce storage needs, such as using efficient output file formats +or compression. ## 60 Day Purge Policy From dd980d40b9681c700143b1c34ae9e73d54c0f36b Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:33:43 -0500 Subject: [PATCH 09/19] fix quota language --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index e5d49a17..bdfc230a 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -64,7 +64,7 @@ manifest will begin on the following Friday. ### Quotas -Quotas will be added on July that with the following limits: +Quotas will be added on July 1, 2026 with the following limits, per user: | Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | |-----------------|-----------------|------------------------|------------------------|--------------| From f46df7bc54cf9efd07fccd3a4ba6d8f7d1608cfe Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:38:59 -0500 Subject: [PATCH 10/19] add how to see usage --- docs/policies/scratch.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index bdfc230a..59159558 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -82,6 +82,13 @@ due to the quota and allow users time to clean up any excess files. We will attempt to notify users that have exceeded the "Soft Quota" within 48 hours using their SMU email address, but it is the responsibility of each user to monitor their own usage. +Users can see their recent usage on ColdFront under [personal storage usage](https://hpcaccess.smu.edu/usage/personal-storage) +or in real-time from the HPC systems with the command + +```bash +lfs quota -h -p $(id -u) /scratch +``` + #### Quota Exceptions Exceptions to the quotas may be considered on a case-by-case basis. From f095952273d5bb1ccf1d34b88bff5cd98b3ebf75 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:41:27 -0500 Subject: [PATCH 11/19] add all of --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 59159558..4a3a3b32 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -109,7 +109,7 @@ be deleted without warning if they are over 60 days old. ### Age determination -Files will be determined to be over 60 days old if the following criteria are met: +Files will be determined to be over 60 days old if all of the following criteria are met: 1. The file's `atime` is greater than or equal to 60 days old. 2. The file's `mtime` is greater than or equal to 60 days old. From 81ae50a28f9c1b350fbd09b71cd8289a23a0463b Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 5 May 2026 20:44:11 -0500 Subject: [PATCH 12/19] add outside of normal usage --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 4a3a3b32..b43c3fd1 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -119,7 +119,7 @@ Definitions: - `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process or through commands and scripts) - `mtime` is "modification time" and it is the last time the file's contents where modified. -Attempts to circumvent the purge policy by modifying either the `atime` or `mtime` is a violation of policy and will be reported. +Attempts to circumvent the purge policy by modifying either the `atime` or `mtime` outside of normal usage is a violation of policy and will be reported. ### Transparency From e6e8dc37f2be6d0e5036220ef4ff6d2b451628de Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 16 Jun 2026 14:23:36 -0500 Subject: [PATCH 13/19] update scratch purge procedures --- docs/policies/scratch.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index b43c3fd1..f3daa946 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -28,8 +28,6 @@ are within the 60-day limit. - All users currently utilizing `$SCRATCH` storage space will receive personalized emails summarizing their their usage with reminders of the upcoming changes. -- At least 5 business days before the automated `$SCRATCH` purge begins, -users will be able to access a manifest of all files that will be deleted. #### Tentative Schedule @@ -59,8 +57,8 @@ The `$SCRATCH` purge automation will begin deleting files that are over 90 days ##### August 3 The `$SCRATCH` purge automation will begin normal operation and will regularly delete files 60 days old. -In most cases, a manifest of eligible files will be generated on Sundays and deletion of the files on that -manifest will begin on the following Friday. +In most cases, `$SCRATCH` purges will run once a week, but may be run more or less frequently to +ensure system stability. ### Quotas @@ -68,10 +66,10 @@ Quotas will be added on July 1, 2026 with the following limits, per user: | Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | |-----------------|-----------------|------------------------|------------------------|--------------| -| 50 TB | 100 TB | 10,000,000 | 11,000,000 | 10 Days | +| 50 TB | 60 TB | 10,000,000 | 11,000,000 | 10 Days | If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they will be able to temporarily exceed this quota, -up to the "Hard Quota" (100TB, 11,000,000 files) for up to the "Grace Period" (10 days). +up to the "Hard Quota" (60TB, 11,000,000 files) for up to the "Grace Period" (10 days). When either the "Hard Quota" is reached or usage has been over the "Soft Quota" for more than the "Grace Period," it will not be possible to create or modify files until the user is under the "Soft Quota." @@ -123,15 +121,18 @@ Attempts to circumvent the purge policy by modifying either the `atime` or `mtim ### Transparency -A manifest of files that are going to be deleted will be available at least 5 days +Whenever possible, a manifest of files that are going to be deleted will be available at least 5 days before the deletion occurs. -These manifests will typically be generated on Sundays. +These manifests will typically be generated once a week. When the files a deleted, a log of the deleted files will also be available. We highly recommend reviewing these lists and moving any files you need to retain. +Files on these manifests may be deleted even the access or modification times change +after the manifest is generated. ### Notifications By default, users will be sent a notice when a new manifest of files is available to their SMU email address. User may opt out of these emails in ColdFront. +More details about opting out will be available before July 1. From 72a9028e0c0745c1a2e1f8c240b596be09b7622c Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 16 Jun 2026 14:26:21 -0500 Subject: [PATCH 14/19] remove line --- docs/policies/scratch.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index f3daa946..b4b220ff 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -124,7 +124,6 @@ Attempts to circumvent the purge policy by modifying either the `atime` or `mtim Whenever possible, a manifest of files that are going to be deleted will be available at least 5 days before the deletion occurs. These manifests will typically be generated once a week. -When the files a deleted, a log of the deleted files will also be available. We highly recommend reviewing these lists and moving any files you need to retain. Files on these manifests may be deleted even the access or modification times change From bc7639b54756e43547026c284aed96e4fb78afe3 Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 16 Jun 2026 15:13:18 -0500 Subject: [PATCH 15/19] Update $SCRATCH filesystem policy details Clarified the purge policy and filesystem usage. --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index b4b220ff..a3907f75 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -3,7 +3,7 @@ The `$SCRATCH` (`/scratch/users/`) filesystem is intended to be temporary, high performance storage for running jobs. It is not intended for persistent storage. -In order to maintain performance and stability for all users, the `$SCRATCH` has a +In order to maintain performance and stability for all users, the `$SCRATCH` filesystem has a **60 day purge policy** where files older than 60 days will be permanently deleted. Files are not backed up in `$SCRATCH`, so important data or scripts need to be moved to `/projects` directories (project directories are not backed up) `$HOME` directories (home directories have a 7 day rolling snapshot based backup) or off of the HPC systems to be retained. From 3cd711bad2a3ce4821e62a884505a03f95a5c52a Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 16 Jun 2026 15:20:16 -0500 Subject: [PATCH 16/19] Fix typo in scratch.md regarding file deletion --- docs/policies/scratch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index a3907f75..c453ec40 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -126,7 +126,7 @@ before the deletion occurs. These manifests will typically be generated once a week. We highly recommend reviewing these lists and moving any files you need to retain. -Files on these manifests may be deleted even the access or modification times change +Files on these manifests may be deleted even if the access or modification times change after the manifest is generated. ### Notifications From 95f6274af8d6590a9590183aef29b27086e645cd Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Tue, 23 Jun 2026 15:34:41 -0500 Subject: [PATCH 17/19] update descriptions --- docs/policies/scratch.md | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index c453ec40..5caa0186 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -36,29 +36,17 @@ the HPC systems. ##### July 1 -The `$SCRATCH` purge automation will begin deleting files that are over 2 years old. +Beginning on July 1, 2026, the `$SCRATCH` purge automation will begin. +Initially the purges will be focused on files older than 2 years old. +The maximum age of the files allowed will slowly be decreased from 2 years to the +stated 60 day policy. -##### July 6 +##### Fall 2026 -The `$SCRATCH` purge automation will begin deleting files that are over 18 months old. - -##### July 9 - -The `$SCRATCH` purge automation will begin deleting files that are over 12 months old. - -##### July 15 - -The `$SCRATCH` purge automation will begin deleting files that are over 6 months old. - -##### July 22 - -The `$SCRATCH` purge automation will begin deleting files that are over 90 days old. - -##### August 3 - -The `$SCRATCH` purge automation will begin normal operation and will regularly delete files 60 days old. -In most cases, `$SCRATCH` purges will run once a week, but may be run more or less frequently to -ensure system stability. +The 60 day purge policy will begin to run on a scheduled basis. +Tentatively, this is planned for mid-August, but may occur later +based on how long it takes to run the initial processing needed to +purge the files much older than 60 days. ### Quotas @@ -89,7 +77,7 @@ lfs quota -h -p $(id -u) /scratch #### Quota Exceptions -Exceptions to the quotas may be considered on a case-by-case basis. +Temporary exceptions to the quotas may be considered on a case-by-case basis. Approval of any requests require documentation of at least: - Your workflow and an explanation of why that workflow is not possible @@ -134,4 +122,4 @@ after the manifest is generated. By default, users will be sent a notice when a new manifest of files is available to their SMU email address. User may opt out of these emails in ColdFront. -More details about opting out will be available before July 1. +More details about opting out will be available soon. From 21214e730e9fc1fcae7a97f4bb05016e5a8b03cd Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Thu, 25 Jun 2026 09:02:54 -0500 Subject: [PATCH 18/19] add purge policy to TOC --- docs/_toc.yml | 5 +++++ docs/policies/policies.md | 6 ++++-- docs/policies/scratch.md | 16 +++++++++++++++- 3 files changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/_toc.yml b/docs/_toc.yml index 2ce5cc0f..0c675934 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -6,6 +6,10 @@ parts: - caption: Announcements chapters: - file: work_deprecation.md +- caption: Policies + chapters: + - file: policies/policies.md + - file: policies/scratch.md - caption: Getting Started chapters: - file: about.md @@ -18,6 +22,7 @@ parts: - url: https://www.smu.edu/provost/research/sponsoredprojects/proposal_development/datamanagementplans title: Data Management Plans - file: policies/policies.md + - file: policies/scratch.md - file: accounts.md - file: coldfront/quick_start.md title: ColdFront diff --git a/docs/policies/policies.md b/docs/policies/policies.md index ee211e97..eb2096e8 100644 --- a/docs/policies/policies.md +++ b/docs/policies/policies.md @@ -40,6 +40,8 @@ Data stored in `$SCRATCH` on M3 or the SuperPod has a time based quota. Data older than 60 days will be purged (deleted) without warning. `$SCRATCH` is intended as high performance, temporary storage for jobs. +For more details see the full [scratch purge policy](scratch.md) + If data is needed for a longer period of time, it should be stored in ColdFront storage allocations (limits vary and require justification) or `$HOME` directories (200 GB). If those are also insufficient, please contact us to discuss options. @@ -52,9 +54,9 @@ Please [contact us](../about/contact.md) to discuss needs and potential options. ## Login Nodes -Running code, tasks, and persitent processes on login nodes is forbidden. The login nodes +Running code, tasks, and persistent processes on login nodes is forbidden. The login nodes are a shared environment and running intensive tasks can make the entire system unstable for -all users. Tasks running on login nodes may be terminated by system adminstrators to preserve +all users. Tasks running on login nodes may be terminated by system administrators to preserve system stability. Some examples of acceptable use of login nodes: diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md index 5caa0186..a2c8f8ad 100644 --- a/docs/policies/scratch.md +++ b/docs/policies/scratch.md @@ -1,6 +1,10 @@ # $SCRATCH Policies -The `$SCRATCH` (`/scratch/users/`) filesystem is intended to be temporary, +:::{important} Beginning **July 1, 2026**, the `$SCRATCH` **60 day purge policy** will begin to be +automatically enforced. More details are below. +::: + +The `$SCRATCH` (aliased as `/scratch/users/`) filesystem is intended to be temporary, high performance storage for running jobs. It is not intended for persistent storage. In order to maintain performance and stability for all users, the `$SCRATCH` filesystem has a @@ -8,6 +12,16 @@ In order to maintain performance and stability for all users, the `$SCRATCH` fil Files are not backed up in `$SCRATCH`, so important data or scripts need to be moved to `/projects` directories (project directories are not backed up) `$HOME` directories (home directories have a 7 day rolling snapshot based backup) or off of the HPC systems to be retained. +On M3, $SCRATCH storage is located at: + +- `/lustre/scratch/client/users/` +- `/scratch/users/` + +On the SuperPOD, $SCRATCH storage is located at + +- `/lustre/smuexa01/client/users/` +- `/scratch/users/` + ## Changes beginning July 1, 2026 - The 60 day purge policy on `$SCRATCH` will be automated. From 15af4d440558759f62158d85775176ad7eafff5e Mon Sep 17 00:00:00 2001 From: John LaGrone Date: Thu, 25 Jun 2026 09:09:07 -0500 Subject: [PATCH 19/19] don't list pages 2x --- docs/_toc.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/_toc.yml b/docs/_toc.yml index 0c675934..f5fdf729 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -21,8 +21,6 @@ parts: title: SMU AI Guidance - url: https://www.smu.edu/provost/research/sponsoredprojects/proposal_development/datamanagementplans title: Data Management Plans - - file: policies/policies.md - - file: policies/scratch.md - file: accounts.md - file: coldfront/quick_start.md title: ColdFront