diff --git a/docs/_toc.yml b/docs/_toc.yml index 2ce5cc0f..f5fdf729 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -6,6 +6,10 @@ parts: - caption: Announcements chapters: - file: work_deprecation.md +- caption: Policies + chapters: + - file: policies/policies.md + - file: policies/scratch.md - caption: Getting Started chapters: - file: about.md @@ -17,7 +21,6 @@ parts: title: SMU AI Guidance - url: https://www.smu.edu/provost/research/sponsoredprojects/proposal_development/datamanagementplans title: Data Management Plans - - file: policies/policies.md - file: accounts.md - file: coldfront/quick_start.md title: ColdFront diff --git a/docs/policies/policies.md b/docs/policies/policies.md index ee211e97..eb2096e8 100644 --- a/docs/policies/policies.md +++ b/docs/policies/policies.md @@ -40,6 +40,8 @@ Data stored in `$SCRATCH` on M3 or the SuperPod has a time based quota. Data older than 60 days will be purged (deleted) without warning. `$SCRATCH` is intended as high performance, temporary storage for jobs. +For more details see the full [scratch purge policy](scratch.md) + If data is needed for a longer period of time, it should be stored in ColdFront storage allocations (limits vary and require justification) or `$HOME` directories (200 GB). If those are also insufficient, please contact us to discuss options. @@ -52,9 +54,9 @@ Please [contact us](../about/contact.md) to discuss needs and potential options. ## Login Nodes -Running code, tasks, and persitent processes on login nodes is forbidden. The login nodes +Running code, tasks, and persistent processes on login nodes is forbidden. The login nodes are a shared environment and running intensive tasks can make the entire system unstable for -all users. Tasks running on login nodes may be terminated by system adminstrators to preserve +all users. Tasks running on login nodes may be terminated by system administrators to preserve system stability. Some examples of acceptable use of login nodes: diff --git a/docs/policies/scratch.md b/docs/policies/scratch.md new file mode 100644 index 00000000..a2c8f8ad --- /dev/null +++ b/docs/policies/scratch.md @@ -0,0 +1,139 @@ +# $SCRATCH Policies + +:::{important} Beginning **July 1, 2026**, the `$SCRATCH` **60 day purge policy** will begin to be +automatically enforced. More details are below. +::: + +The `$SCRATCH` (aliased as `/scratch/users/`) filesystem is intended to be temporary, +high performance storage for running jobs. +It is not intended for persistent storage. +In order to maintain performance and stability for all users, the `$SCRATCH` filesystem has a +**60 day purge policy** where files older than 60 days will be permanently deleted. +Files are not backed up in `$SCRATCH`, so important data or scripts need to be moved to `/projects` directories (project directories are not backed up) `$HOME` directories (home directories have a 7 day rolling snapshot based backup) or +off of the HPC systems to be retained. + +On M3, $SCRATCH storage is located at: + +- `/lustre/scratch/client/users/` +- `/scratch/users/` + +On the SuperPOD, $SCRATCH storage is located at + +- `/lustre/smuexa01/client/users/` +- `/scratch/users/` + +## Changes beginning July 1, 2026 + +- The 60 day purge policy on `$SCRATCH` will be automated. +- A quota of 50TB and 10,000,000 files per user will be added to `$SCRATCH`. + +These changes are meant to improve the performance and availability of the +`$SCRATCH` storage space for all users. + +### Planned 60 day purge policy automation rollout and methodology + +In order to reduce stress on the HPC storage systems, the `$SCRATCH` +purge automations will begin by deleting the oldest files stored in +`$SCRATCH` over several weeks until the age of all remaining files +are within the 60-day limit. + +#### Communications + +- All users currently utilizing `$SCRATCH` storage space will receive +personalized emails summarizing their their usage with reminders of +the upcoming changes. + +#### Tentative Schedule + +Note, the schedule may be slowed or accelerated based on the observed load on +the HPC systems. + +##### July 1 + +Beginning on July 1, 2026, the `$SCRATCH` purge automation will begin. +Initially the purges will be focused on files older than 2 years old. +The maximum age of the files allowed will slowly be decreased from 2 years to the +stated 60 day policy. + +##### Fall 2026 + +The 60 day purge policy will begin to run on a scheduled basis. +Tentatively, this is planned for mid-August, but may occur later +based on how long it takes to run the initial processing needed to +purge the files much older than 60 days. + +### Quotas + +Quotas will be added on July 1, 2026 with the following limits, per user: + +| Soft Quota (TB) | Hard Quota (GB) | Soft Quota (Filecount) | Hard Quota (Filecount) | Grace Period | +|-----------------|-----------------|------------------------|------------------------|--------------| +| 50 TB | 60 TB | 10,000,000 | 11,000,000 | 10 Days | + +If an user exceeds a "Soft Quota" (50TB, 10,000,000 files) they will be able to temporarily exceed this quota, +up to the "Hard Quota" (60TB, 11,000,000 files) for up to the "Grace Period" (10 days). + +When either the "Hard Quota" is reached or usage has been over the "Soft Quota" for more than the "Grace Period," +it will not be possible to create or modify files until the user is under the "Soft Quota." +Once usage is below the "Soft Quota," the "Grace Period" will reset. + +This structure is intended to allow running jobs to complete without being terminated +due to the quota and allow users time to clean up any excess files. +We will attempt to notify users that have exceeded the "Soft Quota" within 48 hours using +their SMU email address, but it is the responsibility of each user to monitor their own usage. + +Users can see their recent usage on ColdFront under [personal storage usage](https://hpcaccess.smu.edu/usage/personal-storage) +or in real-time from the HPC systems with the command + +```bash +lfs quota -h -p $(id -u) /scratch +``` + +#### Quota Exceptions + +Temporary exceptions to the quotas may be considered on a case-by-case basis. +Approval of any requests require documentation of at least: + +- Your workflow and an explanation of why that workflow is not possible +with the default quota amounts. +- A plan and demonstrable process(es) for using the data within the 60 day purge policy. +Requests that cannot demonstrate that the data will be deleted, used, or moved off of +the HPC systems within 60 days will not be approved. +- Reasonable efforts have been made to reduce storage needs, such as using efficient output file formats +or compression. + +## 60 Day Purge Policy + +All files stored in the `$SCRATCH` (`/scratch/users/`) are eligible to +be deleted without warning if they are over 60 days old. + +### Age determination + +Files will be determined to be over 60 days old if all of the following criteria are met: + +1. The file's `atime` is greater than or equal to 60 days old. +2. The file's `mtime` is greater than or equal to 60 days old. + +Definitions: + +- `atime` is "access time" and it the last time the data from a file or directory was accessed (e.g. used directly by a process or through commands and scripts) +- `mtime` is "modification time" and it is the last time the file's contents where modified. + +Attempts to circumvent the purge policy by modifying either the `atime` or `mtime` outside of normal usage is a violation of policy and will be reported. + +### Transparency + +Whenever possible, a manifest of files that are going to be deleted will be available at least 5 days +before the deletion occurs. +These manifests will typically be generated once a week. + +We highly recommend reviewing these lists and moving any files you need to retain. +Files on these manifests may be deleted even if the access or modification times change +after the manifest is generated. + +### Notifications + +By default, users will be sent a notice when a new manifest of files is available +to their SMU email address. +User may opt out of these emails in ColdFront. +More details about opting out will be available soon.