[Feature] Multiple minor improvements, simpler experiment template. (#30)

Author: skandermoalla

.gitignore

@@ -143,6 +143,6 @@ outputs/*
 !outputs/README.md
 
 wandb
-slurm-*.out
+**/*.out
 
 third-party

README.md

@@ -60,16 +60,23 @@ Remember to get back to this root one after finishing each step.
 
 1. Clone the repo with destination `PROJECT_NAME`.
     - If you plan to develop on your local computer, clone it there.
-    - If you plan to develop on your remote server (with direct access over say SSH, e.g. EPFL HaaS), clone it there.
-    - If you plan to develop on CSCS Clariden, clone it there (use an allocation with a compute node, not the login node)
-    - If you plan to develop or deploy on a managed cluster without a build engine
+    - If you plan to develop or deploy on a remote server/cluster without a build engine
       (e.g., EPFL Run:ai clusters, SCITAS clusters), clone on your local machine.
-      (Docker allows cross-platform builds with emulation, but it can be slow.
+      (You will build the image on your local machine then clone there for deployment.
+      Docker allows cross-platform builds with emulation, but it can be slow.
       We would recommend that your local machine is of the same platform as the cluster (e.g. `amd64`, `arm64`),
       or that you have access to a remote Docker engine running on the same platform as the cluster.)
+    - If you plan to develop on a remote server/cluster with a build enginewith direct access over say SSH, e.g. EPFL HaaS)) clone it there.
+      (e.g. EPFL HaaS, CSCS Clariden) clone it there.
     ```
-    git clone <HTTPS/SSH> PROJECT_NAME
+    # For your local machine clone anywhere.
+    # For clusters with scratch filesystems with a cleaning policy, clone in your home directory.
+    # The training artifacts will be later stored on the scratch filesystem and symlinked to this directory.
+    # Also note the creation of a `dev` instance of the repo (And later `run` instance for unattended jobs)
+    mkdir PROJECT_NAME
     cd PROJECT_NAME
+    git clone <HTTPS/SSH> dev
+    cd dev
     # The current directory is referred to as PROJECT_ROOT
     ```
    We will refer to the absolute path to the root of the repository as `PROJECT_ROOT`.

data/README.md

@@ -13,40 +13,41 @@ The directory can be accessed in the experiments with `config.data_dir`.
 Of course, this doesn't mean that the datasets inside `PROJECT_ROOT/data/` need to be physically in the same directory
 as the project.
 You can create symlinks to them.
-This shifts the data path configuration from the code and config,
-which we prefer to be identical across deployment options,
-to the installation steps.
+This shifts the data path configuration from the code and config to the installation steps
+(which we prefer, as it makes the committed code identical across deployment options).
 This is also more convenient than using environment variables to point to individual dataset locations.
 
 Below, you can instruct the users on how to download or link to the data and preprocess it.
 
-When the data is small enough, you can instruct the users to download it in the `PROJECT_ROOT/data/` directory.
-Otherwise, you can provide hints to them on how to download it (or reuse parts of it) in separate storages
-(potentially in a shared storage where some datasets already exist) and then create symlinks to it.
-As these instructions to use separate storage will depend on the installation method, we advise moving
-them to the `installation/*/README.md` file after the development environment installation instructions.
-You can link back to this file for common instructions and information about download links.
-
-* For the local conda installation method, this is only symlinks to other locations in the local filesystem. E.g.
-  ```bash
-  # The data set already exist at /absolute_path/to/some-dataset
-  # FROM the PROJECT_ROOT do
-  ln -s /absolute-path/to/some-dataset data/some-dataset
-  ```
-* For the local deployment option with Docker, and the container deployment on managed clusters,
-  in addition to symlinks, you naturally have to mount the storage where the symlinks points to.
-  The symlink would look like:
-   ```bash
-  # Make sure to consistently mount the volume with in the same location.
-  # The data set already exist at  /absolute-path-in-the-mounted-volume/to/some-dataset
-  # FROM the PROJECT_ROOT do
-  ln -s /absolute-path-in-the-mounted-volume/to/some-dataset data/some-dataset
-  ```
-  And you would edit the `../installation/docker-*/compose.yaml` file for the local deployment option with Docker,
-  otherwise the flags of the cluster client for the managed clusters.
-  mount the individual datasets inside the `PROJECT_ROOT/data/` directory. Avoid nested mounts.
-  Otherwise, you can do like the option below, to mount every shared dataset directory somewhere and symlink to them.
-  (Note: we use symlinks in this case as well to avoid nested mounts which can cause issues.)
+When the data is small enough (a few MBs),
+you can instruct the users (including you) to download it in the `PROJECT_ROOT/data/` directory.
+
+Otherwise, you can provide hints to them on how to download it (or reuse parts of it) in a separate storage
+(likely in a shared storage where some datasets already exist) and then create symlinks to the different parts.
+For managed clusters you need to mount different filesystems remember to add this to the deployment scripts
+and setup files (e.g. `compose.yaml` for deployment with Docker.)
+
+Here are example instructions:
+
+To setup the `data` directory you can download the data anywhere on your system and then symlink to the data from
+the `PROJECT_ROOT/data/` directory.
+
+```bash
+# The data set already exist at /absolute_path/to/some-dataset
+# FROM the PROJECT_ROOT do
+ln -s /absolute-path/to/some-dataset data/some-dataset
+# Do this for each dataset root.
+# TEMPLATE TODO list all dataset roots (it's better to group them and use the groups accordingly in your code).
+```
+
+Be mindful that for the different deployment methods with container engines you will have to mount the filesystems
+where the data is stored (E.g. the local deployment option with Docker, and the container deployment on managed clusters)
+
+`TEMPLATE TODO:` For the local deployment option with Docker you would edit the `../installation/docker-*/compose.yaml`
+file for the local deployment option with Docker,
+for the managed clusters you would edit the flags of the cluster client (`runai`, `srun`, etc.).
+Avoid nested mounts.
+It's better to mount the whole "scratch" filesystem and let the symlinks handle the rest.
 
 ## Description of the data
 

installation/conda-osx-arm64-mps/README.md

@@ -31,6 +31,7 @@ for your future users (and yourself).
    # https://anaconda.org/pytorch/pytorch
    # The hardware acceleration will be determined by the packages you install.
    # E.g. if you install PyTorch with CUDA, set the acceleration to cuda.
+   # Note: new PyTorch versions are only distributed on PyPI (i.e. with `pip`).
    ```
    If you plan to support multiple platforms or hardware accelerations,
    you can duplicate this installation method directory
@@ -61,8 +62,11 @@ for your future users (and yourself).
 Clone the git repository.
 
 ```bash
-git clone <HTTPS/SSH> template-project-name
+# Keep a /dev copy for development and a /run copy for running unattended experiments.
+mkdir template-project-name
 cd template-project-name
+git clone <HTTPS/SSH> dev
+cd dev
 ```
 
 We will refer the absolute path to the root of the repository as `PROJECT_ROOT`.
@@ -131,6 +135,7 @@ Python dependencies are managed by both conda and pip.
   If not available on conda use `brew`.
 - Use `conda` for Python dependencies packaged with more that just Python code (e.g. `pytorch`, `numpy`).
   These will typically be your main dependencies and will likely not change as your project grows.
+  Note: new PyTorch versions are only distributed on PyPI (i.e., with `pip`).
 - Use `pip` for the rest of the Python dependencies (e.g. `tqdm`).
 - For more complex dependencies that may require a custom installation or build,
   manually follow their installation steps.

installation/docker-amd64-cuda/CSCS-Clariden-setup/README.md

@@ -124,7 +124,7 @@ cp _TODO ADD IMAGE_PATH_ $CONTAINER_IMAGES/ADAPTED_NAME.sqsh
 
 #### From a registry (TODO)
 
-### Clone your repository in your scratch directory
+### Clone your repository in your home directory
 
 We strongly suggest having two instances of your project repository.
 
@@ -138,34 +138,25 @@ This guide includes the steps to do it, and there are general details in `data/R
 ```bash
 # SSH to a cluster.
 ssh clariden
-cd $SCRATCH
 # Clone the repo twice with name dev and run (if you already have one, mv it to a different name)
-mkdir template-project-name
-git clone <HTTPS/SSH> template-project-name/dev
-git clone <HTTPS/SSH> template-project-name/run
+mkdir -p $HOME/projects/template-project-name
+cd $HOME/projects/template-project-name
+git clone <HTTPS/SSH> dev
+git clone <HTTPS/SSH> run
 ```
 
 The rest of the instructions should be performed on the cluster from the dev instance of the project.
 ```bash
-cd $SCRATCH/template-project-name/dev/
+cd dev
 # It may also be useful to open a remote code editor on a login node to view the project. (The remote development will happen in another IDE in the container.)
 # Push what you did on your local machine so far (change project name etc) and pull it on the cluster.
 git pull
-cd template-project-name/dev/installation/docker-amd64-cuda
+cd installation/docker-amd64-cuda
 ```
 
-### Note about the examples
-
-The example files were made with username `smoalla` and lab-name `claire`.
-Adapt them accordingly to your username and lab name.
-Run
-```bash
-./template.sh env
-# Edit the .env file with your lab name (you can ignore the rest).
-./template.sh get_cscs_scripts
-```
-to get a copy of the examples in this guide with your username, lab name, etc.
-They will be in `./EPFL-SCITAS-setup/submit-scripts`.
+Example submit scripts are provided in the `example-submit-scripts` directory and are used in the following examples.
+You can copy them to the directory `submit-scripts` which is not tracked by git and edit them to your needs.
+Otherwise, we use shared scripts with shared configurations (including IDE, and shell setups) in `shared-submit-scripts`.
 
 ### A quick test to understand how the template works
 
@@ -176,14 +167,14 @@ and the [`pyxis`](https://github.com/NVIDIA/pyxis) plugin directly integrated in
 
 Run the script to see how the template works.
 ```bash
-cd installation/docker-amd64-cuda//CSCS-Clariden-setup/submit-scripts
+cd installation/docker-amd64-cuda/CSCS-Clariden-setup/submit-scripts
 bash minimal.sh
 ```
 
 When the container starts, its entrypoint does the following:
 
 - It runs the entrypoint of the base image if you specified it in the `compose-base.yaml` file.
-- It expects you specify `PROJECT_ROOT_AT=<location to your project in scratch (dev or run)>`.
+- It expects you specify `PROJECT_ROOT_AT=<location to your project (dev or run)>`.
   and `PROJECT_ROOT_AT` to be the working directory of the container.
   Otherwise, it will issue a warning and set it to the default working directory of the container.
 - It then tries to install the project in editable mode.
@@ -328,9 +319,9 @@ EOL
 We support the [Remote Development](https://www.jetbrains.com/help/pycharm/remote-development-overview.html)
 feature of PyCharm that runs a remote IDE in the container.
 
-The first time connecting you will have to install the IDE in the server in a location mounted from `/scratch` so
-that is stored for future use.
-After that, or if you already have the IDE stored in `/scratch` from a previous project,
+The first time connecting you will have to install the IDE in the server in a location mounted in the container
+that is stored for future use (somewhere in you `$HOME` directory).
+After that, or if you already have the IDE stored in from a previous project,
 the template will start the IDE on its own at the container creation,
 and you will be able to directly connect to it from the JetBrains Gateway client on your local machine.
 
@@ -351,13 +342,13 @@ All the directories will be created automatically.
    variables
     - `JETBRAINS_SERVER_AT`: set it to the `jetbrains-server` directory described above.
     - `PYCHARM_IDE_AT`: don't include it as IDE is not installed yet.
-2. Enable port forwarding for the SSH port.
+2. Add `JETBRAINS_SERVER_AT` in the `--container-mounts`
 3. Then follow the instructions [here](https://www.jetbrains.com/help/pycharm/remote-development-a.html#gateway) and
    install the IDE in your `${JETBRAINS_SERVER_AT}/dist`
-   (something like `/scratch/moalla/jetbrains-server/dist`)
+   (something like `/users/smoalla/jetbrains-server/dist`)
    not in its default location **(use the small "installation options..." link)**.
    For the project directory, it should be in the same location where it was mounted (`${PROJECT_ROOT_AT}`,
-   something like `/scratch/moalla/template-project-name/dev`).
+   something like `/users/smoalla/projects/template-project-name/dev`).
 
 When in the container, locate the name of the PyCharm IDE installed.
 It will be at
@@ -424,7 +415,7 @@ to set up your ssh config file.
 1. In your submit command, set the environment variables for
     - Opening an ssh server `SSH_SERVER=1`.
     - preserving your config `VSCODE_SERVER_AT`.
-2. Enable port forwarding for the SSH connection.
+2. Add `VSCODE_SERVER_AT` to the `--container-mounts`.
 3. Have the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh)
    extension on your local VS Code.
 4. Connect to the ssh host following the

…setup/template-submit-examples/README.md …n-setup/example-submit-scripts/README.mdinstallation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/README.md renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/README.md installation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/README.md renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/README.md

@@ -3,6 +3,4 @@ com.hooks.aws_ofi_nccl.enabled = "true"
 com.hooks.aws_ofi_nccl.variant = "cuda12"
 
 [env]
-FI_CXI_DISABLE_HOST_REGISTER = "1"
-FI_MR_CACHE_MONITOR = "userfaultfd"
 NCCL_DEBUG = "INFO"

…-setup/template-submit-examples/edf.toml …en-setup/example-submit-scripts/edf.tomlinstallation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/edf.toml renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/edf.toml installation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/edf.toml renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/edf.toml

@@ -2,7 +2,9 @@
 
 # Variables used by the entrypoint script
 # Change this to the path of your project (can be the /dev or /run copy)
-export PROJECT_ROOT_AT=$SCRATCH/template-project-name/dev
+export PROJECT_ROOT_AT=$HOME/projects/template-project-name/dev
+export PROJECT_NAME=template-project-name
+export PACKAGE_NAME=template_package_name
 export SLURM_ONE_ENTRYPOINT_SCRIPT_PER_NODE=1
 
 # Enroot + Pyxis
@@ -13,9 +15,9 @@ export SLURM_ONE_ENTRYPOINT_SCRIPT_PER_NODE=1
 srun \
   -J template-minimal \
   --pty \
-  --container-image=$CONTAINER_IMAGES/claire+smoalla+template-project-name+amd64-cuda-root-latest.sqsh \
+  --container-image=$CONTAINER_IMAGES/$(id -gn)+$(id -un)+template-project-name+amd64-cuda-root-latest.sqsh \
   --environment="${PROJECT_ROOT_AT}/installation/docker-amd64-cuda/CSCS-Clariden-setup/submit-scripts/edf.toml" \
-  --container-mounts=$SCRATCH \
+  --container-mounts=$PROJECT_ROOT_AT,$SCRATCH \
   --container-workdir=$PROJECT_ROOT_AT \
   --no-container-mount-home \
   --no-container-remap-root \

…etup/template-submit-examples/minimal.sh …-setup/example-submit-scripts/minimal.shinstallation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/minimal.sh renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/minimal.sh installation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/minimal.sh renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/minimal.sh

@@ -5,27 +5,33 @@
 
 # Variables used by the entrypoint script
 # Change this to the path of your project (can be the /dev or /run copy)
-export PROJECT_ROOT_AT=$SCRATCH/template-project-name/dev
+export PROJECT_ROOT_AT=$HOME/projects/template-project-name/dev
+export PROJECT_NAME=template-project-name
+export PACKAGE_NAME=template_package_name
 export SLURM_ONE_ENTRYPOINT_SCRIPT_PER_NODE=1
 export WANDB_API_KEY_FILE_AT=$HOME/.wandb-api-key
 export HF_TOKEN_AT=$HOME/.hf-token
 export HF_HOME=$SCRATCH/huggingface
 export SSH_SERVER=1
 export NO_SUDO_NEEDED=1
-export JETBRAINS_SERVER_AT=$SCRATCH/jetbrains-server
+# For the first time, mkdir -p $HOME/jetbrains-server, and comment out PYCHARM_IDE_AT
+export JETBRAINS_SERVER_AT=$HOME/jetbrains-server
 #export PYCHARM_IDE_AT=744eea3d4045b_pycharm-professional-2024.1.6-aarch64
 # or
 # export VSCODE_SERVER_AT=$SCRATCH/vscode-server
+# and replace JETBRAINS_SERVER_AT in the container-mounts
 
 srun \
-  --container-image=$CONTAINER_IMAGES/claire+smoalla+template-project-name+amd64-cuda-root-latest.sqsh \
+  --container-image=$CONTAINER_IMAGES/$(id -gn)+$(id -un)+template-project-name+amd64-cuda-root-latest.sqsh \
   --environment="${PROJECT_ROOT_AT}/installation/docker-amd64-cuda/CSCS-Clariden-setup/submit-scripts/edf.toml" \
   --container-mounts=\
+$PROJECT_ROOT_AT,\
 $SCRATCH,\
 $WANDB_API_KEY_FILE_AT,\
 $HOME/.gitconfig,\
 $HF_TOKEN_AT,\
-$HOME/.ssh/authorized_keys \
+$JETBRAINS_SERVER_AT,\
+$HOME/.ssh \
   --container-workdir=$PROJECT_ROOT_AT \
   --no-container-mount-home \
   --no-container-remap-root \

…te-submit-examples/remote-development.sh …ple-submit-scripts/remote-development.shinstallation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/remote-development.sh renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/remote-development.sh installation/docker-amd64-cuda/CSCS-Clariden-setup/template-submit-examples/remote-development.sh renamed to installation/docker-amd64-cuda/CSCS-Clariden-setup/example-submit-scripts/remote-development.sh

@@ -2,8 +2,6 @@
 
 # Enroot + Pyxis
 
-export PROJECT_ROOT_AT=$SCRATCH/template-project-name/dev
-
 srun \
   -J template-test \
   --pty \