benchpark setup: use preexisting spack

[scheibelp]

Add a --spack option to benchpark setup

Like

./bin/benchpark system init --dest=tioga llnl-elcapitan cluster=tioga compiler=rocmcc
./bin/benchpark experiment init --dest=kripke tioga kripke +rocm
./bin/benchpark setup --spack=`pwd`/spack tioga/kripke wkp/
^
(this last command is what changed)

when setting up a second experiment, the user would not specify --spack:

./bin/benchpark experiment init --dest=amg2023 tioga amg2023 +rocm
./bin/benchpark setup tioga/amg2023 wkp/
(no --spack on this)

In this case Benchpark does no configuration of the Spack instance (so for example has no awareness of the benchpark repository). It creates a symlink to the specified Spack instance. The idea is that someone can use any checkout they want of Spack (e.g. a personal branch) and configure it however they like.

If this were used to share installs with several workspaces, or access a binary cache, the experiment builds would not be repeatable unless this Spack instance configured concretizer:reuse:false.

TODOs

• This is probably not working yet, also consider semantics
• If we use a newer Ramble, then benchpark can add ~spack_install_compilers to the ramble.yaml package manager config, and then I think we can avoid any modifications of the shared spack instance
• Docs for the feature

[scheibelp]

@michaelmckinsey1 I've confirmed that the spack instance designated in benchpark setup --spack=... does not get updated based on checkout-versions.yaml. In low-level terms: this is because RuntimeResources.bootstrap() is only called for Ramble/Spack/spack-packages inside of ~/.benchpark (i.e. main.py is the only caller of that function).

(that also means for example that a user can modify the Spack instance in experiments_root after it is generated, and Benchpark would not undo those changes)

[michaelmckinsey1]

Are you still looking to complete these todo's from the description?

• If we use a newer Ramble, then benchpark can add ~spack_install_compilers to the ramble.yaml package manager config, and then I think we can avoid any modifications of the shared spack instance
• Docs for the feature

[michaelmckinsey1]

small nit. The other branch of this if statement basically does the same logic, can we refactor these lines to be after the if-elif?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.20%. Comparing base (8062187) to head (3c97bfa).
⚠️ Report is 41 commits behind head on develop.

Files with missing lines	Patch %	Lines
lib/benchpark/cmd/setup.py	42.85%	4 Missing ⚠️
lib/benchpark/runtime.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1323      +/-   ##
===========================================
+ Coverage    62.43%   64.20%   +1.77%     
===========================================
  Files           49       49              
  Lines         3780     3819      +39     
  Branches       326      338      +12     
===========================================
+ Hits          2360     2452      +92     
+ Misses        1406     1348      -58     
- Partials        14       19       +5     

Files with missing lines	Coverage Δ
lib/benchpark/runtime.py	91.37% <66.66%> (-0.73%)	⬇️
lib/benchpark/cmd/setup.py	86.20% <42.85%> (-2.29%)	⬇️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.