use result files for deployment

[trz42]

This is a long awaited PR for making the bot more agnostic to what it builds for. Specifically this removes code in the deployment procedure that was only correct for deploying tarballs created by build jobs for the (EESSI) software-layer.

It contains necessary changes in the logic but also some improvements to reduce boiler-plate code (handling/reading of information from ini/cfg-type files) and some renaming of variables, functions and log messages.

Below is a breakdown of the major changes

• file tasks/deploy.py

  • uses get_section_from_file (tools/job_metadata.py) to read job metadata and return PR
    section
  • renames function determine_eessi_tarballs to determine_artefacts
    • uses _bot_jobSLURM_JOBID.result file to obtain list of artefacts (previously named tarballs)
    • it relies on what the script bot/check-build.sh in the target GitHub
      repository determines as artefacts, thus it does need to know anything about
      the target GitHub repository
  • renames function check_build_status to check_job_status
    • completely overhauled logic which essentially relies on the value of the
      attribute status in the _bot_jobSLURM_JOBID.result file
    • it relies on what the script bot/check-build.sh in the target GitHub
      repository determines as status, thus it does need to know anything about
      the target GitHub repository
  • renames function upload_tarball to upload_artefact
    • no changes of logic
    • mostly renamed 'tarball' to 'artefact' (except for tarball_prefix* related
      variables; these might be changed in a later PR; a change would also require
      changes to the configurations of bot instances, app.cfg.example and README.md)
    • also changed 'build_target' to 'payload' and made clear that
      'eessi-VERSION-COMPONENT-OS-ARCH' is just a format used within EESSI to
      describe the contents of a 'payload' (or 'build_target')
  • updates function uploaded_before
    • no changes of logic
    • uses 'payload' instead of 'build_target'
  • updates function determine_successful_jobs
    • removes unnecessary function calls: determine_slurm_out
    • removes unnecessary key/value pair in job dictionary: 'slurm_out'
    • switches to using 'artefact' instead of 'eessi_tarball' for variables and
      functions (determine_artefacts)
    • uses renamed and overhauled function check_job_status
    • minor polishing of log messages: 'build' $\rightarrow$ 'job'
  • renames function determine_tarballs_to_deploy to determine_artefacts_to_deploy
    • removes mentioning of 'slurm_out' data in job dictionary (docstring)
    • rephrases 'build*target' to 'payload' (docstring)
    • rephrases '(built/eessi) tarballs' to 'artefacts' (docstring, variables, log
      messages)
    • renames 'tb0' to 'artefact'
  • updates function deploy_built_artefacts
    • uses renamed function determine_artefacts_to_deploy instead of determine_tarballs_to_deploy
    • rephrases 'target' to 'payload'
    • uses renamed function upload_artefact instead of upload_tarball

• file tools/job_metadata.py

  • defines a number of constant strings (to be used in tasks/deploy.py)
  • defines a new function determine_job_id_from_job_directory
  • defines a new function get_section_from_file
  • removes function read_job_metadata_from_file (calls to be replaced by
    get_section_from_file)
  • add constant for 'TEST' section

• file eessi_bot_job_manager.py

  • replace functions read_job_metadata_from_file, read_job_test and read_job_result with function get_section_from_file
  • some polishing: changing imports, removing unused functions, removing unused constants

• file tests/test_tools_job_metadata.py

  • replaced function read_job_metadata_from_file with function get_section_from_file
  • updated log file name accordingly

• files README.md, app.cfg.example, scripts/eessi-upload-to-staging, tasks/deploy.py and tests/test_app.cfg

  • replaced tarball with artefact and TARBALL with ARTEFACT, respectively (wherever it makes sense $\rightarrow$ there is still some use of tarball where it makes sense)

• files README.md, app.cfg.example and tests/test_app.cfg

  • removed unused config settings related to messages that were added to a PR comment when a job had finished
  • those messages are now created by some bot/check-build.sh script in the target repository of a pull request and the bot does not need to know anything about those specific messages ... it just prints whatever the bot/check-build.sh script generates

TODOs

• adjust tests that use removed/changed functions
• test with some example PR
  • tests for software-layer (ie to ensure that deployment still works): see Testing bot PR263 by building and uploading CaDiCaL trz42/software-layer#73
    • found a small bug that is fixed with commit 2e8e0fd; the TEST PR above shows that it is working as intended
  • reran the tests for the software-layer: see Testing bot PR263 by building and uploading CaDiCaL trz42/software-layer#73
    • this revealed that the update requires updates to app.cfg; to ensure that those updates are made we should enhance the event handler and the job manager via a separate PR as described in check for needed configuration settings at the start of bot components #264
  • POSTPONED to when compatibility layer will be rebuilt; tests for compatibility-layer (ie to verify if deployment now also works for other repositories)

[laraPPr]

lgtm