Multithreading apply_macros

[James Bruten (james-bruten-mo)]

PR Summary

Sci/Tech Reviewer: Erica Neininger (@ericaneininger)
Code Reviewer: R Sharp (@r-sharp)

This attempts to introduce some parallelism to apply_macros. Given most of the script involves writing out macros and upgrading apps, all of which are fully independent, adding parallelism is trivial. The only part of the script which can't be easily parallelised is the initial processing of the macros - this part is very fast anyway.
This uses concurrent.futures.ThreadPoolExecutor to implement multi-threading. I've also tested with multi-processing, but this was slightly slower overall. Running with 4 threads for a macro added to gungho has a runtime of 99s, vs 190s with 1 thread.
Handling of errors withing threaded regions has also been checked and this works as expected.

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• This change has been tested appropriately (please describe)

Script run for a test macro with correct output produced and errors raised when appropriate.

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• Code quality standards have been met
• Tests are adequate and have passed
• Security considerations have been addressed
• Performance impact is acceptable

[Erica Neininger (ericaneininger)]

I'm happy with this

[R Sharp (r-sharp)]

One wee Q for the author...

[R Sharp (r-sharp)]

Rather than running executor.shutdown on line 1180, might it be better to allow all the tasks to be attempted and then list the macros that didn't upgrade and stop ?

Where there's one fail, there might be others.