Create reference scripts and master runnner

[MattWellie]

Issue

This repo is fantastic at copying resources in which are created elsewhere

This repo is really bad at making new resources

Solution?

IDK

We could make a top-level script to generate resources, and a number of scripts - each script contains a method to generate one or more related resources, e.g.

generate_X.py: contains a run method which does a couple of things:

1. detects if X already exists, if so, no action
2. if X does not exist, adds jobs to a live Hail Batch to create X
3. If X is created, push to X's home in config

resource_generator.py

from resource_generators import generate_X, generate_Y

# register all current methods
generator_methods = [generate_X, generate_Y]
for generator in generator_methods:
    # call each generator, to add jobs into get_batch(), or do nothing
    generator.run()
    
    # run all collected jobs
    get_batch.run(wait=False)

the run methods shouldn't need input - the script/method will define any external URLs/methods, and the path to write to/any of our current resources to use will be found in config.

The addition of a new method that generates an entry in config will be created as a script and registered as an import in this method. This runner will be submitted to analysis-runner on each commit to main, either all outputs exist (no jobs run), or it queues some jobs to generate those config entries.

I feel like I've passed the Ballmer peak on this one despite being sober. Will have another look tomorrow and see if I can make sense of this