pathogens: Workflows are registered independent of their nextstrain run compatibility

Workflows are now registered under a top level "workflows" key in the
nextstrain-pathogen.yaml file, and each workflow has its own
"compatibility" declarations.  The only current "compatibility"
declaration remains "nextstrain run".

This allows for more meta information about each workflow to be
registered in one place not under the banner of compatibility.

The nextstrain-pathogen.yaml schema changes to a workflow-first
structure like:

  workflows:
    ingest:
      compatibility:
        nextstrain run: yes
    phylogenetic:
      compatibility:
        nextstrain run: no

instead of a compatibility-first structure like:

  compatibility:
    nextstrain run:
      ingest: yes
      phylogenetic: no

as previously introduced¹ (but not yet released).  As it was never
released, don't bother supporting the latter schema at all anymore.

Change first arose out of discussion started by @victorlin in a related
PR.²  Subsequent discussion with @tsibley³ led to dropping support for
the unreleased schema.

¹ <#461>
² <#462 (comment)>
³ <#472 (comment)>

Co-authored-by: Thomas Sibley <tsibley@fredhutch.org>

Author: joverlee521

CHANGES.md

@@ -27,9 +27,10 @@ supported Python version is always bundled with `nextstrain`.
 ## Improvements
 
 * `nextstrain setup <pathogen>` and `nextstrain version --pathogens` now list
-  the available workflows for a pathogen if the pathogen lists the workflows
-  in the top-level `nextstrain-pathogen.yaml` file.
-  ([#461](https://github.com/nextstrain/cli/pull/461))
+  the available workflows (e.g. `ingest`, `phylogenetic`) for a pathogen if the
+  workflows are registered as compatible with `nextstrain run` in the
+  pathogen's `nextstrain-pathogen.yaml` file.
+  ([#461](https://github.com/nextstrain/cli/pull/461), [#472](https://github.com/nextstrain/cli/pull/472))
 
 * Snakemake's storage support downloaded files (stored in `.snakemake/storage/`)
   are now downloaded from AWS Batch builds by default.

doc/changes.md

@@ -31,9 +31,10 @@ supported Python version is always bundled with `nextstrain`.
 ### Improvements
 
 * `nextstrain setup <pathogen>` and `nextstrain version --pathogens` now list
-  the available workflows for a pathogen if the pathogen lists the workflows
-  in the top-level `nextstrain-pathogen.yaml` file.
-  ([#461](https://github.com/nextstrain/cli/pull/461))
+  the available workflows (e.g. `ingest`, `phylogenetic`) for a pathogen if the
+  workflows are registered as compatible with `nextstrain run` in the
+  pathogen's `nextstrain-pathogen.yaml` file.
+  ([#461](https://github.com/nextstrain/cli/pull/461), [#472](https://github.com/nextstrain/cli/pull/472))
 
 * Snakemake's storage support downloaded files (stored in `.snakemake/storage/`)
   are now downloaded from AWS Batch builds by default.

nextstrain/cli/command/run.py

@@ -33,7 +33,7 @@
 from ..errors import UserError
 from ..pathogens import PathogenVersion, UnmanagedPathogen
 from ..runner import aws_batch, docker, singularity
-from ..util import byte_quantity, split_image_name
+from ..util import byte_quantity, split_image_name, warn
 from ..volume import NamedVolume
 from . import build
 
@@ -241,7 +241,21 @@ def run(opts):
         debug(f"Treating {opts.pathogen!r} as unmanaged pathogen directory")
 
     if opts.workflow not in pathogen.registered_workflows():
-        print(f"The {opts.workflow!r} workflow is not registered as a compatible workflow, but trying to run anyways.")
+        warn(cleandoc(f"""
+            The {opts.workflow!r} workflow is not registered by pathogen {opts.pathogen!r}!
+
+            Trying to run it anyways (but it likely won't work)…
+            """))
+        warn()
+
+    elif opts.workflow not in pathogen.compatible_workflows("nextstrain run"):
+        warn(cleandoc(f"""
+            The {opts.workflow!r} workflow is registered by pathogen {opts.pathogen!r}
+            but not marked as compatible with `nextstrain run`!
+
+            Trying to run it anyways (but it likely won't work)…
+            """))
+        warn()
 
     workflow_directory = pathogen.workflow_path(opts.workflow)
 

nextstrain/cli/command/version.py

@@ -72,9 +72,9 @@ def run(opts):
                     if opts.verbose:
                         print("      " + str(version.path))
 
-                    if registered_workflows := version.registered_workflows():
-                        print("      " + "Available workflows:")
-                        for workflow in registered_workflows:
+                    if compatible_workflows := version.compatible_workflows("nextstrain run"):
+                        print("      " + "Compatible workflows:")
+                        for workflow in compatible_workflows:
                             print("        " + workflow)
                     else:
                         print("      " + "No workflows listed, please refer to pathogen docs.")

nextstrain/cli/pathogens.py

@@ -308,21 +308,33 @@ def __init__(self, name_version_url: str, new_setup: bool = False):
 
     def registered_workflows(self) -> Dict[str, Dict]:
         """
-        Parses :attr:`.registration` to return a dict of registered
-        compatible workflows, where the keys are workflow names.
+        Parses :attr:`.registration` to return a dict of registered workflows,
+        where the keys are workflow names.
         """
         if self.registration is None:
             debug("pathogen does not have a registration")
             return {}
 
-        workflows = self.registration.get("compatibility", {}).get("nextstrain run")
+        workflows = self.registration.get("workflows")
         if not isinstance(workflows, dict):
-            debug(f"pathogen registration.compatibility['nextstrain runs'] is not a dict (got a {type(workflows).__name__})")
+            debug(f"pathogen registration.workflows is not a dict (got a {type(workflows).__name__})")
             return {}
 
         return workflows
 
 
+    def compatible_workflows(self, feature: str) -> Dict[str, Dict]:
+        """
+        Filters registered workflows to return a subset of workflows that are
+        compatible with the provided *feature*.
+        """
+        return {
+            name: info
+            for name, info in self.registered_workflows().items()
+            if isinstance(info, dict) and info.get("compatibility", {}).get(feature)
+        }
+
+
     def workflow_path(self, workflow: str) -> Path:
         return self.path / workflow
 
@@ -498,20 +510,13 @@ def test_compatibility() -> SetupTestResult:
             if self.registration is None:
                 return msg + "\n(couldn't read registration)", False
 
-            try:
-                compatibility = self.registration["compatibility"]["nextstrain run"]
-            except (KeyError, IndexError, TypeError):
-                if DEBUGGING:
-                    traceback.print_exc()
-                return msg + "\n(couldn't find 'compatibility: nextstrain run: …' field)", False
-
-            if compatibility:
-                if workflows := self.registered_workflows():
-                    msg += f"\nAvailable workflows: {list(workflows.keys())}"
-                else:
-                    msg += f"\nNo workflows listed, please refer to pathogen docs."
+            if not self.registered_workflows():
+                return msg + "\n(no workflows registered)", False
+
+            if not (compatible_workflows := self.compatible_workflows("nextstrain run")):
+                return msg + "\n(no workflows registered as compatible)", False
 
-            return msg, bool(compatibility)
+            return msg + f"\nCompatible workflows: {list(compatible_workflows.keys())}", True
 
         return [
             ('downloaded',