Define what "dry run" should and should not skip, and review "is dry run" checks #4443
Description
foo.is_dry_run checks are sprinkled throughout tmt code, with no apparent concept behind their placement. Run the provision step in the dryrun mode, and you can find several occurrences of Executing command ... on guest 'None' in the debug log, a sign of code blindly using empty shell of a guest, trying to run commands on it, failing silently.
- What does "dry run" mode mean - no resources created, no resources modified? Can tmt create a workdir? Draw the line.
- Review placement of
is_dry_runchecks (40+ occurrences), and compare with methods that would violate the rules established above. Istestcloudplugin allowed to start an instance? Is it even allowed to create it?
Motivation: I changed how guest facts are discovered, and now I have a terrible time fixing tests that fail because guest facts, in the dryrun mode, have not been discovered. They were not discovered in a different way that they used to be not discovered before my changes, and dryrun tests now fail that "package manager was not discovered". Correct, but that's kind of expected when there's no instance to inspect, so it means parts of the code have clearly different expectations about what information is available all the time. In other words, the failing part of the code should not run in the dryrun mode, and after sprinkling more is_dry_run checks I realized my additions are not based on any kind of conceptual approach, I was just silencing test failures. Guest.start() is skipped in dryrun, Guest.setup() is not. And so on.
Let's define what is allowed, what parts of information space remain empty in the dryrun mode, and let's express that with type annotations, warnings, decorators, tools we have available. E.g. we can emit warnings "Foo has been skipped because of dryrun", as tmt logs like nothing changed while not doing half of the actions; we can use decorators to mark code that is/is not supposed to run in the dryrun mode; and so on.