Add golden file tests

[micahhahn]

This PR adds a new primitive equalToFile to the Expect module. It also adds a prettyPrintSingle method

This primitive allows us to write golden file / snapshot tests in Elm. These are particularly useful at capturing the current outputs of a program and then failing if the outputs change.

NoRedInk in particular has an interest in snapshotting the HTML at various points in elm-tests to a file. I've also added a prettyPrintSingle function to the query module which exposes the internal query pretty print function. I think this will be useful generally for testing to be able to capture focus of a query in a complicated test (especially with elm-program-test). NRI would also see value in dumping the entire HTML of a page to a golden file so that we can do visual diffs of our pages in their various states.

Note that the code its current state is very rough - the kernel code added presumes that the tests are being run in the context of a node environment. That's always true for node-test-runner but AFAIK not always true for elm-test-rs. I'm very open to any ideas that would make this more robust.

I'm also aware that doing the IO direclty inside of the elm-explorations/test probably prevents the test runners from doing proper watching on these golden files. Definitely open to ideas on approaches that would let the actual file IO be handled by the runner itself.

[Janiczek]

Hey @micahhahn, apologies for the radio silence here on this PR!

I've so far only read through the diff, I'll have to try it out etc.

But some thoughts so far:

1. Defining a File module. Could the module name File cause collision issues for test suites in projects that use elm/file as dependency?

   • Perhaps we could make this be Test.File or Test.GoldenFile or Test.SnapshotFile or something similar.

2. Node/Deno. When you say this expects Node (fine for elm-test but not always for elm-test-rs), by this you mean that elm-test-rs can be configured to use Deno, right?

   • That could perhaps be solved for, the JS code could check which environment it's in and use different platform APIs.

3. Browser environments. For old versions of elm-explorations/test, browser test runners like jgrenat/elm-html-test-runner do exist; for the current 2.0.0+ version there's no nice ready-to-use runner, but running tests in the browser is still supported - see an example Ellie.

   • So, how should Expect.equalToFile work in the browser? Should it always fail? Always behave as if the test was skiped? Always pass? Should we try and do some phantom type magic to distinguish between Node / Deno / browser test runners on the type level, and disallow Expect.equalToFile in browser environment?

4. Purpose of the File module. Right now it seems to me it tries to do both generic "read/write files" tasks and specific "deal with .failed.html logic" ones.

   • Maybe the renaming from 1) will later give us some clarity on if the contents need to be refactored somehow.

5. Example usages of Expect.equalToFile. I can imagine how this gets used with raw strings. Can you expand (either in doc comment or here in the PR) on how you imagine this to be used with HTML tests? If you're already using a vendored version of this repo at NRI for snapshot tests, can you show some real-world HTML examples?

   • Maybe we can find a better API (ie. adding htmlEqualToFile : String -> Html -> Expectation and making that do .html specific logic, and making equalToFile instead do .txt logic).
   • Since Expect.equal already works with arbitrary a values, perhaps we could even aim for equalToFile : String -> a -> Expectation...

[Janiczek]

See also (just released by @dillonkearns): https://package.elm-lang.org/packages/dillonkearns/elm-snapshot/latest/

[micahhahn]

Hey @Janiczek! Actually opportune timing to respond as we just changed our direction on this a bit.

I'll start with question 5 as that help illustrate the rest of the discussion.

Our goal is bring most of our visual regression tests into Elm. Our current ones use browser simulation in one form or another and that has inescapably led to a mess of flakes as things aren't fully rendered at the time the snapshot is taken.

Since we use Elm for the entirety of the page and strictly use elm-css for styling it occurred to me that we could simply capture the HTML of a page during a run of elm-program-test and get a fairly high fidelity snapshot of the page with 0 flakes due to Elm's determinism (minus some things like external images etc.).

So my plan was to add a primitive that looks as follows that we could stick anywhere in a ProgramTest:

ensureHtml : String -> ProgramTest model msg effect modalKind -> ProgramTest model msg effect modalKind
ensureHtml filePath =
    if String.endsWith ".html" filePath then
        ProgramTest.ensureView
            (\single ->
                case prettyPrintSingle single of
                    Err err ->
                        Debug.todo err

                    Ok html ->
                        Expect.equalToFile filePath html
            )

    else
        Debug.todo "Error in ensureHtml: filePath must end in \".html\""

This ensureHtml primitive would fail if the HTML didn't strictly match what is in the file and respected an environment variable to force overwriting the cache.

Since writing the original code ~6 months ago our thinking has evolved a little bit. The noise from having all these generated files and pollution in the git tree is pushing us in a different direction where we wouldn't bundle the test with the golden file, but rather just have a way to dump all the HTML files out to a directory that we can hand off to a 3rd party tool for diffing. So I might take a little time and rethink the approach here (we likely will still want a write to write to a file from test code).

[micahhahn]

Ohhh, thanks for the pointer to elm-snapshot, that might just work for our needs.