cabal2nix integration?

[sellout]

In commercialhaskell/stackage#7727 (comment), @sol suggested I run doctest more directly rather than having a test-suite doctests stanza in my Cabal file (which introduces ~75 lines of boilerplate per package). I started down that path, and then eventually remembered why I really run doctest as a test suite¹.

Nixpkgs pulls in packages from Hackage (mostly via Stackage) and creates Nix derivations in a fairly standard way, using cabal2nix. These derivations run test-suites, but not doctest. So, to make sure my tests are run as part of the Nix packaging process, I wrap doctest in a test suite. This is a fairly common pattern for Nix-aware Haskell devs (e.g., from a package that isn’t mine: https://github.com/cdepillabout/termonad/blob/master/termonad.cabal#L156-L166).

I think this is more of a “discussion” than an “issue”, but I’m wondering what the future might look like here.

E.g., hackage2nix (part of cabal2nix) could potentially run cabal doctest on each package², eliminating the need for the test suite, or doctest could provide some tooling to make the test suite approach less verbose.

I’m not sure what the right answer is, but this seems like a reasonable place to ask the question, and see what other ideas people have, and whether any other approaches already exist.

Footnotes

1. This is made feasible by the cabal-doctest package (not to be confused with cabal-doctest in this repo, which adds a doctest subcommand to cabal). ↩

2. Maybe it could somehow identify the subset of packages that have doctests, or the Nixpkgs Haskell infra could include a blacklist of packages that shouldn’t have doctest run on them … or something. ↩

[sol]

In commercialhaskell/stackage#7727 (comment), @sol suggested I run doctest more directly rather than having a test-suite doctests stanza in my Cabal file (which introduces ~75 lines of boilerplate per package). I started down that path, and then eventually remembered why I really run doctest as a test suite1.

Nixpkgs pulls in packages from Hackage (mostly via Stackage) and creates Nix derivations in a fairly standard way, using cabal2nix. These derivations run test-suites, but not doctest. So, to make sure my tests are run as part of the Nix packaging process, I wrap doctest in a test suite. This is a fairly common pattern for Nix-aware Haskell devs (e.g., from a package that isn’t mine: https://github.com/cdepillabout/termonad/blob/master/termonad.cabal#L156-L166).

I think this is more of a “discussion” than an “issue”, but I’m wondering what the future might look like here.

E.g., hackage2nix (part of cabal2nix) could potentially run cabal doctest on each package2, eliminating the need for the test suite

Nix isn’t really my primary focus, and I’m probably not the best person to make recommendations for the Nix Haskell infrastructure. If this discussion is heading in that direction, maybe the cabal2nix repo would be a better place for it? I won’t engage much, but if you do open an issue there, feel free to tag me—I’d love to stay in the loop.

From what I gather, you’re using Nix mostly to pull in dependencies but rely on cabal for development. If that’s the case, I’d expect cabal doctest to work just fine for you both locally and on CI. If it does, that’s what I’d recommend.

or doctest could provide some tooling to make the test suite approach less verbose.

As for making the test suite approach less verbose, I don’t think that’s the best direction.

1. The real challenge is making sure doctest runs with all the necessary options for a given .cabal project, including generating any required files (Paths_, Happy, Alex, etc.).
2. A fast test-debug cycle is key, and ghci (or something built on it, like ghcid, hspec/sensei, or even HLS which gathers options via cabal repl) dramatically speeds that up. For that reason I assume a user already has a way to invoke ghci with the correct options.
3. If a user already has a way to run ghci, they can use the same mechanism to run doctest.

That’s actually by design—doctest is meant to work as a drop-in replacement for ghci, so anywhere you can run ghci, you can run doctest.

2. Maybe it could somehow identify the subset of packages that have doctests, or the Nixpkgs Haskell infra could include a blacklist of packages that shouldn’t have doctest run on them … or something.

Let me differentiate;

1. When I develop a package, then I want to be able to run doctest locally and on CI.
2. When I compile a package as a dependency, or to be installed, I may choose to run tests (and possibly even doctest).

For (1) , as I mentioned before, use cabal doctest if you can.

As for (2), cabal and stack don't do this. But Stackage does run tests, and I think so do Nix Haskell derivations. This is not my primary focus, but it could be neat to standardize on e.g.

-- foo.cabal
name: foo
version: 0.1.0
...
x-doctest: true
...

which could then be used by cabal2nix to decide whether to run doctest or not.