Add available field to package spec in dune-project

[gridbugs]

This lets you generate an opam file with an available field.

[Leonidas-from-XIV]

I discussed the available field in an RFC earlier this year, unfortunately at that time there was not much interest.

[rgrinberg]

Yes, in particular, I never understood the necessity for this construct as opposed to plain old dependencies. Why do we write (available (= :os linux)) vs. (depends conf-linux)?

[Leonidas-from-XIV]

Given there is no such package yet, I wonder what @kit-ty-kate thinks about this approach. It would be fairly trivial to implement, essentially just available: {os = "linux"} and the like for all the other OSes.

[kit-ty-kate]

Yes, in particular, I never understood the necessity for this construct as opposed to plain old dependencies. Why do we write (available (= :os linux)) vs. (depends conf-linux)?

I'm not sure to understand. How would such a conf-linux package be defined if no available field exists?

[Leonidas-from-XIV]

@kit-ty-kate This is not about removing available from opam files, it is about being able to generate opam files from dune-project that have an available field. Currently dune doesn't have support for that and the question is whether it should be added or whether a hypothetical conf-linux package (that uses available) should be created instead.

[gridbugs]

To explain my motivation for adding this, I have a side project which doesn't build on some architectures. To release it to opam-repository I need to add an available field to its opam file to exclude those architectures but I generate opam files from dune-project so I have to add this field this by hand. I know that I will forget to do this every time I release a new version of my project so my options are to stop generating opam files from dune-project (which seems a shame) or to add available to dune-project.

[gridbugs]

Also once dune starts doing package management it's going to become really important to keep the dependencies in dune-project the same as the dependencies in opam files and if opam files aren't generated from dune-project then manually synchronizing dependencies between dune-project and opam is another thing I will forget to do when releasing new versions of my library. I guess alternative would be for me to make a separate tool which auto-generates opam files from dune-project and stop checking in the opam files but that seems like a waste of effort since dune can almost do what I need.

[gridbugs]

Also I get that I could possibly technically simulate available by making some dummy opam packages for the unsupported architectures like conf-arm32 and conf-ppc64 and then add those to conflicts but:

• I've been asked by an opam-repo maintainer on Add llama 0.1.0 opam-repository#24172 to use the available field to restrict architectures so that's clearly the established convention even if technically it's not strictly necessary
• The available field exists in opam and is widely used so I expect people are surprised to find that it's not supported by dune-project while many other opam fields are and I think we should try to avoid surprising users like this
• Even if it wasn't a widely-used feature of opam, users are going to want a way to restrict the architectures that their packages are available on and doing so via dependencies on empty opam packages feels clumsy to me
• Somebody would have to create and then maintain the conf-<arch> packages and that's a bunch of work that can be avoided by supporting available

[Leonidas-from-XIV]

@gridbugs The other option to use .opam.template and write your available there. That's what we do in opam-monorepo in the case of the flags field which the dune generator also doesn't support.

I don't think maintaining conf-<arch> packages is any more work than maintaining available fields in potentially thousands of packages. If anything it seems to me to be less work.

But I think @kit-ty-kate @rgrinberg and I have come up with a pretty decent idea on our call yesterday. It would be implementing available but then also promoting the expression here to the enabled_if stanzas in the project. It's not 100% perfect as the available field uses OPAM variables and enabled_if uses Dune variables but maybe it can be translated in 80% of the cases just fine?

[rgrinberg]

Here's a list of criteria to consider for adding something to a dune-project:

• How does this feature fit in with rest of dune? Dune should strive to provide a small core of features that work well together and are non overlapping. Are there existing features in dune that overlap with what is being added? Every feature that dune has to be described as standalone in dune's manual without reference to other manuals.

• How can dune use this feature to improve/validate/control the build somehow? Passing it along when generating an opam file is not enough of a reason because we make it easy enough to use the opam syntax. If it's too hard to write the template files, we should think of some other way to make it easier.

• Does the feature make sense independently of any package managers? The dune language is agnostic of any package managers.

So in summary, I'm not completely against this feature, but it does raise a few questions with regards to these points that need some consideration.

[gridbugs]

The criteria makes sense and I can see the argument for using enabled_if instead since on unsupported architectures dune shouldn't attempt to build the package and we shouldn't duplicate the availability constraint between enabled_if and available. Also I didn't know about opam templates until now and they will solve my problem. Also I hadn't considered the goal of being agnostic to package managers.

One point of confusion was that in my experience using dune the only reason I've ever added dependencies to dune-project was so that dune could generate an opam file. This along with the fact that many other opam fields can go in dune-project led me to expect available to be supported too and I was surprised when it wasn't. I now understand the reason for it's absence but new dune users who go through the same process as me will be surprised in the same way and it's not clear what users are supposed to do instead. I searched the Generate Opam Files docs for "available" and didn't find anything before starting work on this PR.

With this in mind what if we get dune to recognize the available field but to print an error when it's used and the error messages can suggest using an opam template instead.

[gridbugs]

But I think @kit-ty-kate @rgrinberg and I have come up with a pretty decent idea on our call yesterday. It would be implementing available but then also promoting the expression here to the enabled_if stanzas in the project. It's not 100% perfect as the available field uses OPAM variables and enabled_if uses Dune variables but maybe it can be translated in 80% of the cases just fine?

My concern with traversing the project to find all its enabled_if to generate available is that the behaviour is complex and will largely be hidden from users. Its implementation might not match the user's expectations leading to surprising behaviour when things go wrong. If we do end up doing this I'd encourage making its output verbosely describe the process by which the eventual available constraint was computed so it's less surprising when things go wrong, and hopefully the description can guide users to which enabled_if they need to change to fix the problem.

To simplify the behaviour of generating available from enabled_if what if we allow packages to have a top-level enabled_if statement which we require be at least as restrictive as the conjunction of the enabled_ifs in the project and use the top-level enabled_if alone to generate available. That would give users more direct control over the generated available constraint and it simplifies the mental model for how available is computed since it's just based on a single enabled_if field rather than several. But it would still prevent a package from being available on an arch/os/etc implied to be unavailable by an enabled_if within the project.

• we probably shouldn't call the top-level field enabled_if since it means something different from how enabled_if is used elsewhere. What about package_enabled_if.
• for the types of constraints common to available (but not in general) it's easy to convert between opam filters and dune's blang syntax (used by enabled_if). What if we give users the option to have an available field on their packages instead of a package_enabled_if. It would be checked against the project's enabled_if in the same way. This would improve UX as I claim that dune (accidentally) trains users to expect available to work in dune-project and it would also be generally (as in outside opam) useful as a way for users to declare their package's (machine-checked) availability constraints in a single place.

[rgrinberg]

With this in mind what if we get dune to recognize the available field but to print an error when it's used and the error messages can suggest using an opam template instead.

That would be a useful feature.

[rgrinberg]

You're on the right track with your suggestion of deriving enabled_if for individual stanzas from the available field.

we probably shouldn't call the top-level field enabled_if since it means something different from how enabled_if is used elsewhere. What about package_enabled_if.

I would lean towards using the same name. It's used for the exact same thing, so why introduce another name? If there are some limitations on the generated available field, then so be it.

Instead of checking the toplevel enabled_if against the individual stanzas, I have two alternatives in mind:

• Convert every stanza's enabled_if to (and <enabled-from-package> <user-specified-value>)
• Allow the package level enabled_if to be available as :standard when specifying the enabled_if for individual stanzas.If the user doesn't write anything, we default to :standard.

We have instances where we follow both schemes in the rules already.

[rgrinberg]

Superseded by #12905