Dart Build Hooks Support

[nikeokoronkwo]

Dart recently added support for build hooks in version 3.10. However, there isn't a clear way/path for working with or supporting build hook artifacts during CLI distribution using this package.

I have been working on a Dart CLI application that would benefit from the use of the new build hooks feature (or at least some method of distributing compiled library code alongside my Dart CLI application), but I have been blocked on that aspect because I have setup and will be using this package for distributing the application once built.

It would be much appreciated if there can be a way, or at least some guidance, as to how to support build artifacts in cli_pkg pipelines. Since there is unfortunately no support for static linking (dart-lang/sdk#47718), then the compiled libraries have to be placed in a path accessible to the binary tool. Filed an issue that should be somewhat connected via dart-lang/native#3239

[nex3]

I'm definitely open to something like this, but I don't have the bandwidth at the moment to do a full design on it. I'd be interested in hearing suggestions, though.

[nikeokoronkwo]

I was thinking of two approaches if possible:

1. The ability to specify some of these dependencies as external dependencies in build hooks (maybe we could expose a helper to find the possible path(s) for a given external dependency for a given distribution platform given the dependency’s name and (optionally) version).
2. Support dart build cli having custom paths to their libraries so that distribution for a given format can be configured as their requirements deem fit. This is what the current issue I linked to dart-lang/native is about.

Both have their advantages and disadvantages, and are suitable for different use cases. The former, for instance, is helpful if the third-party library the user is working on already exists in the given registry and the user works with a fixed version of the library, whilst the latter is more suited for libraries with modifications, purposefully-vendored libraries, or libraries developed with the package.

[nex3]

Are either of those things we can do in this package, or would they both require support from the Dart SDK itself?

[nikeokoronkwo]

I think a combination, but primarily the first: we can do that in this package right now, since there is already support for declaring system dependencies in build hooks.

[nex3]

So you're suggesting an API like String findDependencyPath(String name, Version? version = null) which tries to match Dart's build hook logic as much as possible?

[nikeokoronkwo]

I’ve been able to make something for homebrew and chocolatey. Chocolatey by default does not support multiple versions of a dependency, so version pinning is really only on Homebrew (for now)

Build hooks can then use assets under these paths like libraries, static assets, etc

[nex3]

What's the value of specifying a particular version here?

Also, reading the Dart hooks documentation, it looks like currently the only supported asset type is FFI-compatible native libraries, which are intended to be loaded using the @Native annotation. I think maybe a better way to approach this might be to try to understand better how the Dart runtime handles the mapping between asset IDs and physical locations and see if we can hook into that somehow.

[nikeokoronkwo]

What's the value of specifying a particular version here?

For API guarantees, since most compiled libraries will be compiled at a specific version/commit for use normally.

Also, reading the Dart hooks documentation, it looks like currently the only supported asset type is FFI-compatible native libraries, which are intended to be loaded using the @Native annotation.

afaik, C libraries (like Lua, openssl, sqlite, etc) are FFI-Compatible, so they should fit in this category, as they are available in all repositories supported by this package.

I think maybe a better way to approach this might be to try to understand better how the Dart runtime handles the mapping between asset IDs and physical locations and see if we can hook into that somehow.

Could you elaborate a bit more? I assume this should be handled by package:hooks and its runner, so maybe we can check that and see how we can work with that.

[nex3]

I think there are two separate concerns being conflated here:

1. Using assets (especially native libraries) produced by Dart build hooks, either from the local package or from one of its pub dependencies. These assets would presumably either be included in the distributed archive or built as part of the installation process.

2. Using assets (especially native libraries) from pre-packaged non-Dart third-party libraries in a consistent way across distribution methods. Using sqlite as an example, it has packages on Chocolatey and Homebrew (although it's not entirely clear to me if the Homebrew version makes the shared library available or just the binary), and if I understand correctly the goal is to be able to load one of those shared libraries instead of compiling the asset locally.

These are not necessarily related, although it's conceivable we could define infrastructure that abstracts them out and makes it possible to load a shared library from a dependency when possible and from a local build otherwise. There are a lot of complications there, though. There's no guarantee that packages of the same software across multiple repositories have the same name, structure, or even necessarily API. Trying to make too many assumptions about what "sqlite" means between Chocolatey and Homebrew may result in disaster, especially if we add different repositories in the future. If we do do this, I think we have to find a way to shift the burden of configuring each package for each repository onto the user, at which point I'm not sure how much value we're realistically providing.

All of that is probably getting ahead of ourselves, though. The first thing to address, and the only thing covered by this issue's actual title is concern 1: using assets produced directly by Dart build hooks. We'll likely need to have a robust design for that before we can even start to consider concern 2.