Add vector embedding extensions

[impredicative]

I request that various good compatible extensions for vector embeddings be added.

For a discussion on vector embedding extensions, which are very useful for AI/ML, please see this preliminary chat: https://chatgpt.com/share/689e680c-9fdc-8012-835d-e2e334bf802a

To start, in the simple case, it is not necessary for the extension itself to be able to compute the embedding. To start, I can compute the embedding vector myself.

[ncruces]

As your AI chatbot recommend, you can use sqlite-vec for vector search.

Alex's build is rather old, so this repository demonstrates how you can build and use your own binary. It's really not that difficult.

If you are going to use this, I recommend that you build it yourself. If you're happy with the result maybe publish, share and maintain it.

I don't really use vector search so I'd like to avoid the maintenance burden.

[impredicative]

Okay. Vector search is useful for quickly finding similar rows. For example, if I store news article titles, I can find titles that are conceptually similar, not just in words. Maybe you'll find a reason to use it :)

[ncruces]

Yes, Alex's extension really is great. He also demonstrates how to mix vector and full text search:
https://alexgarcia.xyz/blog/2024/sqlite-vec-hybrid-search/index.html

I just mean that it doesn't make sense to make the embed sqlite3.wasm larger for all users of the package, and that, if I don't use it personally and regularly, it doesn't really make much sense for me to publish that package myself.

There are simply too many interesting extensions and configuration options for me to support all of them out-of-the-box.

But I am deeply committed to two ideas:

1. make as much if the API available to Go, so you can create interesting extensions in Go; and
2. make building C extensions as easy as possible.

This includes customizing your own SQLite (running ./configure, changing #defines), and statically linking extensions like Alex's, or others from the SQLite website, etc.

And to make sure this is not pie in the sky, I do many of these things myself:

1. create a bunch of interesting Go extensions
2. offer experimental builds of SQLite
3. demonstrate how you build the Wasm blobs

[impredicative]

In contrast, https://github.com/sqliteai/sqlite-vector (not to be confused with sqlite-vec) is not open source, so it can be skipped.

[ncruces]

I saw that. It should be simple enough to use the examples above to build your own Wasm including it. If you do, please report back any issues/challenges!

To be clear: I am interested, and will be as helpful as possible if you do have concrete issues. But I'm already spread thin, and can't afford more exploratory work.

[ncruces]

The Vec1 Vector Extension by the SQLite authors as been added to the latest release.

It hasn't been formally released upstream, and this port is very much unoptimized.

But please experiment, and if you have a good benchmark to share, that's the first step for me to optimize it.