feat(llm): add API backend support with backend-aware session routing

[marcinbogdanski]

Hello, looks like repo is exploding!

Introduce API backend for LLM support so qmd can be run w/o local models

Draft PR for now, feedback welcome, happy to update to maintainer requirements.

Covered:

• API backend covering:
  • embeddings via /v1/embeddings endpoint
  • query expansion via /v1/chat/completions
  • rerank via /v1/rerank
• backend selection via QMD_LLM_BACKEND="api" env var, current behavior unchanged if unset
• search/vsearch/query session paths update to be backend aware (avoids Llama init/download)
• token-based chunking: in API mode skips tokenization, uses char-based chunking
• tests for new API behavior - require provider api keys or skipped
• some backward compatibility tests

Not covered:

• support per embedding backend/model isolation in index - this is likely required to allow API api provider switching?
• I have separate fix in the works on my end to derive index name from model provider/name
  via short hash and always use it instead of default

Related work / other PRs

• I did a quick look (e.g. PRs Add optional OpenRouter provider for embeddings/search pipeline #133, feat: Add OpenAI embedding and query expansion support #116, Add OpenAI-compatible remote LLM support and remote sanity-check scri… #106, feat: add remote embedding providers (Voyage AI, OpenAI-compatible) #65)
• this PR is intended to handle session-aware backend routing and token-based chunking, which seems out-of-scope in other work at this time

Query Expansion - details

• currently simply prompts completions endpoint to return lex/vec/hyde lines
• it does not use constrained generation
• if result parser returns empty, it will return empty array
• params context and includeLexical supported

Error handling

• query
  • API key missing -> throw
  • network/http/request failure -> throw
  • invalid result shape -> throw
  • valid result but no parseable lex|vec|hyde lines -> return empty array
• Embeddings
  • model override (options param) is ignored
  • missing API key -> throws
  • network/http/request failure -> returns null (consistent with local path)
• rerank
  • model override (options param) is ignored
  • missing API key -> throws
  • network/http/request failure -> throws

Hope this is helpful.

PS. got some free time, happy to update to spec or do other work if you want to dump anything on me, now that I am somewhat familiar with the repo

[marcinbogdanski]

Added a simple guard to protect existing embeddings in the database in case user changes embedding backend/provider.

Problem:

• user starts using qmd via API with model A -> db is populated with embeddings from model A
• user changes API embedding model to model B (or switches back to local model), old model A embeddings still exist in the db
• now new embeddings come from new model B, but database still has old incompatible embeddings from model A

How new guard works:

• if qmd is used only with built-in local models (API never enabled) -> nothing changes, I didn't touch that path
• at any point, if embeddinsg are generated with API model, that information is written to small new table in the database
• on all future embed/vsearch/query commands, qmd first checks if current model/provider setting matches what's saved in the database. If not, an error message is shown to user, to either:
  • A) change env var config to match original embeddings and re-run the command
  • B) use different index with --index parameter
  • C) manually run qmd embed -f to force clear old embeddings and rebuild with new model

I investigated adding model/provider scoping to the relevant tables, but it would be too intrusive IMHO for this PR. I'm happy to look into this again (to provide proper multi-model support in single index file), if such functionality is required.

For now if you want to switch model, you need to re-embed or use separate index.

[Tosko4]

@marcinbogdanski would you be willing to support this FR too: #239

[marcinbogdanski]

@Tosko4 This PR supports Cohere re-ranker, but in a different way thatn #239

This PR implements alternative backend ApiLLM that slots-in as replacement to LlamaCpp (with local models). I also implement minimal additional changes to make this backend swap-able.

Unfortunately at present this all-or-nothing, i.e. you either completely disable local models and do embedding/query-extension/re-ranking via API. Or you stick with local models for all steps. I want to keep this PR minimally intrusive so I can easily re-base to latest qmd.

More fine-grained backend control (switch only re-ranking to API, keep other locally) would require significantly more intrusive refactor in qmd. I could have a look, but would require discussing with @tobi to agree on details and scope to ensure it's aligned with his long term goal for the repo.