Skip to content

Add Boson (Higgs) realtime model support #6835

Description

@helin-mktech

Feature Type

Would make my life easier

Feature Description

Proposal

Boson AI serves Higgs Realtime, a speech-to-speech
model reachable over a WebSocket API modelled on OpenAI's
(docs). I'd like to add
livekit-plugins-boson, exposing it as a LiveKit RealtimeModel.

Opening this per CONTRIBUTING.md's "open an issue first to discuss the viability
and scope" — the implementation is already up as #6340, but I'd rather have the
scope agreed than have a large PR sit unanswered.

Why this needs its own plugin

Higgs speaks a dialect of the OpenAI Realtime protocol, not the protocol, and
the divergences are behavioural rather than cosmetic. Pointing the OpenAI plugin
at the endpoint gives you a session that connects and then misbehaves in ways
that are hard to attribute back to their cause:

  1. session.update replaces the whole session instead of merging into it, so
    a partial update of the kind the OpenAI plugin sends can silently erase
    unrelated fields such as instructions or tools.
  2. The server does not persist sessions. Every connection starts empty, so a
    reconnect has to replay the chat context from the client.
  3. conversation.item.create is a plain insert — it never triggers a reply.
  4. Per-response tools/tool_choice are accepted and ignored. Only
    instructions is honoured per turn, so tools have to be scoped at the session
    level.
  5. Consecutive same-role speech merges into one conversation item, re-sent
    with its accumulated transcript as it grows.
  6. Output is single-modality["text"] or ["audio"], never both.

There is also no insert-at-head primitive: previous_item_id: null always means
append-at-tail, so an update_chat_ctx() that needs to prepend an item (a
context summary, say) has to delete and recreate the remote conversation in order
rather than silently misordering it.

Boson documents much of this themselves — their
migration notes for OpenAI clients
list session fields that Higgs rejects outright (include, prompt, tracing),
a different truncation shape, different voice names, and a different audio
format range.

Scope

One new package, livekit-plugins/livekit-plugins-boson, subclassing the OpenAI
realtime plugin and overriding only where the protocols differ — the same shape
as the xAI plugin.

The connection handling is not forked: _main_task and _run_ws are thin
wrappers that delegate to super(), so the base keeps ownership of the connect,
retry and reconnect flow (including the chat-context replay a reconnect needs).
The wrappers exist only to reclassify the closes Boson reports differently and to
settle pending futures on a terminal failure. What is genuinely Boson-specific is
the session-update payload, conversation synchronisation, response handling, and
error classification.

Nothing in livekit-agents core changes beyond registering the optional extra.

Anyone can verify it

Unlike some provider integrations, this one is self-serve — no sales call, no
allowlist:

  1. Sign in at https://www.boson.ai/workspace with a Google account
  2. New accounts currently get free credit, enough to try this out
  3. Create a key at https://www.boson.ai/workspace/api-key
  4. The endpoint is public and documented: wss://api.boson.ai/v1/realtime

The PR ships examples/agent.py, so trying it costs a key and a microphone:

export BOSON_API_KEY=...
uv run python livekit-plugins/livekit-plugins-boson/examples/agent.py console

Recorded session, if you'd rather watch than set it up:
https://drive.google.com/file/d/1Uc_UMKe-dr_mzXo-Sg-Yqrta8PjwQcMn/view

Status

#6340 is open with the implementation. CI is green (ruff, mypy strict on 3.10
and 3.13, unit tests), the CLA is signed, and the plugin is covered by 93
hermetic unit tests that need no credentials or network.

Is this something you'd accept? I'm happy to rework the approach, split the PR,
or drop anything that doesn't fit the project's conventions — and equally happy
to hear that a separate plugin isn't the direction you want, before more effort
goes into it.

I'm willing to maintain this integration as Boson's API evolves.

Workarounds / Alternatives

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions