soniox: endpoint final can report a span far shorter than its own text (2 words in 180ms)

[aviadr1]

Summary

On the Soniox realtime plugin, a FINAL_TRANSCRIPT emitted at a natural endpoint can carry a start_time/end_time span far shorter than the speech its own text represents. We observed a final whose text was He's trying. but whose span was 0.18s:

{
  "content": " He's trying.",
  "start_time": "2026-08-17T18:46:43.663000Z",
  "end_time":   "2026-08-17T18:46:43.843000Z"
}

Two words in 180ms is not a plausible speaking rate. Longer utterances in the same stream were correct — e.g. 46 words over 7.32s, 53 words over 11.76s — so this is not a constant offset, it affects some finals and not others.

Environment

• livekit-agents / livekit-plugins-soniox 1.6.9
• Soniox model stt-rt-v5, language_hints=["en","he"], no translation config
• Python 3.12

Why it matters

SpeechData.start_time/end_time is the only timing signal for a final, so anything that anchors transcripts on a timeline (captions, clipping, alignment, diarization hand-off) places these utterances wrongly and gives them a near-zero duration.

Suspected cause

send_endpoint_transcript() builds the event from the final accumulator alone:

https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-soniox/livekit/plugins/soniox/stt.py

def send_endpoint_transcript() -> None:
    if final.text:
        ...
        final.to_speech_data(...)   # non_final is never merged here

final only accumulates tokens where token["is_final"] is true, and _TokenAccumulator.update() only advances timing when the token actually carries the keys:

if "start_ms" in token and not self._has_start_time:
    self._has_start_time = True
    self.start_time = float(token["start_ms"])
if "end_ms" in token:
    self.end_time = float(token["end_ms"])

So when an <end>/<fin> token arrives while part of the utterance is still non-final, or when some tokens arrive without start_ms/end_ms, the emitted text can span more audio than the accumulated start_time/end_time do. The interim path already merges both sides via merged_speech_data(...); the endpoint path does not.

Note also that if no token in final ever carried timing, both values stay at their reset() default of 0.0 and the emitted final gets start_time == end_time == start_time_offset.

Suggested fix

Have the endpoint path cover the same tokens its text came from — either merge non_final the way the interim path does, or widen the accumulated span to the tokens actually included in final.text.

Caveat on our setup

We run a thin subclass of SpeechStream that observes incoming frames to emit bounded-latency prefixes; it only peeks at messages and does not modify them or the accumulators. The text and the span above are produced entirely by the stock send_endpoint_transcript() path. Happy to add instrumentation and attach the raw Soniox frames for one of these finals if that would help — say the word and I'll capture them.

[LHMQ878]

I reproduced your exact numbers against the plugin's fake-WS test harness, and I think the suspected cause needs correcting before someone fixes the wrong thing.

non_final is not the problem. In send_endpoint_transcript() both the text and the span come from the same accumulator:

final.to_speech_data(self.start_time_offset, ...)   # text=self.text, start_time=self.start_time, end_time=self.end_time

self.text is only ever appended in _TokenAccumulator.update(), and that same call is the only place start_time/end_time are set. So every token that contributed text also offered its timing — merging non_final would add text and widen the span together, and cannot by itself make the span narrower than the text. The mismatch has to come from tokens whose timing keys are absent, or from timings that aren't monotonic.

Mechanism that reproduces your frame exactly. update() takes start_time from the first token that carries start_ms and overwrites end_time from each token that carries end_ms. If the leading tokens of the utterance arrive without those keys and only a trailing token has them, the span collapses onto that trailing token:

# tokens, all is_final, in one frame, followed by <end>
{"text": " He's",   "language": "en", "is_final": True}                                  # no timing
{"text": " trying", "language": "en", "is_final": True}                                  # no timing
{"text": ".",       "language": "en", "is_final": True, "start_ms": 43663, "end_ms": 43843}

emits

FINAL_TRANSCRIPT: text=" He's trying." span=43.663..43.843 (180ms)

— the same text and the same 180 ms window you observed.

A second, independent mechanism: because end_time is overwritten rather than maxed, a trailing token whose end_ms regresses pulls the end backwards. With " He's" 41000–41400, " trying" 41400–43843 and "." 43663–43700 the emitted span is 41.000..43.700, i.e. 143 ms short. Note merged_speech_data() (the interim path) already guards this with min(...)/max(...); update()/to_speech_data() (the final path) does not, and the class docstring states the chronological ordering as an assumption.

What would settle it. Since you offered to capture the raw frames: for one bad final, the useful thing is the full tokens array of every frame from the previous <end> up to and including the one carrying the <end> — specifically whether each is_final token has start_ms/end_ms at all, and whether those values are monotonic. That distinguishes the two mechanisms above, and they need different fixes (fall back to a neighbouring token's timing vs. make the accumulation monotone).

One more thing worth checking in your capture. is_end_token() treats <fin> and <end> identically, so a <fin> is flushed as if it were an utterance endpoint:

def is_end_token(token: dict) -> bool:
    return token.get("text") in (END_TOKEN, FINALIZED_TOKEN)

With [" He's", " trying", <fin>, " to", " call", <end>] in one frame the harness emits two FINAL_TRANSCRIPTs, " He's trying" (1.0–1.7) and " to call" (1.7–2.1). Each span is internally consistent, so this isn't your 180 ms, but if Soniox emits <fin> for its own periodic finalization rather than only in response to a finalize request, it would explain why some finals in a stream are much shorter than the utterance a human would expect — worth ruling out from the same capture.

(I'm an outside contributor, not a maintainer — happy to send a PR once it's clear which of these the raw frames show.)