Expose model-native user transcripts from GPT-Realtime

[smorimoto]

Summary

I would like LiveKit Agents JS to support emitting user transcripts generated by the same realtime model that handles the conversation, for example gpt-realtime-2, instead of relying only on input_audio_transcription with a separate ASR model such as gpt-4o-mini-transcribe, gpt-4o-transcribe, or gpt-realtime-whisper.

The motivation is not simply to improve transcription quality. The main issue is that the transcript shown to the user can differ from what the realtime model appears to have understood and responded to. This is confusing in voice applications where the visible transcript is treated as the source of truth for the conversation.

Current behaviour

With OpenAI Realtime models in LiveKit Agents JS, user transcription is currently driven by inputAudioTranscription.

For example:

llm: new openai.realtime.RealtimeModel({
  model: "gpt-realtime-2",
  inputAudioTranscription: {
    model: "gpt-4o-mini-transcribe",
    language: "ja",
  },
})

This works, but it means the conversation is effectively using two different recognisers:

• gpt-realtime-2 listens to the audio and decides how to respond.
• gpt-4o-mini-transcribe produces the transcript emitted through UserInputTranscribedEvent / lk.transcription.

In some cases, especially for short utterances, non-English speech, product names, proper nouns, or domain-specific terminology, these two can disagree. The assistant may respond appropriately, while the transcript displayed in the UI says something else.

If inputAudioTranscription is set to null, the realtime model can still understand and respond, but user transcript events are no longer emitted through the normal LiveKit transcription pipeline.

Requested behaviour

It would be useful to have an option for model-native user transcription, where LiveKit emits a best-effort transcript produced by the realtime model itself, using the same user audio item that the model is responding to.

For example, conceptually:

llm: new openai.realtime.RealtimeModel({
  model: "gpt-realtime-2",
  inputAudioTranscription: null,
  userTranscriptionMode: "realtime_model",
})

or:

llm: new openai.realtime.RealtimeModel({
  model: "gpt-realtime-2",
  inputAudioTranscription: null,
  realtimeUserTranscription: {
    enabled: true,
    instructions: "Transcribe the user's utterance in Japanese. Preserve product names and proper nouns where possible.",
  },
})

The resulting transcript should ideally flow through the existing LiveKit mechanisms:

• AgentSessionEventTypes.UserInputTranscribed
• UserInputTranscribedEvent
• RoomIO transcription forwarding
• lk.transcription
• session reports / chat history where appropriate

This would allow applications to keep using the existing LiveKit transcript UI and persistence paths, while avoiding a separate ASR model whose output may not match the realtime model's interpretation.

Possible implementation approach

OpenAI's Realtime API appears to support an out-of-band transcription pattern, where a separate response.create can be issued for the relevant user audio item with instructions to transcribe rather than answer.

I am not asking LiveKit to expose raw OpenAI events directly, but a LiveKit-level abstraction around this would be very helpful.

A possible implementation could be:

1. Keep normal gpt-realtime-2 conversation handling unchanged.
2. When a user audio turn is committed, also request a model-native transcript for that user audio item.
3. Emit the resulting text as a UserInputTranscribedEvent.
4. Ensure the transcript can be associated with the correct realtime item id.
5. Optionally allow applications to choose whether this transcript should update chat history / session reports.

I noticed that item id preservation has recently been worked on in:

• fix(agents): preserve transcription item id #1902

That may make this easier to support cleanly.

Why input_audio_transcription is not enough

This is slightly different from improving input_audio_transcription.

Better ASR models and streaming deltas are useful, but they still leave two separate model paths:

• one model for recognition/transcription,
• another model for realtime understanding and response generation.

For user-facing conversation logs, the most important property is often consistency with the assistant's behaviour. If the assistant clearly understood the user correctly but the displayed transcript is wrong, the user loses trust in the experience.

This is especially noticeable in sales/demo/meeting-style voice agents, where the transcript is later used for review, summaries, CRM notes, or audit trails.

Alternatives considered

Use gpt-4o-transcribe or gpt-realtime-whisper

This may improve transcript quality, but it still does not guarantee consistency with what gpt-realtime-2 used to generate its response.

Disable user transcripts

Setting inputAudioTranscription: null avoids showing an incorrect transcript, but then the application loses user-side transcript events, live captions, conversation history, and downstream session reports.

Implement a separate OpenAI Realtime connection outside LiveKit

This may be possible, but it duplicates audio/session handling and bypasses LiveKit's existing UserInputTranscribedEvent, lk.transcription, and session-report mechanisms. It would be much cleaner if LiveKit Agents provided this as a supported option.

Questions

Would the maintainers be open to supporting model-native user transcription for OpenAI Realtime models?

If so, should this live as:

• a new option on openai.realtime.RealtimeModel,
• a separate transcription strategy,
• an extension of inputAudioTranscription,
• or a more general API for out-of-band realtime model responses?

I would be happy to test this with Japanese speech and domain-specific terminology if that would help.