Realtime models overview

Overview

Realtime models are capable of consuming and producing speech directly, bypassing the need for a voice pipeline with speech-to-text and text-to-speech components. They can be better at understanding the emotional context of input speech, as well as other verbal cues that may not translate well to text transcription. Additionally, the generated speech can include similar emotional aspects and other improvements over what a text-to-speech model can produce.

You can also use supported realtime models in tandem with a TTS instance of your choice, to gain the benefits of realtime speech comprehension while maintaining complete control over speech output.

You can choose a realtime model served through LiveKit Inference, included with LiveKit Cloud. With LiveKit Inference, you use your LiveKit Cloud credentials instead of a provider API key, and LiveKit Cloud bills you for usage. Use a plugin instead if you prefer to manage your own billing, or need access to a provider not currently available through LiveKit Inference.

Turn-based and full-duplex models

Realtime models are either turn-based or full-duplex:

• Turn-based: The model waits for the user to finish speaking, then responds. Most realtime models work this way, including the OpenAI Realtime API.
• Full-duplex: The model listens and speaks at the same time, and decides when each turn starts and stops. OpenAI GPT-Live is a full-duplex model.

Choose a full-duplex model when natural conversational flow matters most. It handles overlapping speech and backchannels more naturally than a turn-based model. Choose a turn-based model, such as the OpenAI Realtime API, when you want to configure turn detection yourself or pair the model with a separate TTS in a realtime hybrid architecture. GPT-Live doesn't support either.

Full-duplex models work differently in a few places. The model handles barge-in, so the framework can stop the audio it plays to the caller, but the model keeps talking until it stops on its own. Output streams continuously, silence included, so LiveKit Agents calculates the turn boundaries from the audio, and conversation items arrive after the audio they describe. These models also can't speak a script word for word, so scripted speech needs a separate TTS.

LiveKit Inference

The following table lists the realtime models available through LiveKit Inference. Refer to the guide for each provider for more details on additional configuration options.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Gemini Live API

OpenAI GPT-Live

OpenAI Realtime API

SpaceXAI Grok Voice Agent API

Plugins

The LiveKit Agents framework also includes plugins for popular realtime models. Plugins require authentication with the provider yourself, usually with an API key, and you manage your own billing. The following table lists the available realtime model providers, along with their availability for Python or Node.js.

LiveKit is open source and welcomes new plugin contributions.

Usage

To set up a realtime model in an AgentSession, pass a realtime model instance to the llm argument, in place of an LLM instance.

LiveKit Inference usage

With LiveKit Inference, use the RealtimeModel class from the inference module. Consult the models list for available models.

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1"),
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    llm: new inference.RealtimeModel({ model: "openai/gpt-realtime-2.1" }),
});

Plugin usage

Realtime model plugins have a constructor method to create a model instance. You can pass this instance directly to an AgentSession or Agent in its constructor.

The following example uses OpenAI GPT-Live. For the equivalent with another provider, see the provider's plugin page.

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.GPTLiveModel(voice="marin")
)

import voice from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.GPTLiveModel({ voice: 'marin' })
});

For additional information about installing and using plugins, see the plugins overview.

Separate TTS configuration

To use a realtime model with a different TTS instance, configure the realtime model to use a text-only response modality and include a TTS instance in your AgentSession configuration. In this setup, the realtime model doesn't generate audio; all speech synthesis is handled by the configured TTS.

This is commonly referred to as a realtime hybrid architecture (or informally as a half-cascade or half-duplex pipeline). Use it when you want both realtime speech understanding and full control over what your agent says. A realtime hybrid architecture has the following strengths:

• Realtime speech comprehension: The realtime model still hears prosody and emotional cues in user speech.
• Output control: A separate TTS lets you speak exact scripts and choose any voice your TTS provider supports.

The tradeoff is that you configure and operate both a realtime model and a TTS, so the setup is closer to a pipeline than a pure realtime agent.

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1", modalities=["text"]),
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a", # Or other TTS instance of your choice
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1', modalities: ['text'] }),
  tts: 'fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a', // Or other TTS instance of your choice
});

session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]), # Or other realtime model plugin
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a" # Or other TTS instance of your choice
)

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({ modalities: ["text"] }), // Or other realtime model plugin
   tts: "fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a" // Or other TTS instance of your choice
});

This feature requires realtime model support for a text-only response modality. Visit the relevant provider page for details about supported features.

Considerations and limitations

Realtime models bring great benefits due to their wider range of audio understanding and expressive output. However, they also have some limitations and considerations to keep in mind. For a side-by-side comparison of all pipeline types, see Pipeline types.

Turn detection and VAD

In general, LiveKit recommends using the built-in turn detection capabilities of the realtime model whenever possible. Accurate turn detection relies on both VAD and context gained from realtime speech-to-text, which, as discussed in the following section, isn't available with realtime models. If you need to use the LiveKit turn detector model, you must also add a separate STT plugin to provide the necessary interim transcripts.

Delayed transcription

Realtime models don't provide interim transcription results, and in general the user input transcriptions can be considerably delayed and often arrive after the agent's response. If you need realtime transcriptions, you should consider an STT-LLM-TTS pipeline or add a separate STT plugin for realtime transcription.

Scripted speech output

Realtime models don't offer a method to directly generate speech from a text script, such as with the say method. You can produce a response with generate_reply(instructions='...') and include specific instructions but the output isn't guaranteed to precisely follow any provided script. If your application requires the use of specific scripts, consider using the model with a separate TTS instance instead.

Loading conversation history

Current models only support loading call history in text format. This limits their ability to interpret emotional context and other verbal cues that may not translate well to text transcription. Additionally, the OpenAI Realtime API becomes more likely to respond in text only after loading extensive history, even if configured to use speech. For OpenAI, it's recommended that you use a separate TTS instance if you need to load conversation history.