OpenAI Realtime API

Overview

The OpenAI Realtime API enables low-latency, multimodal interactions with realtime text, audio, image, and video features. OpenAI realtime models are available in LiveKit Agents through LiveKit Inference and the OpenAI plugin. With LiveKit Inference, you use your LiveKit Cloud credentials instead of a separate OpenAI API key, and LiveKit Cloud bills you for usage. Use the plugin instead if you want to manage your own billing.

LiveKit Inference

Use LiveKit Inference to access OpenAI realtime models without a separate OpenAI API key.

The following table lists the OpenAI realtime models available through LiveKit Inference.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Usage

Use the RealtimeModel class from the inference module to run OpenAI realtime models through LiveKit Inference. You can use this model in the Voice AI quickstart:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1", voice="marin"),
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1', voice: 'marin' }),
});

Authentication uses your LiveKit Cloud credentials. Set LIVEKIT_API_KEY and LIVEKIT_API_SECRET in your .env file, or pass api_key and api_secret directly. In Node.js, these parameters are apiKey and apiSecret.

Parameters

The following are the main parameters for configuring OpenAI realtime models with LiveKit Inference. For a complete list, see the API reference for Python and Node.js.

Usage with separate TTS

To use an OpenAI realtime model through LiveKit Inference with a different TTS instance, set a text-only response modality and add a TTS instance to your AgentSession. This setup is a realtime hybrid architecture.

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1", modalities=["text"]),
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a", # Or other TTS instance of your choice
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1', modalities: ['text'] }),
  tts: 'fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a', // Or other TTS instance of your choice
});

Azure

LiveKit Inference serves openai/gpt-realtime models from both OpenAI and Azure and selects between them for you. You don't configure Azure yourself. The with_azure() method and its azure_deployment, entra_token, and api_version arguments from the Azure plugin don't apply. In Node.js, the equivalents are withAzure(), azureDeployment, entraToken, and apiVersion. To choose a provider, pass provider:

inference.RealtimeModel(model="openai/gpt-realtime-2.1", provider="azure")

new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1', provider: 'azure' });

String descriptors

As a shortcut, you can also pass the model ID directly to the llm argument in your AgentSession:

from livekit.agents import AgentSession

session = AgentSession(
    llm="openai/gpt-realtime-2.1",
)

import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
  llm: 'openai/gpt-realtime-2.1',
});

Plugin

Use the OpenAI plugin to connect directly to the OpenAI Realtime API with your own API key.

Installation

Install the OpenAI plugin:

uv add "livekit-agents[openai]~=1.8"

pnpm add "@livekit/agents-plugin-openai@1.x"

Authentication

The OpenAI plugin requires an OpenAI API key .

Set OPENAI_API_KEY in your .env file.

Usage

Use the OpenAI Realtime API within an AgentSession. For example, you can use it in the Voice AI quickstart.

from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel(voice="marin"),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({ voice: "marin" }),
});

Parameters

This section describes some of the available parameters. For a complete reference of all available parameters, see the plugin reference links in the Additional resources section.

Video input

The OpenAI Realtime API supports live video input. When enabled, the agent receives frames from the user's camera or screen share and sends each frame to the model as an image message.

Video input requires the OpenAI plugin. LiveKit Inference doesn't support image or video input for realtime models.

To enable video, set video_input=True in your RoomOptions:

class VideoAssistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice assistant with live video input from your user.",
            llm=openai.realtime.RealtimeModel(voice="marin"),
        )

server = AgentServer()

@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: JobContext):
    session = AgentSession()

    await session.start(
        agent=VideoAssistant(),
        room=ctx.room,
        room_options=room_io.RoomOptions(
            video_input=True,
        ),
    )

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    room_io,
)
from livekit.plugins import openai

By default, the agent samples one frame per second while the user speaks and one frame every three seconds otherwise. To control the frame rate, pass a video_sampler to the AgentSession.

For more details on video input configuration, see Live video input.

Usage with separate TTS

To use the OpenAI Realtime API with a different TTS instance, configure it with a text-only response modality and include a TTS instance in your AgentSession configuration. This configuration allows you to gain the benefits of realtime speech comprehension while maintaining complete control over the speech output.

session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]),
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a" # Or other TTS instance of your choice
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({ 
      modalities: ["text"]
   }),
   tts: "fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a", // Or other TTS instance of your choice
});

Loading conversation history

If you load conversation history into the model, it might respond with text output even if configured for audio response. To work around this issue, use the model with a separate TTS instance and text-only response modality. You can use the Azure OpenAI TTS plugin to continue using the same voices supported by the Realtime API.

For additional workaround options, see the OpenAI thread  on this topic.

Turn detection

OpenAI's Realtime API includes voice activity detection (VAD)  to automatically detect when a user has started or stopped speaking. This feature is enabled by default.

There are two modes for VAD:

• Semantic VAD (default): Uses a semantic classifier to detect when the user has finished speaking based on their words.
• Server VAD: Uses periods of silence to automatically chunk the audio.

Semantic VAD

Semantic VAD is the default mode. It uses a classifier to determine when the user is done speaking based on their words. This mode is less likely to interrupt users mid-sentence or chunk transcripts prematurely.

from livekit.agents import AgentSession, inference
from openai.types.realtime.realtime_audio_input_turn_detection import SemanticVad

session = AgentSession(
    llm=inference.RealtimeModel(
        model="openai/gpt-realtime-2.1",
        turn_detection=SemanticVad(
            type="semantic_vad",
            eagerness="medium",
            create_response=True,
            interrupt_response=True,
        ),
    ),
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({
    model: 'openai/gpt-realtime-2.1',
    turnDetection: {
      type: 'semantic_vad',
      eagerness: 'medium',
      create_response: true,
      interrupt_response: true,
    },
  }),
});

from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import TurnDetection

session = AgentSession(
    llm=realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="semantic_vad",
            eagerness="medium",
            create_response=True,
            interrupt_response=True,
        )
    ),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({
      turnDetection: {
         type: 'semantic_vad',
         eagerness: 'medium',
         create_response: true,
         interrupt_response: true,
      },
   }),
});

The eagerness property controls how quickly the model responds:

• auto (default) - Equivalent to medium.
• low - Lets users take their time speaking.
• high - Chunks audio as soon as possible.
• medium - Balanced approach.

For more information about turn detection in general, see the Turn detection guide.

Server VAD

Server VAD can be configured with the following properties:

from livekit.agents import AgentSession, inference
from openai.types.realtime.realtime_audio_input_turn_detection import ServerVad

session = AgentSession(
    llm=inference.RealtimeModel(
        model="openai/gpt-realtime-2.1",
        turn_detection=ServerVad(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=500,
            create_response=True,
            interrupt_response=True,
        ),
    ),
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({
    model: 'openai/gpt-realtime-2.1',
    turnDetection: {
      type: 'server_vad',
      threshold: 0.5,
      prefix_padding_ms: 300,
      silence_duration_ms: 500,
      create_response: true,
      interrupt_response: true,
    },
  }),
});

from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import TurnDetection

session = AgentSession(
    llm=realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=500,
            create_response=True,
            interrupt_response=True,
        )
    ),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({
      turnDetection: {
         type: 'server_vad',
         threshold: 0.5,
         prefix_padding_ms: 300,
         silence_duration_ms: 500,
         create_response: true,
         interrupt_response: true,
      },
   }),
});

• threshold: Higher values require louder audio to activate, better for noisy environments.
• prefix_padding_ms: Amount of audio to include before detected speech.
• silence_duration_ms: Duration of silence to detect speech stop (shorter = faster turn detection).

Additional resources

The following resources provide more information about using OpenAI with LiveKit Agents.

Python plugin

ReferenceGitHubPyPI

Node.js plugin

ReferenceGitHubNPM

Voice AI quickstart

Build a simple realtime model voice assistant using the OpenAI Realtime API in less than 10 minutes.

OpenAI docs

OpenAI Realtime API documentation.

OpenAI ecosystem overview

Overview of the entire OpenAI and LiveKit Agents integration.