OpenAI Realtime API integration guide

Overview

OpenAI's Realtime API enables low-latency, multimodal interactions with realtime audio and text processing. Use LiveKit's OpenAI plugin to create an agent that uses the Realtime API.

Quick reference

This section includes a basic usage example and some reference material. For links to more detailed documentation, see Additional resources.

Installation

Install the OpenAI plugin:

pip install "livekit-agents[openai]~=1.2"

pnpm add "@livekit/agents-plugin-openai@1.x"

Authentication

The OpenAI plugin requires an OpenAI API key.

Set OPENAI_API_KEY in your .env file.

Usage

Use the OpenAI Realtime API within an AgentSession. For example, you can use it in the Voice AI quickstart.

from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel(),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel(),
});

Parameters

This section describes some of the available parameters. For a complete reference of all available parameters, see the plugin reference links in the Additional resources section.

Turn detection

OpenAI's Realtime API includes voice activity detection (VAD) to automatically detect when a user has started or stopped speaking. This feature is enabled by default.

There are two modes for VAD:

• Server VAD (default): Uses periods of silence to automatically chunk the audio.
• Semantic VAD: Uses a semantic classifier to detect when the user has finished speaking based on their words.

Server VAD

Server VAD is the default mode and can be configured with the following properties:

from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import TurnDetection

session = AgentSession(
    llm=realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=500,
            create_response=True,
            interrupt_response=True,
        )
    ),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({
      turnDetection: {
         type: "server_vad",
         threshold: 0.5,
         prefix_padding_ms: 300,
         silence_duration_ms: 500,
         create_response: true,
         interrupt_response: true,
      },
   }),
});

• threshold: Higher values require louder audio to activate, better for noisy environments.
• prefix_padding_ms: Amount of audio to include before detected speech.
• silence_duration_ms: Duration of silence to detect speech stop (shorter = faster turn detection).

Semantic VAD

Semantic VAD uses a classifier to determine when the user is done speaking based on their words. This mode is less likely to interrupt users mid-sentence or chunk transcripts prematurely.

from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import TurnDetection

session = AgentSession(
    llm=realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="semantic_vad",
            eagerness="auto",
            create_response=True,
            interrupt_response=True,
        )
    ),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({
      turnDetection: null,
   })
   turnDetection: new livekit.turnDetector.EnglishModel(),
});

The eagerness property controls how quickly the model responds:

• auto (default) - Equivalent to medium.
• low - Lets users take their time speaking.
• high - Chunks audio as soon as possible.
• medium - Balanced approach.

For more information about turn detection in general, see the Turn detection guide.

Usage with separate TTS

To use the OpenAI Realtime API with a different TTS provider, configure it with a text-only response modality and include a TTS plugin in your AgentSession configuration. This configuration allows you to gain the benefits of realtime speech comprehension while maintaining complete control over the speech output.

session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]),
    tts=cartesia.TTS() # Or other TTS plugin of your choice
)

import * as openai from '@livekit/agents-plugin-openai';
import * as elevenlabs from '@livekit/agents-plugin-elevenlabs';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({ 
      modalities: ["text"]
   }),
   tts: new cartesia.TTS(), // Or other TTS plugin of your choice
});

Loading conversation history

If you load conversation history into the model, it might respond with text output even if configured for audio response. To work around this issue, use the model with a separate TTS plugin and text-only response modality. You can use the Azure OpenAI TTS plugin to continue using the same voices supported by the Realtime API.

For additional workaround options, see the OpenAI thread on this topic.

Additional resources

The following resources provide more information about using OpenAI with LiveKit Agents.

Python Agents

ReferenceGitHubPyPI

Node.js Agents

ReferenceGitHubNPM

Voice AI quickstart

Build a simple realtime model voice assistant using the OpenAI Realtime API in less than 10 minutes.

OpenAI docs

OpenAI Realtime API documentation.

OpenAI ecosystem overview

Overview of the entire OpenAI and LiveKit Agents integration.