OpenAI GPT-Live plugin guide

Overview

GPT-Live is a full-duplex voice model from OpenAI. It listens and speaks at the same time, and it decides when each turn starts and stops. Under the default responses delegation, it sends its reasoning and tool work to a backend Responses model.

The plugin implements GPT-Live as a duplex model, and an adapter presents it to the framework as a realtime model. Pass it to AgentSession in place of an LLM.

GPT-Live differs from the OpenAI Realtime API in these ways:

• Server-driven. No client event creates, cancels, or truncates a response. The model starts and ends its own turns.
• Model-controlled barge-in. GPT-Live listens while it speaks, and it decides when to stop. The framework can stop the audio it plays to the user, but the model keeps talking until it stops on its own.
• Delegated reasoning. The voice model handles the conversation. A separate backend model handles the reasoning and decides which tools to call.
• Append-only context. You can load a conversation before the session starts, and append new items after it starts. You can't change or delete an item.
• No half-cascade. GPT-Live has no text-only response modality, so you can't pair it with a TTS plugin in a half-cascade setup.
• Audio only. The plugin doesn't support video or image input, including to the backend model, even when that model supports images on its own.

Installation

GPT-Live is part of the OpenAI plugin:

uv add "livekit-agents[openai]~=1.8"

Authentication

The OpenAI plugin requires an OpenAI API key  on an account with GPT-Live alpha access.

Set OPENAI_API_KEY in your .env file.

Usage

Use GPTLiveModel within an AgentSession. For example, you can use it in the Voice AI quickstart.

from livekit.agents import AgentSession
from livekit.plugins.openai.realtime import GPTLiveModel

session = AgentSession(
    llm=GPTLiveModel(
        voice="marin",
        # backend Responses model that handles the reasoning and the tool calls
        responses_options={
            "model": "gpt-5.6-luna",
            "instructions": "Use tools when current information is required.",
        },
    ),
)

import { voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
  llm: new openai.realtime.GPTLiveModel({
    voice: 'marin',
    // backend Responses model that handles the reasoning and the tool calls
    responsesOptions: {
      model: 'gpt-5.6-luna',
      instructions: 'Use tools when current information is required.',
    },
  }),
});

Parameters

This section describes some of the available parameters. For a complete reference of all available parameters, see the plugin reference links in the Additional resources section.

Backend model options

The responses_options dictionary accepts the following keys. Only model has a plugin default. The plugin leaves out every other key you don't set, so the OpenAI default for that key applies. For those defaults, see the Responses API reference  in the OpenAI documentation.

Voice and backend instructions

Two sets of instructions configure GPT-Live, and you can't change either set after the session starts:

• The voice model takes the agent's instructions: how to converse, when to delegate, and what to say while the work is underway. Set them on the Agent. GPTLiveModel has no instructions parameter.
• The backend model takes the instructions key of responses_options: how to handle delegated work, which tools to use, and what to hand back to the voice model.

The voice model doesn't see the tools, so don't describe them in the persona. Say which requests to delegate and which to answer directly. Without that, the model delegates small talk, or tries to answer a question it has no information about.

from livekit.agents import Agent
from livekit.plugins.openai.realtime import GPTLiveModel

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            # voice persona: the top-level GPT-Live instructions
            instructions=(
                "You are a helpful voice assistant for an online furniture store. "
                "Keep replies short and conversational. "
                "Ask before taking any external action. "
                # when to delegate, and what to say while the work is underway
                "Delegate any question about an order, a delivery, or a price, and say "
                "what you're checking while you wait. Answer greetings, small talk, and "
                "questions about the store yourself."
            ),
            llm=GPTLiveModel(
                responses_options={
                    "instructions": (
                        "You handle the work the voice model delegates. Use tools when "
                        "current information is required, and answer with a short result "
                        "the voice model can read out."
                    ),
                },
            ),
        )

import { voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

class Assistant extends voice.Agent {
  constructor() {
    super({
      // voice persona: the top-level GPT-Live instructions
      instructions:
        'You are a helpful voice assistant for an online furniture store. ' +
        'Keep replies short and conversational. ' +
        'Ask before taking any external action. ' +
        // when to delegate, and what to say while the work is underway
        'Delegate any question about an order, a delivery, or a price, and say ' +
        "what you're checking while you wait. Answer greetings, small talk, and " +
        'questions about the store yourself.',
      llm: new openai.realtime.GPTLiveModel({
        responsesOptions: {
          instructions:
            'You handle the work the voice model delegates. Use tools when ' +
            'current information is required, and answer with a short result ' +
            'the voice model can read out.',
        },
      }),
    });
  }
}

If you set new instructions mid-session, the plugin raises a RealtimeError and the model continues to use the original persona. You can't replace the persona, but you can add standing rules during the session. See Conversation context.

For more information about prompting a realtime voice model, including the preambles it speaks while delegated work is underway, see Using realtime models  in the OpenAI documentation.

Tools

With the default responses delegation, the backend model handles tool calling. The framework runs your @function_tool methods in your agent process and sends each result back to the backend model. Once every call in a batch has a result, the backend model continues, and the voice model speaks the outcome on its own.

Delegated work doesn't block the conversation. The voice model keeps listening and speaking while the backend reasons and calls tools, so use the agent's instructions to tell it what to do in the meantime: acknowledge the request, give updates, or wait quietly for the result.

You can update the tools and the tool choice mid-session, but you can't update the voice or either set of instructions.

The backend model can also call the OpenAI provider tools. These run on OpenAI's servers, not in your agent process. The available tools are WebSearch, FileSearch, and CodeInterpreter. You can use provider tools and function tools together:

from livekit.agents import Agent, RunContext
from livekit.agents.llm import function_tool
from livekit.plugins import openai
from livekit.plugins.openai.realtime import GPTLiveModel

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice assistant for an online furniture store.",
            llm=GPTLiveModel(),
            tools=[openai.tools.WebSearch()],  # replace with any supported provider tool
        )

    @function_tool
    async def check_order_status(self, context: RunContext, order_id: str) -> str:
        """Check the delivery status of an order.

        Args:
            order_id: The order reference, such as A1042.
        """
        return f"Order {order_id} is shipped, arriving Thursday."

import { llm, voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';
import { z } from 'zod';

class Assistant extends voice.Agent {
  constructor() {
    super({
      instructions: 'You are a helpful voice assistant for an online furniture store.',
      llm: new openai.realtime.GPTLiveModel(),
      tools: [
        new openai.tools.WebSearch(), // replace with any supported provider tool
        llm.tool({
          name: 'checkOrderStatus',
          description: 'Check the delivery status of an order.',
          parameters: z.object({
            orderId: z.string().describe('The order reference, such as A1042.'),
          }),
          execute: async ({ orderId }) => {
            return `Order ${orderId} is shipped, arriving Thursday.`;
          },
        }),
      ],
    });
  }
}

Delegation

Delegation controls where the voice model sends its reasoning and tool work. You can't change the target after the session starts. There are two options:

• responses (default): a backend Responses model handles the reasoning and decides which tools to call. Your @function_tool methods still run in your agent process, and the plugin sends each result back to that model.
• client: your own code handles the reasoning instead of a backend model. This mode has no tool channel, so the plugin ignores @function_tool methods and logs a warning if you register any.

Client delegation

Choose client delegation when the reasoning belongs in your own code: your own LLM, your own retrieval stack, or a plain database lookup. No backend model runs, so nothing reaches a second OpenAI model.

The delegated work arrives as a GPTLiveDelegation on the duplex session's delegation_created event. It carries no task and no arguments, only an id to answer against. Work out the request from the conversation: Agent.chat_ctx holds the turns so far, and pending_transcript holds the user's current turn, which hasn't reached the chat context yet.

Produce the answer however you like, but not with a @function_tool, which this mode ignores. Send it back with append_commentary on Agent.duplex_session, passing the id. Commentary is one of three context channels, and it gives the model something to say now in its own words.

from livekit.agents import Agent
from livekit.plugins.openai.realtime import GPTLiveDelegation, GPTLiveModel

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice assistant for an online furniture store.",
            llm=GPTLiveModel(delegation="client"),
        )

    async def on_enter(self) -> None:
        self.duplex_session.on("delegation_created", self._on_delegation)

    def _on_delegation(self, delegation: GPTLiveDelegation) -> None:
        # the user's open turn is the newest part of the request
        answer = look_up_order(delegation.pending_transcript)
        self.duplex_session.append_commentary(answer, delegation_id=delegation.id)

import { voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const ORDERS: Record<string, string> = {
  A1042: 'shipped, arriving Thursday',
  B2231: 'still being packed',
};

class Assistant extends voice.Agent {
  constructor() {
    super({
      instructions: 'You are a helpful voice assistant for an online furniture store.',
      llm: new openai.realtime.GPTLiveModel({ delegation: 'client' }),
    });
  }

  async onEnter(): Promise<void> {
    this.duplexSession.on('delegationCreated', (delegation: openai.realtime.GPTLiveDelegation) => {
      // the user's open turn is the newest part of the request
      const asked = delegation.pendingTranscript;
      console.log(`model delegated while the user said: ${asked}`);
      const orderId = Object.keys(ORDERS).find((o) => asked.toUpperCase().includes(o));
      this.duplexSession.appendCommentary(
        orderId ? `Order ${orderId} is ${ORDERS[orderId]}.` : 'No matching order is on file.',
        { delegationId: delegation.id },
      );
    });
  }
}

Repeated append_commentary calls with the same delegation_id continue the same delegation instead of starting a new one, so you can report progress before the final answer. To pass information the model shouldn't say out loud, call append_thinking with the same id.

Conversation context

A chat_ctx that you pass before the session starts becomes the startup history. The service accepts a maximum of 128 messages and 8192 rendered tokens. It drops the oldest messages first:

from livekit.agents import Agent, ChatContext

def prior_conversation() -> ChatContext:
    chat_ctx = ChatContext.empty()
    chat_ctx.add_message(role="user", content="Hi, I ordered a standing desk last week.")
    chat_ctx.add_message(
        role="assistant", content="Thanks for calling. I have your order A1042 on file."
    )
    return chat_ctx

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice assistant for an online furniture store.",
            chat_ctx=prior_conversation(),
        )

import { llm, voice } from '@livekit/agents';

function priorConversation(): llm.ChatContext {
  const chatCtx = llm.ChatContext.empty();
  chatCtx.addMessage({ role: 'user', content: 'Hi, I ordered a standing desk last week.' });
  chatCtx.addMessage({
    role: 'assistant',
    content: 'Thanks for calling. I have your order A1042 on file.',
  });
  return chatCtx;
}

class Assistant extends voice.Agent {
  constructor() {
    super({
      instructions: 'You are a helpful voice assistant for an online furniture store.',
      chatCtx: priorConversation(),
    });
  }
}

After the session starts, the startup history is append-only. The plugin appends new items for the model to read. If you change or delete an item, the plugin logs a warning and the model keeps the original.

To give the model information from outside the conversation, append it on one of the duplex session's three channels. Each channel takes a maximum of 500 tokens per append, and what you add doesn't reach session.history:

# thinking: silent context, which the model uses only if it becomes relevant
self.duplex_session.append_thinking("The caller is a Priority Plus member.")

# commentary: the model acts on this immediately, in its own words
self.duplex_session.append_commentary(
    "Tell the caller their delivery window just moved to Friday."
)

# instructions: developer guidance for the rest of the session
self.duplex_session.append_instructions("Confirm the delivery address before you book anything.")

// thinking: silent context, which the model uses only if it becomes relevant
this.duplexSession.appendThinking('The caller is a Priority Plus member.');

// commentary: the model acts on this immediately, in its own words
this.duplexSession.appendCommentary('Tell the caller their delivery window just moved to Friday.');

// instructions: developer guidance for the rest of the session
this.duplexSession.appendInstructions('Confirm the delivery address before you book anything.');

Appended instructions tell the model what to do or say for the rest of the session. They don't replace the startup persona, which stays in place. A system or developer message that you add to the chat context goes to the same channel.

As a session approaches the context limit, GPT-Live compacts the conversation history itself and the connection continues. The plugin logs the context-window utilization at debug level with each usage update from the service.

Turns and transcripts

GPT-Live streams audio continuously. The audio can include sounds such as laughter, which the model doesn't transcribe. The framework segments the continuous stream into turns. It uses an adaptive noise gate, which opens when the output is louder than the model's noise floor. This has two effects:

• LiveKit Agents calculates the turn boundaries. The model doesn't mark them. Each continuous stretch of output becomes one turn, and a pause shorter than about half a second doesn't split it.
• Transcripts come after the audio. A turn closes when the model's output goes quiet, and the conversation item for the turn arrives after the speech.

Initiating speech

generate_reply() appends your instruction as commentary, not as a direct response or utterance. The model can refuse a request that doesn't fit the conversation.

If the model hasn't started speaking ten seconds after the request, the reply counts as refused. A newer request, a reconnect, or a closed session ends a pending reply the same way. The framework logs the failure, returns the agent to its listening state, and marks the returned SpeechHandle done with a RealtimeError. Call exception() after the handle is done to check whether the model answered:

class Assistant(Agent):
    async def on_enter(self) -> None:
        handle = self.session.generate_reply(
            instructions=(
                "Greet the caller by picking up where the earlier conversation left off, "
                "and ask how you can help."
            )
        )
        await handle
        if handle.exception() is not None:
            logger.info("the model declined to greet the caller")

class Assistant extends voice.Agent {
  async onEnter(): Promise<void> {
    const handle = this.session.generateReply({
      instructions:
        'Greet the caller by picking up where the earlier conversation left off, ' +
        'and ask how you can help.',
    });
    try {
      await handle.waitForPlayout();
    } catch {
      console.log('the model declined to greet the caller');
    }
  }
}

GPT-Live can't speak a script word for word. session.say() raises an error unless you attach a TTS to the AgentSession. Even then, the model can speak at the same time as the TTS, and it never receives the TTS audio. If your app needs exact wording, use an STT-LLM-TTS pipeline instead.

Turn detection and interruption

The model controls turn detection and barge-in. It listens while it speaks, and it decides when to stop. The LiveKit turn detection options can't change that.

You can't stop the model, but you can stop playing its audio to the user:

• Barge-in needs a VAD. The AgentSession cuts playback with its own interruption detection, which needs a VAD. Pass one explicitly, because the session drops its default VAD for this model. Without a VAD, the agent plays until the model stops on its own.
• The model doesn't know what the user heard. When playback stops, the agent's history records only the part that played, but the model's context still holds the whole turn. GPT-Live doesn't support message truncation, so you can't correct it, and the model can refer to something the user never heard.

Agent handoff

An agent handoff can't reuse a GPT-Live connection. You can't change the voice or either set of instructions during a session. If you hand off to an agent with different instructions or a different chat context, the plugin starts a new session. The new session sends the conversation again as startup history, with the same limits: 128 messages and 8192 tokens. Tools are the exception. The plugin updates them in place, so a handoff that changes only the tools keeps the connection.

Session controls

The duplex session has these other GPT-Live controls:

• mute_input() replaces microphone input with silence, and unmute_input() restores it. The model continues to generate speech while the input is muted.
• append_instructions(text), append_thinking(text), and append_commentary(text) are the three context channels: developer guidance, silent context, and something to say now. Each takes a maximum of 500 tokens.
• session_id is the service's id for the current connection. It changes when the plugin reconnects.

Pricing and usage

OpenAI prices the voice model by session duration and the backend model by token. For the current rate, see Pricing  in the OpenAI documentation. The plugin reports the two separately:

• The voice model reports its cumulative session seconds about once a minute, and the remainder when the session closes. The plugin emits each delta as the session_duration field of a RealtimeModelMetrics event.
• The backend model reports tokens for every response it completes. The plugin emits those as LLMMetrics under the backend model's name.

Read the totals from session.usage, as in the following shutdown callback:

async def log_usage() -> None:
    # gpt-live reports its voice usage about once a minute, and the last of it only on close
    logger.info("usage: %s", session.usage)

ctx.add_shutdown_callback(log_usage)

const logUsage = async () => {
  // gpt-live reports its voice usage about once a minute, and the last of it only on close
  console.log('usage:', session.usage);
};

ctx.addShutdownCallback(logUsage);

For more information, see Metrics and usage data.

Additional resources

The following resources provide more information about using OpenAI GPT-Live with LiveKit Agents.

Python plugin

ReferenceGitHubPyPI

Voice AI quickstart

Get started with LiveKit Agents and OpenAI.

OpenAI Realtime API

Use the OpenAI Realtime API, which supports video input and a text-only modality.

OpenAI ecosystem overview

Overview of the entire OpenAI and LiveKit Agents integration.