Skip to main content

OpenAI STT

How to use OpenAI STT with LiveKit Agents.

On this page

Use in Agent Builder

Create a new agent in your browser using this model

Overview

OpenAI speech-to-text is available in LiveKit Agents through LiveKit Inference and the OpenAI plugin. With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Use the plugin instead if you want to manage your own billing and rate limits. Pricing for LiveKit Inference is available on the pricing page .

LiveKit Inference

Use LiveKit Inference to access OpenAI STT without a separate OpenAI API key.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Model nameModel IDLanguagesEU endpoint
GPT-Live Transcribe
openai/gpt-live-transcribe
afarazbebgbscacscydadeelenesetfafifrglhehihrhuhyidisitiwjakkknkoltlvmimkmrmsnenlnoplptroruskslsrsvswtathtltrukurvizhzh-cnzh-twzh-hkyuecmnfilmulti
EU endpoint

Usage

To use OpenAI, use the STT class from the inference module:

from livekit.agents import AgentSession, inference
session = AgentSession(
stt=inference.STT(
model="openai/gpt-live-transcribe",
language="en"
),
# ... llm, tts, vad, turn_handling, etc.
)
import { AgentSession, inference } from '@livekit/agents';
const session = new AgentSession({
stt: new inference.STT({
model: "openai/gpt-live-transcribe",
language: "en"
}),
// ... llm, tts, vad, turnHandling, etc.
});

Audio format and turn detection

openai/gpt-live-transcribe accepts streaming audio only as 24 kHz PCM or 8 kHz μ-law. The Agents SDK streams PCM, and LiveKit Inference rejects other sample rates instead of resampling them, so if you set sample_rate (sampleRate in Node.js), set it to 24000.

The model has no built-in end-of-turn detection. It emits a final transcript each time your agent finalizes the current audio segment, which the Agents SDK does when it detects the end of speech.

Parameters

model
Required
string

The model to use for the STT. See model IDs for available models.

languageLanguageCode

Language code for the transcription. If not set, the model detects the language.

String descriptors

As a shortcut, you can also pass a model ID string directly to the stt argument in your AgentSession:

from livekit.agents import AgentSession
session = AgentSession(
stt="openai/gpt-live-transcribe:en",
# ... llm, tts, vad, turn_handling, etc.
)
import { AgentSession } from '@livekit/agents';
const session = new AgentSession({
stt: "openai/gpt-live-transcribe:en",
// ... llm, tts, vad, turnHandling, etc.
});

Plugin

LiveKit's plugin support for OpenAI  lets you connect directly to OpenAI's STT API with your own API key.

Available inPython
|
Node.js

Installation

Install the plugin from PyPI:

uv add "livekit-agents[openai]~=1.8"
pnpm add @livekit/agents-plugin-openai@1.x

Authentication

The OpenAI plugin requires an OpenAI API key .

Set OPENAI_API_KEY in your .env file.

Usage

Use OpenAI STT in an AgentSession or as a standalone transcription service. For example, you can use this STT in the Voice AI quickstart.

from livekit.plugins import openai
session = AgentSession(
stt = openai.STT(
model="gpt-4o-mini-transcribe",
),
# ... llm, tts, etc.
)
import { inference, voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';
const vad = new inference.VAD({ model: 'silero' });
const session = new voice.AgentSession({
stt: new openai.STT({
model: 'gpt-realtime-whisper',
vad,
}),
// ... llm, tts, etc.
});
VAD requirement

The gpt-realtime-whisper model doesn't support server-side turn detection. You must pass a vad instance so the plugin can commit the audio buffer at end-of-speech.

Default model change in Node.js

In @livekit/agents-plugin-openai@1.4.1, the default model for openai.STT changed from whisper-1 to gpt-realtime-whisper, which streams transcription over a WebSocket. To restore the previous behavior, set useRealtime: false:

const stt = new openai.STT({ useRealtime: false });

Using an OpenAI-compatible gateway

Only Available inNode.js

Realtime transcription streams over a WebSocket, which you can route through an OpenAI-compatible gateway or proxy, such as LiteLLM , Cloudflare AI Gateway, Helicone, or Portkey. Set the baseURL to your gateway endpoint and set the apiKey:

import { voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';
import * as silero from '@livekit/agents-plugin-silero';
const vad = await silero.VAD.load();
const session = new voice.AgentSession({
stt: new openai.STT({
model: 'gpt-realtime-whisper',
// Connects to wss://your-gateway.example.com/v1/realtime?intent=transcription&model=gpt-realtime-whisper
baseURL: 'https://your-gateway.example.com/v1',
apiKey: process.env.GATEWAY_API_KEY,
vad,
}),
// ... llm, tts, etc.
});

The plugin adds the model as a ?model= query parameter on the connection URL when the host isn't api.openai.com, so the gateway can route each connection to the correct model. An https:// base URL maps to wss://, and http:// maps to ws://.

Connecting directly to OpenAI omits the ?model= parameter by design. In that case, the plugin sends the model in the session configuration instead.

Parameters

This section describes some of the available parameters. See the plugin reference links in the Additional resources section for a complete list of all available parameters.

modelWhisperModels | stringDefault: gpt-4o-mini-transcribe | gpt-realtime-whisper

Model to use for transcription. See OpenAI's documentation for a list of supported models .

Default model varies by SDK:

  • Python: gpt-4o-mini-transcribe
  • Node.js: gpt-realtime-whisper
languageLanguageCodeDefault: en

Language code for the input audio. See OpenAI's documentation for a list of supported languages .

use_realtimebool

Controls whether the plugin streams transcription over a WebSocket connection to the OpenAI Realtime API. When False, the plugin sends audio to the REST transcription API instead. In Node.js, use useRealtime.

Default varies by SDK:

  • Python: True for gpt-realtime-whisper and gpt-live-transcribe, and False for other models.
  • Node.js: true. If you set useRealtime to false and don't set model, the plugin uses whisper-1.
vadVAD

VAD for client-side end-of-speech detection with gpt-realtime-whisper and gpt-live-transcribe, which don't support server-side turn detection. If you don't set it, the plugin uses the default VAD. Pass None in Python or null in Node.js to turn this off and commit the audio yourself.

temperaturefloat

Sampling temperature that controls the randomness of the model's output. Higher values make the output more random, while lower values make it more focused and deterministic. Range of valid values can vary by model.

Valid values are between 0 and 1. Realtime transcription doesn't support this parameter. If realtime transcription is active, the plugin ignores the value and logs a warning.

Realtime transcription is on by default as follows:

  • Python: only for gpt-realtime-whisper and gpt-live-transcribe.
  • Node.js: for all models. Set useRealtime to false to use this parameter.

Additional resources

The following resources provide more information about using OpenAI with LiveKit Agents.