Create a new agent in your browser using this model
Overview
OpenAI speech-to-text is available in LiveKit Agents through LiveKit Inference and the OpenAI plugin. With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Use the plugin instead if you want to manage your own billing and rate limits. Pricing for LiveKit Inference is available on the pricing page .
LiveKit Inference
Use LiveKit Inference to access OpenAI STT without a separate OpenAI API key.
A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.
| Model name | Model ID | Languages | EU endpoint |
|---|---|---|---|
GPT-Live Transcribe | openai/gpt-live-transcribe | afarazbebgbscacscydadeelenesetfafifrglhehihrhuhyidisitiwjakkknkoltlvmimkmrmsnenlnoplptroruskslsrsvswtathtltrukurvizhzh-cnzh-twzh-hkyuecmnfilmulti | EU endpoint |
Usage
To use OpenAI, use the STT class from the inference module:
from livekit.agents import AgentSession, inferencesession = AgentSession(stt=inference.STT(model="openai/gpt-live-transcribe",language="en"),# ... llm, tts, vad, turn_handling, etc.)
import { AgentSession, inference } from '@livekit/agents';const session = new AgentSession({stt: new inference.STT({model: "openai/gpt-live-transcribe",language: "en"}),// ... llm, tts, vad, turnHandling, etc.});
Audio format and turn detection
openai/gpt-live-transcribe accepts streaming audio only as 24 kHz PCM or 8 kHz μ-law. The Agents SDK streams PCM, and LiveKit Inference rejects other sample rates instead of resampling them, so if you set sample_rate (sampleRate in Node.js), set it to 24000.
The model has no built-in end-of-turn detection. It emits a final transcript each time your agent finalizes the current audio segment, which the Agents SDK does when it detects the end of speech.
Parameters
modelstringThe model to use for the STT. See model IDs for available models.
languageLanguageCodeLanguage code for the transcription. If not set, the model detects the language.
String descriptors
As a shortcut, you can also pass a model ID string directly to the stt argument in your AgentSession:
from livekit.agents import AgentSessionsession = AgentSession(stt="openai/gpt-live-transcribe:en",# ... llm, tts, vad, turn_handling, etc.)
import { AgentSession } from '@livekit/agents';const session = new AgentSession({stt: "openai/gpt-live-transcribe:en",// ... llm, tts, vad, turnHandling, etc.});
Plugin
LiveKit's plugin support for OpenAI lets you connect directly to OpenAI's STT API with your own API key.
Installation
Install the plugin from PyPI:
uv add "livekit-agents[openai]~=1.8"
pnpm add @livekit/agents-plugin-openai@1.x
Authentication
The OpenAI plugin requires an OpenAI API key .
Set OPENAI_API_KEY in your .env file.
Usage
Use OpenAI STT in an AgentSession or as a standalone transcription service. For example, you can use this STT in the Voice AI quickstart.
from livekit.plugins import openaisession = AgentSession(stt = openai.STT(model="gpt-4o-mini-transcribe",),# ... llm, tts, etc.)
import { inference, voice } from '@livekit/agents';import * as openai from '@livekit/agents-plugin-openai';const vad = new inference.VAD({ model: 'silero' });const session = new voice.AgentSession({stt: new openai.STT({model: 'gpt-realtime-whisper',vad,}),// ... llm, tts, etc.});
The gpt-realtime-whisper model doesn't support server-side turn detection. You must pass a vad instance so the plugin can commit the audio buffer at end-of-speech.
In @livekit/agents-plugin-openai@1.4.1, the default model for openai.STT changed from whisper-1 to gpt-realtime-whisper, which streams transcription over a WebSocket. To restore the previous behavior, set useRealtime: false:
const stt = new openai.STT({ useRealtime: false });
Using an OpenAI-compatible gateway
Realtime transcription streams over a WebSocket, which you can route through an OpenAI-compatible gateway or proxy, such as LiteLLM , Cloudflare AI Gateway, Helicone, or Portkey. Set the baseURL to your gateway endpoint and set the apiKey:
import { voice } from '@livekit/agents';import * as openai from '@livekit/agents-plugin-openai';import * as silero from '@livekit/agents-plugin-silero';const vad = await silero.VAD.load();const session = new voice.AgentSession({stt: new openai.STT({model: 'gpt-realtime-whisper',// Connects to wss://your-gateway.example.com/v1/realtime?intent=transcription&model=gpt-realtime-whisperbaseURL: 'https://your-gateway.example.com/v1',apiKey: process.env.GATEWAY_API_KEY,vad,}),// ... llm, tts, etc.});
The plugin adds the model as a ?model= query parameter on the connection URL when the host isn't api.openai.com, so the gateway can route each connection to the correct model. An https:// base URL maps to wss://, and http:// maps to ws://.
Connecting directly to OpenAI omits the ?model= parameter by design. In that case, the plugin sends the model in the session configuration instead.
Parameters
This section describes some of the available parameters. See the plugin reference links in the Additional resources section for a complete list of all available parameters.
modelWhisperModels | stringDefault: gpt-4o-mini-transcribe | gpt-realtime-whisperModel to use for transcription. See OpenAI's documentation for a list of supported models .
Default model varies by SDK:
- Python:
gpt-4o-mini-transcribe - Node.js:
gpt-realtime-whisper
languageLanguageCodeDefault: enLanguage code for the input audio. See OpenAI's documentation for a list of supported languages .
use_realtimeboolControls whether the plugin streams transcription over a WebSocket connection to the OpenAI Realtime API. When False, the plugin sends audio to the REST transcription API instead. In Node.js, use useRealtime.
Default varies by SDK:
- Python:
Trueforgpt-realtime-whisperandgpt-live-transcribe, andFalsefor other models. - Node.js:
true. If you setuseRealtimetofalseand don't setmodel, the plugin useswhisper-1.
vadVADVAD for client-side end-of-speech detection with gpt-realtime-whisper and gpt-live-transcribe, which don't support server-side turn detection. If you don't set it, the plugin uses the default VAD. Pass None in Python or null in Node.js to turn this off and commit the audio yourself.
temperaturefloatSampling temperature that controls the randomness of the model's output. Higher values make the output more random, while lower values make it more focused and deterministic. Range of valid values can vary by model.
Valid values are between 0 and 1. Realtime transcription doesn't support this parameter. If realtime transcription is active, the plugin ignores the value and logs a warning.
Realtime transcription is on by default as follows:
- Python: only for
gpt-realtime-whisperandgpt-live-transcribe. - Node.js: for all models. Set
useRealtimetofalseto use this parameter.
Additional resources
The following resources provide more information about using OpenAI with LiveKit Agents.