Azure OpenAI Realtime API and LiveKit

Overview

Azure OpenAI  provides an implementation of OpenAI's Realtime API that enables low-latency, multimodal interactions with realtime audio and text processing through Azure's managed service. OpenAI realtime models on Azure are available in LiveKit Agents through LiveKit Inference and the OpenAI plugin. With LiveKit Inference, you use your LiveKit Cloud credentials instead of your own Azure OpenAI deployment, and LiveKit Cloud bills you for usage. Use the plugin instead to connect to an Azure OpenAI deployment that you manage.

LiveKit Inference

Use LiveKit Inference to access OpenAI Realtime models hosted on Azure without managing Azure resources or providing your own API key.

The following table lists the OpenAI realtime models available through LiveKit Inference.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Usage

Use the RealtimeModel class from the inference module to run OpenAI realtime models through LiveKit Inference. By default, LiveKit Inference selects between OpenAI and Azure for you. To use Azure, set provider to azure:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1", provider="azure"),
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
  llm: new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1', provider: 'azure' }),
});

Authentication uses your LiveKit Cloud credentials. Set LIVEKIT_API_KEY and LIVEKIT_API_SECRET in your .env file, or pass api_key and api_secret directly. In Node.js, these parameters are apiKey and apiSecret.

Parameters

The following are the main parameters for configuring OpenAI realtime models on Azure with LiveKit Inference. For other parameters, see Parameters in the OpenAI Realtime API topic. For a complete list, see the Python API reference.

Plugin

Use the OpenAI plugin to connect directly to an Azure OpenAI deployment that you manage, with your own API key.

Installation

Install the OpenAI plugin from PyPI:

uv add "livekit-agents[openai]~=1.8"

pnpm add @livekit/agents-plugin-openai@1.x

Authentication

The Azure OpenAI plugin requires an Azure OpenAI API key  and your Azure OpenAI endpoint.

Set the following environment variables in your .env file:

AZURE_OPENAI_API_KEY=<your-azure-openai-api-key>
AZURE_OPENAI_ENDPOINT=<your-azure-openai-endpoint>

Usage

Use the Azure OpenAI Realtime API within an AgentSession:

from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel.with_azure(
        azure_deployment="<model-deployment>",
        azure_endpoint="wss://<endpoint>.openai.azure.com/",
        api_key="<api-key>",
    ),
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
    llm: openai.realtime.RealtimeModel.withAzure({
        azureDeployment: "<model-deployment>",
        azureEndpoint: "wss://<endpoint>.openai.azure.com/",
        apiKey: "<api-key>",
    }),
});

For a more comprehensive agent example, see the Voice AI quickstart.

Preview API deployments

Azure deprecated the preview Realtime API on April 30, 2026. The api_version parameter and OPENAI_API_VERSION environment variable target the preview API and are no longer supported. Migrate to a GA deployment to avoid service disruption. To learn more, see Microsoft's Realtime API migration guide .

Parameters

This section describes the Azure-specific parameters. For a complete list of all available parameters, see the plugin reference links in the Additional resources section.

Turn detection

The Azure OpenAI Realtime API includes voice activity detection (VAD)  to automatically detect when a user has started or stopped speaking. This feature is enabled by default

There is one supported mode for VAD:

• Server VAD (default) - Uses periods of silence to automatically chunk the audio

Server VAD

Server VAD is the default mode and can be configured with the following properties:

from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import TurnDetection

session = AgentSession(
    llm=realtime.RealtimeModel(
        turn_detection=TurnDetection(
            type="server_vad",
            threshold=0.5,
            prefix_padding_ms=300,
            silence_duration_ms=200,
            create_response=True,
            interrupt_response=True,
        )
    ),
    # ... vad, tts, stt, etc.
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
   llm: new openai.realtime.RealtimeModel({
      turnDetection: {
         type: 'server_vad',
         threshold: 0.5,
         prefix_padding_ms: 300,
         silence_duration_ms: 200,
         create_response: true,
         interrupt_response: true,
      },
   }),
});

• threshold: Higher values require louder audio to activate, better for noisy environments.
• prefix_padding_ms: Amount of audio to include before detected speech.
• silence_duration_ms: Duration of silence to detect speech stop (shorter = faster turn detection).

Usage with separate TTS

To use the Azure OpenAI Realtime API with a different TTS instance, configure it with a text-only response modality and include a TTS instance in your AgentSession configuration. This configuration allows you to gain the benefits of direct speech understanding while maintaining complete control over the speech output.

session = AgentSession(
    llm=openai.realtime.RealtimeModel.with_azure(
        # ... endpoint and auth params ...,
        modalities=["text"]
    ),
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a" # Or other TTS instance of your choice
)

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
    llm: openai.realtime.RealtimeModel.withAzure({
        // ... endpoint and auth params ...,
        modalities: ["text"]
    }),
    tts: "fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a", // Or other TTS instance of your choice
});

Loading conversation history

If you load conversation history into the model, it might respond with text output even if configured for audio response. To work around this issue, use the model with a separate TTS instance and text-only response modality. You can use the Azure OpenAI TTS plugin to continue using the same voices supported by the Realtime API.

For additional workaround options, see the OpenAI thread  on this topic.

Additional resources

The following resources provide more information about using Azure OpenAI with LiveKit Agents.

Python plugin

ReferenceGitHubPyPI

Node.js plugin

ReferenceGitHubNPM

Azure OpenAI docs

Azure OpenAI service documentation.

Voice AI quickstart

Get started with LiveKit Agents and Azure OpenAI.

Azure ecosystem overview

Overview of the entire Azure AI ecosystem and LiveKit Agents integration.