LiveKit Agents integrations

Overview

LiveKit Agents includes support for a wide variety of AI providers, from the largest research companies to emerging startups.

The open source plugin interface makes it easy to adopt the best AI providers for your app, without needing customized code for each.

Installing plugins

Each provider is available as a separate plugin package, included as an optional dependency on the base SDK for Python.

For example, to install the SDK with the Cartesia, Deepgram, and OpenAI plugins, run the following command:

pip install "livekit-agents[cartesia,deepgram,openai]~=1.0"

You may also install plugins as individual packages. For example, this is equivalent to the previous command:

pip install \
  "livekit-agents~=1.0" \
  "livekit-plugins-cartesia~=1.0" \
  "livekit-plugins-deepgram~=1.0" \
  "livekit-plugins-openai~=1.0"

Using plugins

The AgentSession class accepts plugins as arguments using a standard interface. Each plugin loads its own associated API key from environment variables. For instance, the following code creates an AgentSession that uses the OpenAI, Cartesia, and Deepgram plugins installed in the preceding section:

from livekit.plugins import openai, cartesia, deepgram

session = AgentSession(
    llm=openai.LLM(model="gpt-4o"),
    tts=cartesia.TTS(model="sonic-english"),
    stt=deepgram.STT(model="nova-2"),
)

OpenAI API compatibility

Many providers have standardized around the OpenAI API format for chat completions and more. The LiveKit Agents OpenAI plugin provides easy compatibility with many of these providers through special methods which load the correct API key from environment variables. For instance, to use Cerebras instead of OpenAI, you can use the following code:

from livekit.plugins import openai

session = AgentSession(
    llm=openai.LLM.with_cerebras(model="llama-3.1-70b-versatile"),
    # ... stt, tts, etc ..
)

Core plugins

The following are the core plugin types used in LiveKit Agents, to handle the primary voice AI tasks. Many providers are available for most functions.

Realtime models

Plugins for multimodal speech-to-speech models like the OpenAI Realtime API.

Large language models (LLM)

Plugins for AI models from OpenAI, Anthropic, and more.

Speech-to-text (STT)

Plugins for speech-to-text solutions like Deepgram, Whisper, and more.

Text-to-speech (TTS)

Plugins for text-to-speech solutions like Cartesia, ElevenLabs, and more.

Additional plugins

LiveKit Agents also includes the following additional specialized plugins, which are recommended for most voice AI use cases. Each runs locally and requires no additional API keys.

Silero VAD

Voice activity detection with Silero VAD.

LiveKit turn detector

A custom LiveKit model for improved end-of-turn detection.

Enhanced noise cancellation

LiveKit Cloud enhanced noise cancellation to improve voice AI performance.

Build your own plugin

The LiveKit Agents plugin framework is extensible and community-driven. Your plugin can integrate with new providers or directly load models for local inference. LiveKit especially welcomes new TTS, STT, and LLM plugins.

To learn more, see the guidelines for contributions to the Python and Node.js SDKs.