Pipeline types

Overview

LiveKit Agents supports the following voice pipeline types:

• Cascaded pipeline: Three specialized models (STT, LLM, and TTS) chained in sequence for speech recognition, language understanding, and speech synthesis.
• Turn-based realtime model: A single speech-to-speech model that waits for the user to finish speaking, then responds. The OpenAI Realtime API is turn-based.
• Full-duplex realtime model: A single speech-to-speech model that listens and speaks at the same time. OpenAI GPT-Live is full-duplex.

For most production agents, a cascaded pipeline is the right default. The following sections cover each option and when to choose it.

At a glance

The following table compares the three pipeline types across the dimensions that most often drive architecture selection. The rest of this topic covers each one in more depth.

Cascaded STT-LLM-TTS pipeline

A cascaded pipeline, also called an STT-LLM-TTS or sequential pipeline, strings together three specialized models. Audio flows through them in sequence: speech-to-text (STT) transcribes the user's speech, a large language model (LLM) generates a text response, and text-to-speech (TTS) speaks the response back. Each stage has a clean interface, so you can swap any component independently or change models partway through a session.

Choose this for most production agents. It gives you full control over each stage and is the easiest path to debug and audit.

Strengths

• Modularity: You can mix and match providers for STT, LLM, and TTS, and replace any stage without modifying the others.
• Observability: Every stage produces text or audio you can inspect, log, and audit.
• Mature tool calling: Text-based LLM tool calling is more mature and predictable than audio-native alternatives.
• Realtime transcription: STT produces interim transcripts you can stream to your frontend or store as a record of the conversation.
• Scripted speech: TTS reads exact text, so methods like say() produce predictable output.

Considerations

• Higher total latency: Each stage adds latency. Streaming overlaps the stages and keeps total latency low, but a pipeline still adds more end-to-end delay than a realtime model.
• Loss of vocal nuance: Because STT produces text, prosody and emotional cues in the user's speech don't reach the LLM.

Example

Pass an STT, LLM, and TTS to AgentSession. Each stage accepts a LiveKit Inference model string or a plugin instance.

from livekit.agents import AgentSession, TurnHandlingOptions, inference

session = AgentSession(
    stt="assemblyai/universal-3-6-pro:en",
    llm="google/gemma-4-31b-it",
    tts="fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a",
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(),
    ),
)

import { inference, voice } from '@livekit/agents';

const session = new voice.AgentSession({
  stt: 'assemblyai/universal-3-6-pro:en',
  llm: 'google/gemma-4-31b-it',
  tts: 'fishaudio/s2.1-pro:fa4c9eb3dccc4806b382b40d61c6b10a',
  turnHandling: {
    turnDetection: new inference.TurnDetector(),
  },
});

Realtime models

A realtime model consumes and produces speech directly, in a single model. There's no transcription step on the way in and no separate TTS on the way out.

Realtime models are either turn-based or full-duplex:

• Turn-based: The model waits for the user to finish speaking, then responds. Most realtime models work this way, including the OpenAI Realtime API.
• Full-duplex: The model listens and speaks at the same time, and decides when each turn starts and stops. OpenAI GPT-Live is a full-duplex model.

Choose a realtime model when latency or expressive output matter more than fine-grained control.

Strengths

• Lower end-to-end latency: Combining stages into one model removes the inter-stage hand-offs.
• Expressive output: Generated speech can carry emotion, emphasis, and other prosodic features that text-to-speech models don't capture.
• Richer input understanding: The model hears prosody, tone, and other verbal cues that get lost in transcription.
• Simpler setup: A single model with one provider, instead of three.

Considerations

• Delayed transcripts: Realtime models don't produce interim transcripts. User transcriptions can lag the agent's response. If you need live captions or transcription-driven logic, add a separate STT plugin.
• No scripted speech: The model follows instructions but doesn't read an exact script, so methods like say() aren't supported the same way they are in a pipeline.
• Less provider flexibility: You're committed to a single provider's model for the full speech-to-speech path.
• Harder to audit: Without a text trail at every stage, debugging and compliance review take more work.

For the full list of limitations, see Considerations and limitations on the realtime models page.

To keep realtime speech understanding and add scripted speech, pair a realtime model with a separate TTS in a realtime hybrid architecture. To learn more, see Separate TTS configuration.

Turn-based example

Pass the realtime model to AgentSession as the llm. You don't need an STT or TTS. The following example uses the OpenAI Realtime API through LiveKit Inference:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.RealtimeModel(model="openai/gpt-realtime-2.1"),
)

import { inference, voice } from '@livekit/agents';

const session = new voice.AgentSession({
  llm: new inference.RealtimeModel({ model: 'openai/gpt-realtime-2.1' }),
});

To use the OpenAI plugin instead, with your own OpenAI API key, pass a plugin instance:

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    llm=openai.realtime.RealtimeModel(voice="marin"),
)

import { voice } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
  llm: new openai.realtime.RealtimeModel({ voice: 'marin' }),
});

Full-duplex example

A full-duplex model listens and speaks at the same time, and decides when each turn starts and stops. With LiveKit Inference, use the DuplexModel class in place of RealtimeModel. The following example uses OpenAI GPT-Live through LiveKit Inference:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.DuplexModel(model="openai/gpt-live-1", voice="marin"),
)

import { inference, voice } from '@livekit/agents';

const session = new voice.AgentSession({
  llm: new inference.DuplexModel({ model: 'openai/gpt-live-1', voice: 'marin' }),
});

Latency

Voice conversations feel natural when end-to-end response latency stays under one second. Each architecture has a different latency profile. Pipelines accumulate latency across stages but reduce it through streaming, while realtime models combine stages for a lower baseline. For a per-stage breakdown of voice agent latency, see Sequential pipeline architecture for voice agents .

Additional resources