Speech-to-text (STT) models overview

Overview

STT models, also known as Automated Speech Recognition (ASR) models, are used for realtime transcription or translation of spoken audio. In voice AI, they form the first of three models in the core pipeline: text is transcribed by an STT model, then processed by an LLM model to generate a response which is turned back into speech using a TTS model.

You can choose a model served through LiveKit Inference, included with LiveKit Cloud. With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Use the plugin instead if you prefer to manage billing and rate limits yourself, or need access to a provider not currently available through LiveKit Inference.

LiveKit Inference

The following models are available in LiveKit Inference. Refer to the guide for each model for more details on additional configuration options.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Deepgram

AssemblyAI

Cartesia

Gemini

Speechmatics

SpaceXAI

Plugins

The LiveKit Agents framework also includes a variety of open source plugins for a wide range of STT providers. These plugins require authentication with the provider yourself, usually via an API key. You are responsible for setting up your own account and managing your own billing and credentials. The plugins are listed below, along with their availability for Python or Node.js.

Have another provider in mind? LiveKit is open source and welcomes new plugin contributions.

Usage

To set up STT in an AgentSession, use the STT class from the inference module. Consult the models list for available models and languages.

from livekit.agents import AgentSession, inference

session = AgentSession(
    stt=inference.STT(
        model="assemblyai/universal-3-6-pro",
        language="en"
    ),
    # ... llm, tts, etc.
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    stt: new inference.STT({
        model: "assemblyai/universal-3-6-pro",
        language: "en"
    }),
    // ... llm, tts, etc.
})

Multilingual transcription

If you don't know the language of the input audio, or expect multiple languages to be used simultaneously, use a Deepgram model that supports multi, such as Nova-3. The languages multi covers vary by model. See multilingual transcription.

from livekit.agents import AgentSession

session = AgentSession(
    stt="deepgram/nova-3:multi",
    # ... llm, tts, etc.
)

import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
    stt: "deepgram/nova-3:multi",
    // ... llm, tts, etc.
})

Additional parameters

More configuration options, such as custom vocabulary, are available for each model. To set additional parameters, use the STT class from the inference module. Consult each model reference for examples and available parameters.

Advanced features

The following sections cover more advanced topics common to all STT providers. For more detailed reference on individual provider configuration, consult the model reference or plugin documentation for that provider.

Automatic model selection

If you don't need to use any specific model features, and are only interested in the best model available for a given language, you can specify the language alone with the special model ID auto. LiveKit Inference chooses the best model for the given language automatically.

from livekit.agents import AgentSession

session = AgentSession(
    # Use the best available model for Spanish
    stt="auto:es",   
)

import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
    // Use the best available model for Spanish
    stt: "auto:es",
})

LiveKit Inference supports the following languages. You can pass any of these values using the LanguageCode type, which accepts ISO 639-1, BCP-47, ISO 639-3, language names, and other common formats.

Keyterms

Improve recognition of domain-specific vocabulary, such as names, brands, and technical terms, with a single keyterm set that applies across providers. You can supply static keyterms and enable automatic detection from the live conversation. For details, see Keyterm management.

Custom STT

To create an entirely custom STT, implement the STT node in your agent.

Standalone usage

You can use an STT instance in a standalone fashion, without an AgentSession, using the streaming interface. Use push_frame to add realtime audio frames to the stream, and then consume a stream of SpeechEvent events as output.

Here is an example of a standalone STT app:

import asyncio

from dotenv import load_dotenv

from livekit import agents, rtc
from livekit.agents import AgentServer
from livekit.agents.stt import SpeechEventType, SpeechEvent
from typing import AsyncIterable
from livekit.plugins import (
    deepgram,
)

load_dotenv(".env.local")

server = AgentServer()

@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: agents.JobContext):
    @ctx.room.on("track_subscribed")
    def on_track_subscribed(track: rtc.RemoteTrack):
        print(f"Subscribed to track: {track.name}")

        asyncio.create_task(process_track(track))

    async def process_track(track: rtc.RemoteTrack):
        stt = deepgram.STT(model="nova-2")
        stt_stream = stt.stream()
        audio_stream = rtc.AudioStream(track)

        async with asyncio.TaskGroup() as tg:
            # Create task for processing STT stream
            stt_task = tg.create_task(process_stt_stream(stt_stream))

            # Process audio stream
            async for audio_event in audio_stream:
                stt_stream.push_frame(audio_event.frame)

            # Indicates the end of the audio stream
            stt_stream.end_input()

            # Wait for STT processing to complete
            await stt_task

    async def process_stt_stream(stream: AsyncIterable[SpeechEvent]):
        try:
            async for event in stream:
                if event.type == SpeechEventType.FINAL_TRANSCRIPT:
                    print(f"Final transcript: {event.alternatives[0].text}")
                elif event.type == SpeechEventType.INTERIM_TRANSCRIPT:
                    print(f"Interim transcript: {event.alternatives[0].text}")
                elif event.type == SpeechEventType.START_OF_SPEECH:
                    print("Start of speech")
                elif event.type == SpeechEventType.END_OF_SPEECH:
                    print("End of speech")
        finally:
            await stream.aclose()


if __name__ == "__main__":
    agents.cli.run_app(server)

VAD and StreamAdapter

Some STT providers or models, such as Whisper , don't support streaming input. In these cases, your app must determine when a chunk of audio represents a complete segment of speech. You can do this using VAD together with the StreamAdapter class.

The following example modifies the previous example to use VAD and StreamAdapter to buffer user speech until VAD detects the end of speech:

from livekit import agents, rtc
from livekit.agents import inference
from livekit.plugins import openai

async def process_track(ctx: agents.JobContext, track: rtc.Track):
  whisper_stt = openai.STT()
  vad = inference.VAD(
    model="silero",
    min_speech_duration=0.1,
    min_silence_duration=0.5,
  )
  vad_stream = vad.stream()
  # StreamAdapter buffers audio until VAD emits END_SPEAKING event
  stt = agents.stt.StreamAdapter(whisper_stt, vad_stream)
  stt_stream = stt.stream()
  ...

Speaker diarization

Speaker diarization identifies which speaker is talking when several people share one microphone. STT providers that support diarization label each transcript with a speaker identifier. In Python, you can wrap the STT with MultiSpeakerAdapter to detect the primary speaker and suppress or label background speech. To learn more, including the full list of supported providers, see Speaker diarization.

Provider-specific metadata

When you use LiveKit Inference, some STT providers include additional information with each transcript. This information is available in the metadata field of each SpeechData object in the alternatives list (event.alternatives[0].metadata). If a provider doesn't include any additional information, the field is None (Python) or undefined (Node.js).

The following providers populate metadata today:

Any Inference provider that emits per-transcript data populates metadata.

To read it, access metadata on a transcript alternative. The following example overrides the STT node to inspect the metadata on each final transcript. Interim transcripts can also carry metadata, so filter on the event type you need:

from typing import AsyncIterable

from livekit import rtc
from livekit.agents import Agent, ModelSettings, stt


class MyAgent(Agent):
    async def stt_node(
        self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings
    ) -> AsyncIterable[stt.SpeechEvent]:
        async for event in Agent.default.stt_node(self, audio, model_settings):
            if event.type == stt.SpeechEventType.FINAL_TRANSCRIPT:
                alt = event.alternatives[0]
                # metadata is None unless the provider emits extra data
                if alt.metadata:
                    voice_profile = alt.metadata.get("voice_profile")
                    print(f"voice profile: {voice_profile}")
            yield event

import { stt, voice } from '@livekit/agents';
import type { AudioFrame } from '@livekit/rtc-node';
import { ReadableStream, TransformStream } from 'stream/web';

const agent = voice.Agent.create({
  async sttNode(ctx, audio, modelSettings) {
    const events = await voice.Agent.default.sttNode(ctx.agent, audio, modelSettings);
    if (!events) return null;

    return events.pipeThrough(
      new TransformStream({
        transform(event, controller) {
          if (typeof event !== 'string' && event.type === stt.SpeechEventType.FINAL_TRANSCRIPT) {
            // metadata is typed as Record<string, unknown>, so narrow before use
            const voiceProfile = event.alternatives?.[0]?.metadata?.['voice_profile'];
            if (voiceProfile && typeof voiceProfile === 'object') {
              console.log('voice profile:', voiceProfile);
            }
          }
          controller.enqueue(event);
        },
      }),
    );
  },
});

Language codes

All STT plugins and LiveKit Inference use the LanguageCode type for the language parameter. LanguageCode accepts any common language format and normalizes it automatically to BCP-47 . You don't need to look up the specific format each provider expects. Pass any of the following and the framework handles the conversion:

• ISO 639-1: "en", "es", "fr"
• BCP-47 with region: "en-US", "zh-Hans-CN"
• ISO 639-3: "eng", "spa"
• Language names: "english", "spanish"
• Underscored variants: "en_us" (normalized to "en-US")

For example, all of the following are equivalent:

from livekit.agents import LanguageCode

LanguageCode("english")  # → "en"
LanguageCode("eng")      # → "en"
LanguageCode("en")       # → "en"
LanguageCode("en-US")    # → "en-US"
LanguageCode("en_us")    # → "en-US"

import { normalizeLanguage } from '@livekit/agents';

normalizeLanguage("english")  // → "en"
normalizeLanguage("eng")      // → "en"
normalizeLanguage("en")       // → "en"
normalizeLanguage("en-US")    // → "en-US"
normalizeLanguage("en_us")    // → "en-US"

Additional resources

The following resources cover related topics that might be useful for your app.