AssemblyAI STT | LiveKit Documentation

Create a new agent in your browser using this model

Overview

AssemblyAI speech-to-text is available in LiveKit Agents through LiveKit Inference and the AssemblyAI plugin. With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Use the plugin instead if you want to manage your own billing and rate limits. Pricing for LiveKit Inference is available on the pricing page .

LiveKit Inference

Use LiveKit Inference to access AssemblyAI STT without a separate AssemblyAI API key.

Model name	Model ID	Languages
Universal-3 Pro Streaming	assemblyai/u3-rt-pro	enen-USen-GBen-AUen-CAen-INen-NZeses-ESes-MXes-ARes-COes-CLes-PEes-VEes-ECes-GTes-CUes-BOes-DOes-HNes-PYes-SVes-NIes-CRes-PAes-UYes-PRfrfr-FRfr-CAfr-BEfr-CHdede-DEde-ATde-CHitit-ITit-CHptpt-BRpt-PT
Universal-3.5 Pro Streaming	assemblyai/universal-3-5-pro	enen-USen-GBen-AUen-CAen-INen-NZeses-ESes-MXes-ARes-COes-CLes-PEes-VEes-ECes-GTes-CUes-BOes-DOes-HNes-PYes-SVes-NIes-CRes-PAes-UYes-PRfrfr-FRfr-CAfr-BEfr-CHdede-DEde-ATde-CHitit-ITit-CHptpt-BRpt-PTtrnlsvnodafihiviarhejaurzh
Universal-Streaming	assemblyai/universal-streaming	enen-US
Universal-Streaming-Multilingual	assemblyai/universal-streaming-multilingual	multienen-USen-GBen-AUen-CAen-INen-NZeses-ESes-MXes-ARes-COes-CLes-PEes-VEes-ECes-GTes-CUes-BOes-DOes-HNes-PYes-SVes-NIes-CRes-PAes-UYes-PRfrfr-FRfr-CAfr-BEfr-CHdede-DEde-ATde-CHitit-ITit-CHptpt-BRpt-PT

Usage

To use AssemblyAI, use the STT class from the inference module:

from livekit.agents import AgentSession, inference

session = AgentSession(
    stt=inference.STT(
        model="assemblyai/universal-3-5-pro", 
        language="en"
    ),
    # ... llm, tts, vad, turn_handling, etc.
)

import { AgentSession, inference } from '@livekit/agents';

session = new AgentSession({
    stt: new inference.STT({ 
        model: "assemblyai/universal-3-5-pro", 
        language: "en" 
    }),
    // ... llm, tts, vad, turnHandling, etc.
});

Parameters

model

Required

string

The model to use for the STT. See model IDs for available models.

languageLanguageCode

Language code for the transcription. If not set, the provider default applies. Universal-3.5 Pro and Universal-Streaming Multilingual automatically detect the spoken language and code-switch between supported languages.

extra_kwargsdict

Additional parameters to pass to the AssemblyAI streaming API. Supported fields depend on the selected model. See model parameters for supported fields.

In Node.js this parameter is called modelOptions.

Model parameters

Pass the following parameters inside extra_kwargs (Python) or modelOptions (Node.js).

All models:

Parameter	Type	Default	Notes
`keyterms_prompt`	`list[str]`		List of terms to boost recognition accuracy for.
`language_detection`	`bool`	`True`	Whether to include `language_code` and `language_confidence` in turn messages. Universal-Streaming English defaults to `False`.
`inactivity_timeout`	`float`		Duration of inactivity in seconds before the session closes.
`min_turn_silence`	`int`	`100`	Minimum duration of silence in milliseconds before the model checks for end of turn. For Universal-3.5 Pro this triggers the punctuation-based EOT check. Universal-Streaming uses it as the confident-EOT silence floor. Replaces the deprecated `min_end_of_turn_silence_when_confident`.
`max_turn_silence`	`int`		Maximum duration of silence in milliseconds allowed in a turn before end of turn is triggered.
`vad_threshold`	`float`	`0.3`	Confidence threshold for classifying audio frames as silence. Frames below this value are considered silent. Increase in noisy environments. Universal-Streaming defaults to `0.4`. Valid range: `0.0`–`1.0`.
`domain`	`string`		Enables domain-specific recognition. Set to `medical-v1` to use AssemblyAI's Medical Mode . Works with all streaming models. Supported languages: English, Spanish, German, French. Other languages are ignored with a warning.
`speaker_labels`	`bool`	`False`	Set to `True` to enable speaker diarization.
`max_speakers`	`int`		Maximum number of speakers to detect when `speaker_labels` is enabled. If not set, AssemblyAI detects the number of speakers automatically.

Model-specific parameters:

Parameter	Type	Default	Notes
`prompt`	`str`		Natural-language context about the audio (domain, topic, scenario). When not set, a default prompt optimized for turn detection is used. Mutually exclusive with the model's behavioral instructions, which are managed by AssemblyAI.
`mode`	`str`	`balanced`	Accuracy/latency preset: `min_latency` (fastest time-to-text), `balanced` (recommended for voice agents), or `max_accuracy` (highest accuracy, for note-taking and post-call transcription). The model applies its own per-mode tuning. Silence, partials, and VAD knobs you set explicitly take precedence over the mode's defaults.
`agent_context`	`str`		Your agent's most recent spoken reply (TTS text), used to bias transcription of the user's next turn. See Conversation context. Max ~1500 characters.
`previous_context_n_turns`	`int`	`3`	Maximum number of prior conversation entries carried forward as context for each transcription. The default is applied server-side. `0` disables automatic context carryover. See Conversation context.
`voice_focus`	`str`		Noise suppression that isolates the primary voice before audio reaches the model. Set to `near-field` (headsets, handsets, close-talking mics) or `far-field` (conference rooms, laptop/drive-thru mics, distant capture).
`voice_focus_threshold`	`float`		How aggressively background audio is suppressed when `voice_focus` is set (higher is more aggressive). Only takes effect alongside `voice_focus`. Valid range: `0.0`–`1.0`.
`continuous_partials`	`bool`	`True`	Emit a non-final partial transcript approximately every 3 seconds while speech continues, regardless of silence. Useful for long, uninterrupted turns. The first partial still arrives at the early-partial timing controlled by `interruption_delay`. When `speaker_labels` is enabled, the server disables continuous partials by default.
`interruption_delay`	`int`		Milliseconds before the first early partial is emitted. Lower values produce a faster time-to-first-token for barge-in, and higher values produce more confident first partials. The default is mode-dependent and set by the server based on the `mode` preset. Valid range: `0`–`1000`.

Prompt and Keyterms Prompt

You can use prompt and keyterms_prompt together in the same streaming request. When you use keyterms_prompt, your boosted words are appended to the default prompt (or your custom prompt if provided) automatically.

Parameter	Type	Default	Notes
`format_turns`	`bool`	`False`	Whether to return formatted final transcripts.
`end_of_turn_confidence_threshold`	`float`	`0.01`	Confidence threshold for determining the end of a turn. LiveKit Inference lowers this from AssemblyAI's server default of `0.4` to minimize latency.

String descriptors

As a shortcut, you can also pass a model ID string directly to the stt argument in your AgentSession:

from livekit.agents import AgentSession

session = AgentSession(
    stt="assemblyai/universal-3-5-pro:en",
    # ... llm, tts, vad, turn_handling, etc.
)

import { AgentSession } from '@livekit/agents';

session = new AgentSession({
    stt: "assemblyai/universal-3-5-pro:en",
    // ... llm, tts, vad, turnHandling, etc.
});

Turn detection

Universal-3.5 Pro uses punctuation-based turn detection. It checks for terminal punctuation (. ? !) after periods of silence rather than using a confidence score. To use this for turn detection, set turn_detection="stt" in the turn handling options.

Default parameter differences: The LiveKit plugin defaults to min_turn_silence=100 and max_turn_silence=100. The AssemblyAI API defaults are min_turn_silence=100 and max_turn_silence=1000. When using turn_detection="stt", explicitly set max_turn_silence=1000 to restore AssemblyAI's intended behavior.

Endpointing delay is additive in STT mode: LiveKit's default min_delay (0.5 seconds) in the turn handling endpointing options is applied on top of AssemblyAI's own endpointing. Set endpointing.min_delay to 0 in the turn handling options to avoid extra latency. AssemblyAI's min_turn_silence and max_turn_silence already control the timing.

VAD threshold alignment: Universal-3.5 Pro defaults to a vad_threshold of 0.3. Set LiveKit's Silero activation_threshold to 0.3 as well to ensure consistent barge-in behavior.

Tuning guidance: Experiment with min_turn_silence and max_turn_silence. Settings can vary depending on your use case. Increase min_turn_silence if brief pauses cause the speculative EOT check to fire too early, ending turns on terminal punctuation before the user has finished speaking. Increase max_turn_silence if the forced turn end is cutting off users mid-thought.

For a detailed guide on configuring Universal-3.5 Pro with LiveKit, including entity splitting tradeoffs, VAD threshold alignment, and prompt engineering, see the AssemblyAI LiveKit guide .

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},
    ),
    stt=inference.STT(
        model="assemblyai/universal-3-5-pro",
        extra_kwargs={
            "min_turn_silence": 100,
            "max_turn_silence": 1000,
            "vad_threshold": 0.3,
        }
    ),
    vad=silero.VAD.load(activation_threshold=0.3),
    # ... llm, tts, etc.
)

import { AgentSession, inference } from '@livekit/agents';
import * as silero from '@livekit/agents-plugin-silero';

const session = new AgentSession({
    turnHandling: {
        turnDetection: 'stt',
        endpointing: { minDelay: 0 },
    },
    stt: new inference.STT({
        model: 'assemblyai/universal-3-5-pro',
        modelOptions: {
            min_turn_silence: 100,
            max_turn_silence: 1000,
            vad_threshold: 0.3,
        },
    }),
    vad: await silero.VAD.load({ activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

AssemblyAI includes a custom phrase endpointing model that uses both audio and linguistic information to detect turn boundaries. To use this model for turn detection, set turn_detection="stt" in the turn handling options. You should also provide a VAD plugin for responsive interruption handling.

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
    ),
    stt=inference.STT(
        model="assemblyai/universal-streaming",
        language="en"
    ),
    vad=silero.VAD.load(),  # Recommended for responsive interruption handling
    # ... llm, tts, etc.
)

import { AgentSession, inference } from '@livekit/agents';
import * as silero from '@livekit/agents-plugin-silero';

const session = new AgentSession({
    turnHandling: {
        turnDetection: 'stt',
    },
    stt: new inference.STT({
        model: 'assemblyai/universal-streaming',
        language: 'en',
    }),
    vad: await silero.VAD.load(), // Recommended for responsive interruption handling
    // ... llm, tts, etc.
});

Plugin

LiveKit's plugin support for AssemblyAI lets you connect directly to AssemblyAI's API with your own API key.

Available inPython

Node.js

Installation

Install the plugin:

uv add "livekit-agents[assemblyai]~=1.6"

pnpm add @livekit/agents-plugin-assemblyai@1.x

Authentication

The AssemblyAI plugin requires an AssemblyAI API key .

Set ASSEMBLYAI_API_KEY in your .env file.

Usage

Use AssemblyAI STT in an AgentSession or as a standalone transcription service. For example, you can use this STT in the Voice AI quickstart.

from livekit.plugins import assemblyai, silero

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(activation_threshold=0.3),
    # ... llm, tts, etc.
)

import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';
import * as silero from '@livekit/agents-plugin-silero';

const session = new voice.AgentSession({
    stt: new assemblyai.STT({
        speechModel: 'universal-3-5-pro',
        minTurnSilence: 100,
        maxTurnSilence: 1000,
        vadThreshold: 0.3,
    }),
    vad: await silero.VAD.load({ activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

Parameters

This section describes some of the available parameters. See the plugin reference for a complete list of all available parameters.

Shared parameters

These parameters apply to all AssemblyAI streaming models.

modelstringDefault: universal-3-5-pro

STT model to use. Accepted options are universal-3-5-pro, u3-rt-pro, u3-rt-pro-beta-1, universal-streaming-english, and universal-streaming-multilingual.

keyterms_promptlist[str]

List of terms to boost recognition for.

vad_thresholdfloatDefault: 0.3

AssemblyAI's internal Silero VAD onset threshold. Universal-Streaming defaults to 0.4. For best results, align this with LiveKit's Silero activation_threshold.

language_detectionboolDefault: true

Whether to include language_code and language_confidence in turn messages. Universal-Streaming English defaults to false.

min_turn_silenceintDefault: 100

The minimum duration of silence (in milliseconds) before the model checks for end of turn. The LiveKit plugin defaults this to 100 for all streaming models. Replaces the deprecated min_end_of_turn_silence_when_confident. See the following model-specific sections for how each model uses this parameter.

max_turn_silenceint

The maximum duration of silence (in milliseconds) allowed in a turn before end of turn is triggered. See the following model-specific sections for defaults.

speaker_labelsbool

Enable speaker diarization. When set to True, each transcript event includes a speaker_id identifying the speaker ("A", "B", etc.). Short utterances under ~1 second return speaker_id=None. Use with MultiSpeakerAdapter to detect the primary speaker or format transcripts by speaker.

max_speakersint

Maximum number of speakers to detect. If not set, AssemblyAI detects the number of speakers automatically.

domainstring

Enables domain-specific recognition. Set to medical-v1 to use AssemblyAI's Medical Mode for improved accuracy on medical terminology such as medication names, procedures, conditions, and dosages. Works with all streaming models.

Model-specific parameters

These parameters are supported only by universal-3-5-pro. Passing any of them with a Universal-Streaming model (for example, universal-streaming-english) raises a ValueError.

min_turn_silenceintDefault: 100

Milliseconds of silence before a speculative end-of-turn check. When the check fires, the model looks for terminal punctuation (. ? !) to decide whether the turn has ended. If no terminal punctuation is found, a partial is emitted and the turn continues.

This parameter replaces the now deprecated min_end_of_turn_silence_when_confident.

max_turn_silenceintDefault: 100

Maximum milliseconds of silence before the turn is forced to end, regardless of punctuation. The LiveKit plugin defaults to 100. When using turn_detection="stt", set this to 1000 to match AssemblyAI's API default.

promptstring

Natural-language context about the audio (domain, topic, scenario, conversation details) used to improve transcription accuracy. When not provided, a default prompt optimized for turn detection is used automatically. The transcription behavior (verbatim handling, punctuation, formatting) is built in and managed by AssemblyAI. prompt provides context, not behavioral or formatting instructions.

Note: Start without a prompt to establish baseline performance, then add context only for domain vocabulary the model gets wrong.

modestringDefault: balanced

Accuracy/latency preset forwarded to the model. Accepted values:

min_latency: fastest time-to-text.
balanced: recommended for voice agents (server default).
max_accuracy: highest accuracy, best for note-taking and post-call transcription.

The model applies its own per-mode silence tuning. To let that tuning take effect, the plugin suppresses its default 100 ms min_turn_silence and max_turn_silence windows when a mode is set. Values you pass explicitly for min_turn_silence or max_turn_silence still take precedence over the mode's defaults. Set at construction (connect) time only.

agent_contextstring

Free-text describing your agent's most recent spoken reply, used to bias transcription of the user's next turn. Can be set at construction or updated per turn via update_options(agent_context=...). Max ~1500 characters. See Conversation context.

previous_context_n_turnsintDefault: 3

Maximum number of prior conversation entries (finalized user transcripts plus any agent_context values) carried forward as context for each transcription. The default is applied server-side. Set to 0 to disable automatic context carryover entirely. Valid range: 0–100. Set at construction (connect) time only. It cannot be changed via update_options(). See Conversation context.

agent_context_carryoverboolDefault: False

Only Available inPython

When enabled, your AgentSession automatically pushes each assistant reply into agent_context after every agent turn, biasing transcription of the user's next reply toward what the agent just said. Prior user turns are carried forward by the model automatically regardless of this flag — this flag supplies the assistant side of the conversation. See Conversation context.

Only supported with Universal-3.5 Pro. Enabling it with any other model logs a warning and the flag is ignored.

voice_focusstring

Voice Focus isolates the primary voice and suppresses background noise (chatter, keyboard clicks, fan hum, room echo) before audio reaches the model. Set to near-field for headsets, handsets, and close-talking microphones, or far-field for conference rooms, laptop mics, and other distant-mic setups. Set at construction (connect) time only. See AssemblyAI's Voice Focus docs .

voice_focus_thresholdfloat

Controls how aggressively background audio is suppressed (higher is more aggressive). Only takes effect alongside voice_focus. Valid range: 0.0–1.0. Set at construction (connect) time only.

Using prompts

Test without a prompt first to establish a baseline. You can use prompt and keyterms_prompt together, and boosted terms are appended to the prompt automatically. See AssemblyAI's Prompting and Keyterms guide .

continuous_partialsboolDefault: True

Only Available inPython

When True, the model emits additional partial transcripts at a steady ~3 second cadence during long turns, on top of the baseline partials emitted at the first-partial point (interruption_delay) and at each min_turn_silence silence period. Useful for long, uninterrupted turns where silence-based partials don't fire often enough for downstream consumers. The plugin sets this explicitly when unset, so continuous partials stay on even with speaker_labels enabled, which otherwise disables them by default. Can be updated mid-session via update_options().

interruption_delayint

Only Available inPython

How soon (in milliseconds) the first early partial is emitted. Lower values produce a faster time-to-first-token for barge-in, and higher values produce more confident first partials. The default is mode-dependent and set by the server based on the mode preset. Can be updated mid-session via update_options().

Valid range: 0–1000. The server adds a fixed 256 ms on top of this value, so 0 yields an effective delay of 256 ms and 500 yields 756 ms.

end_of_turn_confidence_thresholdfloatDefault: 0.4

The confidence threshold to use when determining if the end of a turn has been reached. Not applicable to Universal-3.5 Pro.

min_end_of_turn_silence_when_confidentint

The minimum duration of silence (in milliseconds) required to detect end of turn when confident.

Deprecated: This parameter has been renamed to min_turn_silence. Use min_turn_silence instead. Note that the LiveKit plugin defaults min_turn_silence to 100 for all streaming models (not just Universal-3.5 Pro), so the effective default is 100 ms.

max_turn_silenceintDefault: 1280

The maximum duration of silence (in milliseconds) allowed in a turn before end of turn is triggered.

format_turnsboolDefault: false

Whether to return formatted final transcripts. Not applicable to Universal-3.5 Pro (always returns formatted transcripts).

Turn detection

VAD threshold alignment: Universal-3.5 Pro defaults to a vad_threshold of 0.3. Set LiveKit's Silero activation_threshold to 0.3 as well to ensure consistent barge-in behavior.

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},
    ),
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=silero.VAD.load(activation_threshold=0.3),
    # ... llm, tts, etc.
)

import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';
import * as silero from '@livekit/agents-plugin-silero';

const session = new voice.AgentSession({
    turnHandling: {
        turnDetection: 'stt',
        endpointing: { minDelay: 0 },
    },
    stt: new assemblyai.STT({
        speechModel: 'universal-3-5-pro',
        minTurnSilence: 100,
        maxTurnSilence: 1000,
        vadThreshold: 0.3,
    }),
    vad: await silero.VAD.load({ activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

You can also use LiveKit's audio turn detector instead of turn_detection="stt". It operates directly on audio, so it doesn't depend on transcript timing. When you do rely on transcript-based detection (turn_detection="stt"), the plugin defaults (min_turn_silence=100, max_turn_silence=100) are tuned to finalize transcripts as fast as possible. Raising these values (for example, 200–300 ms) can reduce over-segmentation.

For a detailed guide on configuring Universal-3.5 Pro with LiveKit, including entity splitting tradeoffs, VAD threshold alignment, and prompt engineering, see the AssemblyAI LiveKit guide .

AssemblyAI Universal-Streaming includes a custom phrase endpointing model that uses both audio and linguistic information to detect turn boundaries. To use this model for turn detection, set turn_detection="stt" in the turn handling options. You should also provide a VAD plugin for responsive interruption handling.

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
    ),
    stt=assemblyai.STT(
      model="universal-streaming-english",
      end_of_turn_confidence_threshold=0.4,
      min_turn_silence=400,
      max_turn_silence=1280,
    ),
    vad=silero.VAD.load(),  # Recommended for responsive interruption handling
    # ... llm, tts, etc.
)

import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';
import * as silero from '@livekit/agents-plugin-silero';

const session = new voice.AgentSession({
    turnHandling: {
        turnDetection: 'stt',
    },
    stt: new assemblyai.STT({
        speechModel: 'universal-streaming-english',
        endOfTurnConfidenceThreshold: 0.4,
        minTurnSilence: 400,
        maxTurnSilence: 1280,
    }),
    vad: await silero.VAD.load(), // Recommended for responsive interruption handling
    // ... llm, tts, etc.
});

Session information

When a WebSocket session starts, AssemblyAI sends a Begin event that includes a session ID and expiry timestamp. The plugin exposes the following information on the SpeechStream object:

Field	Description
`session_id` (Python) / `sessionId` (Node.js)	UUID string identifying the transcription session. The session ID is also logged automatically at INFO level. Share it with AssemblyAI support when troubleshooting transcription issues.
`expires_at` (Python) / `expiresAt` (Node.js)	Unix timestamp indicating when the session expires.

stream = stt.stream()
async for event in stream:
    # session_id is set before any speech events arrive
    print(stream.session_id)   # e.g. "676d673c-83fc-4d8a-bd95-bfe23b1c5a50"
    print(stream.expires_at)   # e.g. 1773775624

const stream = stt.stream();
for await (const event of stream) {
    // sessionId is set before any speech events arrive
    console.log(stream.sessionId);  // e.g. "676d673c-83fc-4d8a-bd95-bfe23b1c5a50"
    console.log(stream.expiresAt);  // e.g. 1773775624
}

These properties are None (Python) or null (Node.js) until the Begin event is received from AssemblyAI, which happens shortly after the stream starts.

The session ID is also automatically logged:

AssemblyAI session started id=676d673c-83fc-4d8a-bd95-bfe23b1c5a50 expires_at=1773775624

Conversation context

Universal-3.5 Pro includes prior conversation turns as context for each transcription. Speech models rely on surrounding words to choose between similar-sounding interpretations of audio, and a short reply like "yes" or "7pm" has no surrounding words of its own. Prior turns fill that gap. If the agent just asked what time works, an ambiguous short reply is more likely to be transcribed as 7pm. Context is per-session, and closing the stream clears it.

Automatic context carryover

By default, the model carries prior finalized turns forward as context for the next turn. This is on automatically and requires no configuration. The number of prior entries carried forward is controlled by previous_context_n_turns (server default 3). Set it to 0 at construction time to disable carryover entirely.

Agent context

When your agent speaks, pass the text it said (the TTS text) via agent_context so the model knows the question the user is answering. For example, after the agent asks "What's your email address?", agent_context helps the model produce "user@assemblyai.com" instead of "user at assemblyai dot com".

Set agent_context at construction to seed an opening greeting, and update it after each agent reply via update_options(). The most recent value wins (most-recent-turn semantics). Maximum ~1500 characters.

In the Python plugin, set agent_context_carryover=True to have your AgentSession push each assistant reply into agent_context automatically after every agent turn, instead of calling update_options() yourself.

from livekit.agents import inference

stt = inference.STT(
    model="assemblyai/universal-3-5-pro",
    extra_kwargs={
        "agent_context": "Hi! Thanks for calling Acme. How can I help today?",
        "previous_context_n_turns": 3,
    },
)

import { inference } from '@livekit/agents';

const stt = new inference.STT({
    model: 'assemblyai/universal-3-5-pro',
    modelOptions: {
        agent_context: "Hi! Thanks for calling Acme. How can I help today?",
        previous_context_n_turns: 3,
    },
});

from livekit.plugins import assemblyai

stt = assemblyai.STT(
    model="universal-3-5-pro",
    agent_context="Hi! Thanks for calling Acme. How can I help today?",
    previous_context_n_turns=3,
)

# After each agent reply, update the context for the next user turn.
stt.update_options(agent_context="Sure, what's the email on the account?")

import * as assemblyai from '@livekit/agents-plugin-assemblyai';

const stt = new assemblyai.STT({
    speechModel: 'universal-3-5-pro',
    agentContext: "Hi! Thanks for calling Acme. How can I help today?",
    previousContextNTurns: 3,
});

// After each agent reply, update the context for the next user turn.
stt.updateOptions({ agentContext: "Sure, what's the email on the account?" });

For more detail on how Universal-3.5 Pro uses conversation context, see AssemblyAI's LiveKit agent context example .

Speaker diarization

Enable speaker diarization so the STT assigns a speaker identifier to each word or segment. When enabled, transcript events include a speaker_id, and the STT reports capabilities.diarization = True.

With diarization enabled, you can wrap the AssemblyAI STT with MultiSpeakerAdapter for primary speaker detection and transcript formatting.

Enable speaker diarization in the STT constructor:

stt = inference.STT(
    model="assemblyai/universal-3-5-pro",
    extra_kwargs={
        "speaker_labels": True,
    },
)

const stt = new inference.STT({
    model: 'assemblyai/universal-3-5-pro',
    modelOptions: {
        speaker_labels: true,
    },
});

stt = assemblyai.STT(
    model="universal-3-5-pro",
    speaker_labels=True,
)

const stt = new assemblyai.STT({
    speechModel: 'universal-3-5-pro',
    speakerLabels: true,
});

Caution

The Node.js plugin sends speakerLabels to AssemblyAI, but speaker labels are not yet surfaced on transcript events. To get speaker IDs in Node.js, use LiveKit Inference.

Speaker labels are assigned alphabetically ("A", "B", etc.) per session. Short utterances under ~1 second return speaker_id=None.

Additional resources

The following resources provide more information about using AssemblyAI with LiveKit Agents.

Python

Reference GitHub PyPI

Node.js

Reference GitHub NPM

AssemblyAI docs

AssemblyAI's full docs for streaming speech-to-text.

Voice AI quickstart

Get started with LiveKit Agents and AssemblyAI.

AssemblyAI LiveKit guide

Guide to using AssemblyAI Universal-3.5 Pro STT with LiveKit.

Overview

LiveKit Inference

Usage

Parameters

Model parameters

String descriptors

Turn detection

Plugin

Installation

Authentication

Usage

Parameters

Shared parameters

Model-specific parameters

Turn detection

Session information

Conversation context

Automatic context carryover

Agent context

Speaker diarization

Additional resources

Python

Node.js

AssemblyAI docs

Voice AI quickstart

AssemblyAI LiveKit guide

Ask LiveKit