LiveKit docs › Build Agents › Models › STT › AssemblyAI

---

# AssemblyAI STT

> How to use AssemblyAI STT with LiveKit Agents.

- **[Use in Agent Builder](https://cloud.livekit.io/projects/p_/agents/builder/new?stt=assemblyai%2Funiversal-3-6-pro)**: Create a new agent in your browser using assemblyai/universal-3-6-pro

## Overview

AssemblyAI speech-to-text is available in LiveKit Agents through [LiveKit Inference](https://docs.livekit.io/agents/models/inference.md) and the [AssemblyAI plugin](#plugin). With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Use the plugin instead if you want to manage your own billing and rate limits. Pricing for LiveKit Inference is available on the [pricing page](https://livekit.com/pricing/inference#stt).

## LiveKit Inference

Use [LiveKit Inference](https://docs.livekit.io/agents/models/inference.md) to access AssemblyAI STT without a separate AssemblyAI API key.

A checkmark in the **EU endpoint** column in the following tables means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see [Data residency](https://docs.livekit.io/agents/models/inference.md#data-residency).

| Model name | Model ID | Languages | EU endpoint |
| -------- | -------- | --------- | ------ |
| Universal-3.5 Pro | `assemblyai/universal-3-5-pro` | `en`, `en-US`, `en-GB`, `en-AU`, `en-CA`, `en-IN`, `en-NZ`, `es`, `es-ES`, `es-MX`, `es-AR`, `es-CO`, `es-CL`, `es-PE`, `es-VE`, `es-EC`, `es-GT`, `es-CU`, `es-BO`, `es-DO`, `es-HN`, `es-PY`, `es-SV`, `es-NI`, `es-CR`, `es-PA`, `es-UY`, `es-PR`, `fr`, `fr-FR`, `fr-CA`, `fr-BE`, `fr-CH`, `de`, `de-DE`, `de-AT`, `de-CH`, `it`, `it-IT`, `it-CH`, `pt`, `pt-BR`, `pt-PT`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `ur`, `zh` | ✓ |
| Universal-3.6 Pro Streaming | `assemblyai/universal-3-6-pro` | `en`, `en-US`, `en-GB`, `en-AU`, `en-CA`, `en-IN`, `en-NZ`, `es`, `es-ES`, `es-MX`, `es-AR`, `es-CO`, `es-CL`, `es-PE`, `es-VE`, `es-EC`, `es-GT`, `es-CU`, `es-BO`, `es-DO`, `es-HN`, `es-PY`, `es-SV`, `es-NI`, `es-CR`, `es-PA`, `es-UY`, `es-PR`, `fr`, `fr-FR`, `fr-CA`, `fr-BE`, `fr-CH`, `de`, `de-DE`, `de-AT`, `de-CH`, `it`, `it-IT`, `it-CH`, `pt`, `pt-BR`, `pt-PT`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `ur`, `zh`, `ru`, `ko`, `ca`, `gl`, `ro`, `et`, `fa`, `yue`, `af`, `mr`, `zu`, `xh`, `nn` | ✓ |
| Universal-Streaming | `assemblyai/universal-streaming` | `en`, `en-US` | ✓ |
| Universal-Streaming-Multilingual | `assemblyai/universal-streaming-multilingual` | `multi`, `en`, `en-US`, `en-GB`, `en-AU`, `en-CA`, `en-IN`, `en-NZ`, `es`, `es-ES`, `es-MX`, `es-AR`, `es-CO`, `es-CL`, `es-PE`, `es-VE`, `es-EC`, `es-GT`, `es-CU`, `es-BO`, `es-DO`, `es-HN`, `es-PY`, `es-SV`, `es-NI`, `es-CR`, `es-PA`, `es-UY`, `es-PR`, `fr`, `fr-FR`, `fr-CA`, `fr-BE`, `fr-CH`, `de`, `de-DE`, `de-AT`, `de-CH`, `it`, `it-IT`, `it-CH`, `pt`, `pt-BR`, `pt-PT` | ✓ |

### Usage

To use AssemblyAI, use the `STT` class from the `inference` module:

**Python**:

```python
from livekit.agents import AgentSession, inference

session = AgentSession(
    stt=inference.STT(
        model="assemblyai/universal-3-6-pro", 
        language="en"
    ),
    # ... llm, tts, vad, turn_handling, etc.
)

```

---

**Node.js**:

```typescript
import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    stt: new inference.STT({ 
        model: "assemblyai/universal-3-6-pro", 
        language: "en" 
    }),
    // ... llm, tts, vad, turnHandling, etc.
});

```

### Parameters

- **`model`** _(string)_: The model to use for the STT. See [model IDs](#inference) for available models.

- **`language`** _(LanguageCode)_ (optional): [Language code](https://docs.livekit.io/agents/models/stt.md#language-codes) for the transcription. If not set, the provider default applies. Universal-3.6 Pro, Universal-3.5 Pro, and Universal-Streaming Multilingual automatically detect the spoken language and code-switch between supported languages.

- **`extra_kwargs`** _(dict)_ (optional): Additional parameters to pass to the AssemblyAI streaming API. Supported fields depend on the selected model. See [model parameters](#model-parameters) for supported fields.

In Node.js this parameter is called `modelOptions`.

#### Model parameters

Pass the following parameters inside `extra_kwargs` (Python) or `modelOptions` (Node.js).

**All models:**

| Parameter | Type | Default | Notes |
| keyterms_prompt | `list[str]` |  | List of terms to boost recognition accuracy for. |
| language_detection | `bool` | `True` | Whether to include `language_code` and `language_confidence` in turn messages. Universal-Streaming English defaults to `False`. |
| inactivity_timeout | `float` |  | Duration of inactivity in seconds before the session closes. |
| min_turn_silence | `int` | `100` | Minimum duration of silence in milliseconds before the model checks for end of turn. For Universal-3.6 Pro this triggers the punctuation-based EOT check. Universal-Streaming uses it as the confident-EOT silence floor. Replaces the deprecated `min_end_of_turn_silence_when_confident`. |
| max_turn_silence | `int` |  | Maximum duration of silence in milliseconds allowed in a turn before end of turn is triggered. |
| vad_threshold | `float` | `0.3` | Confidence threshold for classifying audio frames as silence. Frames below this value are considered silent. Increase in noisy environments. Universal-Streaming defaults to `0.4`.

Valid range: `0.0`–`1.0`. |
| domain | `string` |  | Enables domain-specific recognition. Set to `medical-v1` to use AssemblyAI's [Medical Mode](https://www.assemblyai.com/docs/streaming/medical-mode). Works with all streaming models. Supported languages: English, Spanish, German, French. Other languages are ignored with a warning. |
| speaker_labels | `bool` | `False` | Set to `True` to enable [speaker diarization](#speaker-diarization). |
| max_speakers | `int` |  | Maximum number of speakers to detect when `speaker_labels` is enabled. If not set, AssemblyAI detects the number of speakers automatically. |

**Model-specific parameters:**

**Universal-3.6 Pro**:

| Parameter | Type | Default | Notes |
| prompt | `str` |  | Natural-language context about the audio (domain, topic, scenario). When not set, a default prompt optimized for turn detection is used. Mutually exclusive with the model's behavioral instructions, which are managed by AssemblyAI. |
| mode | `str` | `balanced` | Accuracy/latency preset: `min_latency` (fastest time-to-text), `balanced` (recommended for voice agents), or `max_accuracy` (highest accuracy, for note-taking and post-call transcription). The model applies its own per-mode tuning. Silence, partials, and VAD knobs you set explicitly take precedence over the mode's defaults. |
| agent_context | `str` |  | Your agent's most recent spoken reply (TTS text), used to bias transcription of the user's next turn. See [Conversation context](#conversation-context). Max 1750 characters. |
| previous_context_n_turns | `int` | `3` | Maximum number of prior conversation entries carried forward as context for each transcription. The default is applied server-side. `0` disables automatic context carryover. See [Conversation context](#conversation-context). |
| voice_focus | `str` |  | Noise suppression that isolates the primary voice before audio reaches the model. Set to `near-field` (headsets, handsets, close-talking mics) or `far-field` (conference rooms, laptop/drive-thru mics, distant capture). |
| voice_focus_threshold | `float` |  | How aggressively background audio is suppressed when `voice_focus` is set (higher is more aggressive). Only takes effect alongside `voice_focus`. Valid range: `0.0`–`1.0`. |
| continuous_partials | `bool` | `True` | Emit a non-final partial transcript approximately every 3 seconds while speech continues, regardless of silence. Useful for long, uninterrupted turns. The first partial still arrives at the early-partial timing controlled by `interruption_delay`. When `speaker_labels` is enabled, the server disables continuous partials by default. |
| interruption_delay | `int` |  | Milliseconds before the first early partial is emitted. Lower values produce a faster time-to-first-token for barge-in, and higher values produce more confident first partials. The default is mode-dependent and set by the server based on the `mode` preset.

Valid range: `0`–`1000`. |

> ℹ️ **Prompt and Keyterms Prompt**
> 
> You can use `prompt` and `keyterms_prompt` together in the same streaming request. When you use `keyterms_prompt`, your boosted words are appended to the default prompt (or your custom `prompt` if provided) automatically.

---

**Universal-Streaming**:

| Parameter | Type | Default | Notes |
| format_turns | `bool` | `False` | Whether to return formatted final transcripts. |
| end_of_turn_confidence_threshold | `float` | `0.01` | Confidence threshold for determining the end of a turn. LiveKit Inference lowers this from AssemblyAI's server default of `0.4` to minimize latency. |

### String descriptors

As a shortcut, you can also pass a [model ID](#inference) string directly to the `stt` argument in your `AgentSession`:

**Python**:

```python
from livekit.agents import AgentSession

session = AgentSession(
    stt="assemblyai/universal-3-6-pro:en",
    # ... llm, tts, vad, turn_handling, etc.
)

```

---

**Node.js**:

```typescript
import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
    stt: "assemblyai/universal-3-6-pro:en",
    // ... llm, tts, vad, turnHandling, etc.
});

```

### Turn detection

**Universal-3.6 Pro**:

Universal-3.6 Pro uses **punctuation-based turn detection**. It checks for terminal punctuation (`.` `?` `!`) after periods of silence rather than using a confidence score. To use this for [turn detection](https://docs.livekit.io/agents/logic/turns.md), set `turn_detection="stt"` in the turn handling options.

**Default parameter differences:** The LiveKit plugin defaults to `min_turn_silence=100` and `max_turn_silence=100`. The AssemblyAI API defaults are `min_turn_silence=100` and `max_turn_silence=1000`. When using `turn_detection="stt"`, explicitly set `max_turn_silence=1000` to restore AssemblyAI's intended behavior.

**Endpointing delay is additive in STT mode:** LiveKit's default `min_delay` (0.5 seconds) in the turn handling endpointing options is applied on top of AssemblyAI's own endpointing. Set `endpointing.min_delay` to `0` in the turn handling options to avoid extra latency. AssemblyAI's `min_turn_silence` and `max_turn_silence` already control the timing.

**VAD threshold alignment:** Universal-3.6 Pro defaults to a `vad_threshold` of `0.3`. Set LiveKit's Silero `activation_threshold` to `0.3` as well to ensure consistent barge-in behavior.

**Tuning guidance:** Experiment with `min_turn_silence` and `max_turn_silence`. Settings can vary depending on your use case. Increase `min_turn_silence` if brief pauses cause the speculative EOT check to fire too early, ending turns on terminal punctuation before the user has finished speaking. Increase `max_turn_silence` if the forced turn end is cutting off users mid-thought.

For a detailed guide on configuring Universal-3.6 Pro with LiveKit, including entity splitting tradeoffs, VAD threshold alignment, and prompt engineering, see the [AssemblyAI LiveKit guide](https://www.assemblyai.com/docs/voice-agents/livekit-u3-rt-pro).

**Python**:

```python
session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},
    ),
    stt=inference.STT(
        model="assemblyai/universal-3-6-pro",
        extra_kwargs={
            "min_turn_silence": 100,
            "max_turn_silence": 1000,
            "vad_threshold": 0.3,
        }
    ),
    vad=inference.VAD(model="silero", activation_threshold=0.3),
    # ... llm, tts, etc.
)

```

---

**Node.js**:

```typescript
import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    turnHandling: {
        turnDetection: 'stt',
        endpointing: { minDelay: 0 },
    },
    stt: new inference.STT({
        model: 'assemblyai/universal-3-6-pro',
        modelOptions: {
            min_turn_silence: 100,
            max_turn_silence: 1000,
            vad_threshold: 0.3,
        },
    }),
    vad: new inference.VAD({ model: 'silero', activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

```

---

**Universal-Streaming**:

AssemblyAI includes a custom phrase endpointing model that uses both audio and linguistic information to detect turn boundaries. To use this model for [turn detection](https://docs.livekit.io/agents/logic/turns.md), set `turn_detection="stt"` in the turn handling options. The session's bundled VAD continues to handle interruption detection, so the agent stays responsive while the STT determines turn boundaries.

**Python**:

```python
session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
    ),
    stt=inference.STT(
        model="assemblyai/universal-streaming",
        language="en"
    ),
    # ... llm, tts, etc.
)

```

---

**Node.js**:

```typescript
import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    turnHandling: {
        turnDetection: 'stt',
    },
    stt: new inference.STT({
        model: 'assemblyai/universal-streaming',
        language: 'en',
    }),
    // ... llm, tts, etc.
});

```

## Plugin

LiveKit's plugin support for AssemblyAI lets you connect directly to AssemblyAI's API with your own API key.

Available in:
- [x] Node.js
- [x] Python

### Installation

Install the plugin:

**Python**:

```shell
uv add "livekit-agents[assemblyai]~=1.8"

```

---

**Node.js**:

```shell
pnpm add @livekit/agents-plugin-assemblyai@1.x

```

### Authentication

The AssemblyAI plugin requires an [AssemblyAI API key](https://www.assemblyai.com/docs/api-reference/overview#authentication).

Set `ASSEMBLYAI_API_KEY` in your `.env` file.

### Usage

Use AssemblyAI STT in an `AgentSession` or as a standalone transcription service. For example, you can use this STT in the [Voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai.md).

**Python**:

```python
from livekit.agents import inference
from livekit.plugins import assemblyai

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-6-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=inference.VAD(model="silero", activation_threshold=0.3),
    # ... llm, tts, etc.
)

```

---

**Node.js**:

```typescript
import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';
import * as silero from '@livekit/agents-plugin-silero';

const session = new voice.AgentSession({
    stt: new assemblyai.STT({
        speechModel: 'universal-3-6-pro',
        minTurnSilence: 100,
        maxTurnSilence: 1000,
        vadThreshold: 0.3,
    }),
    vad: await silero.VAD.load({ activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

```

### Parameters

This section describes some of the available parameters. See the [plugin reference](https://docs.livekit.io/reference/python/livekit/plugins/assemblyai/index.html.md#livekit.plugins.assemblyai.STT) for a complete list of all available parameters.

#### Shared parameters

These parameters apply to all AssemblyAI streaming models.

- **`model`** _(string)_ (optional) - Default: `universal-3-6-pro`: STT model to use. Accepted options are `universal-3-6-pro`, `universal-3-5-pro`, `u3-rt-pro`, `u3-rt-pro-beta-1`, `universal-streaming-english`, and `universal-streaming-multilingual`. The Node.js plugin defaults to `universal-3-5-pro`.

- **`keyterms_prompt`** _(list[str])_ (optional): List of terms to boost recognition for.

- **`vad_threshold`** _(float)_ (optional) - Default: `0.3`: AssemblyAI's internal Silero VAD onset threshold. Universal-Streaming defaults to `0.4`. For best results, align this with LiveKit's Silero `activation_threshold`.

- **`language_detection`** _(bool)_ (optional) - Default: `true`: Whether to include `language_code` and `language_confidence` in turn messages. Universal-Streaming English defaults to `false`.

- **`min_turn_silence`** _(int)_ (optional) - Default: `100`: The minimum duration of silence (in milliseconds) before the model checks for end of turn. The LiveKit plugin defaults this to `100` for **all** streaming models. Replaces the deprecated `min_end_of_turn_silence_when_confident`. See the following model-specific sections for how each model uses this parameter.

- **`max_turn_silence`** _(int)_ (optional): The maximum duration of silence (in milliseconds) allowed in a turn before end of turn is triggered. See the following model-specific sections for defaults.

- **`speaker_labels`** _(bool)_ (optional): Enable speaker diarization. When set to `True`, each transcript event includes a `speaker_id` identifying the speaker (`"A"`, `"B"`, etc.). Short utterances under ~1 second return `speaker_id=None`. Use with [`MultiSpeakerAdapter`](https://docs.livekit.io/agents/models/stt.md#speaker-diarization) to detect the primary speaker or format transcripts by speaker.

- **`max_speakers`** _(int)_ (optional): Maximum number of speakers to detect. If not set, AssemblyAI detects the number of speakers automatically.

- **`domain`** _(string)_ (optional): Enables domain-specific recognition. Set to `medical-v1` to use AssemblyAI's [Medical Mode](https://www.assemblyai.com/docs/streaming/medical-mode) for improved accuracy on medical terminology such as medication names, procedures, conditions, and dosages. Works with all streaming models.

#### Model-specific parameters

**Universal-3.6 Pro**:

These parameters are supported only by the Universal-3 Pro models (`universal-3-6-pro`, `universal-3-5-pro`, and `u3-rt-pro`). Passing any of them with a Universal-Streaming model (for example, `universal-streaming-english`) raises a `ValueError`.

- **`min_turn_silence`** _(int)_ (optional) - Default: `100`: Milliseconds of silence before a speculative end-of-turn check. When the check fires, the model looks for terminal punctuation (`.` `?` `!`) to decide whether the turn has ended. If no terminal punctuation is found, a partial is emitted and the turn continues.

This parameter replaces the now deprecated `min_end_of_turn_silence_when_confident`.

- **`max_turn_silence`** _(int)_ (optional) - Default: `100`: Maximum milliseconds of silence before the turn is forced to end, regardless of punctuation. The LiveKit plugin defaults to `100`. When using `turn_detection="stt"`, set this to `1000` to match AssemblyAI's API default.

- **`prompt`** _(string)_ (optional): Natural-language context about the audio (domain, topic, scenario, conversation details) used to improve transcription accuracy. When not provided, a default prompt optimized for turn detection is used automatically. The transcription behavior (verbatim handling, punctuation, formatting) is built in and managed by AssemblyAI. `prompt` provides context, not behavioral or formatting instructions.

**Note:** Start without a prompt to establish baseline performance, then add context only for domain vocabulary the model gets wrong.

- **`mode`** _(string)_ (optional) - Default: `balanced`: Accuracy/latency preset forwarded to the model. Accepted values:

- `min_latency`: fastest time-to-text.
- `balanced`: recommended for voice agents (server default).
- `max_accuracy`: highest accuracy, best for note-taking and post-call transcription.
The model applies its own per-mode silence tuning. To let that tuning take effect, the plugin suppresses its default `100` ms `min_turn_silence` and `max_turn_silence` windows when a mode is set. Values you pass explicitly for `min_turn_silence` or `max_turn_silence` still take precedence over the mode's defaults. Set at construction (connect) time only.

- **`language_codes`** _(str | list[str])_ (optional): Available in:
- [ ] Node.js
- [x] Python

The languages you expect to hear. Steering improves accuracy for those languages, and the model still code-switches between them. Leave it unset for full multilingual detection across every supported language.

Accepts a single code (`"es"`) or a list (`["en", "es"]`), written as an ISO 639-1 code (`"en"`), a BCP-47 tag (`"en-US"`), or a language name (`"english"`). The plugin normalizes each entry to a bare ISO 639-1 code and drops duplicates. More than 10 unique codes raises a `ValueError`. Update the set mid-session with `update_options()`, and pass an empty list (or `"multi"` on its own) to return to full multilingual detection.

- **`agent_context`** _(string)_ (optional): Free-text describing your agent's most recent spoken reply, used to bias transcription of the user's next turn. Can be set at construction or updated per turn via `update_options(agent_context=...)`. Max 1750 characters. See [Conversation context](#conversation-context).

- **`previous_context_n_turns`** _(int)_ (optional) - Default: `3`: Maximum number of prior conversation entries (finalized user transcripts plus any `agent_context` values) carried forward as context for each transcription. The default is applied server-side. Set to `0` to disable automatic context carryover entirely. Valid range: `0`–`100`. Set at construction (connect) time only. It cannot be changed via `update_options()`. See [Conversation context](#conversation-context).

- **`voice_focus`** _(string)_ (optional): Voice Focus isolates the primary voice and suppresses background noise (chatter, keyboard clicks, fan hum, room echo) before audio reaches the model. Set to `near-field` for headsets, handsets, and close-talking microphones, or `far-field` for conference rooms, laptop mics, and other distant-mic setups. Set at construction (connect) time only. See AssemblyAI's [Voice Focus docs](https://www.assemblyai.com/docs/streaming/voice-focus).

- **`voice_focus_threshold`** _(float)_ (optional): Controls how aggressively background audio is suppressed (higher is more aggressive). Only takes effect alongside `voice_focus`. Valid range: `0.0`–`1.0`. Set at construction (connect) time only.

> 💡 **Using prompts**
> 
> Test without a prompt first to establish a baseline. You can use `prompt` and [`keyterms_prompt`](#keyterms_prompt) together, and boosted terms are appended to the prompt automatically. See AssemblyAI's [Prompting and Keyterms guide](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms).

- **`continuous_partials`** _(bool)_ (optional) - Default: `True`: Available in:
- [ ] Node.js
- [x] Python

When `True`, the model emits additional partial transcripts at a steady ~3 second cadence during long turns, on top of the baseline partials emitted at the first-partial point (`interruption_delay`) and at each `min_turn_silence` silence period. Useful for long, uninterrupted turns where silence-based partials don't fire often enough for downstream consumers. The default is applied server-side. When `speaker_labels` is enabled, the server disables continuous partials by default so turns break cleanly at speaker changes. Can be updated mid-session via `update_options()`.

- **`interruption_delay`** _(int)_ (optional): Available in:
- [ ] Node.js
- [x] Python

How soon (in milliseconds) the first early partial is emitted. Lower values produce a faster time-to-first-token for barge-in, and higher values produce more confident first partials. The default is mode-dependent and set by the server based on the `mode` preset. Can be updated mid-session via `update_options()`.

Valid range: `0`–`1000`. The server adds a fixed 256 ms on top of this value, so `0` yields an effective delay of 256 ms and `500` yields 756 ms.

---

**Universal-Streaming**:

- **`end_of_turn_confidence_threshold`** _(float)_ (optional) - Default: `0.4`: The confidence threshold to use when determining if the end of a turn has been reached. Not applicable to Universal-3.6 Pro.

- **`min_end_of_turn_silence_when_confident`** _(int)_ (optional): The minimum duration of silence (in milliseconds) required to detect end of turn when confident.

**Deprecated:** This parameter has been renamed to `min_turn_silence`. Use `min_turn_silence` instead. Note that the LiveKit plugin defaults `min_turn_silence` to `100` for **all** streaming models (not just Universal-3.6 Pro), so the effective default is `100` ms.

- **`max_turn_silence`** _(int)_ (optional) - Default: `1280`: The maximum duration of silence (in milliseconds) allowed in a turn before end of turn is triggered.

- **`format_turns`** _(bool)_ (optional) - Default: `false`: Whether to return formatted final transcripts. Not applicable to Universal-3.6 Pro (always returns formatted transcripts).

### Turn detection

**Universal-3.6 Pro**:

Universal-3.6 Pro uses **punctuation-based turn detection**. It checks for terminal punctuation (`.` `?` `!`) after periods of silence rather than using a confidence score. To use this for [turn detection](https://docs.livekit.io/agents/logic/turns.md), set `turn_detection="stt"` in the turn handling options.

**Default parameter differences:** The LiveKit plugin defaults to `min_turn_silence=100` and `max_turn_silence=100`. The AssemblyAI API defaults are `min_turn_silence=100` and `max_turn_silence=1000`. When using `turn_detection="stt"`, explicitly set `max_turn_silence=1000` to restore AssemblyAI's intended behavior.

**Endpointing delay is additive in STT mode:** LiveKit's default `min_delay` (0.5 seconds) in the turn handling endpointing options is applied on top of AssemblyAI's own endpointing. Set `endpointing.min_delay` to `0` in the turn handling options to avoid extra latency. AssemblyAI's `min_turn_silence` and `max_turn_silence` already control the timing.

**VAD threshold alignment:** Universal-3.6 Pro defaults to a `vad_threshold` of `0.3`. Set LiveKit's Silero `activation_threshold` to `0.3` as well to ensure consistent barge-in behavior.

**Tuning guidance:** Experiment with `min_turn_silence` and `max_turn_silence`. Settings can vary depending on your use case. Increase `min_turn_silence` if brief pauses cause the speculative EOT check to fire too early, ending turns on terminal punctuation before the user has finished speaking. Increase `max_turn_silence` if the forced turn end is cutting off users mid-thought.

**Python**:

```python
session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
        endpointing={"min_delay": 0},
    ),
    stt=assemblyai.STT(
        model="universal-3-6-pro",
        min_turn_silence=100,
        max_turn_silence=1000,
        vad_threshold=0.3,
    ),
    vad=inference.VAD(model="silero", activation_threshold=0.3),
    # ... llm, tts, etc.
)

```

---

**Node.js**:

```typescript
import { inference, voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';

const session = new voice.AgentSession({
    turnHandling: {
        turnDetection: 'stt',
        endpointing: { minDelay: 0 },
    },
    stt: new assemblyai.STT({
        speechModel: 'universal-3-6-pro',
        minTurnSilence: 100,
        maxTurnSilence: 1000,
        vadThreshold: 0.3,
    }),
    vad: new inference.VAD({ model: 'silero', activationThreshold: 0.3 }),
    // ... llm, tts, etc.
});

```

You can also use LiveKit's [audio turn detector](https://docs.livekit.io/agents/logic/turns/turn-detector.md#audio-turn-detector) instead of `turn_detection="stt"`. It operates directly on audio, so it doesn't depend on transcript timing. When you do rely on transcript-based detection (`turn_detection="stt"`), the plugin defaults (`min_turn_silence=100`, `max_turn_silence=100`) are tuned to finalize transcripts as fast as possible. Raising these values (for example, 200–300 ms) can reduce over-segmentation.

For a detailed guide on configuring Universal-3.6 Pro with LiveKit, including entity splitting tradeoffs, VAD threshold alignment, and prompt engineering, see the [AssemblyAI LiveKit guide](https://www.assemblyai.com/docs/voice-agents/livekit-u3-rt-pro).

---

**Universal-Streaming**:

AssemblyAI Universal-Streaming includes a custom phrase endpointing model that uses both audio and linguistic information to detect turn boundaries. To use this model for [turn detection](https://docs.livekit.io/agents/logic/turns.md), set `turn_detection="stt"` in the turn handling options. The session's bundled VAD continues to handle interruption detection, so the agent stays responsive while the STT determines turn boundaries.

**Python**:

```python
session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
    ),
    stt=assemblyai.STT(
      model="universal-streaming-english",
      end_of_turn_confidence_threshold=0.4,
      min_turn_silence=400,
      max_turn_silence=1280,
    ),
    # ... llm, tts, etc.
)

```

---

**Node.js**:

```typescript
import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';

const session = new voice.AgentSession({
    turnHandling: {
        turnDetection: 'stt',
    },
    stt: new assemblyai.STT({
        speechModel: 'universal-streaming-english',
        endOfTurnConfidenceThreshold: 0.4,
        minTurnSilence: 400,
        maxTurnSilence: 1280,
    }),
    // ... llm, tts, etc.
});

```

### Session information

When a WebSocket session starts, AssemblyAI sends a `Begin` event that includes a session ID and expiry timestamp. The plugin exposes the following information on the `SpeechStream` object:

| Field | Description |
| `session_id` (Python) / `sessionId` (Node.js) | UUID string identifying the transcription session. The session ID is also logged automatically at INFO level. Share it with AssemblyAI support when troubleshooting transcription issues. |
| `expires_at` (Python) / `expiresAt` (Node.js) | Unix timestamp indicating when the session expires. |

**Python**:

```python
stream = stt.stream()
async for event in stream:
    # session_id is set before any speech events arrive
    print(stream.session_id)   # e.g. "676d673c-83fc-4d8a-bd95-bfe23b1c5a50"
    print(stream.expires_at)   # e.g. 1773775624

```

---

**Node.js**:

```typescript
const stream = stt.stream();
for await (const event of stream) {
    // sessionId is set before any speech events arrive
    console.log(stream.sessionId);  // e.g. "676d673c-83fc-4d8a-bd95-bfe23b1c5a50"
    console.log(stream.expiresAt);  // e.g. 1773775624
}

```

These properties are `None` (Python) or `null` (Node.js) until the `Begin` event is received from AssemblyAI, which happens shortly after the stream starts.

The session ID is also automatically logged:

```bash
AssemblyAI session started id=676d673c-83fc-4d8a-bd95-bfe23b1c5a50 expires_at=1773775624

```

## Keyterms

AssemblyAI supports keyterm prompting on all streaming models. To manage keyterms across providers, or detect them automatically from the conversation, see [Keyterm management](https://docs.livekit.io/agents/models/stt/keyterms.md). To set AssemblyAI's `keyterms_prompt` parameter directly, use the [model parameters](#model-parameters).

## Conversation context

Universal-3.6 Pro includes prior conversation turns as context for each transcription. Speech models rely on surrounding words to choose between similar-sounding interpretations of audio, and a short reply like "yes" or "7pm" has no surrounding words of its own. Prior turns fill that gap. If the agent just asked what time works, "7pm" is the more likely transcription of an ambiguous short reply. Context is per-session, and closing the stream clears it.

### Automatic context carryover

By default, the model carries prior **finalized** turns forward as context for the next turn. This is on automatically and requires no configuration. The number of prior entries carried forward is controlled by `previous_context_n_turns` (server default `3`). Set it to `0` at construction time to disable carryover entirely.

### Agent context

`agent_context` is a single string holding what the agent just said. The model uses it to bias transcription of the user's next turn. For example, when the agent asks for the user's email address, the model is more likely to produce "user@acme.com" than a literal "user at acme dot com" transcription.

With an `AgentSession`, this is automatic and on by default for Universal-3.6 Pro: after each agent turn, the reply replaces the current `agent_context` value, truncated to its last 1750 characters if the reply is longer. You can still set the value yourself with `update_options(agent_context=...)`, but the next agent reply overwrites it. To turn the forwarding off, pass `stt_context_options={"forward_chat_context": False}` to the `AgentSession`.

Outside of an `AgentSession`, you manage the value yourself. Set `agent_context` at construction to seed an opening greeting, then call `update_options(agent_context=...)` after each agent reply. Each value replaces the previous one instead of adding to it, and the maximum length is 1750 characters.

**LiveKit Inference**:

**Python**:

```python
from livekit.agents import inference

stt = inference.STT(
    model="assemblyai/universal-3-6-pro",
    extra_kwargs={
        "agent_context": "Hi! Thanks for calling Acme. How can I help today?",
        "previous_context_n_turns": 3,
    },
)

```

---

**Node.js**:

```typescript
import { inference } from '@livekit/agents';

const stt = new inference.STT({
    model: 'assemblyai/universal-3-6-pro',
    modelOptions: {
        agent_context: "Hi! Thanks for calling Acme. How can I help today?",
        previous_context_n_turns: 3,
    },
});

```

---

**Plugin**:

**Python**:

```python
from livekit.plugins import assemblyai

stt = assemblyai.STT(
    model="universal-3-6-pro",
    agent_context="Hi! Thanks for calling Acme. How can I help today?",
    previous_context_n_turns=3,
)

# After each agent reply, update the context for the next user turn.
stt.update_options(agent_context="Sure, what's the email on the account?")

```

---

**Node.js**:

```typescript
import * as assemblyai from '@livekit/agents-plugin-assemblyai';

const stt = new assemblyai.STT({
    speechModel: 'universal-3-6-pro',
    agentContext: "Hi! Thanks for calling Acme. How can I help today?",
    previousContextNTurns: 3,
});

// After each agent reply, update the context for the next user turn.
stt.updateOptions({ agentContext: "Sure, what's the email on the account?" });

```

For more detail on how Universal-3.6 Pro uses conversation context, see AssemblyAI's [LiveKit agent context example](https://www.assemblyai.com/docs/voice-agents/livekit-u3-rt-pro#wiring-agent-context-with-conversation_item_added).

## Speaker diarization

Enable speaker diarization so the STT assigns a speaker identifier to each word or segment. When enabled, transcript events include a `speaker_id`, and the STT reports `capabilities.diarization = True`.

With diarization enabled, you can wrap the AssemblyAI STT with [`MultiSpeakerAdapter`](https://docs.livekit.io/agents/models/stt.md#speaker-diarization) for primary speaker detection and transcript formatting.

Enable speaker diarization in the `STT` constructor:

**LiveKit Inference**:

**Python**:

```python
stt = inference.STT(
    model="assemblyai/universal-3-6-pro",
    extra_kwargs={
        "speaker_labels": True,
    },
)

```

---

**Node.js**:

```typescript
const stt = new inference.STT({
    model: 'assemblyai/universal-3-6-pro',
    modelOptions: {
        speaker_labels: true,
    },
});

```

---

**Plugin**:

**Python**:

```python
stt = assemblyai.STT(
    model="universal-3-6-pro",
    speaker_labels=True,
)

```

---

**Node.js**:

```typescript
const stt = new assemblyai.STT({
    speechModel: 'universal-3-6-pro',
    speakerLabels: true,
});

```

> 🔥 **Caution**
> 
> The Node.js plugin sends `speakerLabels` to AssemblyAI, but speaker labels are not yet surfaced on transcript events. To get speaker IDs in Node.js, use LiveKit Inference.

Speaker labels are assigned alphabetically (`"A"`, `"B"`, etc.) per session. Short utterances under ~1 second return `speaker_id=None`.

## Additional resources

The following resources provide more information about using AssemblyAI with LiveKit Agents.

- **[AssemblyAI docs](https://www.assemblyai.com/docs/streaming/select-the-speech-model)**: AssemblyAI's full docs for streaming speech-to-text.

- **[Voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai.md)**: Get started with LiveKit Agents and AssemblyAI.

- **[AssemblyAI LiveKit guide](https://www.assemblyai.com/docs/voice-agents/livekit-u3-rt-pro)**: Guide to using AssemblyAI Universal-3.6 Pro STT with LiveKit.

---

This document was rendered at 2026-10-07T15:55:38.022Z.
For the latest version of this document, see [https://docs.livekit.io/agents/models/stt/assemblyai.md](https://docs.livekit.io/agents/models/stt/assemblyai.md).

To explore all LiveKit documentation, see [llms.txt](https://docs.livekit.io/llms.txt).