Expressive mode

Overview

Expressive mode lets the LLM control how your agent sounds, not only what it says. Turn it on with a single flag and the framework tells the LLM which delivery tags the active TTS understands, so it adds emotion, pacing, and non-verbal sounds like breaths or laughs inline with its reply. LiveKit renders those tags through the TTS and strips them from the transcript, so the effect lands in the audio while your chat UI stays clean.

The goal is appropriate emotion, not maximum emotion: the agent matches the register of the moment, brightening at good news and easing off at bad. It runs on LiveKit Inference, works with any LLM, and keeps improving as the underlying TTS models do.

Try it out

Give the agent good news or bad news and hear its delivery shift to match. Toggle expressive mode off for the flat version, or switch voices to compare.

How it works

Expressive mode is a translation layer between your LLM and your TTS. When it's on, the framework:

1. Injects the provider's markup guide into the LLM prompt, so the model emits inline delivery tags for emotion, pacing, and non-verbal sounds, chosen from the conversation.
2. Normalizes and converts the markup into the provider's native dialect, fixing common mistakes like a tag that isn't self-closed before it reaches the TTS.
3. Batches sentences into larger chunks before synthesis, so the TTS keeps a consistent emotion across the turn instead of resetting each sentence.
4. Strips the tags from the transcript, so users and chat history see clean text while the TTS receives the marked-up version.

The active provider determines the available delivery tags, so switching providers changes them automatically.

Supported providers

Expressive mode requires an inference.TTS whose model declares a markup dialect. The following providers support it, and each provider's page describes what its delivery covers:

Enable expressive mode

Set expressive=True on your AgentSession. Expressive mode runs on LiveKit Inference, so your tts must be an inference.TTS from a supported provider. It stays off for any unsupported provider.

from livekit.agents import AgentSession, inference

session = AgentSession(
    stt=inference.STT(model="deepgram/nova-3", language="multi"),
    llm=inference.LLM(model="google/gemma-4-31b-it"),
    tts=inference.TTS(model="fishaudio/s2.1-pro", voice="b347db033a6549378b48d00acb0d06cd"),
    expressive=True,
    # ... vad, turn_detection
)

import { voice, inference } from '@livekit/agents';

const session = new voice.AgentSession({
  stt: new inference.STT({ model: 'deepgram/nova-3', language: 'multi' }),
  llm: new inference.LLM({ model: 'google/gemma-4-31b-it' }),
  tts: new inference.TTS({ model: 'fishaudio/s2.1-pro', voice: 'b347db033a6549378b48d00acb0d06cd' }),
  expressive: true,
  // ... vad, turnDetection
});

That's the whole setup. Your prompt still owns what the agent says, while expressive mode owns how it sounds, so keep delivery instructions out of your prompt and let expressive mode handle them.

Customization

Setting expressive=True lets the agent use the provider's full range of expression, chosen to fit the conversation. To narrow or extend that range, pass an ExpressiveOptions dictionary instead of True. There are two controls: speech_steering for delivery and non-verbal sounds, and custom prompt instructions for editing what the LLM is told directly.

Steer delivery and sounds

Use speech_steering to adjust delivery and non-verbal sounds:

session = AgentSession(
    # ... stt, llm, tts
    expressive={
        "speech_steering": {
            "disfluencies": False,
            "nonverbal_sounds": {"laughing": False},
            "pace": "slow",
        },
    },
)

const session = new voice.AgentSession({
  // ... stt, llm, tts
  expressive: {
    speechSteering: {
      disfluencies: false,
      nonverbalSounds: { laughing: false },
      pace: 'slow',
    },
  },
});

The LLM already matches its delivery to the moment, so most keys are opt-out: set disfluencies or nonverbal_sounds only to take an option away. pace sets the overall rate.

In Node.js, speech_steering and nonverbal_sounds are camelCase (speechSteering, nonverbalSounds).

speech_steering accepts:

Customize the prompt

For full control, ExpressiveOptions also accepts tts_instructions_append (ttsInstructionsAppend in Node.js) to add your own rules to the default markup guide, or tts_instructions_template (ttsInstructionsTemplate in Node.js) to replace it entirely:

session = AgentSession(
    # ... stt, llm, tts
    expressive={
        "tts_instructions_append": "Keep laughter rare, and never sound sarcastic.",
    },
)

const session = new voice.AgentSession({
  // ... stt, llm, tts
  expressive: {
    ttsInstructionsAppend: 'Keep laughter rare, and never sound sarcastic.',
  },
});

Read the mood in your frontend

If your app shows the agent's state, like a visualizer or a mood indicator, expressive mode hands the frontend the current mood alongside the transcript. The agent normalizes each delivery to one of eleven moods before publishing it, so you switch on a fixed set rather than free-form text.

Using the useAgentExpression hook

In React, the useAgentExpression hook returns the current mood and the provider's raw expression text behind it:

import { useAgentExpression } from '@livekit/components-react';

function MoodBadge() {
  const { mood, expression } = useAgentExpression();
  return <span>{mood ?? 'neutral'}</span>;
}

mood is one of eleven fixed values, so you can switch on it. An unrecognized label falls back to calm, and the mood clears two agent turns after the last expression. expression is the provider's own wording, best for display rather than switching on.

To drive an audio visualizer's color from the mood, see Expressive Agents.

Reading the attribute directly

When you're not using the React hook (for example, a vanilla JS or non-React client), read the lk.expression attribute off the text stream directly. One value is published per segment, on both the opening and closing header, so it's available as soon as the agent starts speaking:

room.registerTextStreamHandler('lk.transcription', async (reader, participant) => {
  const expression = reader.info.attributes?.['lk.expression'];
  if (expression) {
    // e.g. { expression: 'excited', mood: 'excited' }
    const { mood } = JSON.parse(expression);
    updateMoodIndicator(mood);
  }
  for await (const chunk of reader) {
    appendTranscript(chunk); // clean, tag-free text
  }
});

Fish Audio, Inworld, and Cartesia all publish lk.expression. xAI steers delivery through prosody and sound tags only, so its speech is expressive but it publishes no expression.

Additional resources