Expressive Agents

Overview

Expressive mode lets your agent's speech carry emotion and publishes the mood behind that delivery to every connected client. The useAgentExpression hook reads that mood in React. Map it to a color and pass it to any audio visualizer's color prop. No special mood component is needed: use AgentAudioVisualizerAura (or any variant) with a color you compute yourself. The visualizer then shifts hue as the conversation shifts tone, brightening on good news and cooling on bad.

Start from an existing visualizer

This guide colors the aura visualizer, but the same color prop works on any variant. Copy the component and hook to your project:

pnpm dlx shadcn@latest add @agents-ui/agent-audio-visualizer-aura

Read the agent's mood

Call useAgentExpression from a component inside your AgentSessionProvider. It returns the current mood and the provider's raw expression text behind it:

import { useAgentExpression } from '@livekit/components-react';

function MoodLabel() {
  const { mood, expression } = useAgentExpression();

  return <span title={expression ?? undefined}>{mood ?? 'neutral'}</span>;
}

mood is one of eleven normalized values (excited, happy, playful, curious, surprised, hopeful, empathetic, sad, angry, anxious, or calm), or null when the agent hasn't expressed anything recently. An unrecognized label from the provider falls back to calm.

Each mood decays back to null two agent turns after the agent stops expressing it, so your UI settles instead of freezing on the first thing the agent ever said. See Customize mood decay to change that window.

Map mood to color

Every visualizer in Agents UI accepts a color prop, so a mood-to-color map is enough to drive one. Use warm colors for bright moods and cool ones for heavy moods, keeping subdued moods less saturated so they don't overpower stronger ones:

import type { AgentMood } from '@livekit/components-react';

const MOOD_COLORS: Record<AgentMood, `#${string}`> = {
  angry: '#F5222D',
  excited: '#FF7A45',
  happy: '#FFC53D',
  playful: '#F759AB',
  surprised: '#B37FEB',
  anxious: '#D46B08',
  hopeful: '#52C41A',
  empathetic: '#36CFC9',
  curious: '#6600FF',
  sad: '#2F54EB',
  calm: '#1FD5F9',
};

// Shown when the agent hasn't expressed anything recently.
const NEUTRAL_COLOR: `#${string}` = '#1FD5F9';

The palette is yours to pick. Swap in your brand colors, or map moods you don't want to distinguish onto the same value.

Animate the color transition

Snapping straight to the target color reads as a glitch. Hold the color in a Motion value, animate it toward the target with animate, and convert each tick to hex with chroma-js:

import { useEffect, useState } from 'react';
import { animate, useMotionValue, useMotionValueEvent, useTransform } from 'motion/react';
import chroma from 'chroma-js';
import type { AgentMood } from '@livekit/components-react';

function useMoodColor(
  mood: AgentMood | null,
  moodColors: Record<AgentMood, `#${string}`>,
): `#${string}` {
  const targetColor = mood ? moodColors[mood] : NEUTRAL_COLOR;
  const colorProgress = useMotionValue<string>(targetColor);
  const hexColor = useTransform(colorProgress, (latestRgba) => chroma(latestRgba).hex());
  const [color, setColor] = useState<`#${string}`>(targetColor);

  useMotionValueEvent(hexColor, 'change', (latestHex) => setColor(`#${latestHex.slice(1)}`));

  useEffect(() => {
    const controls = animate(colorProgress, targetColor, { duration: 1, ease: 'linear' });
    return () => controls.stop();
  }, [targetColor, colorProgress]);

  return color;
}

animate interpolates smoothly from the current color to the target, producing intermediate rgba() values. Each frame is converted back to hex with chroma-js, then copied into React state so the visualizer's color prop updates as the animation runs.

Wire it into your visualizer

Pass the computed color straight into AgentAudioVisualizerAura alongside state and audioTrack from your session hooks. The same color prop works on any variant:

'use client';

import { useAgent, useAgentExpression } from '@livekit/components-react';
import { AgentAudioVisualizerAura } from '@/components/agents-ui/agent-audio-visualizer-aura';

export function VoiceAgentInterface() {
  const { microphoneTrack, state } = useAgent();
  const { mood } = useAgentExpression();
  const color = useMoodColor(mood, MOOD_COLORS);

  return (
    <AgentAudioVisualizerAura size="lg" state={state} color={color} audioTrack={microphoneTrack} />
  );
}

Customize mood decay

By default, a mood survives two agent turns before decaying back to null. Pass ttlTurns to hold it longer, or set it to 0 to disable decay entirely and keep the last expressed mood until a new one arrives:

const { mood } = useAgentExpression({ ttlTurns: 4 });

Show the raw expression

expression is the TTS provider's own wording behind the mood. It's free-form and provider-specific, so it's better suited to a tooltip or a log line than a switch statement:

function MoodCaption() {
  const { mood, expression } = useAgentExpression();

  return <p>{expression ?? (mood ? `feeling ${mood}` : 'neutral')}</p>;
}

Related