Skip to main content

Keyframe virtual avatar integration guide

How to use the Keyframe virtual avatar plugin for LiveKit Agents.

On this page

Available inPython

Overview

Keyframe Labs  provides hyper-realistic, emotionally expressive avatars that can participate in live, interactive conversations. You can use the open source Keyframe integration for LiveKit Agents to enable seamless integration of Keyframe avatars into your voice AI app.

Avatar: Lyra

Quick reference

This section includes a basic usage example and some reference material. For links to more detailed documentation, see Additional resources.

Installation

uv add "livekit-agents[keyframe]~=1.8"

Authentication

The Keyframe plugin requires a Keyframe API key .

Set KEYFRAME_API_KEY in your .env file.

Persona setup

The Keyframe plugin requires a persona, a pre-configured avatar identity. You can specify the persona by ID or by slug, passed by persona_id or persona_slug respectively. You must provide exactly one of these.

Persona slug

A slug is a human-readable identifier in the format "public:<persona_name>". Browse available public personas at the Keyframe platform .

avatar = keyframe.AvatarSession(
persona_slug="public:lyra_persona-2.0-live",
)

Persona ID

A persona ID is a UUID that uniquely identifies a persona. You can find persona IDs in the Keyframe platform .

avatar = keyframe.AvatarSession(
persona_id="7b147d15-01a6-4baa-8a9b-689c38b60444",
)

Usage

Use the plugin in an AgentSession. For example, you can use this avatar in the Voice AI quickstart.

from livekit import agents
from livekit.agents import AgentServer, AgentSession
from livekit.plugins import keyframe
server = AgentServer()
@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: agents.JobContext):
session = AgentSession(
# ... stt, llm, tts, etc.
)
avatar = keyframe.AvatarSession(
persona_slug="public:lyra_persona-2.0-live", # or use persona_id. See "Persona setup" for details.
)
# Start the avatar and wait for it to join
await avatar.start(session, room=ctx.room)
# Start your agent session with the user
await session.start(
# ... room, agent, room_options, etc.
)

Preview the avatar in the Agent Console or a frontend starter app that you build.

Parameters

This section describes some of the available parameters. See the plugin reference for a complete list of all available parameters.

api_keystring

Keyframe API key. Defaults to the KEYFRAME_API_KEY environment variable.

persona_idstring

UUID of the Keyframe persona to use. Pass exactly one of persona_id or persona_slug. See Persona setup for details.

persona_slugstring

Slug identifier for a Keyframe persona, for example "public:lyra_persona-2.0-live". Pass exactly one of persona_slug or persona_id. See Persona setup for details.

Emotion control

The Keyframe plugin can change the avatar's facial expression during a conversation. The avatar holds an expression until the next change, and the available expressions depend on the persona's model:

ModelExpressions
persona-2.0-liveneutral, happy, sad, angry, surprised, concerned, confused, disgusted, excited, scared, skeptical
persona-1.5-liveneutral, happy, sad, angry

The plugin API uses the term emotion for an expression, for example in the Emotion type and the set_emotion() method.

You can sync expression changes to the agent's speech, or set an expression directly from your code.

Sync expressions to speech

persona-2.0-live models only

Personas powered by persona-1.5-live support only setting an expression directly.

The LLM includes expression markers in its reply, and the plugin uses them to trigger expression changes in sync with the agent's audio. The timing of each change depends on the timing information reported by your TTS:

TTSWhen the expression changes
Streaming TTS with word timings, such as the Cartesia and ElevenLabs pluginsOn the marked word.
Non-streaming TTSAt the start of the sentence that contains the marker.
Streaming TTS without word timings, such as Fish Audio and Gemini on LiveKit InferenceAt the start of the reply. Only a marker before the first word applies, and the plugin ignores later markers.

On LiveKit Inference, Cartesia reports word timings with extra_kwargs={"add_timestamps": True}, and Inworld with extra_kwargs={"timestamp_type": "WORD"}.

Setup has two parts:

  • Override tts_node with keyframe.emotion_tts_node on every agent in the session, including agents you hand off to.
  • Give the LLM the marker vocabulary with keyframe.emotion_instructions().

Where you pass the emotion_instructions() text depends on your setup, as described in the following sections.

Expressive mode

With expressive mode, one <expr> marker sets both the voice delivery and the avatar's expression. The avatar's expression changes even when the TTS has no matching delivery, such as concerned or skeptical with Cartesia. Pass the instructions as tts_instructions_append:

from livekit.agents import Agent, AgentSession, inference
from livekit.plugins import keyframe
class MyAgent(Agent):
def tts_node(self, text, model_settings):
return keyframe.emotion_tts_node(self, text, model_settings)
session = AgentSession(
tts=inference.TTS(
model="cartesia/sonic-3.5",
voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
# Word timings, so expressions change on the marked word
extra_kwargs={"add_timestamps": True},
),
expressive={"tts_instructions_append": keyframe.emotion_instructions()},
# ... stt, llm, etc.
)

Other TTS

Without expressive mode, add the instructions to the agent's instructions. The markers change only the avatar's expression, and the plugin removes them before synthesis:

from livekit.agents import Agent
from livekit.plugins import keyframe
class MyAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions=(
"You are a friendly voice assistant.\n\n"
+ keyframe.emotion_instructions()
),
)
def tts_node(self, text, model_settings):
return keyframe.emotion_tts_node(self, text, model_settings)

ElevenLabs plugin

With the ElevenLabs plugin and an eleven_v3 or eleven_v4 model, one audio tag such as [happy] sets both the voice delivery and the avatar's expression. Pass style="brackets" to both emotion_instructions() and emotion_tts_node(), and set use_tts_aligned_transcript=True to keep the tags out of the transcript:

from livekit.agents import Agent, AgentSession
from livekit.plugins import elevenlabs, keyframe
class MyAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions=(
"You are a friendly voice assistant.\n\n"
+ keyframe.emotion_instructions(style="brackets")
),
)
def tts_node(self, text, model_settings):
return keyframe.emotion_tts_node(self, text, model_settings, style="brackets")
session = AgentSession(
tts=elevenlabs.TTS(model="eleven_v3"),
use_tts_aligned_transcript=True,
# ... stt, llm, etc.
)

Only tags that exactly match an expression name change the avatar's expression. Other audio tags, such as [whispers], affect only the voice.

Set an expression directly

Use set_emotion() to change the avatar's expression from your code, for example in a tool call. The method works with both persona-2.0-live and persona-1.5-live personas:

await avatar.set_emotion("happy")

A change made before the reply's audio starts, such as from a tool call the LLM makes before answering, appears on the next frame that the avatar renders. A change made while the agent is speaking can appear late, up to the end of the reply. To change the expression on a specific word, sync expressions to speech instead.

To let the LLM change the avatar's expression, wrap set_emotion() in a function tool on your agent. With a persona-1.5-live persona, type the emotion parameter as Literal["neutral", "happy", "sad", "angry"] so that the LLM can choose only expressions the model supports:

from livekit.agents import Agent, RunContext, function_tool
from livekit.plugins import keyframe
from livekit.plugins.keyframe import Emotion
class AvatarAgent(Agent):
def __init__(self, avatar: keyframe.AvatarSession) -> None:
super().__init__(
instructions=(
"You are a friendly voice assistant with an avatar. "
"Use the set_emotion tool to change your facial expression "
"whenever the conversation mood shifts."
),
)
self._avatar = avatar
@function_tool()
async def set_emotion(self, context: RunContext, emotion: Emotion) -> str:
"""Set the avatar's facial expression to match the conversation mood.
Args:
emotion: The expression to show.
"""
await self._avatar.set_emotion(emotion)
return f"Expression set to {emotion}"

Additional resources

The following resources provide more information about using Keyframe with LiveKit Agents.