Agents and handoffs | LiveKit Documentation

Overview

Agents are the core units of a voice AI workflow. They define the instructions, tools, and reasoning behavior that drive a conversation. An agent can transfer control to other agents when different logic or capabilities are required. Create separate agents when you need distinct reasoning behavior or tool access:

Different roles: A moderator agent versus a coaching agent.
Model specialization: A lightweight triage model before escalating to a larger one.
Different permissions: An agent with payment API access versus one handling general inquiries.
Specialized contexts: Agents optimized for particular conversation phases.

Agents

Agents orchestrate the session flow—managing tools, reasoning steps, and control transfers between other agents or tasks.

Defining an agent

Extend the Agent class to define a custom agent.

from livekit.agents import Agent

class HelpfulAssistant(Agent):
    def __init__(self):
        super().__init__(instructions="You are a helpful voice AI assistant.")

    async def on_enter(self) -> None:
        await self.session.generate_reply(instructions="Greet the user and ask how you can help them.")

import { voice } from '@livekit/agents';

class HelpfulAssistant extends voice.Agent {
  constructor() {
    super({
      instructions: 'You are a helpful voice AI assistant.',
    });
  }

  async onEnter(): Promise<void> {
    this.session.generateReply({
      instructions: 'Greet the user and ask how you can help them.',
    });
  }
}

You can also create an instance of Agent class directly:

agent = Agent(instructions="You are a helpful voice AI assistant.")

const agent = new voice.Agent({
  instructions: 'You are a helpful voice AI assistant.',
});

Setting the active agent

The active agent is the agent currently in control of the session. The initial agent is defined in the AgentSession constructor. You can change the active agent using the update_agent method in Python, or a handoff from a tool call.

Specify the initial agent in the AgentSession constructor:

session = AgentSession(
    agent=CustomerServiceAgent()
    # ...
)

await session.start({
  agent: new CustomerServiceAgent(),
  room: ctx.room,
});

To set a new agent, use the update_agent method:

session.update_agent(CustomerServiceAgent())

session.updateAgent(new CustomerServiceAgent());

Agent handoffs

A handoff transfers session control from one agent to another. You can return a different agent from within a tool call to hand off control automatically. This allows the LLM to make decisions about when a handoff should occur. For more information, see tool return value.

from livekit.agents import Agent, function_tool

class CustomerServiceAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are a friendly customer service representative. Help customers with 
            general inquiries, account questions, and technical support. If a customer needs 
            specialized help, transfer them to the appropriate specialist."""
        )

    async def on_enter(self) -> None:
        await self.session.generate_reply(instructions="Greet the user warmly and offer your assistance.")

    @function_tool()
    async def transfer_to_billing(self, context: RunContext):
        """Transfer the customer to a billing specialist for account and payment questions."""
        return BillingAgent(chat_ctx=self.chat_ctx), "Transferring to billing"

    @function_tool()
    async def transfer_to_technical_support(self, context: RunContext):
        """Transfer the customer to technical support for product issues and troubleshooting."""
        return TechnicalSupportAgent(chat_ctx=self.chat_ctx), "Transferring to technical support"

class BillingAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are a billing specialist. Help customers with account questions, 
            payments, refunds, and billing inquiries. Be thorough and empathetic."""
        )

    async def on_enter(self) -> None:
        await self.session.generate_reply(instructions="Introduce yourself as a billing specialist and ask how you can help with their account.")

class TechnicalSupportAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are a technical support specialist. Help customers troubleshoot 
            product issues, setup problems, and technical questions. Ask clarifying questions 
            to diagnose problems effectively."""
        )

    async def on_enter(self) -> None:
        await self.session.generate_reply(instructions="Introduce yourself as a technical support specialist and offer to help with any technical issues.")

import { voice, llm } from '@livekit/agents';

class CustomerServiceAgent extends voice.Agent {
  constructor() {
    super({
      instructions: `You are a friendly customer service representative. Help customers with 
        general inquiries, account questions, and technical support. If a customer needs 
        specialized help, transfer them to the appropriate specialist.`,
      tools: {
        transferToBilling: llm.tool({
          description: 'Transfer the customer to a billing specialist for account and payment questions.',
          execute: async (_, { ctx }) => {
            return llm.handoff({
              agent: new BillingAgent(),
              returns: 'Transferring to billing',
            });
          },
        }),
        transferToTechnicalSupport: llm.tool({
          description: 'Transfer the customer to technical support for product issues and troubleshooting.',
          execute: async (_, { ctx }) => {
            return llm.handoff({
              agent: new TechnicalSupportAgent(),
              returns: 'Transferring to technical support',
            });
          },
        }),
      },
    });
  }

  async onEnter(): Promise<void> {
    this.session.generateReply({
      instructions: 'Greet the user warmly and offer your assistance.',
    });
  }
}

class BillingAgent extends voice.Agent {
  constructor() {
    super({
      instructions: `You are a billing specialist. Help customers with account questions, 
        payments, refunds, and billing inquiries. Be thorough and empathetic.`,
    });
  }

  async onEnter(): Promise<void> {
    this.session.generateReply({
      instructions: 'Introduce yourself as a billing specialist and ask how you can help with their account.',
    });
  }
}

class TechnicalSupportAgent extends voice.Agent {
  constructor() {
    super({
      instructions: `You are a technical support specialist. Help customers troubleshoot 
        product issues, setup problems, and technical questions. Ask clarifying questions 
        to diagnose problems effectively.`,
    });
  }

  async onEnter(): Promise<void> {
    this.session.generateReply({
      instructions: 'Introduce yourself as a technical support specialist and offer to help with any technical issues.',
    });
  }
}

Chat history

When an agent handoff occurs, an AgentHandoff item (or AgentHandoffItem in Node.js) is added to the chat context with the following properties:

old_agent_id: ID of the agent that was active before the handoff.
new_agent_id: ID of the agent that took over session control after the handoff.

Passing state

To store custom state within your session, use the userdata attribute. The type of userdata is up to you, but the recommended approach is to use a dataclass in Python or a typed interface in TypeScript.

from livekit.agents import AgentSession
from dataclasses import dataclass

@dataclass
class MySessionInfo:
    user_name: str | None = None
    age: int | None = None

interface MySessionInfo {
  userName?: string;
  age?: number;
}

To add userdata to your session, pass it in the constructor. You must also specify the type of userdata on the AgentSession itself.

session = AgentSession[MySessionInfo](
    userdata=MySessionInfo(),
    # ... tts, stt, llm, etc.
)

const session = new voice.AgentSession<MySessionInfo>({
  userData: { userName: 'Steve' },
  // ... vad, stt, tts, llm, etc.
});

Userdata is available as session.userdata, and is also available within function tools on the RunContext. The following example shows how to use userdata in an agent workflow that starts with the IntakeAgent.

class IntakeAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are an intake agent. Learn the user's name and age."""
        )
        
    @function_tool()
    async def record_name(self, context: RunContext[MySessionInfo], name: str):
        """Use this tool to record the user's name."""
        context.userdata.user_name = name
        return self._handoff_if_done()
    
    @function_tool()
    async def record_age(self, context: RunContext[MySessionInfo], age: int):
        """Use this tool to record the user's age."""
        context.userdata.age = age
        return self._handoff_if_done()
    
    def _handoff_if_done(self):
        if self.session.userdata.user_name and self.session.userdata.age:
            return CustomerServiceAgent()
        else:
            return None

class CustomerServiceAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You are a friendly customer service representative.")

    async def on_enter(self) -> None:
        userdata: MySessionInfo = self.session.userdata
        await self.session.generate_reply(
            instructions=f"Greet {userdata.user_name} personally and offer your assistance."
        )

import { voice, llm } from '@livekit/agents';
import { z } from 'zod';

class IntakeAgent extends voice.Agent<MySessionInfo> {
  constructor() {
    super({
      instructions: "You are an intake agent. Learn the user's name and age.",
      tools: {
        recordName: llm.tool({
          description: 'Use this tool to record the user\'s name.',
          parameters: z.object({
            name: z.string(),
          }),
          execute: async ({ name }, { ctx }) => {
            ctx.userData.userName = name;
            return this.handoffIfDone(ctx);
          },
        }),
        recordAge: llm.tool({
          description: 'Use this tool to record the user\'s age.',
          parameters: z.object({
            age: z.number(),
          }),
          execute: async ({ age }, { ctx }) => {
            ctx.userData.age = age;
            return this.handoffIfDone(ctx);
          },
        }),
      },
    });
  }

  private handoffIfDone(ctx: voice.RunContext<MySessionInfo>) {
    if (ctx.userData.userName && ctx.userData.age) {
      return llm.handoff({
        agent: new CustomerServiceAgent(),
        returns: 'Information collected, transferring to customer service',
      });
    }
    return 'Please provide both your name and age.';
  }
}

class CustomerServiceAgent extends voice.Agent<MySessionInfo> {
  constructor() {
    super({
      instructions: 'You are a friendly customer service representative.',
    });
  }

  async onEnter(): Promise<void> {
    const userData = this.session.userData;
    this.session.generateReply({
      instructions: `Greet ${userData.userName} personally and offer your assistance.`,
    });
  }
}

Context preservation

By default, each new agent or task starts with a fresh conversation history for their LLM prompt. To include the prior conversation, set the chat_ctx parameter in the Agent or AgentTask constructor. You can either copy the prior agent's chat_ctx, or construct a new one based on custom business logic to provide the appropriate context. For example, see Summarizing context for a helper function that summarizes the prior conversation and passes it to the next agent.

The previous agent or task's instructions aren't preserved during a handoff unless you seed the conversation using chat_ctx.copy(). By default, copy() includes instructions. You can remove them by passing the exclude_instructions parameter. See the following examples.

from livekit.agents import ChatContext, function_tool, Agent

class TechnicalSupportAgent(Agent):
    def __init__(self, chat_ctx: ChatContext):
        super().__init__(
            instructions="""You are a technical support specialist. Help customers troubleshoot 
            product issues, setup problems, and technical questions.""",
            chat_ctx=chat_ctx
        )

class CustomerServiceAgent(Agent):
    # ...

    @function_tool()
    async def transfer_to_technical_support(self):
        """Transfer the customer to technical support for product issues and troubleshooting."""
        await self.session.generate_reply(instructions="Inform the customer that you're transferring them to the technical support team.")
        
        # Pass only the conversation turns, not the previous agent's instructions
        return TechnicalSupportAgent(chat_ctx=self.chat_ctx.copy(exclude_instructions=True))

import { voice, llm } from '@livekit/agents';

class TechnicalSupportAgent extends voice.Agent {
  constructor(chatCtx: llm.ChatContext) {
    super({
      instructions: `You are a technical support specialist. Help customers troubleshoot 
        product issues, setup problems, and technical questions.`,
      chatCtx,
    });
  }
}

class CustomerServiceAgent extends voice.Agent {
  constructor(chatCtx: llm.ChatContext) {
    super({
      // ... instructions, chatCtx, etc.
      tools: {
        transferToTechnicalSupport: llm.tool({
          description: 'Transfer the customer to technical support for product issues and troubleshooting.',
          execute: async (_, { ctx }) => {
            await ctx.session.generateReply({
              instructions: 'Inform the customer that you\'re transferring them to the technical support team.',
            });
            
            return llm.handoff({
              agent: new TechnicalSupportAgent(ctx.session.chatCtx.copy({ excludeInstructions: true })),
              returns: 'Transferring to technical support team',
            });
          },
        }),
      },
    });
  }
}

The complete conversation history for the session is always available in session.history.

Summarizing context

When the prior conversation is long, summarize it before handoff to keep the next agent's context compact. The following helper function filters the chat context down to user and assistant turns, then uses a separate LLM call to generate a brief summary string. It allows you to pass in any LLM instance—including a lighter or faster model—independently of the main voice agent.

In Python, LLMStream.collect() awaits the full response stream and returns a CollectedResponse with text, tool_calls, and usage fields.

from livekit.agents import llm, ChatContext, function_tool, Agent, RunContext

async def summarize_session(summarizer: llm.LLM, chat_ctx: ChatContext) -> str | None:
    """Generate a brief summary of user/assistant turns using a separate LLM call."""
    summary_ctx = ChatContext()
    summary_ctx.add_message(
        role="system",
        content="Summarize the conversation between user and assistant. Keep the summary brief, touching on the main topics and outcomes.",
    )

    n_summarized = 0
    for item in chat_ctx.items:
        if item.type != "message":
            continue
        if item.role not in ("user", "assistant"):
            continue
        if item.extra.get("is_summary") is True:  # avoid summarizing previous summaries
            continue
        text = (item.text_content or "").strip()
        if text:
            summary_ctx.add_message(role="user", content=f"{item.role}: {text}")
            n_summarized += 1

    if n_summarized == 0:
        return None

    response = await summarizer.chat(chat_ctx=summary_ctx).collect()
    return response.text.strip() if response.text else None


class TriageAgent(Agent):
    # ...

    @function_tool()
    async def transfer_to_specialist(self, context: RunContext, topic: str):
        """Hand off to a specialist once triage is complete."""
        summarizer = self.session.llm  # or pass a different model, e.g. openai.LLM(model="gpt-4o-mini")
        summary = await summarize_session(summarizer, self.chat_ctx) if summarizer else None

        # Build a fresh context with only the summary for the next agent
        chat_ctx = ChatContext()
        if summary:
            chat_ctx.add_message(role="system", content=f"Prior conversation summary: {summary}")

        return SpecialistAgent(topic, chat_ctx=chat_ctx)

The LLMStream.collect() function is not available in Node.js. This example iterates the stream manually to achieve the same result.

import { voice, llm } from '@livekit/agents';
import { LLM } from '@livekit/agents/llm';
import { z } from 'zod';

async function summarizeSession(
  summarizerLlm: llm.LLM,
  chatCtx: llm.ChatContext,
): Promise<string | null> {
  const summaryCtx = new llm.ChatContext();
  summaryCtx.addMessage({
    role: 'system',
    content: 'Summarize the conversation between user and assistant. Keep the summary brief, touching on the main topics and outcomes.',
  });

  let nSummarized = 0;
  for (const item of chatCtx.items) {
    if (item.type !== 'message') continue;
    if (item.role !== 'user' && item.role !== 'assistant') continue;
    if (item.extra?.is_summary === true) continue; // avoid summarizing previous summaries
    const text = item.textContent?.trim();
    if (text) {
      summaryCtx.addMessage({ role: 'user', content: `${item.role}: ${text}` });
      nSummarized++;
    }
  }

  if (nSummarized === 0) return null;

  const chunks: string[] = [];
  for await (const chunk of summarizerLlm.chat({ chatCtx: summaryCtx })) {
    if (chunk.delta?.content) chunks.push(chunk.delta.content);
  }
  return chunks.join('').trim() || null;
}

class TriageAgent extends voice.Agent {
  constructor() {
    super({
      // ...
      tools: {
        transferToSpecialist: llm.tool({
          description: 'Hand off to a specialist once triage is complete.',
          parameters: z.object({ topic: z.string() }),
          execute: async ({ topic }, { ctx }) => {
            const sessionLlm = ctx.session.llm;
            const summary =
              sessionLlm instanceof LLM
                ? await summarizeSession(sessionLlm, ctx.session.chatCtx)
                : null;

            // Build a fresh context with only the summary for the next agent
            const chatCtx = new llm.ChatContext();
            if (summary) {
              chatCtx.addMessage({ role: 'system', content: `Prior conversation summary: ${summary}` });
            }

            return llm.handoff({
              agent: new SpecialistAgent(topic, chatCtx),
              returns: `Transferring to ${topic} specialist`,
            });
          },
        }),
      },
    });
  }
}

Other options include copying and truncating the prior context (for example, chat_ctx.copy().truncate(max_items=6)) to carry only the last few turns, or storing key facts in userdata and injecting a short serialized summary (for example, YAML or JSON) as a system message when the next agent starts.

Overriding plugins

You can override any of the plugins used in the session by setting the corresponding attributes in your Agent or AgentTask constructor. This allows you to customize the behavior and properties of the active agent or task in the session by modifying the prompt, TTS, LLM, STT plugins, and more.

For instance, you can change the voice for a specific agent by overriding the tts attribute:

from livekit.agents import Agent, inference

class CustomerServiceManager(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are a customer service manager who can handle escalated issues.",
            tts=inference.TTS(model="cartesia/sonic-3", voice="6f84f4b8-58a2-430c-8c79-688dad597532")
        )

import { voice, inference } from '@livekit/agents';

class CustomerServiceManager extends voice.Agent {
  constructor() {
    super({
      instructions: 'You are a customer service manager who can handle escalated issues.',
      tts: new inference.TTS({ model: 'cartesia/sonic-3', voice: '6f84f4b8-58a2-430c-8c79-688dad597532' }),
    });
  }
}