LiveKit Agents

Introduction

The Agents framework allows you to add a Python or Node.js program to any LiveKit room as a full realtime participant. The SDK includes a complete set of tools and abstractions that make it easy to feed realtime media and data through an AI pipeline that works with any provider, and to publish realtime results back to the room.

If you want to get your hands on the code right away, follow this quickstart guide. It takes just a few minutes to build your first voice agent.

Use cases

Some applications for agents include:

• Multimodal assistant: Talk, text, or screen share with an AI assistant.
• Telehealth: Bring AI into realtime telemedicine consultations, with or without humans in the loop.
• Call center: Deploy AI to the front lines of customer service with inbound and outbound call support.
• Realtime translation: Translate real conversations in realtime.
• NPCs: Add lifelike NPCs backed by language models instead of static scripts.
• Robotics: Put your robot's brain in the cloud, giving it access to the most powerful models.

Framework overview

Your agent code operates as a stateful, realtime bridge between powerful AI models and your users. AI models run in large data centers with fast and reliable internet connections while your users run on consumer phones in unstable network conditions. WebRTC can maintain smooth communications between your users and agents regardless of network instability. LiveKit WebRTC is used between your frontend and your agent, while traditional HTTP and Websocket are used from the agent to the rest of your backend. This provides all of the benefits of WebRTC with none of the complexity.

The agents SDK includes components for all of the tricky parts of building realtime voice AI, such as sending audio input through an STT-LLM-TTS pipeline, reliable turn detection, LLM orchestration, and more. It includes plugins for nearly every AI provider, with more being added all the time. And the entire framework is open source with an active community.

Other framework features include:

• Voice, video, and text: Build agents that can process realtime input and produce output in any modality.
• Agent orchestration: Manage AI tasks and states inside a single session.
• Tool use: Define tools that are compatible with any LLM, and even forward tool calls to your frontend.
• Extensive integrations: Integrate with nearly every AI provider there is for LLMs, STT, TTS, and more.
• State-of-the-art turn detection: Use the custom turn detection model for lifelike conversation flow.
• Made for developers: Build your agents in code, not configuration.
• Production ready: Includes built-in worker orchestration, load balancing, and Kubernetes compatibility.
• Open source: The framework and entire LiveKit ecosystem are open source under the Apache 2.0 license.

How agents connect to LiveKit

When your agent code starts, it first registers with a LiveKit server (either self hosted or LiveKit Cloud to run as a "worker" process. The worker waits until it receives a dispatch request. To fulfill this request, the worker boots a "job" subprocess which joins the room. By default, your workers are dispatched to each new room created in your LiveKit project.

After your agent and user join a room, the agent and your frontend app can communicate using LiveKit WebRTC. This enables reliable and fast realtime communication in any network conditions. LiveKit also includes full support for telephony, so instead of a frontend app your user could simply be on the telephone.

To learn more about how LiveKit works overall, see the Intro to LiveKit guide.

Getting started

Follow these guides to learn more and get started with LiveKit Agents.