OpenAI Realtime API Quickstart

The MultimodalAgent class in the LiveKit Agents framework uses the OpenAI Realtime API for speech-to-speech interactions between AI voice assistants and end users.

If you're not using the OpenAI Realtime API, see the Voice agent with STT, LLM, TTS quickstart.

• OpenAI API Key
• Python 3.9+ or Node 20.17.0

The following steps take you through the process of creating a LiveKit account and using the LiveKit Sandbox to create an agent using the MultimodalAgent class. At the end of the quickstart, you'll have an agent and a frontend client you can use to talk to your agent.

Create an account or sign in to your LiveKit Cloud account.

A sandbox allows you to quickly create and deploy an agent locally, and test the agent using a frontend web client. To create a sandbox, follow these steps:

1. Sign in to LiveKit Sandbox.

2. Select Create app for the Voice assistant template.

3. Follow the Finish setting up your sandbox app instructions provided after you create your sandbox. The instructions include installing the LiveKit CLI and sandbox app.

   After you run the lk app create command, enter the following at the appropriate prompts:

   • For the Select Template prompt, select multimodal-agent-python or multimodal-agent-node.
   • Enter your OpenAI API key at the prompt.

4. Install dependencies and start your agent:

   PythonNode

   cd <your_sandbox_id>
   
   python3 -m venv venv
   source venv/bin/activate
   python3 -m pip install -r requirements.txt
   python3 agent.py dev

   Copy

1. Sign in to LiveKit Sandbox
2. In the Your Sandbox apps section, select Launch for <your_sandbox_id> sandbox.
3. Select Connect and start a conversation with your agent.

• Learn more in the OpenAI Realtime API integration guide.

• Customize your frontend client by using the frontend sandbox as a starting point. To clone the frontend sandbox locally, run the following command:

  lk app create --template voice-assistant-frontend

  Copy

• Create an agent that accepts incoming calls using SIP.

• Create an agent that makes outbound calls using SIP.