Overview
Ollama is an open source LLM engine that you can use to run models locally with an OpenAI-compatible API. Ollama support is available using the OpenAI plugin for LiveKit Agents.
Usage
Install the OpenAI plugin to add Ollama support:
pip install "livekit-agents[openai]~=1.0"
Create an Ollama LLM using the with_ollama
method:
from livekit.plugins import openaisession = AgentSession(llm=openai.LLM.with_ollama(model="llama3.1",base_url="http://localhost:11434/v1",),# ... tts, stt, vad, turn_detection, etc.)
Parameters
This section describes some of the available parameters. See the plugin reference for a complete list of all available parameters.
Ollama model to use. For a list of available models, see Ollama models.
Base URL for the Ollama API.
Controls the randomness of the model's output. Higher values (e.g., 0.8) make the output more random, while lower values (e.g., 0.2) make it more focused and deterministic.
Links
The following links provide more information about the Ollama integration.
Python package
The livekit-plugins-openai
package on PyPI.
Plugin reference
Reference for the with_ollama
method of the OpenAI LLM plugin.
GitHub repo
View the source or contribute to the LiveKit OpenAI LLM plugin.
Ollama docs
Ollama site and documentation.
Voice AI quickstart
Get started with LiveKit Agents and Ollama.