Home AI Agents Telephony Recipes Reference

Ollama integration guide

How to run models locally using Ollama with LiveKit Agents.

Overview

Ollama is an open source LLM engine that you can use to run models locally with an OpenAI-compatible API. Ollama support is available using the OpenAI plugin for LiveKit Agents.

Usage

Install the OpenAI plugin to add Ollama support:

pip install "livekit-agents[openai]~=1.0"

Create an Ollama LLM using the with_ollama method:

from livekit.plugins import openai

session = AgentSession(
    llm=openai.LLM.with_ollama(
        model="llama3.1",
        base_url="http://localhost:11434/v1",
    ),
    # ... tts, stt, vad, turn_detection, etc.
)

Parameters

This section describes some of the available parameters. See the plugin reference for a complete list of all available parameters.

modelstringOptionalDefault: llama3.1

Ollama model to use. For a list of available models, see Ollama models.

base_urlstringOptionalDefault: http://localhost:11434/v1

Base URL for the Ollama API.

temperaturefloatOptional

Controls the randomness of the model's output. Higher values (e.g., 0.8) make the output more random, while lower values (e.g., 0.2) make it more focused and deterministic.

Links

The following links provide more information about the Ollama integration.

Python package

The livekit-plugins-openai package on PyPI.

Plugin reference

Reference for the with_ollama method of the OpenAI LLM plugin.

GitHub repo

View the source or contribute to the LiveKit OpenAI LLM plugin.

Ollama docs

Ollama site and documentation.

Voice AI quickstart

Get started with LiveKit Agents and Ollama.