Ollama LLM plugin guide

Available in

Python

Node.js

Overview

This plugin allows you to use a local Ollama instance as an LLM provider for your voice agents.

Usage

Install the OpenAI plugin to add Ollama support:

pip install "livekit-agents[openai]~=1.2"

Create an Ollama LLM using the with_ollama method:

from livekit.plugins import openai

session = AgentSession(
    llm=openai.LLM.with_ollama(
        model="llama3.1",
        base_url="http://localhost:11434/v1",
    ),
    # ... tts, stt, vad, turn_detection, etc.
)

Parameters

This section describes some of the available parameters. See the plugin reference for a complete list of all available parameters.

modelstringOptionalDefault: llama3.1

Ollama model to use. For a list of available models, see Ollama models.

base_urlstringOptionalDefault: http://localhost:11434/v1

Base URL for the Ollama API.

temperaturefloatOptional

Controls the randomness of the model's output. Higher values (e.g., 0.8) make the output more random, while lower values (e.g., 0.2) make it more focused and deterministic.

Links

The following links provide more information about the Ollama integration.

Python package

The livekit-plugins-openai package on PyPI.

Plugin reference

Reference for the with_ollama method of the OpenAI LLM plugin.

GitHub repo

View the source or contribute to the LiveKit OpenAI LLM plugin.

Ollama docs

Ollama site and documentation.

Voice AI quickstart

Get started with LiveKit Agents and Ollama.