Hosted by LiveKit

Overview

LiveKit hosts a fast, open-weight model through LiveKit Inference and tunes the deployment for the low latency that voice agents need, making it the recommended default LLM for your agents. You don't manage a separate provider API key, and usage and rate limits are handled through LiveKit Cloud. See the pricing page  for current rates.

LiveKit Inference

Use LiveKit Inference to access LiveKit's open-weight hosted models.

A checkmark in the EU endpoint column means the model has a dedicated EU endpoint, in addition to the global endpoint. To learn more, see Data residency.

Usage

To use LiveKit hosted models, use the LLM class from the inference module. You can use this LLM in the Voice AI quickstart:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.LLM(
        model="google/gemma-4-31b-it",
        extra_kwargs={
            "max_completion_tokens": 1000
        }
    ),
    # ... tts, stt, vad, turn_handling, etc.
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    llm: new inference.LLM({ 
        model: "google/gemma-4-31b-it",
        modelOptions: { 
            max_completion_tokens: 1000 
        }
    }),
    // ... tts, stt, vad, turnHandling, etc.
});

Parameters

The following are parameters for configuring LiveKit open-weight hosted models with LiveKit Inference. For model behavior parameters like temperature and max_completion_tokens, see model parameters.

Model parameters

Pass the following parameters inside extra_kwargs (Python) or modelOptions (Node.js). For more details about each parameter in the list, see Inference parameters.

String descriptors

As a shortcut, you can also pass a model ID directly to the llm argument in your AgentSession:

from livekit.agents import AgentSession

session = AgentSession(
    llm="google/gemma-4-31b-it",
    # ... tts, stt, vad, turn_handling, etc.
)

import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
    llm: "google/gemma-4-31b-it",
    // ... tts, stt, vad, turnHandling, etc.
});

Additional resources

The following resources provide more information about using open-weight hosted models with LiveKit Agents.