LiveKit Open-weight Hosted Models

Overview

LiveKit hosts several fast, open-weight models through LiveKit Inference. With LiveKit Inference, your agent runs on LiveKit's infrastructure to minimize latency. No separate provider API key is required, and usage and rate limits are managed through LiveKit Cloud. Pricing for LiveKit Inference is available on the pricing page .

LiveKit Inference

Use LiveKit Inference to access LiveKit's open-weight hosted models.

Usage

To use LiveKit hosted models, use the LLM class from the inference module. You can use this LLM in the Voice AI quickstart:

from livekit.agents import AgentSession, inference

session = AgentSession(
    llm=inference.LLM(
        model="google/gemma-4-31b-it",
        extra_kwargs={
            "max_completion_tokens": 1000
        }
    ),
    # ... tts, stt, vad, turn_handling, etc.
)

import { AgentSession, inference } from '@livekit/agents';

const session = new AgentSession({
    llm: new inference.LLM({ 
        model: "google/gemma-4-31b-it",
        modelOptions: { 
            max_completion_tokens: 1000 
        }
    }),
    // ... tts, stt, vad, turnHandling, etc.
});

Parameters

The following are parameters for configuring LiveKit open-weight hosted models with LiveKit Inference. For model behavior parameters like temperature and max_completion_tokens, see model parameters.

Model parameters

Pass the following parameters inside extra_kwargs (Python) or modelOptions (Node.js). For more details about each parameter in the list, see Inference parameters.

String descriptors

As a shortcut, you can also pass a model ID directly to the llm argument in your AgentSession:

from livekit.agents import AgentSession

session = AgentSession(
    llm="google/gemma-4-31b-it",
    # ... tts, stt, vad, turn_handling, etc.
)

import { AgentSession } from '@livekit/agents';

const session = new AgentSession({
    llm: "google/gemma-4-31b-it",
    // ... tts, stt, vad, turnHandling, etc.
});

Additional resources

The following resources provide more information about using open-weight hosted models with LiveKit Agents.