Testing and evaluation

Overview

Testing and evaluation help ensure your agent behaves as expected as your application evolves. LiveKit Agents includes tools for validating individual behaviors during development and evaluating complete conversations before deployment.

Behavioral tests verify specific interactions and expected outcomes. They integrate with your existing test suite using pytest  (Python) or Vitest  (Node.js), making them suitable for unit and integration testing.

Agent simulations run end-to-end conversations between your agent and an LLM-driven user, then evaluate the results across the full interaction. Use simulations to test multi-turn behavior, reproduce edge cases, and compare changes to your agent over time.

Together, these tools help you validate changes, identify regressions, and iterate on your agent without breaking existing functionality.

Testing options

LiveKit Agents supports several approaches to testing, depending on what you want to validate.

What to test

Test your agent in the following areas:

• Expected behavior: Does your agent respond with the right intent and tone for common use cases?
• Tool usage: Are tools called with the expected arguments and context?
• Error handling: How does your agent respond to invalid input or tool failures?
• Grounding: Does your agent stay factual and avoid hallucinating information?
• Misuse resistance: How does your agent respond to attempts to misuse or manipulate it?

Use the test framework for turn-level behaviors such as tool usage and error handling. Use simulations to evaluate behaviors that span multiple turns, such as conversation flow, memory, and misuse resistance.

Example test

Here is a simple behavioral test for the agent created in the voice AI quickstart. It ensures that the agent responds with a friendly greeting and offers assistance.

from livekit.agents import AgentSession, inference

from agent import Assistant

@pytest.mark.asyncio
async def test_assistant_greeting() -> None:
    async with (
        inference.LLM(model="google/gemma-4-31b-it") as llm,
        AgentSession(llm=llm) as session,
    ):
        await session.start(Assistant())

        result = await session.run(user_input="Hello")

        await result.expect.next_event().is_message(role="assistant").judge(
            llm, intent="Makes a friendly introduction and offers assistance."
        )

        result.expect.no_more_events()

import { inference, initializeLogger, voice } from '@livekit/agents';
import { describe, it, beforeAll, afterAll } from 'vitest';
// Import your agent class
import { Agent } from './agent';

// Initialize logger to suppress CLI output
initializeLogger({ pretty: false, level: 'warn' });

const { AgentSession } = voice;

describe('Assistant', () => {
  let session: voice.AgentSession;
  let llm: inference.LLM;

  beforeAll(async () => {
    llm = new inference.LLM({ model: 'google/gemma-4-31b-it' });
    session = new AgentSession({ llm });
    await session.start({ agent: new Agent() });
  });

  afterAll(async () => {
    await session?.close();
  });

  it('should greet and offer assistance', async () => {
    const result = await session.run({ userInput: 'Hello' }).wait();

    await result.expect
      .nextEvent()
      .isMessage({ role: 'assistant' })
      .judge(llm, {
        intent: 'Makes a friendly introduction and offers assistance.',
      });

    result.expect.noMoreEvents();
  });
});

For the full testing API, including setup, assertions, mocking, and multi-turn testing, see Test framework.

Verbose output

Environment variables can turn on detailed output for each agent execution.

LIVEKIT_EVALS_VERBOSE=1 uv run pytest -s -o log_cli=true <your-test-file>

LIVEKIT_EVALS_VERBOSE=1

Sample verbose output:

evals/test_agent.py::test_offers_assistance 
+ RunResult(
   user_input=`Hello`
   events:
     [0] ChatMessageEvent(item={'role': 'assistant', 'content': ['Hi there! How can I assist you today?']})
)
- Judgment succeeded for `Hi there! How can I assist...`: `The message provides a friendly greeting and explicitly offers assistance, fulfilling the intent.`
PASSED

stdout | conversation-history.test.ts > RunResult > should greet user by name

+ RunResult {
    userInput: "What's my name?"
    events: [
      [0] { type: "message", role: "assistant", content: "Your name is Alice.", interrupted: false }
    ]
  }

stdout | conversation-history.test.js > RunResult > should greet user by name
- Judgment succeeded for `Your name is Alice.`: `The message explicitly states the user's name is Alice, fulfilling the intent to remember and mention the user's name.`

Integrating with CI

The testing helpers work live against your LLM provider to test real agent behavior. If you're using LiveKit Inference, set LIVEKIT_API_KEY and LIVEKIT_API_SECRET in your CI environment. If you're using a plugin directly, set the appropriate provider API keys instead. Testing does not make a LiveKit room connection.

For GitHub Actions, see the guide on using secrets in GitHub Actions .

Agent simulations also fit into CI. A checked-in scenarios.yaml is reproducible, so you can run the same scenarios on every change. Simulations run on LiveKit Cloud through the authenticated CLI rather than against your LLM provider directly.

Considerations

The following considerations apply when testing agents:

• get_job_context() is unavailable in test environments and raises a RuntimeError when called. If your agent uses get_job_context(), avoid testing code paths that invoke it, or mock the call using unittest.mock (Python-only).

• When testing agents that use task groups, consider testing each task in isolation as well as the overall flow. Test transitions between tasks, regression to previous steps, and proper completion with summarized results. For specific guidelines, see Best practices for testing task groups.

• Agent simulations are in beta and currently support Python agents only. Simulations run on LiveKit Cloud, in parallel up to your project's concurrency limit.

Third-party testing tools

First-party simulations run in text mode. To test the full audio pipeline or monitor deployed agents in production, consider these third-party services:

Additional resources

These examples and resources provide more help with testing and evaluation.

Python agent evals

Drive-thruFront-desk agent

Node.js agent evals

Drive-thruFront-desk agent

Agent starter project

Starter project with a complete testing integration.

Agent starter project (Node.js)

Starter project with a complete testing integration.

Testing framework API reference (Python)

API reference for the RunResult class.

Testing framework API reference (Node.js)

API reference for the RunResult class.

Agent simulations

Run your agent end to end against a simulated user and grade the whole conversation.