Agent simulations

Overview

Simulations evaluate your agent by running it against an LLM-driven simulated user that plays out a scenario from start to finish, then judges whether the agent met your expectations. Unlike the test framework, which asserts on individual turns, simulations evaluate the conversation as a whole: the simulated user pursues a goal, your agent responds with its real logic and tools, and the result includes a pass or fail verdict and the full transcript.

Run a few scenarios while you iterate, or a larger batch to catch regressions before you ship. Simulations execute in parallel on LiveKit Cloud, up to your project's maximum concurrency limit.

Requirements

Before you begin, make sure you have the following:

• LiveKit CLI v2.16.4 or later. See Install the CLI to install or upgrade lk.
• LiveKit Agents 1.6.6 or later (Python).
• A LiveKit Cloud project. Simulations run on LiveKit Cloud using your project's credentials, so the CLI must be authenticated to a project.

How it works

A simulation run has three components:

• The simulated user. An LLM follows the scenario's instructions (a persona and a goal for the simulated user) and chats with your agent until the conversation reaches a natural end.
• Your agent. By default, the CLI starts your real agent as a local worker and dispatches jobs for the simulated rooms to it. Your entrypoint, tools, and conversation logic all run unchanged. To run against an agent that's already running, use --agent-name.
• The judge. When the conversation ends, the simulator judges the transcript against the scenario's agent_expectations and records a verdict. You can layer your own check on top to grade against real end state. See Grade on final state.

By default, the simulated user interacts over text (see Text and audio modes), so a run exercises your LLM, tools, and logic without the STT and TTS pipeline. This makes runs fast, cheap, and deterministic enough to put in CI.

Run a simulation

Run simulations from your agent's project directory with the LiveKit CLI:

lk agent simulate -n 10

With no scenario file, the CLI generates scenarios from your agent's source. Because this uploads your code to LiveKit Cloud for the generator to read, the CLI asks you to confirm first. It then starts your agent, dispatches the generated scenarios, and reports results live, with a link to the run in the dashboard.

Command options

lk agent simulate accepts the following options:

The entrypoint is auto-detected, or you can pass it as a positional argument: lk agent simulate agent.py.

Run against a live agent

By default, lk agent simulate starts your agent as a local worker, registers it under a temporary name, dispatches jobs for the simulated rooms to it, and stops the worker when the run finishes.

To run against an agent that's already running instead of spawning one locally, pass --agent-name:

lk agent simulate --scenarios scenarios.yaml --agent-name my-agent

Pass the agent's registered name, or an empty string ("") to target the project's default agent that auto-joins every room. Running against a live agent requires --scenarios, since there's no local source to generate scenarios from.

Iterate with scenarios.yaml

Generating from source is a fast way to bootstrap a set of scenarios, but the real workflow is iterative: capture the scenarios you care about in a scenarios.yaml file, run them, refine the ones that surface bugs, and re-run. A checked-in scenario file is the source of truth. It's reproducible, reviewable, and runnable in CI.

lk agent simulate --scenarios scenarios.yaml

A scenario file is a named group of scenarios:

name: Room booking
scenarios:
  - label: Book a king room for one night
    instructions: >
      You are Jordan Reyes (email jordan.reyes@example.com, phone 5550142).
      Book a king room for the night of 2026-06-09, checking out the 10th,
      just you. No breakfast, no late checkout. Pay with the card ending 4242.
    agent_expectations: Room booked successfully
    tags:
      feature: room_booking
    userdata:
      guest:
        first_name: Jordan
        last_name: Reyes
      room_type: king

Each scenario has the following fields:

Connect scenarios to your agent

The userdata in a scenario is only useful if your agent reads it. In your entrypoint, call ctx.simulation_context() to detect a simulated run and swap in deterministic, per-scenario state. It returns a SimulationContext during a simulation, or None in production.

The simulation context is available immediately from the job's dispatch attributes, so you can call ctx.simulation_context() as soon as your entrypoint runs. In most agents, that means calling it right after you connect:

from livekit.agents import AgentServer, JobContext, mock_tools

server = AgentServer()


@server.rtc_session(on_simulation_end=on_simulation_end)
async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    tool_mocks = {}
    if sim := ctx.simulation_context():
        # Seed deterministic state from the scenario's userdata.
        calendar = build_fake_calendar(sim.userdata()["available_slots"])
        tool_mocks = build_tool_mocks(calendar)
    else:
        calendar = production_calendar()

    session = AgentSession(userdata=Userdata(cal=calendar), ...)

    # Mock the agent's tools under simulation so runs are reproducible. The LLM
    # still sees the real tool schemas; only execution is intercepted.
    mock_tools(MyAgent, tool_mocks, session=session)
    await session.start(agent=MyAgent(), room=ctx.room)

Passing session to mock_tools keeps mocks active for the session's lifetime. To learn more, see Mock tools for a running session.

This keeps your production code path untouched: in a real session simulation_context() returns None and your agent connects to its real backends. The front-desk example  seeds a deterministic calendar this way, and the hotel receptionist example  seeds a SQLite database. Both are useful references for wiring simulations into a production-shaped agent.

Grade on the final state

The simulator's verdict is an LLM evaluation of the conversation. That isn't always sufficient, for example, a polished conversation can still book the wrong room. Register an on_simulation_end callback to validate your agent's final state and fail the simulation if it doesn't match the expected result:

from livekit.agents import SimulationContext


async def on_simulation_end(ctx: SimulationContext) -> None:
    expected = ctx.userdata().get("expected_state")
    if not expected:
        return  # grade on the conversation alone

    session = ctx.job_context.primary_session
    if not booking_matches(session.userdata.db, expected):
        ctx.fail(reason="final DB state diverged from the expected booking")

Key points:

• Your check can only fail a simulation. It can't override a failed simulator verdict. The final result is the logical AND of the simulator's verdict and your check. Calling ctx.fail() fails a simulation the simulator passed, but it can't pass one the simulator failed. If you don't call ctx.fail(), the simulator's verdict stands.
• Use ctx.simulator_verdict to inspect the simulator's decision, including its success flag and reason. It's only available inside on_simulation_end.
• Access your agent's final state through ctx.job_context.primary_session, which contains the same room, session, and user data your agent accumulated during the simulation. Compare it against the expected state defined in the scenario's user data.

This pattern turns simulation into an evaluation by checking both the conversation and the resulting application state. A simulation passes only if both are correct.

Text and audio modes

Simulations run in text mode by default. The simulated user exchanges text with your agent, so the run covers your LLM, tools, and conversation logic without the STT and TTS pipeline. Text mode is faster, cheaper, and more deterministic, which makes it the right default for iteration and CI.

If you need to branch on the mode, read sim.simulation_mode from the SimulationContext.

Additional resources