Agent Simulations

Overview

Agent Simulations evaluate your agent against an LLM-driven simulated user that follows a scenario from start to finish, then judge whether the agent met your expectations. Unlike unit tests, which assert on individual turns, simulations evaluate the conversation as a whole. The simulated user pursues a goal, your agent responds with its real logic and tools, and the result includes a pass or fail verdict and the full transcript.

Agent Simulations run in two modes. In text mode the simulated user exchanges text with your agent, testing your LLM, tools, and conversation logic. In audio mode it speaks and listens over a real audio track, testing the whole STT-LLM-TTS pipeline and measuring how the call sounds.

Run a few scenarios while you iterate, or a larger batch to catch regressions before you ship. Agent Simulations executes in parallel on LiveKit Cloud, subject to two separate concurrency limits: a per-run limit that controls how many of a run's simulations execute at once (15 by default), and a per-project limit of 30 simulations executing at once across all runs.

Run text simulations on every commit, and save audio runs for a nightly or pre-release pass. To learn more, see Run simulations in CI.

Agent Simulations execute in parallel on LiveKit Cloud, subject to two separate concurrency limits. A per-run limit controls how many of a run's simulations execute at once, 15 by default. A per-project limit of 30 running simulations applies across all runs.

Requirements

Before you begin, make sure you have the following:

• LiveKit CLI v2.16.4 or later for Python, or v2.16.7 or later for Node.js. Audio simulations require v2.18.3 or later. To install or upgrade lk, see Install the CLI.
• LiveKit Agents 1.6.6 or later (Python), or @livekit/agents 1.6.0 or later (Node.js).
• A LiveKit Cloud project. Agent Simulations run on LiveKit Cloud using your project's credentials, so the CLI must be authenticated to a project.

How it works

A simulation run has three components:

• The simulated user. An LLM follows the scenario's instructions (a persona and a goal for the simulated user) and chats with your agent until the conversation reaches a natural end.
• Your agent. By default, the CLI starts your real agent as a local worker and dispatches jobs for the simulated rooms to it. Your entrypoint, tools, and conversation logic all run unchanged. To run against an agent that's already running, use --agent-name.
• The judge. When the conversation ends, the simulator judges the transcript against the scenario's agent_expectations and records a verdict. You can layer your own check on top to grade against real end state. To learn more, see Grade on final state.

By default, the simulated user interacts over text, so a run exercises your LLM, tools, and logic without the STT and TTS pipeline. To learn more, see Text mode.

Every simulation spends tokens on the simulated user, on your agent, and on the judge. That spend is what lets a simulation catch behavior the other checks in the testing lifecycle can't reach.

Run a simulation

Run simulations from your agent's project directory with the LiveKit CLI. Each mode and action has its own subcommand, and text runs a simulation in the default text mode:

lk agent simulate text -n 10

With no scenario file, the CLI generates scenarios from your agent source. Because this uploads your code to LiveKit Cloud for the generator to read, the CLI asks you to confirm first. After you confirm, it starts your agent, dispatches the generated scenarios, and reports results live, with a link to the run in the dashboard.

The CLI runs your agent as a local worker by default. It registers the worker under a temporary name, dispatches jobs for the simulated rooms to it, and stops it when the run finishes. Your entrypoint, tools, and conversation logic run unchanged.

Run against a live agent

To grade an agent that's already running instead of one spawned from your working directory, pass --agent-name with the registered name for the agent:

lk agent simulate text --agent-name my-agent

To target the default agent for the project, the one that auto-joins every room, pass an empty string "". Running against a live agent requires a scenario file, since there's no local source to generate scenarios from. The CLI reads ./scenarios.yaml from your working directory, or pass --scenarios to name a different file.

For every subcommand and option, see lk agent simulate in the CLI reference.

Work with scenarios.yaml

Generating from source bootstraps a set of scenarios, but the workflow is iterative. Capture the scenarios you care about in a scenarios.yaml file, run them, refine the ones that surface bugs, and run them again. A checked-in scenario file is reproducible, reviewable, and runnable from automation.

Run every scenario in the file with a single command:

lk agent simulate text

The file carries an ID, a name, and a list of scenarios. Each scenario holds the instructions the simulated user follows and the expectations the judge grades against. To learn more about the format and the fields, see Write a scenario. To decide which scenarios the file needs, see Build a scenario set. To run it automatically, see Run simulations in CI.

Connect scenarios to your agent

To use scenario userdata, your agent must read it from the simulation context. In your entrypoint, call ctx.simulation_context() (Python) or ctx.simulationContext() (Node.js) to detect a simulated run and use deterministic, per-scenario state. It returns a SimulationContext during a simulation, or None (Python) / undefined (Node.js) in production. Your agent receives userdata as decoded JSON, with the keys exactly as written in your scenarios.yaml file. For example, read room_type rather than roomType.

The simulation context is available immediately from the job's dispatch attributes, so you can read it as soon as your entrypoint runs. In most agents, that means reading it right after you connect:

from livekit.agents import AgentServer, JobContext, mock_tools

server = AgentServer()


@server.rtc_session(on_simulation_end=on_simulation_end)
async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    tool_mocks = {}
    if sim := ctx.simulation_context():
        # Seed deterministic state from the scenario's userdata.
        inventory = build_fake_inventory(sim.userdata()["available_rooms"])
        tool_mocks = build_tool_mocks(inventory)
    else:
        inventory = production_inventory()

    session = AgentSession(userdata=Userdata(inventory=inventory), ...)

    # Mock the agent's tools under simulation so runs are reproducible. The LLM
    # still sees the real tool schemas; only execution is intercepted.
    mock_tools(MyAgent, tool_mocks, session=session)
    await session.start(agent=MyAgent(), room=ctx.room)

import { type JobContext, defineAgent, inference, tool, voice } from '@livekit/agents';
import { z } from 'zod';

export default defineAgent({
  entry: async (ctx: JobContext) => {
    await ctx.connect();

    // Under a simulation, seed deterministic state from the scenario's userdata;
    // otherwise use your real backend. The tool below reads `inventory`, so the
    // LLM sees the same tool schemas either way. Only the data changes.
    // Userdata values are typed `unknown`, so `buildFakeInventory(rooms: unknown)` validates.
    const sim = ctx.simulationContext();
    const inventory = sim
      ? buildFakeInventory(sim.userdata()['available_rooms'])
      : productionInventory();

    const agent = voice.Agent.create({
      instructions: 'You are a hotel booking assistant.',
      tools: [
        tool({
          name: 'getAvailability',
          description: 'List room types available on a date.',
          parameters: z.object({ date: z.string() }),
          execute: async ({ date }) => inventory.roomsFor(date),
        }),
      ],
    });

    const session = new voice.AgentSession({
      llm: new inference.LLM({ model: 'openai/gpt-4.1-mini' }),
      // Plus your usual stt, tts, and turn detection.
      // A text simulation drops them automatically.
    });

    await session.start({ agent, room: ctx.room });
  },
});

Checking for a simulation context keeps the production code path unchanged. In a real session, the context is absent, so the agent connects to its real backends. The front-desk example  (Python) seeds a deterministic calendar this way, and the hotel receptionist example  (Python) seeds a SQLite database. Both are useful references for wiring simulations into a production-shaped agent.

Grade on the final state

The simulator's verdict is an LLM evaluation of the conversation. That isn't always sufficient, for example, a polished conversation can still book the wrong room. Register an on_simulation_end (Python) or onSimulationEnd (Node.js) callback to validate your agent's final state and fail the simulation if it doesn't match the expected result:

from livekit.agents import SimulationContext


async def on_simulation_end(ctx: SimulationContext) -> None:
    expected = ctx.userdata().get("expected_state")
    if not expected:
        return  # grade on the conversation alone

    session = ctx.job_context.primary_session
    if not booking_matches(session.userdata.db, expected):
        ctx.fail(reason="final DB state diverged from the expected booking")

import { type JobContext, type SimulationContext, defineAgent } from '@livekit/agents';

// Node.js has no equivalent of Python's `ctx.job_context.primary_session`, so
// keep a handle to the state you want to grade, keyed by the job.
const gradedState = new WeakMap<JobContext, Db>();

export default defineAgent({
  entry: async (ctx: JobContext) => {
    const sim = ctx.simulationContext();
    const db = sim ? seedDb(sim.userdata()) : productionDb();
    if (sim) gradedState.set(ctx, db);
    // Start the session as usual; the agent's tools read and write `db`.
  },
  onSimulationEnd: (ctx: SimulationContext) => {
    const expected = ctx.userdata()['expected_state'];
    if (!expected) return; // grade on the conversation alone

    const db = gradedState.get(ctx.jobContext);
    if (db && !bookingMatches(db, expected)) {
      ctx.fail('final DB state diverged from the expected booking');
    }
  },
});

Key points:

• Your check can only fail a simulation. It can't override a failed simulator verdict. The final result is the logical AND of the simulator's verdict and your check. Calling ctx.fail() fails a simulation the simulator passed, but it can't pass one the simulator failed. If you don't call ctx.fail(), the simulator's verdict stands.
• Use ctx.simulator_verdict (Python) or ctx.simulatorVerdict (Node.js) to inspect the simulator's decision, including its success flag and reason. It's only available inside the callback.
• Access the final agent state to compare it against the expected state defined in the scenario user data. In Python, read ctx.job_context.primary_session for the room, session, and user data your agent accumulated. Node.js has no equivalent accessor, so keep a reference to the state you want to grade, such as the backend you seeded from userdata(), and read it in the callback as shown in the preceding example.

This pattern turns simulation into an evaluation by checking both the conversation and the state your agent ended in. A simulation passes only if both are correct.

Text mode

Agent Simulations run in text mode by default. The simulated user exchanges text with your agent, so the run tests your LLM, tools, and conversation logic without the STT and TTS pipeline. Text mode is faster, more cost effective, and more deterministic. Use it for CI runs and for any scenario that doesn't exercise speech.

Under a text simulation, the framework automatically disables STT, TTS, VAD, and audio input and output, so your agent runs its LLM and tools unchanged without any audio setup.

To check which mode a simulation is running in, read sim.simulation_mode (Python) or sim.simulationMode (Node.js) from the SimulationContext. It returns a SimulationMode enum value. Compare it against SimulationMode.SIMULATION_MODE_TEXT (Python) or SimulationMode.TEXT (Node.js). An unspecified mode resolves to text.

Audio mode runs the same scenarios through your agent's full media pipeline and scores the call itself. To learn more, see Audio simulations.

Audio simulations

In audio mode, the simulated user speaks your scenario aloud, listens to your agent, and interrupts as a real caller would. Your agent runs its full STT-LLM-TTS pipeline against that audio, and the run scores the aspects that only speech exposes:

• Turn-taking. Whether the agent starts speaking before the caller finishes, or leaves a caller who has finished waiting.
• Interruption handling. Whether the agent yields to a barge-in and correctly distinguishes a brief acknowledgment from a turn.
• Transcription. Numbers, spelled-out names, addresses, and confirmation codes, in both directions.
• Perceived speed and speech quality. The latency the caller hears, and whether names and amounts are pronounced correctly.

Run a scenario file in audio mode with the audio subcommand:

lk agent simulate audio

How an audio simulation runs

Scenarios, the judge, and the pass or fail verdict work the same way in both text and audio modes. Audio mode adds the following behavior:

• The simulated user joins the room as a participant. It publishes an audio track and subscribes to your agent's, so your agent sees a caller rather than a text stream.
• Your configured audio models run. STT, TTS, and VAD use the models configured for your agent, and their providers are billed accordingly. An agent with no STT or TTS configured has nothing to exercise in audio mode.
• The run executes in real time at normal inference priority. This means measured response times match production. Text runs are dispatched to LiveKit Inference as low-priority batch load because nothing is waiting on the results.
• Usage is metered in audio turns. Audio turns are tracked separately from text turns.

What an audio run measures

Alongside the verdict, an audio run measures call quality per turn and aggregated over the run, so you can track real-world quality over time rather than correctness alone. These metrics appear in the dashboard and in the exported JSON.

Responsiveness. The primary metric is the end-to-end latency the caller heard, reported at p50, p95, and p99. A positive value indicates a gap before the agent responded, while a negative value means the agent talked over the caller. Track this separately from the latency the agent reports. The agent measures when it started producing audio, while the caller measures when they heard it. The difference between the two is what the user perceives. The run also breaks the pipeline down stage by stage, reporting STT and endpointing delay, LLM time-to-first-token and time-to-first-sentence, tokens per second, and TTS time-to-first-byte. This helps identify the stage responsible for slow responses.

Turn-taking. A turn-taking score with the specific failures behind it: end-of-turn mispredictions where the agent started speaking before the caller finished, time to yield after a barge-in, false interruptions where the agent stopped when there was no interruption, the share of overlapping speech, unfilled silences after a natural pause, and caller turns that the agent never answered.

Speech accuracy. Word and character error rates in both directions: what the agent heard compared with what the caller said, and what the caller heard compared with what the agent said. Key entities, such as names, IDs, confirmation codes, and amounts, are scored separately, with recall distinguishing between an entity the agent never recognized and one it recognized but later lost.

Conversation quality. The overall score, conciseness, and whole-call issue flags such as unnecessary tool calls, information loss, redundant statements, and poor question quality. These combine into a conversation-progression score. Because these measures are judged from the dialog, they're reported for both modes.

Simulate a degraded connection

Real callers connect from noisy environments and unreliable networks. These flags degrade the simulated user's audio so you can test how your agent responds:

Combine them to model a worst-case caller:

lk agent simulate audio --background-noise --packet-loss

Every option on text also applies to audio. For example, run a degraded audio pass against an already-running agent:

lk agent simulate audio --agent-name my-agent --low-quality-microphone

Export a run

To analyze a finished run outside the CLI, export it as JSON. The export includes the run, its summary, and the exact per-job chat contexts, which is useful for archiving results as a build artifact or comparing behavior between runs:

lk agent simulate export <run-id> > run.json

To reopen a previous run in the terminal, pass its ID to view:

lk agent simulate view <run-id>

To find a run ID, list the most recent runs for the project:

lk agent simulate list

Example agents and scenarios

These open source agents run simulations as part of their own test setup. Read them to see complete scenario files alongside the agent code they grade.

Additional resources

These topics cover writing scenarios, choosing which ones to commit, and running them.