Build a scenario set

Overview

A single scenario shows whether your agent handled one conversation. A committed set of scenarios verifies that the behavior you depend on still holds as your agent changes.

Your scenarios live in a scenarios.yaml file committed with your agent code. Each scenario names an outcome your agent must reach, and the file as a whole is what every run verifies. Because the file lives in your repository, it's reviewable in a diff, it grows as your agent grows, and it runs the same way on your machine and in automation.

Agent Simulations grade a whole conversation rather than a single turn, so this set is the evidence that your agent still behaves as you expect. Keep it focused as it grows.

This topic covers building the set, organizing it, and keeping it reproducible. To learn more about the scenario format, see Write a scenario.

Requirements

Before you begin, make sure you have a LiveKit Cloud project and an authenticated CLI. For versions and setup, see Requirements.

What belongs in a scenario

Every scenario runs a whole conversation, spending tokens on the simulated user, your agent, and the judge. That's what makes outcome-based grading possible. Add scenarios selectively, so each one tests something the others don't.

Add a scenario when the answer depends on how a conversation unfolds:

• Multi-turn flow. Whether the agent reaches the goal when the caller changes their mind, backtracks, or supplies information out of order.
• Memory across turns. Whether details gathered early survive to the end of the call.
• Misuse resistance. Whether the agent holds its instructions under pressure.
• End state. Whether the conversation produced the right booking, transfer, or record.

Use a unit test instead when you can state the expected result for a single turn. Turn-level assertions on messages, tool calls, and arguments run in a normal test suite, are more cost effective than a simulation, and return in seconds. Use the Agent Console while you're still developing the behavior.

Start from generated scenarios

Writing scenarios from an empty file means inventing every case yourself. Generate a first pass from your agent source instead, then keep what's useful:

lk agent simulate text -n 10

The CLI reads your agent source, generates ten scenarios, and runs them. To keep the batch, press s in the run view and save it as scenarios.yaml in your project. On later runs, pass that file with --scenarios scenarios.yaml.

Generated scenarios are a starting point. Read them before you commit them. The generator infers intent from your code, so it produces plausible conversations rather than the ones your users actually have. Delete the scenarios that don't matter and rewrite the expectations that are too vague to grade. To learn more about best practices, see Write a scenario.

Organize your scenarios

When the file grows past a handful of scenarios, it needs structure to stay navigable when a run fails.

Keep scenario IDs stable

Every scenario carries an id, and the file itself carries one:

id: SCNG_9nF2kQwPbXtR
name: Room booking
scenarios:
  - id: SCN_4mTvH7yLdKpZ
    label: Book a king room for one night
    instructions: |
      PERSONA: Jordan Reyes, booking a one-night stay. Friendly, answers one
      question at a time, and volunteers nothing unasked.
      OPENING LINE: "Hi, I'd like to book a king room for the 9th."
      FACTS (reveal each only when asked, one per turn):
      - checking in 2026-06-09, checking out the 10th
      - one guest
      - email: jordan.reyes@example.com
      - phone: 5550142
      DO, IN ORDER:
      1. Book the room and answer each question one at a time.
      2. Don't end the call until the agent confirms the booking.
    agent_expectations: >
      Books one king room under Jordan Reyes, checking in 2026-06-09 and
      checking out 2026-06-10, and confirms the booking to the caller. Booking
      a different room type or different dates, or ending the call without a
      confirmed booking, is a fail.

If you edit the wording of a scenario without changing what it tests, keep the ID. The ID lets LiveKit correlate runs of the same scenario over time. Within a sampled run, the ID also groups the samples of each scenario.

An edit that changes what the scenario tests, such as the caller goal or the pass condition, makes it a new scenario. Delete its id line, and let the CLI generate a new one that keeps the history of the old scenario separate. Each ID must also be unique within the file.

The CLI requires IDs. When the file or any scenario in it lacks one, the CLI offers to add the missing IDs before the run starts, preserving your comments and formatting. Accept the offer and commit the file. Scenarios you save from a generated run already have IDs.

Group scenarios with tags

Tags are arbitrary key/value pairs used to group and filter runs:

scenarios:
  - id: SCN_4mTvH7yLdKpZ
    label: Book a king room for one night
    tags:
      feature: room_booking
      suite: smoke

Two tagging schemes are worth applying from the start. Tag by feature so a failure points at the part of the agent that owns it. Tag by suite so you can separate a fast subset from the full set, and run only the subset when a full run takes too long.

Decide what to cover

A file that only covers the conversations that go well passes until a user does something unusual. Cover these areas:

• The paths that carry traffic. The three or four things most callers actually do.
• The edges of each path. Missing information, changes of mind, and requests your agent has to decline.
• Tool failures. What your agent says when a backend is down or returns nothing.
• Misuse. Attempts to talk your agent out of its instructions.
• Anything that broke once. Every production failure you've fixed belongs here permanently.

Scenarios drawn from real failures are the most durable. A file written in one sitting reflects the cases you could think of at the time, while one grown from production keeps pace with how callers actually use your agent.

Make scenarios reproducible

A scenario is only useful if the same run produces the same verdict. Two things break that.

Dates are the first. A scenario that books a room "next Tuesday" grades differently every week, so write absolute dates and pin the clock your agent reads. To learn more, see Pin time-sensitive scenarios.

Live backends are the second. A scenario that reads your real database depends on whatever that database holds at the time. Seed deterministic state from userdata instead, and read it through the simulation context so your production path stays unchanged. To learn more, see Connect scenarios to your agent.

Run them automatically

After you commit the file, configure your pipeline to run text simulations on every commit and audio simulations on a nightly or pre-release schedule. To learn more, see Run simulations in CI.

Additional resources

These topics cover writing the individual scenarios, finding new ones, and the layers that run beside them.