Build a scenario set

Overview

A single scenario shows whether your agent handled one conversation. A committed set of scenarios verifies that the behavior you depend on still holds as your agent changes.

Your scenarios live in a scenarios.yaml file committed with your agent code. Each scenario names an outcome your agent must reach, and the file as a whole is what every run verifies. Because the file lives in your repository, it's reviewable in a diff, it grows as your agent grows, and it runs the same way on your machine and in automation.

Agent Simulations grade a whole conversation rather than a single turn, so this set is the evidence that your agent still behaves as you expect. Keep it focused as it grows.

This topic covers building the set, organizing it, and keeping it reproducible. To learn more about the scenario format, see Write a scenario.

Requirements

Before you begin, make sure you have a LiveKit Cloud project and an authenticated CLI. For versions and setup, see Requirements.

What belongs in a scenario

Every scenario runs a whole conversation, spending tokens on the simulated user, your agent, and the judge. That's what makes outcome-based grading possible. Add scenarios selectively, so each one tests something the others don't.

Add a scenario when the answer depends on how a conversation unfolds:

• Multi-turn flow. Whether the agent reaches the goal when the caller changes their mind, backtracks, or supplies information out of order.
• Memory across turns. Whether details gathered early survive to the end of the call.
• Misuse resistance. Whether the agent holds its instructions under pressure.
• End state. Whether the conversation produced the right booking, transfer, or record.

Use a unit test instead when you can state the expected result for a single turn. Turn-level assertions on messages, tool calls, and arguments run in a normal test suite, are more cost effective than a simulation, and return in seconds. Use the Agent Console while you're still developing the behavior.

Start from generated scenarios

Writing scenarios from an empty file means inventing every case yourself. Generate a first pass from your agent source instead, then keep what's useful:

lk agent simulate text -n 10

The CLI reads your agent source, generates ten scenarios, runs them, and offers to save them to scenarios.yaml in your project. Accept the offer, and later runs read that file automatically.

Generated scenarios are a starting point. Read them before you commit them. The generator infers intent from your code, so it produces plausible conversations rather than the ones your users actually have. Delete the scenarios that don't matter and rewrite the expectations that are too vague to grade. To learn more about best practices, see Write a scenario.

Grow your scenarios from real conversations

The scenarios worth having are the ones that describe something that actually went wrong. Once your agent is running, derive scenarios from recorded sessions rather than inventing them:

lk agent simulate generate <session-id>

Each session becomes a scenario appended to scenarios.yaml. The command creates scenarios without running them. To learn more about this workflow, see Derive a scenario from a session.

Organize your scenarios

When the file grows past a handful of scenarios, it needs structure to stay navigable when a run fails.

Keep scenario IDs stable

Every scenario carries an id, and the file itself carries one:

id: SCNG_9nF2kQwPbXtR
name: Room booking
scenarios:
  - id: SCN_4mTvH7yLdKpZ
    label: Book a king room for one night
    instructions: >
      You are Jordan Reyes (email jordan.reyes@example.com, phone 5550142).
      Book a king room for the night of 2026-06-09, checking out the 10th.
    agent_expectations: Room booked successfully

Labels, instructions, and expectations all change over the life of a scenario. The ID doesn't, so it's what correlates runs of the same scenario over time. The CLI offers to insert IDs when a file is missing them, preserving your comments and formatting. Accept the offer and commit the file.

Don't reuse an ID for a different situation. A scenario you rewrite from scratch is a new scenario, and giving it a fresh ID keeps the history of the old one intact.

Group scenarios with tags

Tags are arbitrary key/value pairs used to group and filter runs:

scenarios:
  - id: SCN_4mTvH7yLdKpZ
    label: Book a king room for one night
    tags:
      feature: room_booking
      suite: smoke

Two tagging schemes are worth applying from the start. Tag by feature so a failure points at the part of the agent that owns it. Tag by suite so you can separate a fast subset from the full set, and run only the subset when a full run takes too long.

Decide what to cover

A file that only covers the conversations that go well passes until a user does something unusual. Cover these areas:

• The paths that carry traffic. The three or four things most callers actually do.
• The edges of each path. Missing information, changes of mind, and requests your agent has to decline.
• Tool failures. What your agent says when a backend is down or returns nothing.
• Misuse. Attempts to talk your agent out of its instructions.
• Anything that broke once. Every production failure you've fixed belongs here permanently.

Scenarios drawn from real failures are the most durable. A file written in one sitting reflects the cases you could think of at the time, while one grown from production keeps pace with how callers actually use your agent.

Make scenarios reproducible

A scenario is only useful if the same run produces the same verdict. Two things break that.

Dates are the first. A scenario that books a room "next Tuesday" grades differently every week, so write absolute dates and pin the clock your agent reads. To learn more, see Pin time-sensitive scenarios.

Live backends are the second. A scenario that reads your real database depends on whatever that database holds at the time. Seed deterministic state from userdata instead, and read it through the simulation context so your production path stays unchanged. To learn more, see Connect scenarios to your agent.

Run them automatically

After you commit the file, configure your pipeline to run text simulations on every commit and audio simulations on a nightly or pre-release schedule. To learn more, see Run simulations in CI.

Additional resources

These topics cover writing the individual scenarios, finding new ones, and the layers that run beside them.