Derive a scenario from a session

Overview

A recorded session shows what happened, but it doesn't give you a reproducible way to test whether a change fixes the underlying issue.

Deriving a scenario turns a session into a reproducible test case. LiveKit reads the session, generates instructions that reproduce the caller's goal, and defines the outcome the agent should have reached. You can run the scenario, evaluate the result, update your agent, and run it again to verify the fix.

You can also save a scenario from a previous simulation run. Commit saved scenarios alongside your existing scenarios to test for regressions in later releases.

This workflow connects production behavior with pre-production testing. Agent insights helps you identify issues in real sessions, while scenario generation lets you turn those issues into repeatable tests.

Requirements

Before you begin, make sure you have a LiveKit Cloud project and an authenticated CLI. For versions and setup, see Requirements.

Deriving a scenario requires a recorded session. To learn more, see Agent insights.

Find a session worth testing

Start in your project session list, where each recorded session carries its transcript, traces, logs, and audio in one timeline.

A session is worth deriving from when it shows a failure your agent should have avoided:

• A goal the caller never reached. The conversation ends without the booking, transfer, or answer the caller came for.
• A tool your agent called wrongly. The right tool with the wrong arguments, or the wrong tool entirely.
• An answer that wasn't grounded. Confident statements your data doesn't support.
• A conversation that stalled. Repeated clarifying questions, or the same information collected twice.

Make a note of the session ID, which you need for the steps that follow. To keep a scenario from a simulation run instead of a recorded session, see Keep a scenario from a run.

Create the scenario

Derive the scenario from the dashboard or from the CLI. Both produce the same thing: one scenario holding the instructions for the simulated user and the criterion the judge grades against.

From the dashboard

Use the following steps to derive a scenario from a recorded session:

1. Navigate to your Sessions  page.
2. Select a session → Agent insights.
3. Select Turn into a test.

The dialog box shows the derived simulated user and pass criteria, and downloads a scenario.yaml file containing both.

The downloaded file holds a single scenario, but scenarios: is a list. Paste the scenario into the scenarios.yaml file you already keep in your repository.

Keep a scenario from a run

A scenario you have already run is sometimes worth keeping. The run might have generated it from your agent source and found a real bug, or you might have written it by hand for a single run and now want it in your committed set. This path doesn't derive anything, because the scenario already exists. You're saving a copy of it.

Open the run in the dashboard, select the scenario, and select Turn into a test. The dialog shows the simulated user and the pass criteria, takes a name, and downloads a scenario.yaml file holding that one scenario. Paste it into the file you keep in your repository.

This path skips the next section, Run the scenario. The run you took the scenario from already showed you how your agent handled it, so there's no failure left to reproduce. Go straight to editing the scenario, and hold a generated one to the same standard as one you wrote yourself. To learn more, see Keep a generated scenario.

Run the scenario

Run the scenario before you change anything, to confirm that it reproduces the failure. A scenario that passes on the first run hasn't captured the problem, so it can't confirm that a later fix works:

lk agent simulate text

The CLI reads ./scenarios.yaml from your working directory and starts your agent as a local worker, resolving the entrypoint from your project. It then runs the simulated user against that agent, judges the transcript, and prints a link to the run in the dashboard. Your entrypoint, tools, and conversation logic run unchanged. To run a file you downloaded without merging it first, name it explicitly with --scenarios scenario.yaml.

To grade a deployed agent instead of a local worker, name the agent:

lk agent simulate text --agent-name my-agent

Pass --agent-name "" to target the default project agent, the one that auto-joins every room.

When the scenario passes but production failed, something that shaped the real conversation is missing from the simulated one. The usual causes are these:

• Audio. The failure came from transcription, turn-taking, or interruption handling, none of which a text run exercises. Run the scenario with lk agent simulate audio instead.
• State. Your agent reached a different backend, or the same backend holding different data.
• Detail. The derived instructions are shorter than the real conversation and leave out the turn that caused the problem.

Editing the scenario addresses each of these causes.

Generalize the scenario

A derived scenario describes one call by one caller. Editing widens it to cover the behavior rather than the call.

Start with agent_expectations. The derived expectation states what should have happened in that call, which is often narrower or vaguer than the rule you actually want enforced. Write the outcome you require, specifically enough that a judge reading only the transcript can decide it.

Then widen instructions past the literal call. The simulated user follows them turn by turn, so replace the details that don't matter with the behavior that does. The scenario then covers a class of callers rather than one person. A caller who changed a date once becomes a caller who changes their mind.

Replace live data with userdata so the scenario grades the same way every run, and read it through the simulation context so your production path stays unchanged. To learn more, see Connect scenarios to your agent.

Finally, tag the scenario with the feature that owns the failure so later runs group it with the other scenarios for that feature.

The following scenario shows the result of all four edits: a sharpened expectation, widened instructions, seeded userdata with the end state to grade against, and a feature tag.

- id: SCN_4mTvH7yLdKpZ
  label: Caller changes the checkout date mid-booking
  instructions: >
    You are Jordan Reyes. Book a king room for 2026-06-09, then change
    the checkout date to the 12th before you confirm. Pay with the card
    ending 4242.
  agent_expectations: >
    Booking is confirmed for 2026-06-09 to 2026-06-12 at the price for
    three nights. The agent does not re-ask for the card.
  tags:
    feature: room_booking
  userdata:
    available_rooms:
      "2026-06-09": [king, queen]
    expected_state:
      booking:
        room_type: king
        check_in: "2026-06-09"
        check_out: "2026-06-12"

Fix your agent and verify

Change your agent, then run the scenario again. Check the instructions, the prompts, and the tool definitions. A failure that looks like a reasoning problem is often a tool whose description doesn't say when to use it.

Once the scenario passes, run the whole file rather than the one scenario you were working on. A fix aimed at one conversation can change how your agent handles related ones:

lk agent simulate text

Commit the scenario

Add the scenario to the scenarios.yaml file you commit with your agent code. Use it to re-check the agent's behavior after your next change.

Once the file is in your repository, the scenario runs alongside the rest of your scenarios in CI, so a return of the same failure fails the job. To learn more about organizing them, see Build a scenario set.

Additional resources

These topics cover writing and organizing scenarios.