Write a scenario

Overview

A scenario is a script for one simulated caller: the persona they play, what they want, and what counts as your agent handling the call correctly. The simulator reads instructions to drive that caller turn by turn, and the judge reads agent_expectations to decide whether the run passed.

Specific, well-defined scenarios are better at finding bugs. A vague persona leads to an unfocused conversation, while a vague expectation gives the judge nothing concrete to grade against. If the scenario isn't specific, the simulator can't reliably exercise the behavior you want to test.

This topic covers the scenario format and how to write one scenario well. To decide which scenarios your committed set needs, see Build a scenario set. To run them automatically, see Run simulations in CI. For how a simulation runs, see Agent Simulations.

The examples come from the hotel receptionist agent , a production-shaped agent with 100 scenarios.

The scenario format

A scenario file has an ID, a name, and a list of scenarios:

id: SCNG_9nF2kQwPbXtR
name: Room booking
scenarios:
  - id: SCN_4mTvH7yLdKpZ
    label: Book a king room for one night
    instructions: >
      You are Jordan Reyes (email jordan.reyes@example.com, phone 5550142).
      Book a king room for the night of 2026-06-09, checking out the 10th,
      just you. No breakfast, no late checkout. Pay with the card ending 4242.
    agent_expectations: Room booked successfully
    tags:
      feature: room_booking
    userdata:
      guest:
        first_name: Jordan
        last_name: Reyes
      room_type: king
      available_rooms:
        "2026-06-09":
          - king
          - queen
      expected_state:
        booking:
          room_type: king
          check_in: "2026-06-09"
          check_out: "2026-06-10"

Each scenario has the following fields:

Start from a caller goal

Begin with the reason a real person contacts your agent. For example, they might want to book a room, cancel an order, or check on a claim. That goal becomes the basis for a family of scenarios rather than a single one. Write the straightforward version first, then add the complications that can arise on a real call:

• The caller changes their mind partway through, such as booking a room and then asking to change the name on the reservation.
• The caller wants two things that conflict, such as a room with a full kitchen and an ocean view when no room offers both.
• The caller states a detail incorrectly, then corrects themselves a moment later.
• The caller asks for something outside policy and doesn't accept the first refusal.

Each variant is its own scenario with its own expectation. Separate scenarios identify which complication broke the agent. A single long conversation shows only that the agent failed somewhere.

Script the simulated user

The instructions field is the prompt an LLM follows turn by turn, so treat it as direction for an actor. A point-form script holds the simulated user to the plan more reliably than a paragraph of description. For example, the hotel receptionist scenarios use a fixed set of blocks:

The simulator doesn't parse these headings. They're a prompting convention, and any consistent set of blocks works. Use the same ones across a scenario file so every simulated user behaves predictably.

The following scenario books a room and then complicates it with an amendment:

- label: Book a room, then change the name on it
  instructions: |
    PERSONA: Jordan Reyes, booking a stay and slightly distracted. Friendly,
    answers one question at a time, and volunteers nothing unasked.
    OPENING LINE: "Hi, I'd like to book a king room for the 9th."
    FACTS (reveal each only when asked, one per turn):
    - checking in 2026-06-09, checking out the 10th
    - one guest
    - no breakfast, no late checkout
    - email: jordan.reyes@example.com
    - phone: 5550142
    - paying with the Visa ending 4242
    DO, IN ORDER:
    1. Book the room and answer each question one at a time.
    2. Once the agent reads the booking back and confirms it, explain that the
       stay is for a colleague: "sorry, can you put it under Nina Patel?"
    3. Don't end the call until the agent confirms the name changed.
    REACTIONS:
    - If the agent proposes cancelling and rebooking instead of amending the
      reservation, ask whether the confirmation code stays the same.
  agent_expectations: >
    Books a king room for the night of 2026-06-09 under Jordan Reyes, then
    changes the guest name on that same reservation to Nina Patel and confirms
    the change to the caller. A pass is a single booking that ends in the name
    Nina Patel. Leaving the original name in place, or ending the call with two
    bookings, is a fail.
  tags:
    feature: room_booking
    workflow: amend_booking

State expectations as outcomes

Keep agent_expectations focused on what has to be true when the call ends, not on how the agent gets there. "A single booking that ends in the name Nina Patel" is something the judge can check. "Handles the call well" isn't. Writing the exact words you expect back is the opposite mistake, because the judge should accept equivalent phrasing when the outcome is the same.

When a pass and a fail read similarly in a transcript, name the fail case explicitly. For example, if the caller misspeaks a phone number and corrects it in the same breath, the expectation has to say which number is correct:

agent_expectations: >
  Captures the corrected value each time rather than the first thing said, and
  reads key values back for confirmation: check-in June 13th (not the 30th),
  three guests (not two), and a phone number ending 0197 (not 0190). Booking the
  30th, two guests, or the 0190 number is a fail.

For a flow that ends in a concrete change to your data, pair the expectation with an assertion on the resulting state so a convincing conversation can't pass on its own. To learn more, see Grade on the final state.

Expect a refusal when a refusal is correct

A scenario isn't a request for your agent to comply. Some of the most useful ones describe a caller asking for something the agent has to decline, and the expectation records that refusal as the pass condition. In the hotel receptionist example, one caller wants a six-week stay past the 30-night maximum. Another presses to confirm a 16-guest group block on the spot, and a third tries to talk the agent out of its system prompt. In each, declining is the pass condition.

Write these expectations as precisely as the successful ones. Say what the agent declines and what it does instead: offer an alternative, escalate to a manager, or record a callback. An agent that refuses and abandons the caller still fails. Scenarios where two requests conflict work the same way: the pass condition is naming the conflict and letting the caller choose, and inventing an option that satisfies both is the fail.

Account for realistic caller behavior

Scenarios written as clean transactions miss most of what breaks a live call. Direct the simulated user to speak the way callers do:

• State a number wrong and correct it in the same breath, then check the read-back for the corrected value.
• Deliver every detail in one dense opening turn, and push back if the agent re-asks for something already given.
• Correct a single value mid-read-back without restarting the booking.
• Spell a name that sounds like a more common one, and insist on the spelling.

These matter most in audio mode, where a self-correction also has to survive transcription. Run the same scenarios with a degraded connection to see which ones still pass.

Keep each scenario to one complication

A scenario that tests a cancellation policy, an upgrade offer, and a card decline at once makes it harder to identify what went wrong. A scenario that fails should point to one behavior, rather than requiring a transcript review to determine which behavior broke.

Give each scenario a label that names the situation rather than the feature, so a failure in the run output is legible without opening the transcript. A label such as "Caller corrects a date during the read-back" identifies the behavior at fault. A label such as "Booking test 4" doesn't. For grouping scenarios once the set grows, see Group scenarios with tags.

Keep a generated scenario

Generating scenarios from your agent source produces drafts rather than finished tests. Most are worth discarding, but a generated scenario sometimes describes a case you hadn't considered. Keep that one instead of rewriting it from scratch.

The CLI offers once to save generated scenarios to scenarios.yaml, when they first arrive. Accept the offer to keep the whole batch, or choose Not now and press s later to save them under a name you pick. Either way the CLI writes the generated scenarios to a temporary file, so declining the offer doesn't lose them.

To keep one scenario out of a run rather than the whole batch, open the run in the dashboard, select the job, and select Turn into a test. That produces a single-scenario file you can paste into your committed set. To learn more, see Derive a scenario from a session.

A generated scenario still has to meet the same bar as one you wrote. Before you commit it:

• Rewrite instructions as a script. Generated instructions tend to describe the caller rather than direct them. To learn more, see Script the simulated user.
• Restate agent_expectations as an outcome. Generated expectations often describe the conversation instead of the result the agent has to reach. To learn more, see State expectations as outcomes.
• Split it if it carries more than one complication. To learn more, see Keep each scenario to one complication.
• Give it a label that names the situation. Generated labels tend to name the feature under test, such as Booking test 4. A label such as Caller corrects a date during the read-back identifies the behavior at fault straight from the run output.

Additional resources

These topics and examples cover how a simulation runs, choosing which scenarios to commit, and complete scenario files to read.