Run simulations in CI

Overview

A committed scenarios.yaml file lets you automate simulations. The same scenarios behave consistently on every execution, and a regression in multi-turn behavior fails the job the way a broken unit test does.

This topic covers when to run simulations and how to integrate them into your continuous integration (CI) pipeline. To learn how to write the scenarios themselves, see Write a scenario. To learn which scenarios your committed set needs, see Build a scenario set.

Requirements

Before you begin, make sure you have a LiveKit Cloud project and an authenticated CLI. For versions and setup, see Requirements.

When to run simulations

A simulation runs a whole conversation and grades it with a judge, so it costs more than a unit test and takes longer. Use these guidelines to decide when to run each mode.

Keep simulations out of your development loop. While you're building a behavior, the feedback has to be immediate. Use the Agent Console to talk to your agent directly, and unit tests for turn-level assertions that finish in seconds. Move to a simulation once the behavior works and you want to verify it across a full conversation.

Run text simulations on every commit. Text mode drops the STT and TTS pipeline, so a run exercises your LLM, tools, and conversation logic at a cost that fits a per-commit job. At this cadence, a regression surfaces in the same change that introduced it.

Reserve audio simulations for a schedule. Audio runs execute in real time and call your STT and TTS providers on every turn. Run them nightly, before a release, or when a change affects speech, turn-taking, or interruption handling.

Each mode suits a different point in your pipeline:

GitHub Actions

The CLI needs no CI-specific configuration. It drops its interactive display when stdout isn't a terminal or when the CI environment variable is present, so the job log reads line by line, and it exits non-zero when any scenario fails, so a regression fails the job without extra scripting.

Authenticate the CLI with LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET from repository secrets. Because the CLI starts your real agent as a local worker, the workflow also has to install the dependencies for your agent and provide any API keys it reads:

name: Agent Simulations

on:
  pull_request:

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      LIVEKIT_URL: ${{ secrets.LIVEKIT_URL }}
      LIVEKIT_API_KEY: ${{ secrets.LIVEKIT_API_KEY }}
      LIVEKIT_API_SECRET: ${{ secrets.LIVEKIT_API_SECRET }}
      # Any keys your agent plugins read.
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv sync

      - name: Install the LiveKit CLI
        run: curl -sSL https://get.livekit.io/cli | bash

      - name: Run simulations
        run: lk agent simulate text --concurrency 5

name: Agent Simulations

on:
  pull_request:

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      LIVEKIT_URL: ${{ secrets.LIVEKIT_URL }}
      LIVEKIT_API_KEY: ${{ secrets.LIVEKIT_API_KEY }}
      LIVEKIT_API_SECRET: ${{ secrets.LIVEKIT_API_SECRET }}
      # Any keys your agent plugins read.
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci

      - name: Install the LiveKit CLI
        run: curl -sSL https://get.livekit.io/cli | bash

      - name: Run simulations
        run: lk agent simulate text agent.ts

The run reads ./scenarios.yaml automatically. To grade a deployed agent instead of one spawned from the checkout, add --agent-name with the registered agent name, or "" to target the default project agent.

Schedule the audio pass

Run audio as a separate workflow rather than adding it to the text job. The steps are identical apart from the subcommand, so only the triggers change. The following example runs an audio pass every day at 06:00 UTC, and lets you start one on demand:

on:
  # Adds a Run workflow button on the Actions tab, for an audio pass on demand.
  workflow_dispatch:
  # Runs automatically at 06:00 UTC every day.
  schedule:
    - cron: '0 6 * * *'

# ... same steps, with:
#   run: lk agent simulate audio --concurrency 5

GitHub reads the cron value in UTC rather than your local timezone, so convert the hour you want before you set it.

To learn more about what an audio run measures, see Audio simulations.

Triage a failed run

A failed job prints a link to the run in the dashboard, where the transcript for each scenario shows what the simulated user said and how your agent answered.

From the terminal, list recent runs and reopen any of them:

lk agent simulate list
lk agent simulate view <run-id>

To analyze a run outside the CLI, export it as JSON. The export includes the run, its summary, and the exact per-job chat contexts, which is useful for archiving results as a build artifact:

lk agent simulate export <run-id> > run.json

The transcript tells you which side is at fault: your agent, or the scenario describing it. For guidance on failed runs, see Keep your scenarios passing.

Keep your scenarios passing

By default, any failing scenario fails the whole job. A scenario your agent has never passed can therefore block builds until someone fixes or removes it.

When you add a scenario that fails, take one of these actions:

• Fix the agent. The correct outcome when the scenario describes real, required behavior.
• Sharpen the expectation. Vague agent_expectations values produce judgments that vary between runs. To learn more, see State expectations as outcomes.
• Leave it out of the committed file. Keep aspirational scenarios in a separate file and run them on demand until you implement the behavior.

Rewriting the scenario is often the fix. A failing scenario whose instructions wander or whose expectation describes the conversation instead of the result grades unreliably however the agent behaves. To learn more, see Write a scenario.

Pin time-sensitive scenarios to absolute dates as well. Otherwise a scenario that passes today starts failing months later for reasons unrelated to your agent.

Additional resources

The following resources contain additional information about simulations and testing.