Run a saved test

Overview

Turn into a test on a session or a simulation job downloads a scenario.yaml: one scenario file holding the simulated user's instructions and the criterion the judge grades against. Running it replays that conversation against your agent and gives you a pass or fail.

You need the LiveKit CLI, authenticated to the project the test came from. See Requirements for versions.

Run it locally

From your agent's project directory:

lk agent simulate --scenarios scenario.yaml

The CLI starts your agent as a local worker, runs the simulated user against it, judges the transcript, and prints a link to the run in the dashboard. Your entrypoint, tools, and conversation logic run unchanged.

To run against an agent that's already deployed instead of a local one, name it:

lk agent simulate --scenarios scenario.yaml --agent-name my-agent

Pass --agent-name "" to target the project's default agent, the one that auto-joins every room.

Downloaded files are single-scenario groups, but scenarios: is a list: paste new tests into the file you already keep in the repo and one command runs all of them.

Run it in CI

lk agent simulate drops its interactive UI whenever the CI environment variable is set, and exits non-zero when a scenario fails. GitHub Actions sets CI=true on every run, so a job needs only credentials and the command.

Add LIVEKIT_API_KEY and LIVEKIT_API_SECRET as repository secrets, then commit your scenario file and this workflow:

name: Agent simulations

on:
  pull_request:

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      LIVEKIT_URL: wss://<your-project>.livekit.cloud
      LIVEKIT_API_KEY: ${{ secrets.LIVEKIT_API_KEY }}
      LIVEKIT_API_SECRET: ${{ secrets.LIVEKIT_API_SECRET }}
    steps:
      - uses: actions/checkout@v4

      - name: Install the LiveKit CLI
        run: curl -sSL https://get.livekit.io/cli | bash

      - name: Run simulations
        run: >
          lk agent simulate
          --scenarios scenario.yaml
          --agent-name ""
          --baseline <baseline-run-id>

Two flags carry the weight:

• --agent-name targets a deployed agent, so CI grades what's running rather than spawning a worker from the checkout.
• --baseline <run-id> names a finished run you consider good. Only scenarios that pass there and fail here fail the job, so a test your agent has never passed doesn't block every pull request. The flag applies to non-interactive runs only.

Additional resources