Overview
Every simulation run appears on the Simulations page in LiveKit Cloud, whether you start it from your terminal or a CI pipeline. The CLI prints a link to each run as it starts. In the dashboard, you can compare a run with earlier ones, see why each scenario passed or failed, and read the transcript of each simulated conversation.
The dashboard is for reviewing runs. To start a run, use the CLI. To learn more, see Run a simulation.
Requirements
The Simulations page requires Agent observability to be turned on for your project. A run that executes while it's off gets no summary, score, or detected issues, because the summary is written from the stored transcript of each scenario. To learn how to turn it on, see Agent insights.
Permissions
Every built-in project role can view simulation runs, including their summaries and scores. Agent insights for each scenario follows the same permissions as for a session, so only project admins can read the transcripts, traces, logs, and judge results.
Find a run
The Simulations page lists the runs from the past 30 days by default. Use the time range selector to customize the time window, or drag across the x-axis of the time-series chart to select a specific range. You can view runs from the past 60 days.
The transcripts, traces, and logs under Agent insights for each scenario follow the 30-day retention window for agent observability data. In a run older than 30 days, each scenario shows Agent insights expired, but the results and scores for the run remain. To keep transcripts longer, export the run before they expire.
Run history
The Run history table lists one run per row, newest first. The table has the following columns:
| Column | Description |
|---|---|
| Run | The simulation run ID. Select it to open the run. |
| Outcomes | How many scenarios passed, failed, or haven't finished yet. |
| Score | The overall score for the run. To learn more, see Quality. |
| Issues detected | The number of issues in the run summary. To learn more, see Issues detected. |
| Scenario group | The scenario file the run used. Runs that generated their scenarios from your agent source show Generated. To learn more, see Scenario groups. |
| Simulated scenarios | The number of simulations in the run. In a run with samples, each sample counts separately. |
| Agent | The agent the run tested. |
| Mode | Text or Voice, for a text or audio run. |
| Duration | How long the run took, or how long it has been running so far. |
| Started | When the run started. The table is sorted by this column. |
| Status | The state of the run. It shows Generating while LiveKit Cloud prepares the scenarios, Running while the simulations run and the summary is written, and then Completed, Failed, or Cancelled. A run with failed scenarios still shows Completed. Failed means the run itself didn't finish. |
To show only runs with a given status, select Filters.
Read a run
A run page has two tabs. The Summary tab covers the run as a whole, and the Simulated scenarios tab covers each conversation in it. While the run is in progress, the page updates as scenarios finish.
The page header shows the status of the run and an Add to CI button, which generates a GitHub Actions workflow for your scenarios. To learn more, see Generate a workflow in the dashboard. A run started from GitHub Actions links to the workflow run that started it, labeled with the commit and branch.
Summary
When a run finishes, the Summary tab shows two written assessments of it: Improvements and Going well. The text links to the scenarios it mentions. Some links go straight to a specific message in the transcript. Below the assessments, the tab shows the quality score for the run and any issues detected in it.
If the run failed or was cancelled, the tab shows the reason instead.
Quality
The Overall score for the run combines the following measures into one percentage:
- Task completion. The share of scenarios that passed.
- Conciseness. Whether the agent's replies were brief enough to follow.
- Conversation progression. Whether the conversation moved forward, without lost information, redundant statements, weak questions, or unnecessary tool calls.
For an audio run, the overall score also includes the following measures:
- Turn-taking. How cleanly the agent and the caller handed the conversation back and forth, without slow replies, talking over the caller, or unanswered turns.
- Transcript accuracy. How much of what the caller said the agent transcribed correctly.
- Entity recognition. The share of key entities the caller said, such as names, IDs, and amounts, that the agent heard correctly.
In an audio run, the score also subtracts a penalty for false interruptions, where the agent stopped speaking for an interruption that wasn't one.
To learn more, see What an audio run measures.
Issues detected
Issues detected lists, in ranked order, the problems the summary found that you can fix to raise the scores for the run. Each issue has the following actions:
- View details opens the issue, with a Recommended action, a prompt for a coding agent, and the conversations from the run that show the problem. Select a conversation to open that scenario.
- Copy fix prompt copies a prompt that states the issue and the suggested change, to paste into a coding agent.
Read the conversations an issue cites before you apply its recommended action. A problem that shows up in a single conversation might come from the scenario rather than your agent.
Simulated scenarios
The Simulated scenarios tab lists each scenario in the run, with failed scenarios first. To find a scenario, search by its label or the reason it failed, or show only Passed or Failed scenarios. Use the up and down arrow keys to move through the list.
Select a scenario to see its result:
- Why this scenario test failed or Why this scenario test passed. The reason for the verdict, at the top of the result. It's the reason the judge gave, or, when your
on_simulation_endcheck fails the scenario, the reason you passed toctx.fail(). - User behavior and Agent expectation. The
instructionsthe simulated user followed and theagent_expectationsthe judge graded against. To learn more, see Write a scenario. - Quality. The overall score for this conversation, built from the same measures as the run score. If the agent misheard a name, number, or code, Entity recognition lists each one with a link to its turn in the transcript. To see every reading the simulation measured from the call, select View all metrics.
- Agent insights. The transcript, traces, logs, and metrics from the conversation, on one timeline. In an audio run, it also plays the recording of the conversation. The transcript and recording appear shortly after the conversation ends. Its Metrics tab reports latency from the agent's traces, which is separate from View all metrics. If it shows no data, see An empty Agent insights tab. To learn more, see Agent insights.
- Judge results. The results of any evaluations your agent runs during the session, such as a
JudgeGroupin itson_session_endcallback. It's empty if your agent runs none.
Repeated scenarios
In a run with --samples, the list still holds one entry per scenario, marked with its verdict across all samples. To see the result of each sample, hover over the scenario.
When you select a repeated scenario, a tab for each sample appears at the top of its result, labeled Run 1, Run 2, and so on. Each tab holds the verdict, transcript, and metrics for that sample.
The Passed and Failed counts in the list count each sample separately, so a scenario whose samples disagree appears under both. The dashboard doesn't show pass@k and pass^k. To read them, open the run with lk agent simulate view. To learn more, see Read pass@k and pass^k.
View all metrics
To see every reading for a scenario, select View all metrics on its quality score. The metrics view groups the readings into the following sections:
| Section | What it measures | Available for |
|---|---|---|
| Response timing | The delay before the caller heard each reply, at p50, p95, and p99, and the end-to-end latency the agent reported. | Audio runs |
| Conversation health | Awkward silences, unanswered turns, early starts, overlapping speech, time to yield after a barge-in, and false interruptions. | Audio runs |
| Component latency | STT delay, LLM time to first token and first sentence, TTS time to first byte, and the LLM decode rate. | All runs |
| Speech recognition | Word and character error rates and entity scores for what the agent heard, compared with what the caller said. | Audio runs |
| Speech synthesis | Word and character error rates and entity scores for the agent's speech, transcribed back to text and compared with the reply it meant to say. | Audio runs |
| Latency distribution | A chart of how latency varied from turn to turn for each pipeline stage, showing the median, the typical range, and which turns were slow. | All runs |
| Turns | A row per turn, with its timings and scores. If any turn was flagged, such as for an Early start or Awkward silence, a Flags column marks it. If the agent or the caller misheard any entities, a Misheard entities column lists them. | All runs |
A section appears only when the run measured it. A test by phone has no metrics the agent reports about itself, such as component latency. To learn more about each reading, see What an audio run measures.
Scenario groups
A scenario group is a scenario file as LiveKit Cloud tracks it across runs, named after the top-level name in your scenarios.yaml file. The group page collects the recent runs of that file in one place. Use it to track how the pass rate and quality score change for the same set of conversations. A score that drops while every scenario still passes shows quality drifting before it causes a failure. Runs that generate their scenarios from your agent source don't belong to a group.
To open a group, select it in the Recent scenario groups section on the Simulations page, or select its name in the Scenario group column. The group page tracks the file across runs in the following panels:
- Latest run. The outcomes, overall score, issue count, and summary for the most recent run. Select View run to open it.
- Run history. The pass rate and score for the latest finished run, the change from the run before it, and a list of recent runs.
- Scenarios. Each scenario in the latest run, grouped by Failed, Running, and Passed. Select a scenario to open its result.
Additional resources
These topics cover running simulations and acting on their results.
Agent Simulations
Run your agent against simulated users, in text or audio mode.
Run simulations in CI
Automate your scenarios, and choose how often to run each mode.
Derive a scenario from a session
Turn a recorded session into a scenario.
Agent insights
Transcripts, traces, logs, and audio for each session.