Skip to main content

Review simulation runs in LiveKit Cloud

Read scores, detected issues, metrics, and transcripts for your simulation runs in the LiveKit Cloud dashboard.

On this page

Overview

Every simulation run appears on the Simulations  page in LiveKit Cloud, whether you start it from your terminal or a CI pipeline. The CLI prints a link to each run as it starts. In the dashboard, you can compare a run with earlier ones, see why each scenario passed or failed, and read the transcript of each simulated conversation.

The dashboard is for reviewing runs. To start a run, use the CLI. To learn more, see Run a simulation.

Requirements

The Simulations page requires Agent observability to be turned on for your project. A run that executes while it's off gets no summary, score, or detected issues, because the summary is written from the stored transcript of each scenario. To learn how to turn it on, see Agent insights.

Permissions

Every built-in project role can view simulation runs, including their summaries and scores. Agent insights for each scenario follows the same permissions as for a session, so only project admins can read the transcripts, traces, logs, and judge results.

Find a run

The Simulations page lists the runs from the past 30 days by default. Use the time range selector to customize the time window, or drag across the x-axis of the time-series chart to select a specific range. You can view runs from the past 60 days.

Agent insights expire after 30 days

The transcripts, traces, and logs under Agent insights for each scenario follow the 30-day retention window for agent observability data. In a run older than 30 days, each scenario shows Agent insights expired, but the results and scores for the run remain. To keep transcripts longer, export the run before they expire.

Run history

The Run history table lists one run per row, newest first. The table has the following columns:

Column Description
RunThe simulation run ID. Select it to open the run.
OutcomesHow many scenarios passed, failed, or haven't finished yet.
ScoreThe overall score for the run. To learn more, see Quality.
Issues detectedThe number of issues in the run summary. To learn more, see Issues detected.
Scenario groupThe scenario file the run used. Runs that generated their scenarios from your agent source show Generated. To learn more, see Scenario groups.
Simulated scenariosThe number of simulations in the run. In a run with samples, each sample counts separately.
AgentThe agent the run tested.
ModeText or Voice, for a text or audio run.
DurationHow long the run took, or how long it has been running so far.
StartedWhen the run started. The table is sorted by this column.
StatusThe state of the run. It shows Generating while LiveKit Cloud prepares the scenarios, Running while the simulations run and the summary is written, and then Completed, Failed, or Cancelled. A run with failed scenarios still shows Completed. Failed means the run itself didn't finish.

To show only runs with a given status, select Filters.

Read a run

A run page has two tabs. The Summary tab covers the run as a whole, and the Simulated scenarios tab covers each conversation in it. While the run is in progress, the page updates as scenarios finish.

The page header shows the status of the run and an Add to CI button, which generates a GitHub Actions workflow for your scenarios. To learn more, see Generate a workflow in the dashboard. A run started from GitHub Actions links to the workflow run that started it, labeled with the commit and branch.

Summary

When a run finishes, the Summary tab shows two written assessments of it: Improvements and Going well. The text links to the scenarios it mentions. Some links go straight to a specific message in the transcript. Below the assessments, the tab shows the quality score for the run and any issues detected in it.

If the run failed or was cancelled, the tab shows the reason instead.

Quality

The Overall score for the run combines the following measures into one percentage:

  • Task completion. The share of scenarios that passed.
  • Conciseness. Whether the agent's replies were brief enough to follow.
  • Conversation progression. Whether the conversation moved forward, without lost information, redundant statements, weak questions, or unnecessary tool calls.

For an audio run, the overall score also includes the following measures:

  • Turn-taking. How cleanly the agent and the caller handed the conversation back and forth, without slow replies, talking over the caller, or unanswered turns.
  • Transcript accuracy. How much of what the caller said the agent transcribed correctly.
  • Entity recognition. The share of key entities the caller said, such as names, IDs, and amounts, that the agent heard correctly.

In an audio run, the score also subtracts a penalty for false interruptions, where the agent stopped speaking for an interruption that wasn't one.

To learn more, see What an audio run measures.

Issues detected

Issues detected lists, in ranked order, the problems the summary found that you can fix to raise the scores for the run. Each issue has the following actions:

  • View details opens the issue, with a Recommended action, a prompt for a coding agent, and the conversations from the run that show the problem. Select a conversation to open that scenario.
  • Copy fix prompt copies a prompt that states the issue and the suggested change, to paste into a coding agent.
Check the examples first

Read the conversations an issue cites before you apply its recommended action. A problem that shows up in a single conversation might come from the scenario rather than your agent.

Simulated scenarios

The Simulated scenarios tab lists each scenario in the run, with failed scenarios first. To find a scenario, search by its label or the reason it failed, or show only Passed or Failed scenarios. Use the up and down arrow keys to move through the list.

Select a scenario to see its result:

  • Why this scenario test failed or Why this scenario test passed. The reason for the verdict, at the top of the result. It's the reason the judge gave, or, when your on_simulation_end check fails the scenario, the reason you passed to ctx.fail().
  • User behavior and Agent expectation. The instructions the simulated user followed and the agent_expectations the judge graded against. To learn more, see Write a scenario.
  • Quality. The overall score for this conversation, built from the same measures as the run score. If the agent misheard a name, number, or code, Entity recognition lists each one with a link to its turn in the transcript. To see every reading the simulation measured from the call, select View all metrics.
  • Agent insights. The transcript, traces, logs, and metrics from the conversation, on one timeline. In an audio run, it also plays the recording of the conversation. The transcript and recording appear shortly after the conversation ends. Its Metrics tab reports latency from the agent's traces, which is separate from View all metrics. If it shows no data, see An empty Agent insights tab. To learn more, see Agent insights.
  • Judge results. The results of any evaluations your agent runs during the session, such as a JudgeGroup in its on_session_end callback. It's empty if your agent runs none.

Repeated scenarios

In a run with --samples, the list still holds one entry per scenario, marked with its verdict across all samples. To see the result of each sample, hover over the scenario.

When you select a repeated scenario, a tab for each sample appears at the top of its result, labeled Run 1, Run 2, and so on. Each tab holds the verdict, transcript, and metrics for that sample.

The Passed and Failed counts in the list count each sample separately, so a scenario whose samples disagree appears under both. The dashboard doesn't show pass@k and pass^k. To read them, open the run with lk agent simulate view. To learn more, see Read pass@k and pass^k.

View all metrics

To see every reading for a scenario, select View all metrics on its quality score. The metrics view groups the readings into the following sections:

Section What it measures Available for
Response timingThe delay before the caller heard each reply, at p50, p95, and p99, and the end-to-end latency the agent reported.Audio runs
Conversation healthAwkward silences, unanswered turns, early starts, overlapping speech, time to yield after a barge-in, and false interruptions.Audio runs
Component latencySTT delay, LLM time to first token and first sentence, TTS time to first byte, and the LLM decode rate.All runs
Speech recognitionWord and character error rates and entity scores for what the agent heard, compared with what the caller said.Audio runs
Speech synthesisWord and character error rates and entity scores for the agent's speech, transcribed back to text and compared with the reply it meant to say.Audio runs
Latency distributionA chart of how latency varied from turn to turn for each pipeline stage, showing the median, the typical range, and which turns were slow.All runs
TurnsA row per turn, with its timings and scores. If any turn was flagged, such as for an Early start or Awkward silence, a Flags column marks it. If the agent or the caller misheard any entities, a Misheard entities column lists them.All runs

A section appears only when the run measured it. A test by phone has no metrics the agent reports about itself, such as component latency. To learn more about each reading, see What an audio run measures.

Scenario groups

A scenario group is a scenario file as LiveKit Cloud tracks it across runs, named after the top-level name in your scenarios.yaml file. The group page collects the recent runs of that file in one place. Use it to track how the pass rate and quality score change for the same set of conversations. A score that drops while every scenario still passes shows quality drifting before it causes a failure. Runs that generate their scenarios from your agent source don't belong to a group.

To open a group, select it in the Recent scenario groups section on the Simulations page, or select its name in the Scenario group column. The group page tracks the file across runs in the following panels:

  • Latest run. The outcomes, overall score, issue count, and summary for the most recent run. Select View run to open it.
  • Run history. The pass rate and score for the latest finished run, the change from the run before it, and a list of recent runs.
  • Scenarios. Each scenario in the latest run, grouped by Failed, Running, and Passed. Select a scenario to open its result.

Additional resources

These topics cover running simulations and acting on their results.