Agent Simulations | LiveKit Documentation

Overview

A simulation run tests an agent against a set of scenarios, and a judge model grades each conversation. To run your own scenarios, send them in scenarioGroup when you create a run. To have LiveKit generate scenarios from your agent code instead, set numSimulations and upload the code.

To upload your agent's code:

Pack the agent project directory into a .tar.gz archive.
Send the archive to the url in the presignedPostRequest of the response, as a multipart/form-data POST. Include each entry of values as a form field, and the archive in a field named file.
Confirm the upload with your agent's entrypoint file, such as agent.py, in codeEntrypoint. LiveKit generates the scenarios from the code.

Reading a run needs the Read role on the project. Creating, confirming, cancelling, and generating a scenario need the Write role.

To call these endpoints from the CLI, use the lk agent simulate list, get, and cancel commands, for example lk agent simulate list --user-auth.

The simulation run object

idstring

The id value.

projectIdstring

The projectId value.

statusSimulationRun.Status

The status value.

To learn more, see SimulationRun.Status.

agentDescriptionstring

The agentDescription value.

errorstring

The error value.

createdAttimestamp

The createdAt value.

jobsArray<Job>

The jobs value.

To learn more, see Job.

agentNamestring

The agentName value.

scenarioGroupScenarioGroup

The scenarioGroup value.

To learn more, see ScenarioGroup.

endedAttimestamp

The endedAt value.

jobCountint32

The jobCount value.

passedCountint32

The passedCount value.

failedCountint32

The failedCount value.

numSimulationsint32

The numSimulations value.

usageSimulationRun.Usage

Aggregate usage across all jobs in this run (sum of Job.usage).

To learn more, see SimulationRun.Usage.

concurrencyint32

Maximum simulate jobs running in parallel for this run (0 = server default).

modeSimulationMode

Conversation mode for every job in this run; unspecified = TEXT.

To learn more, see SimulationMode.

metricsRunMetrics

Run-level aggregates over this run's jobs.

To learn more, see RunMetrics.

summaryZstdstring (byte)

Zstd-compressed SimulationRunSummary: decompress, then proto.Unmarshal. Compressed by default so the blob ships as-is on every hop.

issueCountint32

How many issues the stored summary flagged. Present only once a summary landed — absent and 0 differ ("never summarized" vs "no issues") — and List responses carry it without the summary blob itself.

ciCI

The pipeline this run came from; unset when it did not come from one.

To learn more, see CI.

samplingSamplingOptions

Absent when the run ran each scenario once and required it to pass.

To learn more, see SamplingOptions.

The simulation run object

{
  "id": string,
  "projectId": string,
  "status": SimulationRun.Status,
  "agentDescription": string,
  "error": string,
  "createdAt": timestamp,
  "jobs": Array<Job>,
  "agentName": string,
  "scenarioGroup": ScenarioGroup,
  "endedAt": timestamp,
  "jobCount": int32,
  "passedCount": int32,
  "failedCount": int32,
  "numSimulations": int32,
  "usage": SimulationRun.Usage,
  "concurrency": int32,
  "mode": SimulationMode,
  "metrics": RunMetrics,
  "summaryZstd": string (byte),
  "issueCount": int32,
  "ci": CI,
  "sampling": SamplingOptions
}

List simulation runs

GET

/v0/projects/{projectId}/simulation-runs

Lists the runs of a project, newest first. You can filter the runs by status, agent, conversation mode, and time window. Each run includes its counts (jobCount, passedCount, failedCount, and issueCount), but not its jobs or its summary.

Returns a list of SimulationRun objects, with a pageInfo cursor.

Path parameters

projectId

Required

string

The projectId path parameter.

Query parameters

page.cursorstring

The cursor of the page to return: the nextCursor of the previous page. When empty, the list starts at the first item.

page.pageSizeint32

The maximum number of items to return. When 0, the list uses its default page size.

statusSimulationRun.Status

A run status. Returns only the runs in this status.

To learn more, see SimulationRun.Status.

agentNamestring

An agent name. Returns only the runs of the agent with this exact name.

startTimestring (date-time)

The start of the window on the creation time of the run, inclusive. When unset, the window starts 45 days before endTime. The window can be at most 45 days. A longer window returns InvalidArgument.

endTimestring (date-time)

The end of the window on the creation time of the run, exclusive. When unset, the window ends now.

modeSimulationMode

A conversation mode. Returns only the runs in this mode.

To learn more, see SimulationMode.

Request

curl -X GET "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN"

Response

{
  "items": [
    {
      "id": "SR_8kQmv3TdwZpL",
      "projectId": "p_f9779j5lpvh",
      "status": "STATUS_COMPLETED",
      "agentDescription": "A billing support agent that answers questions about invoices and plans.",
      "error": "",
      "createdAt": "2026-01-15T01:30:15Z",
      "jobs": [],
      "agentName": "support-agent",
      "scenarioGroup": {
        "name": "Billing questions",
        "scenarios": [],
        "id": ""
      },
      "endedAt": "2026-01-15T01:33:57Z",
      "jobCount": 12,
      "passedCount": 10,
      "failedCount": 2,
      "numSimulations": 12,
      "usage": {
        "textTurnsCount": 0,
        "audioTurnsCount": 108
      },
      "concurrency": 4,
      "mode": "SIMULATION_MODE_AUDIO",
      "metrics": {
        "scenarioPassRate": 0.8333333,
        "overallScore": 0.8125,
        "stt": {
          "wer": 0.0523,
          "words": 1530,
          "wordErrors": 80,
          "cer": 0.0216,
          "chars": 7980,
          "charErrors": 172,
          "sttDelayMs": 335,
          "entityRecognition": 0.9583333,
          "entitiesUttered": 24,
          "entitiesRecognized": 23,
          "entityRecall": 1,
          "entitiesAcquiredUttered": 12,
          "entitiesAcquiredRecognized": 12
        },
        "llm": {
          "ttftMs": 940,
          "ttfsMs": 1265,
          "tokensPerSecond": 128.4
        },
        "tts": {
          "ttfbMs": 251,
          "wer": 0.0152,
          "words": 2240,
          "wordErrors": 34,
          "cer": 0.008,
          "chars": 12150,
          "charErrors": 97,
          "entityRecognition": 0.9722222,
          "entitiesUttered": 36,
          "entitiesRecognized": 35,
          "entityRecall": 1,
          "entitiesAcquiredUttered": 24,
          "entitiesAcquiredRecognized": 24
        },
        "conversation": {
          "turnTakingScore": 0.88,
          "heardE2eLatencyP50Ms": 1310,
          "heardE2eLatencyP95Ms": 1960,
          "heardE2eLatencyP99Ms": 2380,
          "timeToYieldMs": 380,
          "eotMispredictionCount": 0,
          "overlapRatio": 0.03,
          "overlapSpeechMs": 640,
          "totalSpeechMs": 21400,
          "awkwardSilenceCount": 0,
          "unansweredTurns": 0,
          "falseInterruptionCount": 0,
          "e2eLatencyMs": 1225
        },
        "conciseness": 0.875,
        "conversationProgression": 0.9166667
      },
      "summaryZstd": "",
      "issueCount": 1
    }
  ],
  "pageInfo": {
    "nextCursor": "",
    "hasMore": false
  }
}

Create simulation run

POST

/v0/projects/{projectId}/simulation-runs

Creates a simulation run. With a scenarioGroup, the run starts immediately and the response has no upload target. Without a scenarioGroup, the run waits in PENDING_UPLOAD for the agent source that the API generates scenarios from. Upload the source to the returned presigned POST, then call ConfirmSimulationSourceUpload. A run that stays pending for one hour fails. This RPC is not idempotent: a retry starts a second run.

Path parameters

projectId

Required

string

The projectId path parameter.

Request body

agentNamestring

The name of the agent under test. Each job dispatches this agent into the simulation room, so the agent must be deployed or running as a worker. When empty, the run uses the default agent of the project. The run records the name, and ListSimulationRuns can filter on it.

numSimulationsint32

The number of scenarios to generate from the agent source when scenarioGroup is unset.

regionstring

Reserved. Must be empty. The API chooses where the run happens.

scenarioGroupScenarioGroup

The scenarios to run, with the contents of a scenarios.yaml file. When set, the run plays only these scenarios and does not generate numSimulations scenarios. The run needs no source upload and starts immediately.

To learn more, see ScenarioGroup.

concurrencyint32

The maximum number of jobs that run in parallel. The maximum value is 20. A larger value returns InvalidArgument. The concurrency quota of the project can run fewer jobs at a time. When unset, the API uses its default.

modeSimulationMode

The conversation mode of each job. Unset means TEXT.

To learn more, see SimulationMode.

backgroundNoiseboolean

When true, the simulated user has background noise. Applies only in AUDIO mode.

lowQualityMicrophoneboolean

When true, the simulated user has a low-quality microphone. Applies only in AUDIO mode.

packetLossboolean

When true, the audio of the simulated user has packet loss. Applies only in AUDIO mode.

ciCI

The CI pipeline that started the run. Use it to link a run to its commit.

To learn more, see CI.

Response fields

simulationRunIdstring

The simulationRunId value.

presignedPostRequestPresignedPostRequest

The presigned POST to upload the agent source archive that the API generates scenarios from. After the upload, call ConfirmSimulationSourceUpload. Unset when the run has a scenarioGroup.

To learn more, see PresignedPostRequest.

Request

curl -X POST "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "agentName": "support-agent",
    "numSimulations": 12,
    "mode": "SIMULATION_MODE_AUDIO",
    "concurrency": 4
  }'

Response

{
  "simulationRunId": "SR_8kQmv3TdwZpL",
  "presignedPostRequest": {
    "url": "https://example-bucket.s3.us-east-1.amazonaws.com/",
    "values": {
      "key": "p_f9779j5lpvh/SR_8kQmv3TdwZpL/source.tar.gz",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAIOSFODNN7EXAMPLE/20260115/us-east-1/s3/aws4_request",
      "x-amz-date": "20260115T013015Z",
      "policy": "eyJleHBpcmF0aW9uIjoiMjAyNi0wMS0xNVQwMjozMDoxNVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJsaXZla2l0LXNpbXVsYXRpb24tc291cmNlcyJ9LHsia2V5IjoicF9mOTc3OWo1bHB2aC9TUl84a1FtdjNUZHdacEwvc291cmNlLnRhci5neiJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFJT1NGT0ROTjdFWEFNUExFLzIwMjYwMTE1L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QifSx7IngtYW16LWRhdGUiOiIyMDI2MDExNVQwMTMwMTVaIn0sWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMCw1MjQyODgwMF1dfQ==",
      "x-amz-signature": "823412d1eacb67956220e532959f0104603057c88704863ca38e7cd188fda812"
    }
  }
}

Get simulation run

GET

/v0/projects/{projectId}/simulation-runs/{simulationRunId}

Returns one run with its jobs and aggregate metrics. The response does not include the compressed summary (summaryZstd). To get the summary, call GetSimulationRunSummary.

Returns the SimulationRun object.

Path parameters

projectId

Required

string

The projectId path parameter.

simulationRunId

Required

string

The simulationRunId path parameter.

Request

curl -X GET "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs/${SIMULATION_RUN_ID}" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN"

Response

{
  "run": {
    "id": "SR_8kQmv3TdwZpL",
    "projectId": "p_f9779j5lpvh",
    "status": "STATUS_COMPLETED",
    "agentDescription": "A billing support agent that answers questions about invoices and plans.",
    "error": "",
    "createdAt": "2026-01-15T01:30:15Z",
    "jobs": [
      {
        "id": "SRJ_4hVnc8RtKqmW",
        "status": "STATUS_COMPLETED",
        "instructions": "You are a customer whose monthly invoice rose from $40 to $115. Ask why, and push back once if the first answer is vague.",
        "error": "",
        "agentExpectations": "The agent names the line item that changed, gives the amount, and offers to send the invoice. It does not promise a refund.",
        "label": "Customer asks why an invoice went up",
        "tags": [],
        "roomName": "sim-SRJ_4hVnc8RtKqmW-7Qa2WkPt",
        "startedAt": "2026-01-15T01:30:52Z",
        "endedAt": "2026-01-15T01:31:24Z",
        "roomId": "RM_x4Lq9BvTdnCe",
        "usage": {
          "textTurnsCount": 0,
          "audioTurnsCount": 9
        },
        "metrics": {
          "taskCompletion": 1,
          "overallScore": 0.95,
          "stt": {
            "wer": 0.04,
            "words": 125,
            "wordErrors": 5,
            "cer": 0.0215,
            "chars": 652,
            "charErrors": 14,
            "sttDelayMs": 310,
            "entityRecognition": 1,
            "entitiesUttered": 2,
            "entitiesRecognized": 2,
            "entityRecall": 1,
            "entitiesAcquiredUttered": 1,
            "entitiesAcquiredRecognized": 1
          },
          "llm": {
            "ttftMs": 852,
            "ttfsMs": 1140,
            "tokensPerSecond": 115.8
          },
          "tts": {
            "ttfbMs": 238,
            "wer": 0.016,
            "words": 187,
            "wordErrors": 3,
            "cer": 0.0079,
            "chars": 1012,
            "charErrors": 8,
            "entityRecognition": 1,
            "entitiesUttered": 3,
            "entitiesRecognized": 3,
            "entityRecall": 1,
            "entitiesAcquiredUttered": 2,
            "entitiesAcquiredRecognized": 2
          },
          "conversation": {
            "turnTakingScore": 0.92,
            "heardE2eLatencyP50Ms": 1240,
            "heardE2eLatencyP95Ms": 1810,
            "heardE2eLatencyP99Ms": 2050,
            "timeToYieldMs": 380,
            "eotMispredictionCount": 0,
            "overlapRatio": 0.03,
            "overlapSpeechMs": 640,
            "totalSpeechMs": 21400,
            "awkwardSilenceCount": 0,
            "unansweredTurns": 0,
            "falseInterruptionCount": 0,
            "e2eLatencyMs": 1180
          },
          "turns": [
            {
              "index": 2,
              "role": "ASSISTANT",
              "startMs": 5180,
              "endMs": 11240,
              "sttDelayMs": 290,
              "llmTtftMs": 840,
              "llmTtfsMs": 1120,
              "llmTps": 117.2,
              "ttsTtfbMs": 231,
              "e2eLatencyMs": 1190,
              "heardE2eLatencyMs": 1260,
              "timeToYieldMs": 0,
              "turnTakingScore": 1,
              "conciseness": 1,
              "eotMisprediction": false,
              "awkwardSilence": false,
              "unanswered": false
            }
          ],
          "judgeModel": "google/gemma-4-31b-it",
          "hasRemoteSession": true,
          "t0": "2026-01-15T01:30:53.412Z",
          "conciseness": 0.875,
          "unnecessaryToolCalls": false,
          "informationLoss": false,
          "redundantStatements": false,
          "poorQuestionQuality": false,
          "conversationProgression": 1
        },
        "scenarioId": "SCN_7Qa2WkPtR8mz",
        "sample": 1
      }
    ],
    "agentName": "support-agent",
    "scenarioGroup": {
      "name": "Billing questions",
      "scenarios": [
        {
          "label": "Customer asks why an invoice went up",
          "instructions": "You are a customer whose monthly invoice rose from $40 to $115. Ask why, and push back once if the first answer is vague.",
          "agentExpectations": "The agent names the line item that changed, gives the amount, and offers to send the invoice. It does not promise a refund.",
          "tags": {},
          "userdata": "",
          "id": "SCN_7Qa2WkPtR8mz"
        }
      ],
      "id": ""
    },
    "endedAt": "2026-01-15T01:33:57Z",
    "jobCount": 12,
    "passedCount": 10,
    "failedCount": 2,
    "numSimulations": 12,
    "usage": {
      "textTurnsCount": 0,
      "audioTurnsCount": 108
    },
    "concurrency": 4,
    "mode": "SIMULATION_MODE_AUDIO",
    "metrics": {
      "scenarioPassRate": 0.8333333,
      "overallScore": 0.8125,
      "stt": {
        "wer": 0.0523,
        "words": 1530,
        "wordErrors": 80,
        "cer": 0.0216,
        "chars": 7980,
        "charErrors": 172,
        "sttDelayMs": 335,
        "entityRecognition": 0.9583333,
        "entitiesUttered": 24,
        "entitiesRecognized": 23,
        "entityRecall": 1,
        "entitiesAcquiredUttered": 12,
        "entitiesAcquiredRecognized": 12
      },
      "llm": {
        "ttftMs": 940,
        "ttfsMs": 1265,
        "tokensPerSecond": 128.4
      },
      "tts": {
        "ttfbMs": 251,
        "wer": 0.0152,
        "words": 2240,
        "wordErrors": 34,
        "cer": 0.008,
        "chars": 12150,
        "charErrors": 97,
        "entityRecognition": 0.9722222,
        "entitiesUttered": 36,
        "entitiesRecognized": 35,
        "entityRecall": 1,
        "entitiesAcquiredUttered": 24,
        "entitiesAcquiredRecognized": 24
      },
      "conversation": {
        "turnTakingScore": 0.88,
        "heardE2eLatencyP50Ms": 1310,
        "heardE2eLatencyP95Ms": 1960,
        "heardE2eLatencyP99Ms": 2380,
        "timeToYieldMs": 380,
        "eotMispredictionCount": 0,
        "overlapRatio": 0.03,
        "overlapSpeechMs": 640,
        "totalSpeechMs": 21400,
        "awkwardSilenceCount": 0,
        "unansweredTurns": 0,
        "falseInterruptionCount": 0,
        "e2eLatencyMs": 1225
      },
      "conciseness": 0.875,
      "conversationProgression": 0.9166667
    },
    "summaryZstd": "",
    "issueCount": 1
  }
}

Count simulation runs

GET

/v0/projects/{projectId}/simulation-runs:counts

Returns a histogram of the runs in the window [startTime, endTime). It takes the same filters as ListSimulationRuns, so the histogram and the list describe the same set of runs. Both bounds are required, and endTime must be after startTime.

Path parameters

projectId

Required

string

The projectId path parameter.

Query parameters

startTimestring (date-time)

The start of the window on the creation time of the run, inclusive. Required.

endTimestring (date-time)

The end of the window on the creation time of the run, exclusive. Required. Must be after startTime.

statusSimulationRun.Status

A run status. Counts only the runs in this status.

To learn more, see SimulationRun.Status.

agentNamestring

An agent name. Counts only the runs of the agent with this exact name.

modeSimulationMode

A conversation mode. Counts only the runs in this mode.

To learn more, see SimulationMode.

Response fields

bucketsArray<Bucket>

The buckets that hold at least one run, in ascending order of bucketStart. The response does not include empty buckets. Each run is in exactly one bucket, so the sum of the counts is the number of runs in the window.

To learn more, see Bucket.

intervalduration

The width of each bucket. The API chooses the width from the window: the window has at most 60 buckets, and a bucket is at least one minute wide. Use the width to add the empty buckets, with a count of 0.

Request

curl -X GET "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs:counts" \
  -G \
  --data-urlencode "startTime=$START_TIME" \
  --data-urlencode "endTime=$END_TIME" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN"

Response

{
  "buckets": [
    {
      "bucketStart": "2026-01-15T01:00:00Z",
      "count": "12"
    }
  ],
  "interval": "3600s"
}

Cancel simulation run

POST

/v0/projects/{projectId}/simulation-runs/{simulationRunId}:cancel

Stops a run that has not finished. When the run is already finished, the call succeeds and has no effect.

Path parameters

projectId

Required

string

The projectId path parameter.

simulationRunId

Required

string

The simulationRunId path parameter.

Request

curl -X POST "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs/${SIMULATION_RUN_ID}:cancel" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN"

Response

{}

Confirm simulation source upload

POST

/v0/projects/{projectId}/simulation-runs/{simulationRunId}:confirmSourceUpload

Confirms that the agent source is uploaded and starts scenario generation. The run goes to GENERATING, then to RUNNING when the scenarios exist. The call is valid only while the run is PENDING_UPLOAD, so a repeated call fails.

Path parameters

projectId

Required

string

The projectId path parameter.

simulationRunId

Required

string

The simulationRunId path parameter.

Request body

codeEntrypointstring

The entrypoint of the uploaded source, for example agent.py.

Request

curl -X POST "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs/${SIMULATION_RUN_ID}:confirmSourceUpload" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "codeEntrypoint": "agent.py"
  }'

Response

{}

Get simulation run summary

GET

/v0/projects/{projectId}/simulation-runs/{simulationRunId}/summary

Returns the graded outcome of a finished run: the pass and fail counts, what went well, what to improve, the flagged issues with suggestions, and the chat history of each job. Returns FailedPrecondition while the run is in progress. Returns NotFound when the run finished without a summary.

Returns the SimulationRunSummary object.

Path parameters

projectId

Required

string

The projectId path parameter.

simulationRunId

Required

string

The simulationRunId path parameter.

Request

curl -X GET "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/simulation-runs/${SIMULATION_RUN_ID}/summary" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN"

Response

{
  "summary": {
    "passed": 10,
    "failed": 2,
    "goingWell": "The agent names the line item that changed and gives the amount <ref job=\"SRJ_4hVnc8RtKqmW\" item=\"item_4f21a7c9b80e\">rather than describing the invoice in general</ref>, and it offers to send the invoice without being asked.",
    "toImprove": "Treat agreement to discuss cancellation as a question, not as permission to start one. Ask for a yes before any account change.",
    "issues": [
      {
        "description": "The agent began a cancellation after the customer only asked what would happen <ref job=\"SRJ_qp7XwLs2EdBr\" item=\"item_9c3e1d5a72bf\">to the rest of the term</ref>.",
        "suggestion": "Make the account change a separate yes/no step, and record the answer before calling the tool.",
        "label": "Consent"
      }
    ],
    "chatHistory": {
      "SRJ_4hVnc8RtKqmW": {
        "items": [
          {
            "message": {
              "id": "item_4f21a7c9b80e",
              "role": "ASSISTANT",
              "content": [
                {
                  "text": "Your plan stayed the same. The change is session minutes: 4,210 this month against 1,100 last month, at $0.018 a minute. I can send you the invoice."
                }
              ],
              "interrupted": false,
              "extra": {},
              "metrics": {
                "startedSpeakingAt": "2026-01-15T01:31:02.614Z",
                "stoppedSpeakingAt": "2026-01-15T01:31:09.127Z",
                "llmNodeTtft": 0.412,
                "ttsNodeTtfb": 0.238,
                "e2eLatency": 1.18,
                "llmNodeTps": 117.2,
                "llmNodeTtfs": 0.627
              },
              "createdAt": "2026-01-15T01:31:02Z"
            }
          }
        ]
      }
    }
  }
}

Generate scenario from session

POST

/v0/projects/{projectId}/scenarios:generateFromSession

Generates a scenario (label, instructions, and expectations) from a recorded room session. Use the scenario in the scenarioGroup of a later run. The API does not save the scenario: the response is the only copy. The session must have a recorded chat history. Each call runs an LLM on the chat history, so it counts against the scenario-generation quota of the project. A call can take up to about one minute.

Returns the Scenario object.

Path parameters

projectId

Required

string

The projectId path parameter.

Request body

roomIdstring

The ID of the room session (RM_...) to generate the scenario from.

regionstring

Ignored. The API finds the session recording by project and room.

Request

curl -X POST "https://beta-api.livekit.cloud/v0/projects/${PROJECT_ID}/scenarios:generateFromSession" \
  -H "Authorization: Bearer $LIVEKIT_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "roomId": "RM_u52jUOHjoTIF"
  }'

Response

{
  "scenario": {
    "label": "Customer checks on a late order",
    "instructions": "You are a customer whose order 48213 still says processing. Give the order number when asked, and ask when it will arrive.",
    "agentExpectations": "The agent asks for the order number, looks it up, and gives the status and the delivery date.",
    "tags": {},
    "userdata": "{\"orderId\":\"48213\"}",
    "id": "SCN_2bHx6NqWe9Ld"
  }
}

Types

Types used by these endpoints, alongside the simulation run object.

agent.AgentHandoff

idstring

The id value.

oldAgentIdstring

The oldAgentId value.

newAgentIdstring

The newAgentId value.

createdAttimestamp

The createdAt value.

AgentConfigUpdate

idstring

The id value.

instructionsstring

The instructions value.

toolsAddedArray<string>

The toolsAdded value.

toolsRemovedArray<string>

The toolsRemoved value.

createdAttimestamp

The createdAt value.

Bucket

bucketStarttimestamp

The bucketStart value.

countint64

The count value.

ChatContent

textstring

The text value.

ChatContext

itemsArray<ChatItem>

The items value.

To learn more, see ChatItem.

ChatItem

agentConfigUpdateAgentConfigUpdate

The agentConfigUpdate value.

At most one of agentConfigUpdate, agentHandoff, functionCall, functionCallOutput, and message is set.

To learn more, see AgentConfigUpdate.

agentHandoffagent.AgentHandoff

The agentHandoff value.

At most one of agentConfigUpdate, agentHandoff, functionCall, functionCallOutput, and message is set.

To learn more, see agent.AgentHandoff.

functionCallFunctionCall

The functionCall value.

At most one of agentConfigUpdate, agentHandoff, functionCall, functionCallOutput, and message is set.

To learn more, see FunctionCall.

functionCallOutputFunctionCallOutput

The functionCallOutput value.

At most one of agentConfigUpdate, agentHandoff, functionCall, functionCallOutput, and message is set.

To learn more, see FunctionCallOutput.

messageChatMessage

The message value.

At most one of agentConfigUpdate, agentHandoff, functionCall, functionCallOutput, and message is set.

To learn more, see ChatMessage.

ChatMessage

idstring

The id value.

roleChatRole

The role value.

To learn more, see ChatRole.

contentArray<ChatContent>

The content value.

To learn more, see ChatContent.

interruptedboolean

The interrupted value.

transcriptConfidencenumber

The transcriptConfidence value.

extraMap<string, string>

The extra value.

metricsMetricsReport

The metrics value.

To learn more, see MetricsReport.

createdAttimestamp

The createdAt value.

ChatRole

This type has these values:

DEVELOPER
SYSTEM
USER
ASSISTANT

CI

The pipeline this run came from. Set by the client from its CI environment; absent for runs not started by a pipeline.

providerstring

E.g. github_actions

commitShastring

The commitSha value.

refstring

Branch or tag.

pullRequeststring

The pullRequest value.

runUrlstring

Link back to the CI job.

actorstring

The actor value.

Conversation

Audio only.

turnTakingScorenumber

0-1.

heardE2eLatencyP50Msint32

Floor-transfer offset; a gap is > 0, an overlap < 0.

heardE2eLatencyP95Msint32

The heardE2eLatencyP95Ms value.

heardE2eLatencyP99Msint32

The heardE2eLatencyP99Ms value.

timeToYieldMsinteger

Agent audio continuing past a barge-in.

eotMispredictionCountinteger

Agent started before the caller's turn ended.

overlapRationumber

Overlapping speech / total speech.

overlapSpeechMsinteger

Pooling stats for overlap_ratio

totalSpeechMsinteger

The totalSpeechMs value.

awkwardSilenceCountinteger

Gaps past the natural-pause threshold.

unansweredTurnsinteger

The simulated party spoke, the agent never responded.

falseInterruptionCountinteger

Needs the RemoteSession; agent paused for a non-interruption.

e2eLatencyMsinteger

Needs the RemoteSession; the agent's own claim.

FunctionCall

idstring

The id value.

callIdstring

The callId value.

argumentsstring

The arguments value.

namestring

The name value.

createdAttimestamp

The createdAt value.

FunctionCallOutput

idstring

The id value.

namestring

The name value.

callIdstring

The callId value.

outputstring

The output value.

isErrorboolean

The isError value.

createdAttimestamp

The createdAt value.

Issue

descriptionstring

The description value.

suggestionstring

The suggestion value.

labelstring

The label value.

Job

idstring

The id value.

statusJob.Status

The status value.

To learn more, see Job.Status.

instructionsstring

The instructions value.

errorstring

The error value.

agentExpectationsstring

The agentExpectations value.

labelstring

The label value.

tagsArray<string>

The tags value.

roomNamestring

The roomName value.

startedAttimestamp

The startedAt value.

endedAttimestamp

The endedAt value.

roomIdstring

The roomId value.

usageJob.Usage

The usage value.

To learn more, see Job.Usage.

metricsJobMetrics

Quality metrics for this job's call. Unset when the job produced none.

To learn more, see JobMetrics.

scenarioIdstring

The Scenario.id this job ran; empty for generated runs. Every sample of one scenario shares it, so a run's jobs group back into scenarios.

sampleint32

1-based sample of that scenario within the run; always 1 when the run's samples is 1.

Job.Status

This type has these values:

STATUS_PENDING
STATUS_RUNNING
STATUS_COMPLETED
STATUS_FAILED
STATUS_CANCELLED

Job.Usage

textTurnsCountint32

The textTurnsCount value.

audioTurnsCountint32

The audioTurnsCount value.

JobMetrics

taskCompletionnumber

The scenario verdict as 1/0.

overallScorenumber

Weighted mean of the judged and heard scores.

sttSTT

The stt value.

To learn more, see STT.

llmLLM

The llm value.

To learn more, see LLM.

ttsTTS

The tts value.

To learn more, see TTS.

conversationConversation

The conversation value.

To learn more, see Conversation.

turnsArray<Turn>

One entry per turn, both speakers.

To learn more, see Turn.

judgeModelstring

Text judge for judged scores; "" if none ran.

hasRemoteSessionboolean

false = waveform-only capture (e.g. SIP)

t0timestamp

Audio only; call start (first speech edge); turn times relative to it.

concisenessnumber

Judged over the authored dialog (text and audio); needs the text judge. judged {0, 0.5, 1} per agent turn; job = mean.

unnecessaryToolCallsboolean

Whole-call issue flags; true = present, absent = not judged.

informationLossboolean

The informationLoss value.

redundantStatementsboolean

The redundantStatements value.

poorQuestionQualityboolean

The poorQuestionQuality value.

conversationProgressionnumber

0-1, derived from the flags above.

LLM

ttftMsinteger

Needs the RemoteSession; time to first token.

ttfsMsinteger

Needs the RemoteSession; time to first sentence, the smallest speakable unit.

tokensPerSecondnumber

Needs the RemoteSession.

MetricsReport

startedSpeakingAttimestamp

The startedSpeakingAt value.

stoppedSpeakingAttimestamp

The stoppedSpeakingAt value.

transcriptionDelaynumber

The transcriptionDelay value.

endOfTurnDelaynumber

The endOfTurnDelay value.

onUserTurnCompletedDelaynumber

The onUserTurnCompletedDelay value.

llmNodeTtftnumber

The llmNodeTtft value.

ttsNodeTtfbnumber

The ttsNodeTtfb value.

e2eLatencynumber

The e2eLatency value.

llmNodeTpsnumber

The llmNodeTps value.

llmNodeTtfsnumber

The llmNodeTtfs value.

PresignedPostRequest

urlstring

The url value.

valuesMap<string, string>

The values value.

RunMetrics

Aggregates over this run's jobs, so a run summary does not have to pull every job and turn. Reuses the JobMetrics group shapes.

scenarioPassRatenumber

Share of jobs whose scenario verdict passed.

overallScorenumber

Mean over scored jobs.

sttSTT

The stt value.

To learn more, see STT.

llmLLM

The llm value.

To learn more, see LLM.

ttsTTS

The tts value.

To learn more, see TTS.

conversationConversation

The conversation value.

To learn more, see Conversation.

concisenessnumber

Means over the jobs that report them.

conversationProgressionnumber

The conversationProgression value.

SamplingOptions

Runs each scenario more than once, so a flaky sample is distinguishable from a broken scenario. Both fields are required when it is present.

samplesint32

Samples per scenario, > 1. The run holds scenarios × samples jobs, subject to the usual job cap.

passRatenumber

The share of a scenario's samples that must pass for that scenario to pass, 0-1. A scenario needs round(rate × samples) of them, so 1 requires every sample and 0.67 of 3 requires 2.

Scenario

A single scenario, mirroring one entry in a scenarios.yaml file. Scenarios are no longer stored server-side; the yaml file is the source of truth.

labelstring

The label value.

instructionsstring

The instructions value.

agentExpectationsstring

The agentExpectations value.

tagsMap<string, string>

Set as participant attributes on the simulation participant's access token.

userdatastring

Arbitrary JSON-encoded object, surfaced to the agent under test as SimulationContext userdata (e.g. inputs and benchmark target state).

idstring

SCN_-prefixed guid. Stable across edits to label, instructions, or expectations, so runs of the same scenario can be correlated over time.

ScenarioGroup

A named group of scenarios, mirroring a whole scenarios.yaml file.

namestring

The name value.

scenariosArray<Scenario>

The scenarios value.

To learn more, see Scenario.

idstring

SCNG_-prefixed guid. Stable identity of the scenarios file, independent of its name.

SimulationMode

How the simulated user interacts with the agent under test.

This type has these values:

SIMULATION_MODE_UNSPECIFIED
SIMULATION_MODE_TEXT
SIMULATION_MODE_AUDIO

SimulationRun.Status

This type has these values:

STATUS_PENDING_UPLOAD
STATUS_GENERATING
STATUS_RUNNING
STATUS_SUMMARIZING
STATUS_COMPLETED
STATUS_FAILED
STATUS_CANCELLED

SimulationRun.Usage

textTurnsCountint32

The textTurnsCount value.

audioTurnsCountint32

The audioTurnsCount value.

SimulationRunSummary

passedint32

The passed value.

failedint32

The failed value.

goingWellstring

The goingWell value.

toImprovestring

The toImprove value.

issuesArray<Issue>

The issues value.

To learn more, see Issue.

chatHistoryMap<string, ChatContext>

The chatHistory value.

To learn more, see ChatContext.

STT

The agent's perception of the caller. Audio only.

wernumber

Needs the RemoteSession; word_errors / words.

wordsinteger

Needs the RemoteSession.

wordErrorsinteger

Needs the RemoteSession.

cernumber

Needs the RemoteSession; char_errors / chars.

charsinteger

Needs the RemoteSession.

charErrorsinteger

Needs the RemoteSession.

sttDelayMsinteger

Needs the RemoteSession; agent-reported STT/endpointing delay.

entityRecognitionnumber

Key entities (names, IDs, amounts) the caller said. Needs the text judge. recognized / uttered.

entitiesUtteredinteger

The entitiesUttered value.

entitiesRecognizedinteger

The entitiesRecognized value.

entityRecallnumber

Retention: an entity is acquired at the first saying heard right, and these count only the sayings after that, so recall separates keeping an entity from ever getting it. acquired_recognized / acquired_uttered

entitiesAcquiredUtteredinteger

The entitiesAcquiredUttered value.

entitiesAcquiredRecognizedinteger

The entitiesAcquiredRecognized value.

TTS

The agent's voice. Audio only.

ttfbMsinteger

Needs the RemoteSession; provider byte-level TTFB.

wernumber

Needs the RemoteSession; word_errors / words.

wordsinteger

Needs the RemoteSession.

wordErrorsinteger

Needs the RemoteSession.

cernumber

Needs the RemoteSession; char_errors / chars.

charsinteger

Needs the RemoteSession.

charErrorsinteger

Needs the RemoteSession.

entityRecognitionnumber

Key entities the agent said, as the caller heard them. Needs the text judge. recognized / uttered.

entitiesUtteredinteger

The entitiesUttered value.

entitiesRecognizedinteger

The entitiesRecognized value.

entityRecallnumber

acquired_recognized / acquired_uttered

entitiesAcquiredUtteredinteger

The entitiesAcquiredUttered value.

entitiesAcquiredRecognizedinteger

The entitiesAcquiredRecognized value.

Turn

Rows exist for text and audio. The groups above hold the means and pools over these; anything not measured for a turn stays unset.

indexinteger

1-based, conversation order.

roleChatRole

ASSISTANT = the agent under test, USER = simulator persona.

To learn more, see ChatRole.

startMsinteger

Audio only; relative to t0; unset if the turn could not be aligned to the audio.

endMsinteger

Audio only.

sttDelayMsinteger

Needs the RemoteSession.

llmTtftMsinteger

Needs the RemoteSession.

llmTtfsMsinteger

Needs the RemoteSession.

llmTpsnumber

Needs the RemoteSession.

ttsTtfbMsinteger

Needs the RemoteSession.

e2eLatencyMsinteger

Needs the RemoteSession.

heardE2eLatencyMsint32

The heardE2eLatencyMs value.

timeToYieldMsinteger

The timeToYieldMs value.

turnTakingScorenumber

The turnTakingScore value.

concisenessnumber

Needs the text judge.

eotMispredictionboolean

The eotMisprediction value.

awkwardSilenceboolean

The awkwardSilence value.

unansweredboolean

The unanswered value.

Overview

The simulation run object

List simulation runs

Create simulation run

Get simulation run

Count simulation runs

Cancel simulation run

Confirm simulation source upload

Get simulation run summary

Generate scenario from session

Types

agent.AgentHandoff

AgentConfigUpdate

Bucket

ChatContent

ChatContext

ChatItem

ChatMessage

ChatRole

CI

Conversation

FunctionCall

FunctionCallOutput

Issue

Job

Job.Status

Job.Usage

JobMetrics

LLM

MetricsReport

PresignedPostRequest

RunMetrics

SamplingOptions

Scenario

ScenarioGroup

SimulationMode

SimulationRun.Status

SimulationRun.Usage

SimulationRunSummary

STT

TTS

Turn

Ask LiveKit