Module livekit.plugins.zoom_ai
Zoom AI plugin for LiveKit Agents.
Speech-to-text with Zoom Scribe: STT streams over Scribe Live and
STT(mode="fast") transcribes one utterance per request with Scribe Fast.
See https://developers.zoom.us/docs/ai-services/scribe/ for more information.
Classes
class STT (*,
mode: "Literal['live', 'fast']" = 'live',
api_key: NotGivenOr[str] = NOT_GIVEN,
language: str = 'en-US',
model: str | None = None,
language_detection: str | None = None,
turn_detection: "Literal['server', 'none']" = 'server',
silence_duration_ms: int | None = 350,
diarization: bool = False,
word_time_offsets: bool = False,
channel_separation: bool = False,
vocabulary: list[str] | None = None,
text_polishing: bool = False,
extra_kwargs: dict[str, Any] | None = None,
base_url: str = 'https://api.zoom.us/v2/aiservices/scribe',
live_url: NotGivenOr[str] = NOT_GIVEN,
fast_url: NotGivenOr[str] = NOT_GIVEN,
http_session: aiohttp.ClientSession | None = None)-
Expand source code
class STT(stt.STT): """Zoom Scribe speech-to-text: streaming (Scribe Live) or per-utterance (Scribe Fast). Scribe Live detects the end of each turn itself, so use STT turn detection:: session = AgentSession( stt=zoom_ai.STT(), turn_handling={"turn_detection": "stt"}, ... ) Without it, LiveKit's VAD and turn detector add their own delay on top of Scribe's, and a turn can be committed before a later final arrives. To let LiveKit's VAD (and turn detector) decide when the user is done instead, turn Scribe's VAD off with ``turn_detection="none"``. Each time LiveKit ends the user's speech, the plugin commits the audio so far and Scribe returns its final transcript:: session = AgentSession( stt=zoom_ai.STT(turn_detection="none"), vad=silero.VAD.load(), turn_handling={"turn_detection": inference.TurnDetector()}, # or "vad" ... ) In Live mode, ``recognize()`` (batch recognition) uses Scribe Fast. """ def __init__( self, *, mode: Literal["live", "fast"] = "live", api_key: NotGivenOr[str] = NOT_GIVEN, language: str = "en-US", model: str | None = None, language_detection: str | None = None, turn_detection: Literal["server", "none"] = "server", silence_duration_ms: int | None = DEFAULT_SILENCE_DURATION_MS, diarization: bool = False, word_time_offsets: bool = False, channel_separation: bool = False, vocabulary: list[str] | None = None, text_polishing: bool = False, extra_kwargs: dict[str, Any] | None = None, base_url: str = BASE_URL, live_url: NotGivenOr[str] = NOT_GIVEN, fast_url: NotGivenOr[str] = NOT_GIVEN, http_session: aiohttp.ClientSession | None = None, ) -> None: """Create a Zoom Scribe STT. Options map to the Scribe API fields of the same name. For what each feature does, and its availability, pricing and enablement, see https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/ Args: mode: ``"live"`` (streaming, default) or ``"fast"`` (one request per utterance). api_key: Zoom AI Services API key (from https://www.zoom.ai/). Defaults to ``$ZOOM_AI_API_KEY``. It is sent on every request and reconnect, so it must stay valid for as long as the agent runs. language: A Scribe locale, e.g. ``"en-US"``; base codes such as ``"en"`` map to their locale. A regional variant Scribe doesn't list resolves to the locale of its base language (``"en-GB"`` to ``"en-US"``); anything else is sent as given, for Scribe to validate. For the supported languages of each model, see https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/ model: Scribe speech recognition model (``config.model``). ``None`` uses Scribe's default model. language_detection: Scribe language detection mode (``config.language_detection.mode``; Live supports ``"continuous"``). ``None`` turns it off. A detected language Scribe reports is in ``SpeechData.language``; when Scribe reports none, it is empty. With ``zoom-scribe-multilingual`` in Live, set it to get each final's language. turn_detection: Live only: who ends each turn. ``"server"`` (default): Scribe's voice activity detection, with ``silence_duration_ms``; use ``turn_handling={"turn_detection": "stt"}``. ``"none"``: Scribe's voice activity detection is off (``turn_detection.type``), and each STT flush from LiveKit (at the end of speech from its VAD, or at the end of turn from its turn detector) sends ``input_audio_buffer.commit``, for which Scribe returns the final transcript. Use it with a VAD and ``turn_handling={"turn_detection": "vad"}`` or a turn detector model, not ``"stt"``: in this mode the STT sends no start or end of speech events, and finals have no ``speech_end_time`` (LiveKit's VAD times the turn). Every flush gets one final; a flush after only digital silence gets an empty one without a commit. silence_duration_ms: Live only: milliseconds of silence that end a turn in Scribe's voice activity detection; not used with ``turn_detection="none"``. Defaults to 350, tuned for voice agents; lower is faster, higher keeps sentences with longer pauses together. For transcription without an agent, a longer value such as 600 can improve accuracy. ``None`` uses Scribe's default (see the Scribe docs). diarization: Label each final transcript with a speaker. Off by default. With one speaker in the utterance, ``SpeechData.speaker_id`` is Scribe's label. With several, ``speaker_id`` is empty and ``metadata["speaker_segments"]`` has each one's span. In Fast mode labels are per request, so they aren't consistent across utterances. ``MultiSpeakerAdapter`` works with Live mode only. word_time_offsets: Fast only: word timings (``config.word_time_offsets``), in ``SpeechData.words``. Off by default. channel_separation: Fast only: recognize each channel of stereo audio separately (``config.channel_separation``); each segment's channel is in ``metadata["channel_segments"]``. Off by default. vocabulary: Words and phrases that customize recognition (``config.vocabulary.phrases``). text_polishing: Scribe's text polishing (``outputs.text_polishing``). The polished text is the transcript, and the recognized text is in ``metadata["transcript"]``; if polishing fails, the recognized text is used. Word timings aren't returned with polished text. ``en-US`` only, and not with language detection. Off by default. extra_kwargs: Scribe request fields to add or override, shaped like the request, e.g. ``{"config": {"language_detection": {"candidate_languages": ["en-US", "fr-FR"]}}}`` to restrict language detection. Merged into the Live ``session.update`` and the Fast request, over the options above, so Scribe fields this plugin has no option for can be used. ``audio.format`` stays ``pcm16``. base_url: Scribe API base URL; Live uses ``<base>/live`` (as ``wss``) and Fast uses ``<base>/transcribe``. A plaintext (``http``/``ws``) URL to a host other than localhost logs a warning. live_url: Override the Scribe Live WebSocket URL. fast_url: Override the Scribe Fast HTTP URL. http_session: Optional aiohttp session; the job's session is used otherwise. """ if mode not in ("live", "fast"): raise ValueError(f'mode must be "live" or "fast", got {mode!r}') if turn_detection not in ("server", "none"): raise ValueError(f'turn_detection must be "server" or "none", got {turn_detection!r}') if mode == "fast" and turn_detection == "none": logger.warning( 'turn_detection="none" is ignored with mode="fast": Scribe Fast transcribes ' "one utterance per request" ) if extra_kwargs is not None and not isinstance(extra_kwargs, dict): raise ValueError(f"extra_kwargs must be a dict, got {type(extra_kwargs).__name__}") extra_kwargs = dict(extra_kwargs or {}) super().__init__( capabilities=stt.STTCapabilities( streaming=mode == "live", interim_results=False, # not in the GA Scribe Live API diarization=diarization or bool((extra_kwargs.get("config") or {}).get("diarization")), offline_recognize=True, # With Scribe's VAD off, LiveKit flushes the stream to end each turn. manual_flush=mode == "live" and turn_detection == "none", ) ) api_key = api_key if is_given(api_key) else os.environ.get("ZOOM_AI_API_KEY", "") if not api_key: raise ValueError("Zoom AI API key is required: pass api_key or set ZOOM_AI_API_KEY") self._api_key = api_key self._mode = mode base = base_url.rstrip("/") scheme, sep, rest = base.partition("://") if not sep or scheme.lower() not in ("http", "https"): raise ValueError(f"base_url must be an http(s) URL, got {base_url!r}") _check_url(base, "base_url") ws_scheme = "wss" if scheme.lower() == "https" else "ws" self._live_url = _check_url( live_url if is_given(live_url) else f"{ws_scheme}://{rest}/live", "live_url" ) self._fast_url = _check_url( fast_url if is_given(fast_url) else base + "/transcribe", "fast_url" ) self._session = http_session self._opts = STTOptions( language=_to_locale(language), turn_detection=turn_detection, diarization=diarization, silence_duration_ms=silence_duration_ms, word_time_offsets=word_time_offsets, channel_separation=channel_separation, vocabulary=vocabulary, text_polishing=text_polishing, model=model, language_detection=language_detection, extra_kwargs=extra_kwargs, ) self._streams = weakref.WeakSet[SpeechStream]() @property def model(self) -> str: """The Scribe model set with ``model``, else the mode: ``"scribe-live"`` or ``"scribe-fast"``.""" return self._opts.model or f"scribe-{self._mode}" @property def provider(self) -> str: """The provider name, ``"Zoom"``.""" return "Zoom" @property def _http(self) -> aiohttp.ClientSession: if not self._session: self._session = utils.http_context.http_session() return self._session def _headers(self, tracking_id: str) -> dict[str, str]: return { "Authorization": f"Bearer {self._api_key}", "User-Agent": USER_AGENT, "x-zm-trackingid": tracking_id, } def _fast_request(self, locale: str) -> dict[str, Any]: """The Fast request fields besides the audio: the options, then extra_kwargs.""" config: dict[str, Any] = {"language": locale, **_recognition_config(self._opts)} if self._opts.word_time_offsets: config["word_time_offsets"] = True if self._opts.channel_separation: config["channel_separation"] = True request: dict[str, Any] = {"config": config} if self._opts.text_polishing: request["outputs"] = {"text_polishing": {"enabled": True}} request = _merge(request, self._opts.extra_kwargs) request.pop("file", None) # the plugin uploads the audio return request # ---------------------------------------------------------------- Scribe Fast async def _recognize_impl( self, buffer: AudioBuffer, *, language: NotGivenOr[str] = NOT_GIVEN, conn_options: APIConnectOptions, ) -> stt.SpeechEvent: locale = _to_locale(language) if is_given(language) else self._opts.language request = self._fast_request(locale) view = _RequestView.of(request) # Upload the utterance as multipart form data: `file`, then one JSON field per # top-level request field (`config`, `outputs`, ...). form = aiohttp.FormData() form.add_field( "file", rtc.combine_audio_frames(buffer).to_wav_bytes(), filename="utterance.wav", content_type="audio/wav", ) for field, value in request.items(): form.add_field(field, json.dumps(value)) # Each request has its own tracking ID; it is the request_id of an error, so a # failure can be matched in Zoom's logs. tracking_id = _tracking_id() try: async with self._http.post( self._fast_url, data=form, headers=self._headers(tracking_id), timeout=aiohttp.ClientTimeout(total=30, sock_connect=conn_options.timeout), ) as resp: body = await resp.text() if resp.status != 200: # Zoom explains a refusal (a bad API key, say) in the JSON body. reason = _zoom_reason(body) raise APIStatusError( f"Zoom Scribe Fast error (HTTP {resp.status})" + (f": {reason}" if reason else ""), status_code=resp.status, request_id=tracking_id, body=body, ) try: result = json.loads(body) except json.JSONDecodeError: raise APIStatusError( "Zoom Scribe Fast returned a non-JSON response", status_code=resp.status, request_id=tracking_id, ) from None except asyncio.TimeoutError as e: raise APITimeoutError() from e except aiohttp.ClientError as e: raise APIConnectionError() from e text = _clean((result.get("result") or {}).get("text_display")) segments = (result.get("result") or {}).get("segments") or [] # The detected language, when Scribe reports one. Scribe Fast reports it as # result.language with one-time detection; with continuous detection, production # omits it from the result and the segments (as the docs say), so the language is # then empty. A segment's language is used if Scribe ever sends one. detected = (result.get("result") or {}).get("language") or next( (s["language"] for s in segments if s.get("language")), None ) speaker_id: str | None = None metadata: dict[str, Any] | None = None polished = _polished_text(result) if view.text_polishing else None if polished is not None: metadata = {"transcript": text} text = polished if view.diarization: # One speaker: speaker_id. Several: speaker_id is empty and each one's span is # in metadata["speaker_segments"] (as Scribe Live does). speakers = {sg["speaker"] for sg in segments if sg.get("speaker")} if len(speakers) == 1: speaker_id = next(iter(speakers)) elif len(speakers) > 1: metadata = { **(metadata or {}), "speaker_segments": [ {k: s.get(k) for k in ("speaker", "text", "start", "end")} for s in segments ], } if view.channel_separation: metadata = { **(metadata or {}), "channel_segments": [ {k: s.get(k) for k in ("channel", "text", "start", "end")} for s in segments ], } words = None # Word timings belong to the recognized text, not the polished one. if view.word_time_offsets and polished is None: words = [ TimedString(text=w.get("word", ""), start_time=w["start"], end_time=w["end"]) for s in segments for w in s.get("words") or [] if "start" in w and "end" in w ] or None return stt.SpeechEvent( type=stt.SpeechEventType.FINAL_TRANSCRIPT, request_id=result.get("request_id", ""), alternatives=[ stt.SpeechData( language=_reported_language(detected, view.language_detection, locale), text=text, start_time=segments[0].get("start", 0.0) if segments else 0.0, end_time=segments[-1].get("end", 0.0) if segments else 0.0, speaker_id=speaker_id, words=words, metadata=metadata, ) ], ) # ---------------------------------------------------------------- Scribe Live def stream( self, *, language: NotGivenOr[str] = NOT_GIVEN, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, ) -> SpeechStream: """Open a Scribe Live streaming session. Args: language: Override the STT's language for this stream. conn_options: Connection timeout and retry options. Raises: NotImplementedError: When the STT was created with ``mode="fast"``. """ if self._mode != "live": raise NotImplementedError( 'mode="fast" is not streaming; AgentSession wraps it with a VAD' ) opts = dataclasses.replace(self._opts) if is_given(language): opts.language = _to_locale(language) stream = SpeechStream(stt=self, opts=opts, conn_options=conn_options) self._streams.add(stream) return stream async def aclose(self) -> None: """Close every Live stream opened by :meth:`stream`. The HTTP session is left open: it is either supplied by the caller or owned by the job's HTTP context. """ streams = list(self._streams) self._streams.clear() # return_exceptions=True so one stream failing to close doesn't leave the rest open await asyncio.gather(*(stream.aclose() for stream in streams), return_exceptions=True) def update_options( self, *, language: NotGivenOr[str] = NOT_GIVEN, silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN, ) -> None: """Change options; open Live streams reconnect with the new session config. Args: language: Scribe locale, e.g. ``"en-US"``. silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's default. Not used with ``turn_detection="none"``. A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. """ if not is_given(language) and not is_given(silence_duration_ms): return if is_given(language): self._opts.language = _to_locale(language) if is_given(silence_duration_ms): self._opts.silence_duration_ms = silence_duration_ms for stream in self._streams: stream.update_options(language=language, silence_duration_ms=silence_duration_ms)Zoom Scribe speech-to-text: streaming (Scribe Live) or per-utterance (Scribe Fast).
Scribe Live detects the end of each turn itself, so use STT turn detection::
session = AgentSession( stt=zoom_ai.STT(), turn_handling={"turn_detection": "stt"}, ... )Without it, LiveKit's VAD and turn detector add their own delay on top of Scribe's, and a turn can be committed before a later final arrives.
To let LiveKit's VAD (and turn detector) decide when the user is done instead, turn Scribe's VAD off with
turn_detection="none". Each time LiveKit ends the user's speech, the plugin commits the audio so far and Scribe returns its final transcript::session = AgentSession( stt=zoom_ai.STT(turn_detection="none"), vad=silero.VAD.load(), turn_handling={"turn_detection": inference.TurnDetector()}, # or "vad" ... )In Live mode,
recognize()(batch recognition) uses Scribe Fast.Create a Zoom Scribe STT.
Options map to the Scribe API fields of the same name. For what each feature does, and its availability, pricing and enablement, see https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/
Args
mode"live"(streaming, default) or"fast"(one request per utterance).api_key- Zoom AI Services API key (from https://www.zoom.ai/). Defaults to
$ZOOM_AI_API_KEY. It is sent on every request and reconnect, so it must stay valid for as long as the agent runs. language- A Scribe locale, e.g.
"en-US"; base codes such as"en"map to their locale. A regional variant Scribe doesn't list resolves to the locale of its base language ("en-GB"to"en-US"); anything else is sent as given, for Scribe to validate. For the supported languages of each model, see https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/ model- Scribe speech recognition model (
config.model).Noneuses Scribe's default model. language_detection- Scribe language detection mode
(
config.language_detection.mode; Live supports"continuous").Noneturns it off. A detected language Scribe reports is inSpeechData.language; when Scribe reports none, it is empty. Withzoom-scribe-multilingualin Live, set it to get each final's language. turn_detection- Live only: who ends each turn.
"server"(default): Scribe's voice activity detection, withsilence_duration_ms; useturn_handling={"turn_detection": "stt"}."none": Scribe's voice activity detection is off (turn_detection.type), and each STT flush from LiveKit (at the end of speech from its VAD, or at the end of turn from its turn detector) sendsinput_audio_buffer.commit, for which Scribe returns the final transcript. Use it with a VAD andturn_handling={"turn_detection": "vad"}or a turn detector model, not"stt": in this mode the STT sends no start or end of speech events, and finals have nospeech_end_time(LiveKit's VAD times the turn). Every flush gets one final; a flush after only digital silence gets an empty one without a commit. silence_duration_ms- Live only: milliseconds of silence that end a turn in
Scribe's voice activity detection; not used with
turn_detection="none". Defaults to 350, tuned for voice agents; lower is faster, higher keeps sentences with longer pauses together. For transcription without an agent, a longer value such as 600 can improve accuracy.Noneuses Scribe's default (see the Scribe docs). diarization- Label each final transcript with a speaker. Off by default.
With one speaker in the utterance,
SpeechData.speaker_idis Scribe's label. With several,speaker_idis empty andmetadata["speaker_segments"]has each one's span. In Fast mode labels are per request, so they aren't consistent across utterances.MultiSpeakerAdapterworks with Live mode only. word_time_offsets- Fast only: word timings (
config.word_time_offsets), inSpeechData.words. Off by default. channel_separation- Fast only: recognize each channel of stereo audio
separately (
config.channel_separation); each segment's channel is inmetadata["channel_segments"]. Off by default. vocabulary- Words and phrases that customize recognition
(
config.vocabulary.phrases). text_polishing- Scribe's text polishing (
outputs.text_polishing). The polished text is the transcript, and the recognized text is inmetadata["transcript"]; if polishing fails, the recognized text is used. Word timings aren't returned with polished text.en-USonly, and not with language detection. Off by default. extra_kwargs- Scribe request fields to add or override, shaped like the
request, e.g.
{"config": {"language_detection": {"candidate_languages": ["en-US", "fr-FR"]}}}<code> to restrict language detection. Merged into the Live </code>session.updateand the Fast request, over the options above, so Scribe fields this plugin has no option for can be used.audio.formatstayspcm16. base_url- Scribe API base URL; Live uses
<base>/live(aswss) and Fast uses<base>/transcribe. A plaintext (http/ws) URL to a host other than localhost logs a warning. live_url- Override the Scribe Live WebSocket URL.
fast_url- Override the Scribe Fast HTTP URL.
http_session- Optional aiohttp session; the job's session is used otherwise.
Ancestors
- livekit.agents.stt.stt.STT
- abc.ABC
- EventEmitter
- typing.Generic
Instance variables
prop model : str-
Expand source code
@property def model(self) -> str: """The Scribe model set with ``model``, else the mode: ``"scribe-live"`` or ``"scribe-fast"``.""" return self._opts.model or f"scribe-{self._mode}"The Scribe model set with
model, else the mode:"scribe-live"or"scribe-fast". prop provider : str-
Expand source code
@property def provider(self) -> str: """The provider name, ``"Zoom"``.""" return "Zoom"The provider name,
"Zoom".
Methods
async def aclose(self) ‑> None-
Expand source code
async def aclose(self) -> None: """Close every Live stream opened by :meth:`stream`. The HTTP session is left open: it is either supplied by the caller or owned by the job's HTTP context. """ streams = list(self._streams) self._streams.clear() # return_exceptions=True so one stream failing to close doesn't leave the rest open await asyncio.gather(*(stream.aclose() for stream in streams), return_exceptions=True)Close every Live stream opened by :meth:
stream.The HTTP session is left open: it is either supplied by the caller or owned by the job's HTTP context.
def stream(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
conn_options: APIConnectOptions = APIConnectOptions(max_retry=3, retry_interval=2.0, timeout=10.0, liveness_timeout=6.0)) ‑> livekit.plugins.zoom_ai.stt.SpeechStream-
Expand source code
def stream( self, *, language: NotGivenOr[str] = NOT_GIVEN, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, ) -> SpeechStream: """Open a Scribe Live streaming session. Args: language: Override the STT's language for this stream. conn_options: Connection timeout and retry options. Raises: NotImplementedError: When the STT was created with ``mode="fast"``. """ if self._mode != "live": raise NotImplementedError( 'mode="fast" is not streaming; AgentSession wraps it with a VAD' ) opts = dataclasses.replace(self._opts) if is_given(language): opts.language = _to_locale(language) stream = SpeechStream(stt=self, opts=opts, conn_options=conn_options) self._streams.add(stream) return streamOpen a Scribe Live streaming session.
Args
language- Override the STT's language for this stream.
conn_options- Connection timeout and retry options.
Raises
NotImplementedError- When the STT was created with
mode="fast".
def update_options(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN) ‑> None-
Expand source code
def update_options( self, *, language: NotGivenOr[str] = NOT_GIVEN, silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN, ) -> None: """Change options; open Live streams reconnect with the new session config. Args: language: Scribe locale, e.g. ``"en-US"``. silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's default. Not used with ``turn_detection="none"``. A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. """ if not is_given(language) and not is_given(silence_duration_ms): return if is_given(language): self._opts.language = _to_locale(language) if is_given(silence_duration_ms): self._opts.silence_duration_ms = silence_duration_ms for stream in self._streams: stream.update_options(language=language, silence_duration_ms=silence_duration_ms)Change options; open Live streams reconnect with the new session config.
Args
language- Scribe locale, e.g.
"en-US". silence_duration_ms- Silence (ms) that ends a turn;
Noneuses Scribe's default. Not used withturn_detection="none".
A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it.
Inherited members
class SpeechStream (*,
stt: STT,
opts: STTOptions,
conn_options: APIConnectOptions)-
Expand source code
class SpeechStream(stt.SpeechStream): """One Scribe Live session; reconnects on option changes and server-side session ends.""" _carried: stt.SpeechData | None def __init__(self, *, stt: STT, opts: STTOptions, conn_options: APIConnectOptions) -> None: super().__init__(stt=stt, conn_options=conn_options, sample_rate=SAMPLE_RATE) self._zoom = stt self._opts = opts # turn_detection can't change after the stream starts (see update_options). self._manual_commit = opts.turn_detection == "none" # What the current session was configured with (options plus extra_kwargs). self._view = _RequestView.of(self._session_update()) self._reconnect_event = asyncio.Event() self._audio_duration = 0.0 self._speech_started = False # turn_detection="none" (manual commit) state: # - what is still owed a final, in order: "commit" (a flush that sent a commit), # "silent" (a flush after only digital silence: no commit, answered once the finals # before it arrive) and "drain" (the plugin's own commit before an options # reconnect: its final isn't an answer to any flush); self._flushes: collections.deque[Literal["commit", "silent", "drain"]] = collections.deque() # - transcript of drain commits, delivered with the final of the next flush so it # can't end a turn early (e.g. commit_user_turn() waiting on that flush); self._carried = None # - non-silent audio sent to the current session since its last commit; self._uncommitted = False # - an options reconnect is waiting: input is held back instead of being sent; self._draining = False # - input held back while draining, sent first on the next session, in order; a # frame leaves it only once sent, so a failed send keeps it for the retry. self._held: collections.deque[rtc.AudioFrame | SpeechStream._FlushSentinel] = ( collections.deque() ) # Serializes sends with the drain commit, so no audio follows it on the old session. self._send_lock = asyncio.Lock() self._tracking_id: str | None = None # the current session's x-zm-trackingid self._session_updated = False # Scribe accepted the current session's settings # Time between the start of this _run attempt and the current Scribe session. # The base class adds whole attempts to start_time_offset when it retries. self._session_offset = 0.0 def update_options( self, *, language: NotGivenOr[str] = NOT_GIVEN, silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN, ) -> None: """Change this stream's options; it reconnects with the new session config. Args: language: Scribe locale, e.g. ``"en-US"``. silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's default. A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. With ``turn_detection="none"``, audio sent since the last flush is committed first, and input that arrives meanwhile is sent to the new session; ``silence_duration_ms`` isn't used in that mode, so changing only it doesn't reconnect. """ if not is_given(language) and not is_given(silence_duration_ms): return if is_given(language): self._opts.language = _to_locale(language) if is_given(silence_duration_ms): self._opts.silence_duration_ms = silence_duration_ms if self._manual_commit and not is_given(language): logger.warning( 'silence_duration_ms is not used with turn_detection="none"; ' "the Zoom Scribe Live session is unchanged" ) return self._reconnect_event.set() def _session_update(self) -> dict[str, Any]: """The session.update for this stream's options.""" update: dict[str, Any] = { "type": "session.update", "language": self._opts.language, "audio": {"format": "pcm16"}, } config = _recognition_config(self._opts) if config: update["config"] = config if self._opts.turn_detection == "server" and self._opts.silence_duration_ms is not None: update["turn_detection"] = {"silence_duration_ms": self._opts.silence_duration_ms} if self._opts.text_polishing: update["outputs"] = {"text_polishing": {"enabled": True}} update = _merge(update, self._opts.extra_kwargs) # Fields the plugin depends on, whatever extra_kwargs says. update["type"] = "session.update" update["audio"] = {**(update.get("audio") or {}), "format": "pcm16"} if self._opts.turn_detection == "none": # Scribe's VAD is off; each flush from LiveKit commits the turn. update["turn_detection"] = {"type": "none"} polishing = (update.get("outputs") or {}).get("text_polishing") if isinstance(polishing, dict) and polishing.get("enabled"): # Live returns only the polished text unless the transcript is asked for too, # and the transcript is the fallback when polishing fails. polishing["include_transcript"] = True return update async def _connect_ws(self) -> aiohttp.ClientWebSocketResponse: started_at = time.perf_counter() # Each session has its own tracking ID. It is logged on connect and is the # request_id of the session's errors, so a failure can be matched in Zoom's logs. tracking_id = self._tracking_id = _tracking_id() self._session_updated = False try: ws = await asyncio.wait_for( self._zoom._http.ws_connect( self._zoom._live_url, protocols=("live-asr",), headers=self._zoom._headers(tracking_id), heartbeat=30.0, # notice a dead connection; audio keeps Scribe's session alive ), self._conn_options.timeout, ) except aiohttp.WSServerHandshakeError as e: # aiohttp discards the response body, where Zoom gives its reason (e.g. "Invalid # Access token"), so only the status is reported. `from None`: the handshake # error's repr keeps the request headers (the API key). raise APIStatusError( f"Zoom refused the connection (HTTP {e.status})", status_code=e.status, request_id=tracking_id, ) from None except (aiohttp.ClientError, asyncio.TimeoutError) as e: raise APIConnectionError("failed to connect to Zoom Scribe Live") from e try: update = self._session_update() self._view = _RequestView.of(update) await ws.send_str(json.dumps(update)) except (aiohttp.ClientError, ConnectionError) as e: await ws.close() raise APIConnectionError("failed to configure the Zoom Scribe Live session") from e logger.debug("connected to Zoom Scribe Live (tracking_id=%s)", tracking_id) self._report_connection_acquired(time.perf_counter() - started_at, False) return ws def _turn_open(self) -> bool: """A turn whose final transcript a reconnect would lose.""" if self._manual_commit: return any(kind != "silent" for kind in self._flushes) return self._speech_started def _send_final(self, data: stt.SpeechData, request_id: str = "") -> None: """Emit a final transcript event with the supplied speech data.""" self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.FINAL_TRANSCRIPT, request_id=request_id, alternatives=[data], ) ) def _answer_flush(self, data: stt.SpeechData | None = None, request_id: str = "") -> None: """Answer a flush (turn_detection="none"), with any carried drain transcript first.""" if data is None: data = self._carried or stt.SpeechData( language=_reported_language( None, self._view.language_detection, self._opts.language ), text="", ) elif self._carried is not None: data = _join_speech(self._carried, data) self._carried = None self._send_final(data, request_id) def _carry(self, data: stt.SpeechData) -> None: """Keep a drain commit's transcript for the final of the next flush.""" if data.text: self._carried = data if self._carried is None else _join_speech(self._carried, data) async def _commit( self, ws: aiohttp.ClientWebSocketResponse, kind: Literal["commit", "drain"] = "commit" ) -> None: """Commit the audio sent since the last commit; Scribe answers with one final.""" self._flushes.append(kind) try: await ws.send_str(json.dumps({"type": "input_audio_buffer.commit"})) except BaseException: self._flushes.pop() # not sent: the caller keeps the flush for the next session raise self._uncommitted = False async def _start_drain(self, ws: aiohttp.ClientWebSocketResponse) -> None: """Prepare a turn_detection="none" session for an options reconnect. Audio sent since the last flush is committed so its final isn't lost (a commit covering only digital silence is skipped), and further input is held back for the next session. The reconnect then waits for every outstanding final. """ async with self._send_lock: self._draining = True if self._uncommitted: await self._commit(ws, "drain") async def _wait_for_turn_end(self) -> None: """Return once no turn is open, or after ``RECONNECT_HOLD_S``.""" deadline = time.monotonic() + RECONNECT_HOLD_S while self._turn_open() and time.monotonic() < deadline: await asyncio.sleep(0.05) def _end_open_utterance(self, *, closing: bool = False) -> None: """Close an utterance cut off by a reconnect so turn detection doesn't wait forever.""" # With turn_detection="none", answer each flush the old session left unanswered # (with an empty final, or the carried transcript), so a flush (e.g. # commit_user_turn()) doesn't wait for one. A cut-off drain commit owes no answer. for kind in self._flushes: if kind != "drain": self._answer_flush() self._flushes.clear() if closing and self._carried is not None: self._send_final(self._carried) # no flush is left to deliver it with self._carried = None self._uncommitted = False # audio not committed on the old session is gone with it self._draining = False if self._speech_started: self._speech_started = False self._event_ch.send_nowait(stt.SpeechEvent(type=stt.SpeechEventType.END_OF_SPEECH)) async def _run(self) -> None: self._session_offset = 0.0 closing = False closed = asyncio.Event() # Scribe confirmed our session.close last_audio_at = time.monotonic() # last audio sent, input or silence last_input_at = time.monotonic() # last input audio sent gap_filled = False # silence already sent for the current input gap session_audio_s = 0.0 # audio sent on the current Scribe session, keepalive included async def send_task(ws: aiohttp.ClientWebSocketResponse) -> None: nonlocal closing, last_audio_at, last_input_at, gap_filled, session_audio_s anchored = False async def send(frame: rtc.AudioFrame | SpeechStream._FlushSentinel) -> None: nonlocal last_audio_at, last_input_at, gap_filled, session_audio_s, anchored if isinstance(frame, self._FlushSentinel): # LiveKit ended the user's speech. With Scribe's VAD off, commit the # audio so far for its final; otherwise Scribe's VAD ends the turn. if self._manual_commit: if self._uncommitted: await self._commit(ws) elif self._flushes: # Only digital silence since the last commit (e.g. end_input() # after a long pause): Scribe can return made-up text for # that, so this flush gets an empty final instead, after the # finals still due for earlier commits. self._flushes.append("silent") else: self._answer_flush() return if not frame.samples_per_channel: return # Each frame is sent as it arrives (10-20 ms in an agent). The docs suggest # about 100 ms per message, but smaller messages end turns slightly sooner. data = frame.data.tobytes() await ws.send_bytes(data) # Counted once sent: a frame whose send fails is sent again on the retry. self._audio_duration += frame.duration if self._manual_commit and not self._uncommitted and data.strip(b"\0"): self._uncommitted = True last_audio_at = last_input_at = time.monotonic() gap_filled = False # Scribe's audio_*_ms count from the session's first audio. Audio queued # while connecting goes out in a burst, so anchor that 0 ms at the # earliest "now minus audio sent", which is when that audio was captured. session_audio_s += frame.duration captured_at = time.time() - session_audio_s if not anchored or captured_at < self.start_time: self.start_time = captured_at anchored = True async def send_held() -> None: # Each held frame leaves the queue only once sent, so if this session # fails, the retry resumes from the first unsent one. while self._held: await send(self._held[0]) self._held.popleft() try: async with self._send_lock: await send_held() # held back while the previous session drained async for frame in self._input_ch: async with self._send_lock: if self._draining: self._held.append(frame) # for the next session continue self._held.append(frame) # kept until sent, for a retry await send_held() # Input ended while a reconnect was waiting: there is no next session, so # send what was held to this one. async with self._send_lock: self._draining = False await send_held() closing = True await ws.send_str(json.dumps({"type": "session.close"})) except (aiohttp.ClientError, ConnectionError) as e: if closing or self._zoom._http.closed: return raise APIConnectionError("Zoom Scribe Live connection closed unexpectedly") from e # Wait for the remaining final transcript and session.closed, but not forever. try: await asyncio.wait_for(closed.wait(), self._conn_options.timeout) except asyncio.TimeoutError: logger.warning("Zoom Scribe Live did not confirm session.close; closing") await ws.close() async def keepalive_task(ws: aiohttp.ClientWebSocketResponse) -> None: # Scribe Live ends a session after 30 s without audio (e.g. the user muted), and it # has no keepalive message, so send a little silence when the input has gone quiet. # If input stops while a turn is open, send enough silence for Scribe's VAD to end # the turn now rather than at the next keepalive. nonlocal last_audio_at, session_audio_s, gap_filled keepalive = bytes(SAMPLE_RATE * 2 // 10) # 100 ms try: while not closing: await asyncio.sleep(INPUT_GAP_S / 2) if closing: break now = time.monotonic() silence = b"" if ( # Only Scribe's VAD needs silence to end a turn; else the commit does. not self._manual_commit and self._speech_started and not gap_filled and now - last_input_at >= INPUT_GAP_S ): window_ms = self._view.silence_duration_ms or 1000 silence = bytes(SAMPLE_RATE * 2 * (window_ms + 100) // 1000) gap_filled = True elif now - last_audio_at >= KEEPALIVE_INTERVAL_S: silence = keepalive if silence: await ws.send_bytes(silence) self._audio_duration += len(silence) / (2 * SAMPLE_RATE) session_audio_s += len(silence) / (2 * SAMPLE_RATE) last_audio_at = now except (aiohttp.ClientError, ConnectionError) as e: if closing or self._zoom._http.closed: return raise APIConnectionError("Zoom Scribe Live connection closed unexpectedly") from e async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: while True: msg = await ws.receive() if msg.type in ( aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING, ): if closing: closed.set() return raise APIStatusError( "Zoom Scribe Live connection closed unexpectedly", status_code=ws.close_code or -1, request_id=self._tracking_id, body=f"{msg.data=} {msg.extra=}", ) if msg.type != aiohttp.WSMsgType.TEXT: continue try: ev = json.loads(msg.data) except json.JSONDecodeError: logger.warning("ignoring non-JSON message from Zoom Scribe") continue if ev.get("type") == "session.closed" and not closing: # Server-side end (session time limit, idle timeout): # raise a retryable error so the base class reconnects with a new session. raise APIStatusError( f"Zoom Scribe Live {ev['type']} ({ev.get('reason')})", status_code=-1, request_id=self._tracking_id, body=ev, retryable=True, ) try: if self._process_event(ev): closed.set() return # session.closed after our session.close except APIStatusError: raise except Exception: logger.exception("failed to process Zoom Scribe message") while True: self._end_open_utterance() connected_at = time.time() session_audio_s = 0.0 self._reconnect_event.clear() # this connect reads the current options ws = await self._connect_ws() tasks = [ asyncio.create_task(send_task(ws)), asyncio.create_task(recv_task(ws)), asyncio.create_task(keepalive_task(ws)), ] io_done: asyncio.Future[list[None]] = asyncio.gather(*tasks[:2]) wait_reconnect = asyncio.create_task(self._reconnect_event.wait()) hold: asyncio.Task[None] | None = None # waits for the turn in progress to end reconnect = False try: waiters: set[asyncio.Future[Any]] = {io_done, wait_reconnect, tasks[2]} while True: done, _ = await asyncio.wait(waiters, return_when=asyncio.FIRST_COMPLETED) for task in done: if task is wait_reconnect or task is hold: continue if isinstance(task.exception(), APIConnectionError): # A failed send usually means Scribe closed the socket; prefer the # error event it sent first (e.g. an unsupported language). await asyncio.wait({tasks[1]}, timeout=1.0) if tasks[1].done() and isinstance(tasks[1].exception(), APIStatusError): tasks[1].result() task.result() # surface errors from I/O and keepalive if io_done in done: break # input ended and the session closed cleanly if hold is not None and hold in done: reconnect = True # the turn ended: reconnect with the new config break if wait_reconnect in done: # Options changed: reconnect with the new config, once the turn in # progress (if any) has its final transcript. waiters.discard(wait_reconnect) if self._manual_commit: try: await self._start_drain(ws) except (aiohttp.ClientError, ConnectionError) as e: raise APIConnectionError( "Zoom Scribe Live connection closed unexpectedly" ) from e if not self._turn_open(): reconnect = True break hold = asyncio.create_task(self._wait_for_turn_end()) waiters.add(hold) continue waiters.discard(tasks[2]) # keepalive stopped while closing; keep waiting finally: await utils.aio.gracefully_cancel(*tasks, wait_reconnect, *([hold] if hold else [])) io_done.cancel() with contextlib.suppress(asyncio.CancelledError): io_done.exception() # retrieve it so asyncio doesn't log it at shutdown await ws.close() if not reconnect: # Scribe closed without a final for the last turn self._end_open_utterance(closing=True) return # Scribe's audio_*_ms restart at 0 on the new session; keep stream-relative # timestamps moving forward, as the base class does when it retries. self._session_offset += time.time() - connected_at def _speech_end_time(self, end_ms: int) -> float | None: """Wall-clock end of speech for a final ending at ``end_ms``. Scribe ends a segment after ``silence_duration_ms`` of silence, so ``audio_end_ms`` is that much later than the end of speech. With ``silence_duration_ms=None`` the window is Scribe's, so ``audio_end_ms`` is used as is. With ``turn_detection="none"`` the segment ends where the flush committed it, after LiveKit's VAD silence (and any turn detector hold), not at the end of speech. Only LiveKit's VAD knows when speech ended, so no time is reported. """ if not end_ms or self._manual_commit: return None silence_ms = self._view.silence_duration_ms or 0 return self.start_time + max(0, end_ms - silence_ms) / 1000 def _process_event(self, ev: dict[str, Any]) -> bool: kind = ev.get("type") if kind == "input_audio_buffer.speech_started": # With turn_detection="none", Scribe opens its next turn right after each # commit; LiveKit's VAD owns the user's turn. if not self._speech_started and not self._manual_commit: self._speech_started = True start_ms = ev.get("audio_start_ms") self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.START_OF_SPEECH, speech_start_time=self.start_time + start_ms / 1000 if start_ms is not None else None, ) ) elif kind == "transcription.completed": text = _clean(ev.get("transcript")) start_ms, end_ms = ev.get("audio_start_ms") or 0, ev.get("audio_end_ms") or 0 speaker_id: str | None = None metadata: dict[str, Any] | None = None if self._view.text_polishing: polished = _polished_text(ev) if polished is not None: metadata = {"transcript": text} text = polished if self._view.diarization: # One speaker: speaker_id. Several: Scribe sends `speaker_segments`, and # speaker_id is left empty (one field can't name several speakers). spans = ev.get("speaker_segments") or [] if spans: metadata = {**(metadata or {}), "speaker_segments": spans} else: speaker_id = ev.get("speaker") or None answers: Literal["commit", "silent", "drain"] = "commit" if self._manual_commit: # This final answers the oldest commit ("silent" entries never get one). answers = self._flushes.popleft() if self._flushes else "commit" elif text and not self._speech_started: # No speech_started for this transcript (e.g. it was lost in a reconnect): # open the turn so turn_detection="stt" still commits it. self._speech_started = True self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.START_OF_SPEECH, speech_start_time=self.start_time + start_ms / 1000 if "audio_start_ms" in ev else None, ) ) data = stt.SpeechData( language=_reported_language( ev.get("language"), self._view.language_detection, self._opts.language ), text=text, start_time=start_ms / 1000 + self.start_time_offset + self._session_offset, end_time=end_ms / 1000 + self.start_time_offset + self._session_offset, speaker_id=speaker_id, metadata=metadata, ) if self._manual_commit: # Each final answers a flush, so an empty one is sent too: it tells LiveKit # the flush is done (e.g. commit_user_turn() waits). A drain commit's # transcript waits for the next flush's final instead. if answers == "drain": self._carry(data) else: self._answer_flush(data, ev.get("item_id", "")) # Then answer the silence-only flushes that were waiting behind this one. while self._flushes and self._flushes[0] == "silent": self._flushes.popleft() self._answer_flush() elif text: self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.FINAL_TRANSCRIPT, request_id=ev.get("item_id", ""), alternatives=[data], speech_end_time=self._speech_end_time(end_ms), ) ) # Scribe's final closes the utterance: end of speech comes after the transcript, # so turn_detection="stt" commits the turn with the text already in hand. if self._speech_started: self._speech_started = False # speech_end_time lets turn_detection="stt" count its endpointing delay from # when the user stopped speaking, not from when the final arrived. self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.END_OF_SPEECH, speech_end_time=self._speech_end_time(end_ms), ) ) self._report_usage() elif kind == "session.updated": self._session_updated = True # Scribe never rejects diarization; it reports whether labels will come. if self._view.diarization and ev.get("diarization_enabled") is False: logger.warning("Zoom Scribe Live can't label speakers for this session") elif kind == "error": err = ev.get("error") or {} code = err.get("code") if err.get("fatal"): # Scribe rejects a session's settings (e.g. an unsupported language) with a # fatal error in place of session.updated, then closes. A new session would be # rejected the same way, so that isn't retried, and a FallbackAdapter can take # over. A fatal error after session.updated is retried with a new session. raise APIStatusError( f"Zoom Scribe Live error {code}: {err.get('message')}", status_code=-1, request_id=self._tracking_id, body=err, retryable=self._session_updated, ) # A non-fatal error leaves the session running. (A non-fatal invalid_config, e.g. # text polishing for an unsupported locale, is followed by Scribe's own fatal # error.) Scribe's message is kept out of the log text in case it quotes user # content. logger.warning( "Zoom Scribe Live error %s (non-fatal, tracking_id=%s)", code, self._tracking_id, extra={"lk.pii.message": err.get("message")}, ) elif kind == "session.closed": self._report_usage() # audio sent after the last final transcript return True return False def _report_usage(self) -> None: if self._audio_duration > 0: self._event_ch.send_nowait( stt.SpeechEvent( type=stt.SpeechEventType.RECOGNITION_USAGE, recognition_usage=stt.RecognitionUsage(audio_duration=self._audio_duration), ) ) self._audio_duration = 0.0One Scribe Live session; reconnects on option changes and server-side session ends.
Args: sample_rate : int or None, optional The desired sample rate for the audio input. If specified, the audio input will be automatically resampled to match the given sample rate before being processed for Speech-to-Text. If not provided (None), the input will retain its original sample rate.
Ancestors
- livekit.agents.stt.stt.RecognizeStream
- abc.ABC
Methods
def update_options(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN) ‑> None-
Expand source code
def update_options( self, *, language: NotGivenOr[str] = NOT_GIVEN, silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN, ) -> None: """Change this stream's options; it reconnects with the new session config. Args: language: Scribe locale, e.g. ``"en-US"``. silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's default. A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. With ``turn_detection="none"``, audio sent since the last flush is committed first, and input that arrives meanwhile is sent to the new session; ``silence_duration_ms`` isn't used in that mode, so changing only it doesn't reconnect. """ if not is_given(language) and not is_given(silence_duration_ms): return if is_given(language): self._opts.language = _to_locale(language) if is_given(silence_duration_ms): self._opts.silence_duration_ms = silence_duration_ms if self._manual_commit and not is_given(language): logger.warning( 'silence_duration_ms is not used with turn_detection="none"; ' "the Zoom Scribe Live session is unchanged" ) return self._reconnect_event.set()Change this stream's options; it reconnects with the new session config.
Args
language- Scribe locale, e.g.
"en-US". silence_duration_ms- Silence (ms) that ends a turn;
Noneuses Scribe's default.
A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. With
turn_detection="none", audio sent since the last flush is committed first, and input that arrives meanwhile is sent to the new session;silence_duration_msisn't used in that mode, so changing only it doesn't reconnect.