Module livekit.plugins.zoom_ai

Zoom AI plugin for LiveKit Agents.

Speech-to-text with Zoom Scribe: STT streams over Scribe Live and STT(mode="fast") transcribes one utterance per request with Scribe Fast.

See https://developers.zoom.us/docs/ai-services/scribe/ for more information.

Classes

class STT (*,
mode: "Literal['live', 'fast']" = 'live',
api_key: NotGivenOr[str] = NOT_GIVEN,
language: str = 'en-US',
model: str | None = None,
language_detection: str | None = None,
turn_detection: "Literal['server', 'none']" = 'server',
silence_duration_ms: int | None = 350,
diarization: bool = False,
word_time_offsets: bool = False,
channel_separation: bool = False,
vocabulary: list[str] | None = None,
text_polishing: bool = False,
extra_kwargs: dict[str, Any] | None = None,
base_url: str = 'https://api.zoom.us/v2/aiservices/scribe',
live_url: NotGivenOr[str] = NOT_GIVEN,
fast_url: NotGivenOr[str] = NOT_GIVEN,
http_session: aiohttp.ClientSession | None = None)
Expand source code
class STT(stt.STT):
    """Zoom Scribe speech-to-text: streaming (Scribe Live) or per-utterance (Scribe Fast).

    Scribe Live detects the end of each turn itself, so use STT turn detection::

        session = AgentSession(
            stt=zoom_ai.STT(),
            turn_handling={"turn_detection": "stt"},
            ...
        )

    Without it, LiveKit's VAD and turn detector add their own delay on top of Scribe's,
    and a turn can be committed before a later final arrives.

    To let LiveKit's VAD (and turn detector) decide when the user is done instead, turn
    Scribe's VAD off with ``turn_detection="none"``. Each time LiveKit ends the user's
    speech, the plugin commits the audio so far and Scribe returns its final transcript::

        session = AgentSession(
            stt=zoom_ai.STT(turn_detection="none"),
            vad=silero.VAD.load(),
            turn_handling={"turn_detection": inference.TurnDetector()},  # or "vad"
            ...
        )

    In Live mode, ``recognize()`` (batch recognition) uses Scribe Fast.
    """

    def __init__(
        self,
        *,
        mode: Literal["live", "fast"] = "live",
        api_key: NotGivenOr[str] = NOT_GIVEN,
        language: str = "en-US",
        model: str | None = None,
        language_detection: str | None = None,
        turn_detection: Literal["server", "none"] = "server",
        silence_duration_ms: int | None = DEFAULT_SILENCE_DURATION_MS,
        diarization: bool = False,
        word_time_offsets: bool = False,
        channel_separation: bool = False,
        vocabulary: list[str] | None = None,
        text_polishing: bool = False,
        extra_kwargs: dict[str, Any] | None = None,
        base_url: str = BASE_URL,
        live_url: NotGivenOr[str] = NOT_GIVEN,
        fast_url: NotGivenOr[str] = NOT_GIVEN,
        http_session: aiohttp.ClientSession | None = None,
    ) -> None:
        """Create a Zoom Scribe STT.

        Options map to the Scribe API fields of the same name. For what each feature
        does, and its availability, pricing and enablement, see
        https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/

        Args:
            mode: ``"live"`` (streaming, default) or ``"fast"`` (one request per utterance).
            api_key: Zoom AI Services API key (from https://www.zoom.ai/). Defaults to
                ``$ZOOM_AI_API_KEY``. It is sent on every request and reconnect, so it
                must stay valid for as long as the agent runs.
            language: A Scribe locale, e.g. ``"en-US"``; base codes such as ``"en"`` map to
                their locale. A regional variant Scribe doesn't list resolves to the locale
                of its base language (``"en-GB"`` to ``"en-US"``); anything else is sent as
                given, for Scribe to validate. For the supported languages of each model, see
                https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/
            model: Scribe speech recognition model (``config.model``). ``None`` uses
                Scribe's default model.
            language_detection: Scribe language detection mode
                (``config.language_detection.mode``; Live supports ``"continuous"``). ``None``
                turns it off. A detected language Scribe reports is in
                ``SpeechData.language``; when Scribe reports none, it is empty. With
                ``zoom-scribe-multilingual`` in Live, set it to get each final's language.
            turn_detection: Live only: who ends each turn. ``"server"`` (default):
                Scribe's voice activity detection, with ``silence_duration_ms``; use
                ``turn_handling={"turn_detection": "stt"}``. ``"none"``: Scribe's voice
                activity detection is off (``turn_detection.type``), and each STT flush
                from LiveKit (at the end of speech from its VAD, or at the end of turn from
                its turn detector) sends ``input_audio_buffer.commit``, for which Scribe
                returns the final transcript. Use it with a VAD and
                ``turn_handling={"turn_detection": "vad"}`` or a turn detector model, not
                ``"stt"``: in this mode the STT sends no start or end of speech events,
                and finals have no ``speech_end_time`` (LiveKit's VAD times the turn).
                Every flush gets one final; a flush after only digital silence gets an
                empty one without a commit.
            silence_duration_ms: Live only: milliseconds of silence that end a turn in
                Scribe's voice activity detection; not used with ``turn_detection="none"``.
                Defaults to 350, tuned for voice agents;
                lower is faster, higher keeps sentences with longer pauses together. For
                transcription without an agent, a longer value such as 600 can improve
                accuracy. ``None`` uses Scribe's default (see the Scribe docs).
            diarization: Label each final transcript with a speaker. Off by default.
                With one speaker in the utterance, ``SpeechData.speaker_id`` is Scribe's
                label. With several, ``speaker_id`` is empty and
                ``metadata["speaker_segments"]`` has each one's span. In Fast mode labels
                are per request, so they aren't consistent across utterances.
                ``MultiSpeakerAdapter`` works with Live mode only.
            word_time_offsets: Fast only: word timings (``config.word_time_offsets``), in
                ``SpeechData.words``. Off by default.
            channel_separation: Fast only: recognize each channel of stereo audio
                separately (``config.channel_separation``); each segment's channel is in
                ``metadata["channel_segments"]``. Off by default.
            vocabulary: Words and phrases that customize recognition
                (``config.vocabulary.phrases``).
            text_polishing: Scribe's text polishing (``outputs.text_polishing``). The
                polished text is the transcript, and the recognized text is in
                ``metadata["transcript"]``; if polishing fails, the recognized text is
                used. Word timings aren't returned with polished text. ``en-US`` only, and not
                with language detection. Off by default.
            extra_kwargs: Scribe request fields to add or override, shaped like the
                request, e.g. ``{"config": {"language_detection": {"candidate_languages":
                ["en-US", "fr-FR"]}}}`` to restrict language detection. Merged into the Live ``session.update`` and the Fast
                request, over the options above, so Scribe fields this plugin has no option
                for can be used. ``audio.format`` stays ``pcm16``.
            base_url: Scribe API base URL; Live uses ``<base>/live`` (as ``wss``) and Fast
                uses ``<base>/transcribe``. A plaintext (``http``/``ws``) URL to a host other
                than localhost logs a warning.
            live_url: Override the Scribe Live WebSocket URL.
            fast_url: Override the Scribe Fast HTTP URL.
            http_session: Optional aiohttp session; the job's session is used otherwise.
        """
        if mode not in ("live", "fast"):
            raise ValueError(f'mode must be "live" or "fast", got {mode!r}')
        if turn_detection not in ("server", "none"):
            raise ValueError(f'turn_detection must be "server" or "none", got {turn_detection!r}')
        if mode == "fast" and turn_detection == "none":
            logger.warning(
                'turn_detection="none" is ignored with mode="fast": Scribe Fast transcribes '
                "one utterance per request"
            )
        if extra_kwargs is not None and not isinstance(extra_kwargs, dict):
            raise ValueError(f"extra_kwargs must be a dict, got {type(extra_kwargs).__name__}")
        extra_kwargs = dict(extra_kwargs or {})
        super().__init__(
            capabilities=stt.STTCapabilities(
                streaming=mode == "live",
                interim_results=False,  # not in the GA Scribe Live API
                diarization=diarization
                or bool((extra_kwargs.get("config") or {}).get("diarization")),
                offline_recognize=True,
                # With Scribe's VAD off, LiveKit flushes the stream to end each turn.
                manual_flush=mode == "live" and turn_detection == "none",
            )
        )
        api_key = api_key if is_given(api_key) else os.environ.get("ZOOM_AI_API_KEY", "")
        if not api_key:
            raise ValueError("Zoom AI API key is required: pass api_key or set ZOOM_AI_API_KEY")
        self._api_key = api_key
        self._mode = mode
        base = base_url.rstrip("/")
        scheme, sep, rest = base.partition("://")
        if not sep or scheme.lower() not in ("http", "https"):
            raise ValueError(f"base_url must be an http(s) URL, got {base_url!r}")
        _check_url(base, "base_url")
        ws_scheme = "wss" if scheme.lower() == "https" else "ws"
        self._live_url = _check_url(
            live_url if is_given(live_url) else f"{ws_scheme}://{rest}/live", "live_url"
        )
        self._fast_url = _check_url(
            fast_url if is_given(fast_url) else base + "/transcribe", "fast_url"
        )
        self._session = http_session
        self._opts = STTOptions(
            language=_to_locale(language),
            turn_detection=turn_detection,
            diarization=diarization,
            silence_duration_ms=silence_duration_ms,
            word_time_offsets=word_time_offsets,
            channel_separation=channel_separation,
            vocabulary=vocabulary,
            text_polishing=text_polishing,
            model=model,
            language_detection=language_detection,
            extra_kwargs=extra_kwargs,
        )
        self._streams = weakref.WeakSet[SpeechStream]()

    @property
    def model(self) -> str:
        """The Scribe model set with ``model``, else the mode: ``"scribe-live"`` or ``"scribe-fast"``."""
        return self._opts.model or f"scribe-{self._mode}"

    @property
    def provider(self) -> str:
        """The provider name, ``"Zoom"``."""
        return "Zoom"

    @property
    def _http(self) -> aiohttp.ClientSession:
        if not self._session:
            self._session = utils.http_context.http_session()
        return self._session

    def _headers(self, tracking_id: str) -> dict[str, str]:
        return {
            "Authorization": f"Bearer {self._api_key}",
            "User-Agent": USER_AGENT,
            "x-zm-trackingid": tracking_id,
        }

    def _fast_request(self, locale: str) -> dict[str, Any]:
        """The Fast request fields besides the audio: the options, then extra_kwargs."""
        config: dict[str, Any] = {"language": locale, **_recognition_config(self._opts)}
        if self._opts.word_time_offsets:
            config["word_time_offsets"] = True
        if self._opts.channel_separation:
            config["channel_separation"] = True
        request: dict[str, Any] = {"config": config}
        if self._opts.text_polishing:
            request["outputs"] = {"text_polishing": {"enabled": True}}
        request = _merge(request, self._opts.extra_kwargs)
        request.pop("file", None)  # the plugin uploads the audio
        return request

    # ---------------------------------------------------------------- Scribe Fast
    async def _recognize_impl(
        self,
        buffer: AudioBuffer,
        *,
        language: NotGivenOr[str] = NOT_GIVEN,
        conn_options: APIConnectOptions,
    ) -> stt.SpeechEvent:
        locale = _to_locale(language) if is_given(language) else self._opts.language
        request = self._fast_request(locale)
        view = _RequestView.of(request)
        # Upload the utterance as multipart form data: `file`, then one JSON field per
        # top-level request field (`config`, `outputs`, ...).
        form = aiohttp.FormData()
        form.add_field(
            "file",
            rtc.combine_audio_frames(buffer).to_wav_bytes(),
            filename="utterance.wav",
            content_type="audio/wav",
        )
        for field, value in request.items():
            form.add_field(field, json.dumps(value))
        # Each request has its own tracking ID; it is the request_id of an error, so a
        # failure can be matched in Zoom's logs.
        tracking_id = _tracking_id()
        try:
            async with self._http.post(
                self._fast_url,
                data=form,
                headers=self._headers(tracking_id),
                timeout=aiohttp.ClientTimeout(total=30, sock_connect=conn_options.timeout),
            ) as resp:
                body = await resp.text()
                if resp.status != 200:
                    # Zoom explains a refusal (a bad API key, say) in the JSON body.
                    reason = _zoom_reason(body)
                    raise APIStatusError(
                        f"Zoom Scribe Fast error (HTTP {resp.status})"
                        + (f": {reason}" if reason else ""),
                        status_code=resp.status,
                        request_id=tracking_id,
                        body=body,
                    )
                try:
                    result = json.loads(body)
                except json.JSONDecodeError:
                    raise APIStatusError(
                        "Zoom Scribe Fast returned a non-JSON response",
                        status_code=resp.status,
                        request_id=tracking_id,
                    ) from None
        except asyncio.TimeoutError as e:
            raise APITimeoutError() from e
        except aiohttp.ClientError as e:
            raise APIConnectionError() from e

        text = _clean((result.get("result") or {}).get("text_display"))
        segments = (result.get("result") or {}).get("segments") or []
        # The detected language, when Scribe reports one. Scribe Fast reports it as
        # result.language with one-time detection; with continuous detection, production
        # omits it from the result and the segments (as the docs say), so the language is
        # then empty. A segment's language is used if Scribe ever sends one.
        detected = (result.get("result") or {}).get("language") or next(
            (s["language"] for s in segments if s.get("language")), None
        )
        speaker_id: str | None = None
        metadata: dict[str, Any] | None = None
        polished = _polished_text(result) if view.text_polishing else None
        if polished is not None:
            metadata = {"transcript": text}
            text = polished
        if view.diarization:
            # One speaker: speaker_id. Several: speaker_id is empty and each one's span is
            # in metadata["speaker_segments"] (as Scribe Live does).
            speakers = {sg["speaker"] for sg in segments if sg.get("speaker")}
            if len(speakers) == 1:
                speaker_id = next(iter(speakers))
            elif len(speakers) > 1:
                metadata = {
                    **(metadata or {}),
                    "speaker_segments": [
                        {k: s.get(k) for k in ("speaker", "text", "start", "end")} for s in segments
                    ],
                }
        if view.channel_separation:
            metadata = {
                **(metadata or {}),
                "channel_segments": [
                    {k: s.get(k) for k in ("channel", "text", "start", "end")} for s in segments
                ],
            }
        words = None
        # Word timings belong to the recognized text, not the polished one.
        if view.word_time_offsets and polished is None:
            words = [
                TimedString(text=w.get("word", ""), start_time=w["start"], end_time=w["end"])
                for s in segments
                for w in s.get("words") or []
                if "start" in w and "end" in w
            ] or None
        return stt.SpeechEvent(
            type=stt.SpeechEventType.FINAL_TRANSCRIPT,
            request_id=result.get("request_id", ""),
            alternatives=[
                stt.SpeechData(
                    language=_reported_language(detected, view.language_detection, locale),
                    text=text,
                    start_time=segments[0].get("start", 0.0) if segments else 0.0,
                    end_time=segments[-1].get("end", 0.0) if segments else 0.0,
                    speaker_id=speaker_id,
                    words=words,
                    metadata=metadata,
                )
            ],
        )

    # ---------------------------------------------------------------- Scribe Live
    def stream(
        self,
        *,
        language: NotGivenOr[str] = NOT_GIVEN,
        conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
    ) -> SpeechStream:
        """Open a Scribe Live streaming session.

        Args:
            language: Override the STT's language for this stream.
            conn_options: Connection timeout and retry options.

        Raises:
            NotImplementedError: When the STT was created with ``mode="fast"``.
        """
        if self._mode != "live":
            raise NotImplementedError(
                'mode="fast" is not streaming; AgentSession wraps it with a VAD'
            )
        opts = dataclasses.replace(self._opts)
        if is_given(language):
            opts.language = _to_locale(language)
        stream = SpeechStream(stt=self, opts=opts, conn_options=conn_options)
        self._streams.add(stream)
        return stream

    async def aclose(self) -> None:
        """Close every Live stream opened by :meth:`stream`.

        The HTTP session is left open: it is either supplied by the caller or owned by the
        job's HTTP context.
        """
        streams = list(self._streams)
        self._streams.clear()
        # return_exceptions=True so one stream failing to close doesn't leave the rest open
        await asyncio.gather(*(stream.aclose() for stream in streams), return_exceptions=True)

    def update_options(
        self,
        *,
        language: NotGivenOr[str] = NOT_GIVEN,
        silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN,
    ) -> None:
        """Change options; open Live streams reconnect with the new session config.

        Args:
            language: Scribe locale, e.g. ``"en-US"``.
            silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's
                default. Not used with ``turn_detection="none"``.

        A turn in progress keeps its session until Scribe's final transcript ends it (at
        most 10 s), so the reconnect doesn't lose it.
        """
        if not is_given(language) and not is_given(silence_duration_ms):
            return
        if is_given(language):
            self._opts.language = _to_locale(language)
        if is_given(silence_duration_ms):
            self._opts.silence_duration_ms = silence_duration_ms
        for stream in self._streams:
            stream.update_options(language=language, silence_duration_ms=silence_duration_ms)

Zoom Scribe speech-to-text: streaming (Scribe Live) or per-utterance (Scribe Fast).

Scribe Live detects the end of each turn itself, so use STT turn detection::

session = AgentSession(
    stt=zoom_ai.STT(),
    turn_handling={"turn_detection": "stt"},
    ...
)

Without it, LiveKit's VAD and turn detector add their own delay on top of Scribe's, and a turn can be committed before a later final arrives.

To let LiveKit's VAD (and turn detector) decide when the user is done instead, turn Scribe's VAD off with turn_detection="none". Each time LiveKit ends the user's speech, the plugin commits the audio so far and Scribe returns its final transcript::

session = AgentSession(
    stt=zoom_ai.STT(turn_detection="none"),
    vad=silero.VAD.load(),
    turn_handling={"turn_detection": inference.TurnDetector()},  # or "vad"
    ...
)

In Live mode, recognize() (batch recognition) uses Scribe Fast.

Create a Zoom Scribe STT.

Options map to the Scribe API fields of the same name. For what each feature does, and its availability, pricing and enablement, see https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/

Args

mode
"live" (streaming, default) or "fast" (one request per utterance).
api_key
Zoom AI Services API key (from https://www.zoom.ai/). Defaults to $ZOOM_AI_API_KEY. It is sent on every request and reconnect, so it must stay valid for as long as the agent runs.
language
A Scribe locale, e.g. "en-US"; base codes such as "en" map to their locale. A regional variant Scribe doesn't list resolves to the locale of its base language ("en-GB" to "en-US"); anything else is sent as given, for Scribe to validate. For the supported languages of each model, see https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/
model
Scribe speech recognition model (config.model). None uses Scribe's default model.
language_detection
Scribe language detection mode (config.language_detection.mode; Live supports "continuous"). None turns it off. A detected language Scribe reports is in SpeechData.language; when Scribe reports none, it is empty. With zoom-scribe-multilingual in Live, set it to get each final's language.
turn_detection
Live only: who ends each turn. "server" (default): Scribe's voice activity detection, with silence_duration_ms; use turn_handling={"turn_detection": "stt"}. "none": Scribe's voice activity detection is off (turn_detection.type), and each STT flush from LiveKit (at the end of speech from its VAD, or at the end of turn from its turn detector) sends input_audio_buffer.commit, for which Scribe returns the final transcript. Use it with a VAD and turn_handling={"turn_detection": "vad"} or a turn detector model, not "stt": in this mode the STT sends no start or end of speech events, and finals have no speech_end_time (LiveKit's VAD times the turn). Every flush gets one final; a flush after only digital silence gets an empty one without a commit.
silence_duration_ms
Live only: milliseconds of silence that end a turn in Scribe's voice activity detection; not used with turn_detection="none". Defaults to 350, tuned for voice agents; lower is faster, higher keeps sentences with longer pauses together. For transcription without an agent, a longer value such as 600 can improve accuracy. None uses Scribe's default (see the Scribe docs).
diarization
Label each final transcript with a speaker. Off by default. With one speaker in the utterance, SpeechData.speaker_id is Scribe's label. With several, speaker_id is empty and metadata["speaker_segments"] has each one's span. In Fast mode labels are per request, so they aren't consistent across utterances. MultiSpeakerAdapter works with Live mode only.
word_time_offsets
Fast only: word timings (config.word_time_offsets), in SpeechData.words. Off by default.
channel_separation
Fast only: recognize each channel of stereo audio separately (config.channel_separation); each segment's channel is in metadata["channel_segments"]. Off by default.
vocabulary
Words and phrases that customize recognition (config.vocabulary.phrases).
text_polishing
Scribe's text polishing (outputs.text_polishing). The polished text is the transcript, and the recognized text is in metadata["transcript"]; if polishing fails, the recognized text is used. Word timings aren't returned with polished text. en-US only, and not with language detection. Off by default.
extra_kwargs
Scribe request fields to add or override, shaped like the request, e.g. {"config": {"language_detection": {"candidate_languages": ["en-US", "fr-FR"]}}}<code> to restrict language detection. Merged into the Live </code>session.update and the Fast request, over the options above, so Scribe fields this plugin has no option for can be used. audio.format stays pcm16.
base_url
Scribe API base URL; Live uses <base>/live (as wss) and Fast uses <base>/transcribe. A plaintext (http/ws) URL to a host other than localhost logs a warning.
live_url
Override the Scribe Live WebSocket URL.
fast_url
Override the Scribe Fast HTTP URL.
http_session
Optional aiohttp session; the job's session is used otherwise.

Ancestors

  • livekit.agents.stt.stt.STT
  • abc.ABC
  • EventEmitter
  • typing.Generic

Instance variables

prop model : str
Expand source code
@property
def model(self) -> str:
    """The Scribe model set with ``model``, else the mode: ``"scribe-live"`` or ``"scribe-fast"``."""
    return self._opts.model or f"scribe-{self._mode}"

The Scribe model set with model, else the mode: "scribe-live" or "scribe-fast".

prop provider : str
Expand source code
@property
def provider(self) -> str:
    """The provider name, ``"Zoom"``."""
    return "Zoom"

The provider name, "Zoom".

Methods

async def aclose(self) ‑> None
Expand source code
async def aclose(self) -> None:
    """Close every Live stream opened by :meth:`stream`.

    The HTTP session is left open: it is either supplied by the caller or owned by the
    job's HTTP context.
    """
    streams = list(self._streams)
    self._streams.clear()
    # return_exceptions=True so one stream failing to close doesn't leave the rest open
    await asyncio.gather(*(stream.aclose() for stream in streams), return_exceptions=True)

Close every Live stream opened by :meth:stream.

The HTTP session is left open: it is either supplied by the caller or owned by the job's HTTP context.

def stream(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
conn_options: APIConnectOptions = APIConnectOptions(max_retry=3, retry_interval=2.0, timeout=10.0, liveness_timeout=6.0)) ‑> livekit.plugins.zoom_ai.stt.SpeechStream
Expand source code
def stream(
    self,
    *,
    language: NotGivenOr[str] = NOT_GIVEN,
    conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
) -> SpeechStream:
    """Open a Scribe Live streaming session.

    Args:
        language: Override the STT's language for this stream.
        conn_options: Connection timeout and retry options.

    Raises:
        NotImplementedError: When the STT was created with ``mode="fast"``.
    """
    if self._mode != "live":
        raise NotImplementedError(
            'mode="fast" is not streaming; AgentSession wraps it with a VAD'
        )
    opts = dataclasses.replace(self._opts)
    if is_given(language):
        opts.language = _to_locale(language)
    stream = SpeechStream(stt=self, opts=opts, conn_options=conn_options)
    self._streams.add(stream)
    return stream

Open a Scribe Live streaming session.

Args

language
Override the STT's language for this stream.
conn_options
Connection timeout and retry options.

Raises

NotImplementedError
When the STT was created with mode="fast".
def update_options(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN) ‑> None
Expand source code
def update_options(
    self,
    *,
    language: NotGivenOr[str] = NOT_GIVEN,
    silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN,
) -> None:
    """Change options; open Live streams reconnect with the new session config.

    Args:
        language: Scribe locale, e.g. ``"en-US"``.
        silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's
            default. Not used with ``turn_detection="none"``.

    A turn in progress keeps its session until Scribe's final transcript ends it (at
    most 10 s), so the reconnect doesn't lose it.
    """
    if not is_given(language) and not is_given(silence_duration_ms):
        return
    if is_given(language):
        self._opts.language = _to_locale(language)
    if is_given(silence_duration_ms):
        self._opts.silence_duration_ms = silence_duration_ms
    for stream in self._streams:
        stream.update_options(language=language, silence_duration_ms=silence_duration_ms)

Change options; open Live streams reconnect with the new session config.

Args

language
Scribe locale, e.g. "en-US".
silence_duration_ms
Silence (ms) that ends a turn; None uses Scribe's default. Not used with turn_detection="none".

A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it.

Inherited members

class SpeechStream (*,
stt: STT,
opts: STTOptions,
conn_options: APIConnectOptions)
Expand source code
class SpeechStream(stt.SpeechStream):
    """One Scribe Live session; reconnects on option changes and server-side session ends."""

    _carried: stt.SpeechData | None

    def __init__(self, *, stt: STT, opts: STTOptions, conn_options: APIConnectOptions) -> None:
        super().__init__(stt=stt, conn_options=conn_options, sample_rate=SAMPLE_RATE)
        self._zoom = stt
        self._opts = opts
        # turn_detection can't change after the stream starts (see update_options).
        self._manual_commit = opts.turn_detection == "none"
        # What the current session was configured with (options plus extra_kwargs).
        self._view = _RequestView.of(self._session_update())
        self._reconnect_event = asyncio.Event()
        self._audio_duration = 0.0
        self._speech_started = False
        # turn_detection="none" (manual commit) state:
        # - what is still owed a final, in order: "commit" (a flush that sent a commit),
        #   "silent" (a flush after only digital silence: no commit, answered once the finals
        #   before it arrive) and "drain" (the plugin's own commit before an options
        #   reconnect: its final isn't an answer to any flush);
        self._flushes: collections.deque[Literal["commit", "silent", "drain"]] = collections.deque()
        # - transcript of drain commits, delivered with the final of the next flush so it
        #   can't end a turn early (e.g. commit_user_turn() waiting on that flush);
        self._carried = None
        # - non-silent audio sent to the current session since its last commit;
        self._uncommitted = False
        # - an options reconnect is waiting: input is held back instead of being sent;
        self._draining = False
        # - input held back while draining, sent first on the next session, in order; a
        #   frame leaves it only once sent, so a failed send keeps it for the retry.
        self._held: collections.deque[rtc.AudioFrame | SpeechStream._FlushSentinel] = (
            collections.deque()
        )
        # Serializes sends with the drain commit, so no audio follows it on the old session.
        self._send_lock = asyncio.Lock()
        self._tracking_id: str | None = None  # the current session's x-zm-trackingid
        self._session_updated = False  # Scribe accepted the current session's settings
        # Time between the start of this _run attempt and the current Scribe session.
        # The base class adds whole attempts to start_time_offset when it retries.
        self._session_offset = 0.0

    def update_options(
        self,
        *,
        language: NotGivenOr[str] = NOT_GIVEN,
        silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN,
    ) -> None:
        """Change this stream's options; it reconnects with the new session config.

        Args:
            language: Scribe locale, e.g. ``"en-US"``.
            silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's
                default.

        A turn in progress keeps its session until Scribe's final transcript ends it (at
        most 10 s), so the reconnect doesn't lose it. With ``turn_detection="none"``, audio
        sent since the last flush is committed first, and input that arrives meanwhile is
        sent to the new session; ``silence_duration_ms`` isn't used in that mode, so
        changing only it doesn't reconnect.
        """
        if not is_given(language) and not is_given(silence_duration_ms):
            return
        if is_given(language):
            self._opts.language = _to_locale(language)
        if is_given(silence_duration_ms):
            self._opts.silence_duration_ms = silence_duration_ms
            if self._manual_commit and not is_given(language):
                logger.warning(
                    'silence_duration_ms is not used with turn_detection="none"; '
                    "the Zoom Scribe Live session is unchanged"
                )
                return
        self._reconnect_event.set()

    def _session_update(self) -> dict[str, Any]:
        """The session.update for this stream's options."""
        update: dict[str, Any] = {
            "type": "session.update",
            "language": self._opts.language,
            "audio": {"format": "pcm16"},
        }
        config = _recognition_config(self._opts)
        if config:
            update["config"] = config
        if self._opts.turn_detection == "server" and self._opts.silence_duration_ms is not None:
            update["turn_detection"] = {"silence_duration_ms": self._opts.silence_duration_ms}
        if self._opts.text_polishing:
            update["outputs"] = {"text_polishing": {"enabled": True}}
        update = _merge(update, self._opts.extra_kwargs)
        # Fields the plugin depends on, whatever extra_kwargs says.
        update["type"] = "session.update"
        update["audio"] = {**(update.get("audio") or {}), "format": "pcm16"}
        if self._opts.turn_detection == "none":
            # Scribe's VAD is off; each flush from LiveKit commits the turn.
            update["turn_detection"] = {"type": "none"}
        polishing = (update.get("outputs") or {}).get("text_polishing")
        if isinstance(polishing, dict) and polishing.get("enabled"):
            # Live returns only the polished text unless the transcript is asked for too,
            # and the transcript is the fallback when polishing fails.
            polishing["include_transcript"] = True
        return update

    async def _connect_ws(self) -> aiohttp.ClientWebSocketResponse:
        started_at = time.perf_counter()
        # Each session has its own tracking ID. It is logged on connect and is the
        # request_id of the session's errors, so a failure can be matched in Zoom's logs.
        tracking_id = self._tracking_id = _tracking_id()
        self._session_updated = False
        try:
            ws = await asyncio.wait_for(
                self._zoom._http.ws_connect(
                    self._zoom._live_url,
                    protocols=("live-asr",),
                    headers=self._zoom._headers(tracking_id),
                    heartbeat=30.0,  # notice a dead connection; audio keeps Scribe's session alive
                ),
                self._conn_options.timeout,
            )
        except aiohttp.WSServerHandshakeError as e:
            # aiohttp discards the response body, where Zoom gives its reason (e.g. "Invalid
            # Access token"), so only the status is reported. `from None`: the handshake
            # error's repr keeps the request headers (the API key).
            raise APIStatusError(
                f"Zoom refused the connection (HTTP {e.status})",
                status_code=e.status,
                request_id=tracking_id,
            ) from None
        except (aiohttp.ClientError, asyncio.TimeoutError) as e:
            raise APIConnectionError("failed to connect to Zoom Scribe Live") from e
        try:
            update = self._session_update()
            self._view = _RequestView.of(update)
            await ws.send_str(json.dumps(update))
        except (aiohttp.ClientError, ConnectionError) as e:
            await ws.close()
            raise APIConnectionError("failed to configure the Zoom Scribe Live session") from e
        logger.debug("connected to Zoom Scribe Live (tracking_id=%s)", tracking_id)
        self._report_connection_acquired(time.perf_counter() - started_at, False)
        return ws

    def _turn_open(self) -> bool:
        """A turn whose final transcript a reconnect would lose."""
        if self._manual_commit:
            return any(kind != "silent" for kind in self._flushes)
        return self._speech_started

    def _send_final(self, data: stt.SpeechData, request_id: str = "") -> None:
        """Emit a final transcript event with the supplied speech data."""
        self._event_ch.send_nowait(
            stt.SpeechEvent(
                type=stt.SpeechEventType.FINAL_TRANSCRIPT,
                request_id=request_id,
                alternatives=[data],
            )
        )

    def _answer_flush(self, data: stt.SpeechData | None = None, request_id: str = "") -> None:
        """Answer a flush (turn_detection="none"), with any carried drain transcript first."""
        if data is None:
            data = self._carried or stt.SpeechData(
                language=_reported_language(
                    None, self._view.language_detection, self._opts.language
                ),
                text="",
            )
        elif self._carried is not None:
            data = _join_speech(self._carried, data)
        self._carried = None
        self._send_final(data, request_id)

    def _carry(self, data: stt.SpeechData) -> None:
        """Keep a drain commit's transcript for the final of the next flush."""
        if data.text:
            self._carried = data if self._carried is None else _join_speech(self._carried, data)

    async def _commit(
        self, ws: aiohttp.ClientWebSocketResponse, kind: Literal["commit", "drain"] = "commit"
    ) -> None:
        """Commit the audio sent since the last commit; Scribe answers with one final."""
        self._flushes.append(kind)
        try:
            await ws.send_str(json.dumps({"type": "input_audio_buffer.commit"}))
        except BaseException:
            self._flushes.pop()  # not sent: the caller keeps the flush for the next session
            raise
        self._uncommitted = False

    async def _start_drain(self, ws: aiohttp.ClientWebSocketResponse) -> None:
        """Prepare a turn_detection="none" session for an options reconnect.

        Audio sent since the last flush is committed so its final isn't lost (a commit
        covering only digital silence is skipped), and further input is held back for the
        next session. The reconnect then waits for every outstanding final.
        """
        async with self._send_lock:
            self._draining = True
            if self._uncommitted:
                await self._commit(ws, "drain")

    async def _wait_for_turn_end(self) -> None:
        """Return once no turn is open, or after ``RECONNECT_HOLD_S``."""
        deadline = time.monotonic() + RECONNECT_HOLD_S
        while self._turn_open() and time.monotonic() < deadline:
            await asyncio.sleep(0.05)

    def _end_open_utterance(self, *, closing: bool = False) -> None:
        """Close an utterance cut off by a reconnect so turn detection doesn't wait forever."""
        # With turn_detection="none", answer each flush the old session left unanswered
        # (with an empty final, or the carried transcript), so a flush (e.g.
        # commit_user_turn()) doesn't wait for one. A cut-off drain commit owes no answer.
        for kind in self._flushes:
            if kind != "drain":
                self._answer_flush()
        self._flushes.clear()
        if closing and self._carried is not None:
            self._send_final(self._carried)  # no flush is left to deliver it with
            self._carried = None
        self._uncommitted = False  # audio not committed on the old session is gone with it
        self._draining = False
        if self._speech_started:
            self._speech_started = False
            self._event_ch.send_nowait(stt.SpeechEvent(type=stt.SpeechEventType.END_OF_SPEECH))

    async def _run(self) -> None:
        self._session_offset = 0.0
        closing = False
        closed = asyncio.Event()  # Scribe confirmed our session.close
        last_audio_at = time.monotonic()  # last audio sent, input or silence
        last_input_at = time.monotonic()  # last input audio sent
        gap_filled = False  # silence already sent for the current input gap
        session_audio_s = 0.0  # audio sent on the current Scribe session, keepalive included

        async def send_task(ws: aiohttp.ClientWebSocketResponse) -> None:
            nonlocal closing, last_audio_at, last_input_at, gap_filled, session_audio_s
            anchored = False

            async def send(frame: rtc.AudioFrame | SpeechStream._FlushSentinel) -> None:
                nonlocal last_audio_at, last_input_at, gap_filled, session_audio_s, anchored
                if isinstance(frame, self._FlushSentinel):
                    # LiveKit ended the user's speech. With Scribe's VAD off, commit the
                    # audio so far for its final; otherwise Scribe's VAD ends the turn.
                    if self._manual_commit:
                        if self._uncommitted:
                            await self._commit(ws)
                        elif self._flushes:
                            # Only digital silence since the last commit (e.g. end_input()
                            # after a long pause): Scribe can return made-up text for
                            # that, so this flush gets an empty final instead, after the
                            # finals still due for earlier commits.
                            self._flushes.append("silent")
                        else:
                            self._answer_flush()
                    return
                if not frame.samples_per_channel:
                    return
                # Each frame is sent as it arrives (10-20 ms in an agent). The docs suggest
                # about 100 ms per message, but smaller messages end turns slightly sooner.
                data = frame.data.tobytes()
                await ws.send_bytes(data)
                # Counted once sent: a frame whose send fails is sent again on the retry.
                self._audio_duration += frame.duration
                if self._manual_commit and not self._uncommitted and data.strip(b"\0"):
                    self._uncommitted = True
                last_audio_at = last_input_at = time.monotonic()
                gap_filled = False
                # Scribe's audio_*_ms count from the session's first audio. Audio queued
                # while connecting goes out in a burst, so anchor that 0 ms at the
                # earliest "now minus audio sent", which is when that audio was captured.
                session_audio_s += frame.duration
                captured_at = time.time() - session_audio_s
                if not anchored or captured_at < self.start_time:
                    self.start_time = captured_at
                    anchored = True

            async def send_held() -> None:
                # Each held frame leaves the queue only once sent, so if this session
                # fails, the retry resumes from the first unsent one.
                while self._held:
                    await send(self._held[0])
                    self._held.popleft()

            try:
                async with self._send_lock:
                    await send_held()  # held back while the previous session drained
                async for frame in self._input_ch:
                    async with self._send_lock:
                        if self._draining:
                            self._held.append(frame)  # for the next session
                            continue
                        self._held.append(frame)  # kept until sent, for a retry
                        await send_held()
                # Input ended while a reconnect was waiting: there is no next session, so
                # send what was held to this one.
                async with self._send_lock:
                    self._draining = False
                    await send_held()
                closing = True
                await ws.send_str(json.dumps({"type": "session.close"}))
            except (aiohttp.ClientError, ConnectionError) as e:
                if closing or self._zoom._http.closed:
                    return
                raise APIConnectionError("Zoom Scribe Live connection closed unexpectedly") from e
            # Wait for the remaining final transcript and session.closed, but not forever.
            try:
                await asyncio.wait_for(closed.wait(), self._conn_options.timeout)
            except asyncio.TimeoutError:
                logger.warning("Zoom Scribe Live did not confirm session.close; closing")
                await ws.close()

        async def keepalive_task(ws: aiohttp.ClientWebSocketResponse) -> None:
            # Scribe Live ends a session after 30 s without audio (e.g. the user muted), and it
            # has no keepalive message, so send a little silence when the input has gone quiet.
            # If input stops while a turn is open, send enough silence for Scribe's VAD to end
            # the turn now rather than at the next keepalive.
            nonlocal last_audio_at, session_audio_s, gap_filled
            keepalive = bytes(SAMPLE_RATE * 2 // 10)  # 100 ms
            try:
                while not closing:
                    await asyncio.sleep(INPUT_GAP_S / 2)
                    if closing:
                        break
                    now = time.monotonic()
                    silence = b""
                    if (
                        # Only Scribe's VAD needs silence to end a turn; else the commit does.
                        not self._manual_commit
                        and self._speech_started
                        and not gap_filled
                        and now - last_input_at >= INPUT_GAP_S
                    ):
                        window_ms = self._view.silence_duration_ms or 1000
                        silence = bytes(SAMPLE_RATE * 2 * (window_ms + 100) // 1000)
                        gap_filled = True
                    elif now - last_audio_at >= KEEPALIVE_INTERVAL_S:
                        silence = keepalive
                    if silence:
                        await ws.send_bytes(silence)
                        self._audio_duration += len(silence) / (2 * SAMPLE_RATE)
                        session_audio_s += len(silence) / (2 * SAMPLE_RATE)
                        last_audio_at = now
            except (aiohttp.ClientError, ConnectionError) as e:
                if closing or self._zoom._http.closed:
                    return
                raise APIConnectionError("Zoom Scribe Live connection closed unexpectedly") from e

        async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None:
            while True:
                msg = await ws.receive()
                if msg.type in (
                    aiohttp.WSMsgType.CLOSED,
                    aiohttp.WSMsgType.CLOSE,
                    aiohttp.WSMsgType.CLOSING,
                ):
                    if closing:
                        closed.set()
                        return
                    raise APIStatusError(
                        "Zoom Scribe Live connection closed unexpectedly",
                        status_code=ws.close_code or -1,
                        request_id=self._tracking_id,
                        body=f"{msg.data=} {msg.extra=}",
                    )
                if msg.type != aiohttp.WSMsgType.TEXT:
                    continue
                try:
                    ev = json.loads(msg.data)
                except json.JSONDecodeError:
                    logger.warning("ignoring non-JSON message from Zoom Scribe")
                    continue
                if ev.get("type") == "session.closed" and not closing:
                    # Server-side end (session time limit, idle timeout):
                    # raise a retryable error so the base class reconnects with a new session.
                    raise APIStatusError(
                        f"Zoom Scribe Live {ev['type']} ({ev.get('reason')})",
                        status_code=-1,
                        request_id=self._tracking_id,
                        body=ev,
                        retryable=True,
                    )
                try:
                    if self._process_event(ev):
                        closed.set()
                        return  # session.closed after our session.close
                except APIStatusError:
                    raise
                except Exception:
                    logger.exception("failed to process Zoom Scribe message")

        while True:
            self._end_open_utterance()
            connected_at = time.time()
            session_audio_s = 0.0
            self._reconnect_event.clear()  # this connect reads the current options
            ws = await self._connect_ws()
            tasks = [
                asyncio.create_task(send_task(ws)),
                asyncio.create_task(recv_task(ws)),
                asyncio.create_task(keepalive_task(ws)),
            ]
            io_done: asyncio.Future[list[None]] = asyncio.gather(*tasks[:2])
            wait_reconnect = asyncio.create_task(self._reconnect_event.wait())
            hold: asyncio.Task[None] | None = None  # waits for the turn in progress to end
            reconnect = False
            try:
                waiters: set[asyncio.Future[Any]] = {io_done, wait_reconnect, tasks[2]}
                while True:
                    done, _ = await asyncio.wait(waiters, return_when=asyncio.FIRST_COMPLETED)
                    for task in done:
                        if task is wait_reconnect or task is hold:
                            continue
                        if isinstance(task.exception(), APIConnectionError):
                            # A failed send usually means Scribe closed the socket; prefer the
                            # error event it sent first (e.g. an unsupported language).
                            await asyncio.wait({tasks[1]}, timeout=1.0)
                            if tasks[1].done() and isinstance(tasks[1].exception(), APIStatusError):
                                tasks[1].result()
                        task.result()  # surface errors from I/O and keepalive
                    if io_done in done:
                        break  # input ended and the session closed cleanly
                    if hold is not None and hold in done:
                        reconnect = True  # the turn ended: reconnect with the new config
                        break
                    if wait_reconnect in done:
                        # Options changed: reconnect with the new config, once the turn in
                        # progress (if any) has its final transcript.
                        waiters.discard(wait_reconnect)
                        if self._manual_commit:
                            try:
                                await self._start_drain(ws)
                            except (aiohttp.ClientError, ConnectionError) as e:
                                raise APIConnectionError(
                                    "Zoom Scribe Live connection closed unexpectedly"
                                ) from e
                        if not self._turn_open():
                            reconnect = True
                            break
                        hold = asyncio.create_task(self._wait_for_turn_end())
                        waiters.add(hold)
                        continue
                    waiters.discard(tasks[2])  # keepalive stopped while closing; keep waiting
            finally:
                await utils.aio.gracefully_cancel(*tasks, wait_reconnect, *([hold] if hold else []))
                io_done.cancel()
                with contextlib.suppress(asyncio.CancelledError):
                    io_done.exception()  # retrieve it so asyncio doesn't log it at shutdown
                await ws.close()
            if not reconnect:
                # Scribe closed without a final for the last turn
                self._end_open_utterance(closing=True)
                return
            # Scribe's audio_*_ms restart at 0 on the new session; keep stream-relative
            # timestamps moving forward, as the base class does when it retries.
            self._session_offset += time.time() - connected_at

    def _speech_end_time(self, end_ms: int) -> float | None:
        """Wall-clock end of speech for a final ending at ``end_ms``.

        Scribe ends a segment after ``silence_duration_ms`` of silence, so ``audio_end_ms``
        is that much later than the end of speech. With ``silence_duration_ms=None`` the
        window is Scribe's, so ``audio_end_ms`` is used as is.

        With ``turn_detection="none"`` the segment ends where the flush committed it, after
        LiveKit's VAD silence (and any turn detector hold), not at the end of speech. Only
        LiveKit's VAD knows when speech ended, so no time is reported.
        """
        if not end_ms or self._manual_commit:
            return None
        silence_ms = self._view.silence_duration_ms or 0
        return self.start_time + max(0, end_ms - silence_ms) / 1000

    def _process_event(self, ev: dict[str, Any]) -> bool:
        kind = ev.get("type")
        if kind == "input_audio_buffer.speech_started":
            # With turn_detection="none", Scribe opens its next turn right after each
            # commit; LiveKit's VAD owns the user's turn.
            if not self._speech_started and not self._manual_commit:
                self._speech_started = True
                start_ms = ev.get("audio_start_ms")
                self._event_ch.send_nowait(
                    stt.SpeechEvent(
                        type=stt.SpeechEventType.START_OF_SPEECH,
                        speech_start_time=self.start_time + start_ms / 1000
                        if start_ms is not None
                        else None,
                    )
                )
        elif kind == "transcription.completed":
            text = _clean(ev.get("transcript"))
            start_ms, end_ms = ev.get("audio_start_ms") or 0, ev.get("audio_end_ms") or 0
            speaker_id: str | None = None
            metadata: dict[str, Any] | None = None
            if self._view.text_polishing:
                polished = _polished_text(ev)
                if polished is not None:
                    metadata = {"transcript": text}
                    text = polished
            if self._view.diarization:
                # One speaker: speaker_id. Several: Scribe sends `speaker_segments`, and
                # speaker_id is left empty (one field can't name several speakers).
                spans = ev.get("speaker_segments") or []
                if spans:
                    metadata = {**(metadata or {}), "speaker_segments": spans}
                else:
                    speaker_id = ev.get("speaker") or None
            answers: Literal["commit", "silent", "drain"] = "commit"
            if self._manual_commit:
                # This final answers the oldest commit ("silent" entries never get one).
                answers = self._flushes.popleft() if self._flushes else "commit"
            elif text and not self._speech_started:
                # No speech_started for this transcript (e.g. it was lost in a reconnect):
                # open the turn so turn_detection="stt" still commits it.
                self._speech_started = True
                self._event_ch.send_nowait(
                    stt.SpeechEvent(
                        type=stt.SpeechEventType.START_OF_SPEECH,
                        speech_start_time=self.start_time + start_ms / 1000
                        if "audio_start_ms" in ev
                        else None,
                    )
                )
            data = stt.SpeechData(
                language=_reported_language(
                    ev.get("language"), self._view.language_detection, self._opts.language
                ),
                text=text,
                start_time=start_ms / 1000 + self.start_time_offset + self._session_offset,
                end_time=end_ms / 1000 + self.start_time_offset + self._session_offset,
                speaker_id=speaker_id,
                metadata=metadata,
            )
            if self._manual_commit:
                # Each final answers a flush, so an empty one is sent too: it tells LiveKit
                # the flush is done (e.g. commit_user_turn() waits). A drain commit's
                # transcript waits for the next flush's final instead.
                if answers == "drain":
                    self._carry(data)
                else:
                    self._answer_flush(data, ev.get("item_id", ""))
                # Then answer the silence-only flushes that were waiting behind this one.
                while self._flushes and self._flushes[0] == "silent":
                    self._flushes.popleft()
                    self._answer_flush()
            elif text:
                self._event_ch.send_nowait(
                    stt.SpeechEvent(
                        type=stt.SpeechEventType.FINAL_TRANSCRIPT,
                        request_id=ev.get("item_id", ""),
                        alternatives=[data],
                        speech_end_time=self._speech_end_time(end_ms),
                    )
                )
            # Scribe's final closes the utterance: end of speech comes after the transcript,
            # so turn_detection="stt" commits the turn with the text already in hand.
            if self._speech_started:
                self._speech_started = False
                # speech_end_time lets turn_detection="stt" count its endpointing delay from
                # when the user stopped speaking, not from when the final arrived.
                self._event_ch.send_nowait(
                    stt.SpeechEvent(
                        type=stt.SpeechEventType.END_OF_SPEECH,
                        speech_end_time=self._speech_end_time(end_ms),
                    )
                )
            self._report_usage()
        elif kind == "session.updated":
            self._session_updated = True
            # Scribe never rejects diarization; it reports whether labels will come.
            if self._view.diarization and ev.get("diarization_enabled") is False:
                logger.warning("Zoom Scribe Live can't label speakers for this session")
        elif kind == "error":
            err = ev.get("error") or {}
            code = err.get("code")
            if err.get("fatal"):
                # Scribe rejects a session's settings (e.g. an unsupported language) with a
                # fatal error in place of session.updated, then closes. A new session would be
                # rejected the same way, so that isn't retried, and a FallbackAdapter can take
                # over. A fatal error after session.updated is retried with a new session.
                raise APIStatusError(
                    f"Zoom Scribe Live error {code}: {err.get('message')}",
                    status_code=-1,
                    request_id=self._tracking_id,
                    body=err,
                    retryable=self._session_updated,
                )
            # A non-fatal error leaves the session running. (A non-fatal invalid_config, e.g.
            # text polishing for an unsupported locale, is followed by Scribe's own fatal
            # error.) Scribe's message is kept out of the log text in case it quotes user
            # content.
            logger.warning(
                "Zoom Scribe Live error %s (non-fatal, tracking_id=%s)",
                code,
                self._tracking_id,
                extra={"lk.pii.message": err.get("message")},
            )
        elif kind == "session.closed":
            self._report_usage()  # audio sent after the last final transcript
            return True
        return False

    def _report_usage(self) -> None:
        if self._audio_duration > 0:
            self._event_ch.send_nowait(
                stt.SpeechEvent(
                    type=stt.SpeechEventType.RECOGNITION_USAGE,
                    recognition_usage=stt.RecognitionUsage(audio_duration=self._audio_duration),
                )
            )
            self._audio_duration = 0.0

One Scribe Live session; reconnects on option changes and server-side session ends.

Args: sample_rate : int or None, optional The desired sample rate for the audio input. If specified, the audio input will be automatically resampled to match the given sample rate before being processed for Speech-to-Text. If not provided (None), the input will retain its original sample rate.

Ancestors

  • livekit.agents.stt.stt.RecognizeStream
  • abc.ABC

Methods

def update_options(self,
*,
language: NotGivenOr[str] = NOT_GIVEN,
silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN) ‑> None
Expand source code
def update_options(
    self,
    *,
    language: NotGivenOr[str] = NOT_GIVEN,
    silence_duration_ms: NotGivenOr[int | None] = NOT_GIVEN,
) -> None:
    """Change this stream's options; it reconnects with the new session config.

    Args:
        language: Scribe locale, e.g. ``"en-US"``.
        silence_duration_ms: Silence (ms) that ends a turn; ``None`` uses Scribe's
            default.

    A turn in progress keeps its session until Scribe's final transcript ends it (at
    most 10 s), so the reconnect doesn't lose it. With ``turn_detection="none"``, audio
    sent since the last flush is committed first, and input that arrives meanwhile is
    sent to the new session; ``silence_duration_ms`` isn't used in that mode, so
    changing only it doesn't reconnect.
    """
    if not is_given(language) and not is_given(silence_duration_ms):
        return
    if is_given(language):
        self._opts.language = _to_locale(language)
    if is_given(silence_duration_ms):
        self._opts.silence_duration_ms = silence_duration_ms
        if self._manual_commit and not is_given(language):
            logger.warning(
                'silence_duration_ms is not used with turn_detection="none"; '
                "the Zoom Scribe Live session is unchanged"
            )
            return
    self._reconnect_event.set()

Change this stream's options; it reconnects with the new session config.

Args

language
Scribe locale, e.g. "en-US".
silence_duration_ms
Silence (ms) that ends a turn; None uses Scribe's default.

A turn in progress keeps its session until Scribe's final transcript ends it (at most 10 s), so the reconnect doesn't lose it. With turn_detection="none", audio sent since the last flush is committed first, and input that arrives meanwhile is sent to the new session; silence_duration_ms isn't used in that mode, so changing only it doesn't reconnect.