Module livekit.plugins.atmee
Atmee avatar plugin for LiveKit Agents.
Bring your own LiveKit voice agent; Atmee renders its avatar into your room. Any Atmee avatar renders by its id, v1 or v2 (create avatars in the Atmee studio or with the Atmee API)::
from livekit.plugins import atmee
avatar = atmee.AvatarSession(avatar_id="...") # ATMEE_API_KEY in the environment
await avatar.start(session, room=ctx.room)
await session.start(agent=..., room=ctx.room) # the agent's TTS drives the avatar's video
See https://docs.livekit.io/agents/models/avatar/plugins/atmee/ for more information.
Classes
class AtmeeAPI (api_key: NotGivenOr[str] = NOT_GIVEN,
api_url: NotGivenOr[str] = NOT_GIVEN,
*,
conn_options: APIConnectOptions = APIConnectOptions(max_retry=3, retry_interval=2.0, timeout=10.0, liveness_timeout=6.0),
session: aiohttp.ClientSession | None = None)-
Expand source code
class AtmeeAPI: """Async client for the Atmee API. Args: api_key: your ``sk_atmee_...`` key; defaults to ``ATMEE_API_KEY``. api_url: API base; defaults to ``ATMEE_API_URL`` or ``https://api.atmanity.us``. conn_options: retry/timeout policy. ``timeout`` bounds the connect of every request and each attempt of the calls that are safe to repeat (reading and ending a session, reading an avatar), which are retried ``max_retry`` times on transport failures and 5xx answers. Starting a session is never repeated and has a budget of its own. session: an ``aiohttp.ClientSession`` to reuse. Inside a LiveKit job the worker's shared session is used; outside (scripts, tests) a private one is created and closed by :meth:`aclose`. """ def __init__( self, api_key: NotGivenOr[str] = NOT_GIVEN, api_url: NotGivenOr[str] = NOT_GIVEN, *, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, session: aiohttp.ClientSession | None = None, ) -> None: key = api_key or os.getenv("ATMEE_API_KEY") if not key: raise AtmeeException( "ATMEE_API_KEY must be set (or pass api_key=...); create one at " "https://www.atmee.ai/studio/api-keys" ) self._api_key = key self._api_url = (api_url or os.getenv("ATMEE_API_URL") or DEFAULT_API_URL).rstrip("/") _check_api_url(self._api_url) self._conn_options = conn_options self._session = session self._owns_session = False # --- avatar sessions ------------------------------------------------- async def create_avatar_session( self, avatar_id: str, *, livekit_url: str, livekit_token: str, agent_identity: str | None = None, max_duration_seconds: int | None = None, metadata: dict[str, Any] | None = None, wait_for: WaitFor = "initializing", avatar_version: NotGivenOr[str] = NOT_GIVEN, stop: asyncio.Event | None = None, ) -> AvatarSessionInfo: """Render ``avatar_id`` into the room ``livekit_token`` grants. ``livekit_token`` is a LiveKit access token you minted with YOUR project's secret for the avatar participant: ``kind=agent``, a ``roomJoin`` grant for your room, and the ``lk.publish_on_behalf`` attribute naming your agent (:class:`AvatarSession` does this for you). Returns as soon as the rendering worker acknowledged the start, or once the avatar joined with ``wait_for="avatar_joined"``. Any avatar renders as what it is, v1 or v2; the request names no generation. The call first reads the avatar's generation to know how long the start may take: a v1 start may take up to 1 minute (3 with ``wait_for="avatar_joined"``). The first v2 session after a quiet period can take minutes, so a v2 start may take up to 5 (7.5), and while the API answers ``avatar_warming`` or ``avatar_start_interrupted`` (nothing billed) it asks again after the ``Retry-After`` given. Once ``stop`` is set, the call asks no more: it raises an ``AtmeeException`` with ``code="closed"`` instead of asking again. A request already sent is still awaited, since it may open a session that must then be ended. """ _check_wait_for(wait_for) _warn_avatar_version(avatar_version) payload: dict[str, Any] = {"livekitUrl": livekit_url, "livekitToken": livekit_token} if agent_identity: payload["agentIdentity"] = agent_identity if max_duration_seconds is not None: payload["maxDurationSeconds"] = int(max_duration_seconds) if metadata: payload["metadata"] = metadata params: dict[str, str] | None = ( {"waitFor": wait_for} if wait_for != "initializing" else None ) # Never retried on a timeout or a 5xx: the POST reserves and bills a # session as soon as the server accepts it, so asking again could open # a second render for the same room. Only the answers that say nothing # was reserved (`avatar_warming`, `avatar_start_interrupted`) are asked # again, within the start budget. loop = asyncio.get_running_loop() budget = await self._start_budget(avatar_id, wait_for) deadline = loop.time() + budget while True: if stop is not None and stop.is_set(): raise AtmeeException("the avatar session start was stopped", code="closed") try: data = await self._request( "POST", f"/v1/avatars/{avatar_id}/avatar_sessions", json=payload, params=params, retry=False, total_timeout=max(1.0, deadline - loop.time()), ) break except AtmeeAvatarWarmingError as e: pause = e.retry_after if e.retry_after is not None else _WARMING_RETRY_DEFAULT pause = min(max(pause, _WARMING_RETRY_BOUNDS[0]), _WARMING_RETRY_BOUNDS[1]) if loop.time() + pause >= deadline: raise logger.info( "atmee avatar render not started yet; asking again", extra={"code": e.code, "retry_in": pause}, ) # nothing is reserved while waiting: a stop ends the wait at once if stop is None: await asyncio.sleep(pause) else: try: await asyncio.wait_for(stop.wait(), timeout=pause) except asyncio.TimeoutError: pass session_id = data.get("sessionId") if not isinstance(session_id, str) or not session_id: raise AtmeeException( f"the Atmee API accepted the avatar session but returned no sessionId: {data!r}", code="invalid_response", ) return AvatarSessionInfo( session_id=session_id, status=str(data.get("status", "")), avatar_participant_identity=str(data.get("avatarParticipantIdentity", "")), agent_identity=str(data.get("agentIdentity", "")), room_name=str(data.get("roomName", "")), max_duration_seconds=int(data.get("maxDurationSeconds") or 0), raw=data, ) async def _start_budget(self, avatar_id: str, wait_for: WaitFor) -> float: """How long a start of ``avatar_id`` may take, from its generation.""" try: info = await self._get_avatar( avatar_id, deadline=asyncio.get_running_loop().time() + self._conn_options.timeout ) except AtmeeException as e: # A key without read access to the avatar, or a slow read: allow # for the slower generation; the start reports real failures. logger.debug( "could not read the atmee avatar's generation; allowing for a v2 start", extra={"avatar_id": avatar_id, "error": str(e)}, ) return _START_TOTAL_TIMEOUT["v2"][wait_for] generation = "v2" if info.engine in _V2_ENGINES else "v1" return _START_TOTAL_TIMEOUT[generation][wait_for] async def end_avatar_session(self, session_id: str) -> dict[str, Any]: """Stop billing for a session now. Idempotent: ending an ended session answers normally with ``alreadyEnded: true``. Does not remove the avatar from your room — :meth:`AvatarSession.aclose` does that.""" return await self._request("POST", f"/v1/avatar_sessions/{session_id}/end") async def get_avatar_session(self, session_id: str) -> dict[str, Any]: return await self._request("GET", f"/v1/avatar_sessions/{session_id}") # --- avatars ----------------------------------------------------------- async def create_avatar( self, name: str, image: str | Path | bytes, *, description: str | None = None, content_type: str | None = None, avatar_version: NotGivenOr[str] = NOT_GIVEN, ) -> str: """Deprecated: create avatars in the Atmee studio or with the Atmee API. Creates a v1 avatar from a portrait (a file path, the image bytes or an ``https://`` URL) and returns its id. """ logger.warning( "create_avatar is deprecated, create avatars in the Atmee studio or with the Atmee API" ) if isinstance(image, str) and image.startswith(("http://", "https://")): manifest = {"schemaVersion": 1, "name": name, "assets": {"image": {"url": image}}} if description: manifest["description"] = description created = await self._request( "POST", "/v1/avatars", json=manifest, retry=False, total_timeout=120.0 ) else: if isinstance(image, bytes): blob = image filename = "portrait" + (mimetypes.guess_extension(content_type or "") or ".jpg") else: blob, filename = await asyncio.to_thread(Path(image).read_bytes), Path(image).name form = aiohttp.FormData() form.add_field("name", name) if description: form.add_field("description", description) ctype = content_type or mimetypes.guess_type(filename)[0] or "image/jpeg" form.add_field("file", blob, filename=filename, content_type=ctype) created = await self._request( "POST", "/v1/avatars", data=form, retry=False, total_timeout=120.0 ) return str(created.get("avatarId") or created.get("id") or "") async def get_avatar(self, avatar_id: str) -> AvatarInfo: return await self._get_avatar(avatar_id) async def _get_avatar(self, avatar_id: str, *, deadline: float | None = None) -> AvatarInfo: return _avatar_info( await self._request("GET", f"/v1/avatars/{avatar_id}", deadline=deadline) ) # --- plumbing ------------------------------------------------------------ def _ensure_session(self) -> aiohttp.ClientSession: if self._session is not None and not self._session.closed: return self._session try: self._session = utils.http_context.http_session() except RuntimeError: # Outside a LiveKit job (a script creating avatars, tests): own one. self._session = aiohttp.ClientSession() self._owns_session = True return self._session async def aclose(self) -> None: """Close the HTTP session if this client created it.""" if self._owns_session and self._session is not None and not self._session.closed: await self._session.close() self._session = None self._owns_session = False async def __aenter__(self) -> AtmeeAPI: return self async def __aexit__(self, *exc: object) -> None: await self.aclose() async def _request( self, method: str, path: str, *, json: dict[str, Any] | None = None, data: aiohttp.FormData | None = None, params: dict[str, str] | None = None, retry: bool = True, total_timeout: float | None = None, deadline: float | None = None, ) -> dict[str, Any]: """One API call with the plugin's retry policy: transport errors and 5xx answers are retried up to ``conn_options.max_retry`` times after the first attempt; a 503 ``no_capacity`` or ``avatar_warming`` and every 4xx are final. ``retry=False`` for calls that are not idempotent (creating a session). Each attempt is bounded by ``total_timeout``, ``conn_options.timeout`` when none is given; ``deadline`` (event-loop time) bounds the whole call, retries and the pauses between them included.""" loop = asyncio.get_running_loop() attempts = self._conn_options.max_retry + 1 if retry else 1 per_attempt = self._conn_options.timeout if total_timeout is None else total_timeout last_error: Exception | None = None for attempt in range(attempts): attempt_timeout = per_attempt if deadline is not None: remaining = deadline - loop.time() if remaining <= 0: # the budget ran out: report a timeout, keep the last failure as its cause raise AtmeeException(f"{method} {path} timed out") from last_error attempt_timeout = min(per_attempt, remaining) try: async with self._ensure_session().request( method, f"{self._api_url}{path}", headers={"X-Api-Key": self._api_key, "Accept": "application/json"}, json=json, data=data, params=params, timeout=aiohttp.ClientTimeout( sock_connect=self._conn_options.timeout, total=attempt_timeout ), ) as response: if response.ok: if response.status == 204 or response.content_length == 0: return {} try: payload = await response.json(content_type=None) except ValueError as e: raise AtmeeException( "the Atmee API returned an invalid JSON response", status_code=response.status, code="invalid_response", ) from e return payload if isinstance(payload, dict) else {"data": payload} raise await _error_from_response(response) except AtmeeException as e: if e.status_code >= 500 and not isinstance( e, AtmeeNoCapacityError | AtmeeAvatarWarmingError ): last_error = e else: raise except (aiohttp.ClientError, asyncio.TimeoutError) as e: last_error = AtmeeException(f"could not reach the Atmee API: {e}") if attempt < attempts - 1: logger.warning( "atmee api call failed; retrying", extra={"path": path, "attempt": attempt + 1, "error": str(last_error)}, ) pause = self._conn_options.retry_interval if deadline is not None: pause = min(pause, max(0.0, deadline - loop.time())) await asyncio.sleep(pause) assert last_error is not None raise last_errorAsync client for the Atmee API.
Args
api_key- your
sk_atmee_…key; defaults toATMEE_API_KEY. api_url- API base; defaults to
ATMEE_API_URLorhttps://api.atmanity.us. conn_options- retry/timeout policy.
timeoutbounds the connect of every request and each attempt of the calls that are safe to repeat (reading and ending a session, reading an avatar), which are retriedmax_retrytimes on transport failures and 5xx answers. Starting a session is never repeated and has a budget of its own. session- an
aiohttp.ClientSessionto reuse. Inside a LiveKit job the worker's shared session is used; outside (scripts, tests) a private one is created and closed by :meth:aclose.
Methods
async def aclose(self) ‑> None-
Expand source code
async def aclose(self) -> None: """Close the HTTP session if this client created it.""" if self._owns_session and self._session is not None and not self._session.closed: await self._session.close() self._session = None self._owns_session = FalseClose the HTTP session if this client created it.
async def create_avatar(self,
name: str,
image: str | Path | bytes,
*,
description: str | None = None,
content_type: str | None = None,
avatar_version: NotGivenOr[str] = NOT_GIVEN) ‑> str-
Expand source code
async def create_avatar( self, name: str, image: str | Path | bytes, *, description: str | None = None, content_type: str | None = None, avatar_version: NotGivenOr[str] = NOT_GIVEN, ) -> str: """Deprecated: create avatars in the Atmee studio or with the Atmee API. Creates a v1 avatar from a portrait (a file path, the image bytes or an ``https://`` URL) and returns its id. """ logger.warning( "create_avatar is deprecated, create avatars in the Atmee studio or with the Atmee API" ) if isinstance(image, str) and image.startswith(("http://", "https://")): manifest = {"schemaVersion": 1, "name": name, "assets": {"image": {"url": image}}} if description: manifest["description"] = description created = await self._request( "POST", "/v1/avatars", json=manifest, retry=False, total_timeout=120.0 ) else: if isinstance(image, bytes): blob = image filename = "portrait" + (mimetypes.guess_extension(content_type or "") or ".jpg") else: blob, filename = await asyncio.to_thread(Path(image).read_bytes), Path(image).name form = aiohttp.FormData() form.add_field("name", name) if description: form.add_field("description", description) ctype = content_type or mimetypes.guess_type(filename)[0] or "image/jpeg" form.add_field("file", blob, filename=filename, content_type=ctype) created = await self._request( "POST", "/v1/avatars", data=form, retry=False, total_timeout=120.0 ) return str(created.get("avatarId") or created.get("id") or "")Deprecated: create avatars in the Atmee studio or with the Atmee API.
Creates a v1 avatar from a portrait (a file path, the image bytes or an
https://URL) and returns its id. async def create_avatar_session(self,
avatar_id: str,
*,
livekit_url: str,
livekit_token: str,
agent_identity: str | None = None,
max_duration_seconds: int | None = None,
metadata: dict[str, Any] | None = None,
wait_for: WaitFor = 'initializing',
avatar_version: NotGivenOr[str] = NOT_GIVEN,
stop: asyncio.Event | None = None) ‑> livekit.plugins.atmee.api.AvatarSessionInfo-
Expand source code
async def create_avatar_session( self, avatar_id: str, *, livekit_url: str, livekit_token: str, agent_identity: str | None = None, max_duration_seconds: int | None = None, metadata: dict[str, Any] | None = None, wait_for: WaitFor = "initializing", avatar_version: NotGivenOr[str] = NOT_GIVEN, stop: asyncio.Event | None = None, ) -> AvatarSessionInfo: """Render ``avatar_id`` into the room ``livekit_token`` grants. ``livekit_token`` is a LiveKit access token you minted with YOUR project's secret for the avatar participant: ``kind=agent``, a ``roomJoin`` grant for your room, and the ``lk.publish_on_behalf`` attribute naming your agent (:class:`AvatarSession` does this for you). Returns as soon as the rendering worker acknowledged the start, or once the avatar joined with ``wait_for="avatar_joined"``. Any avatar renders as what it is, v1 or v2; the request names no generation. The call first reads the avatar's generation to know how long the start may take: a v1 start may take up to 1 minute (3 with ``wait_for="avatar_joined"``). The first v2 session after a quiet period can take minutes, so a v2 start may take up to 5 (7.5), and while the API answers ``avatar_warming`` or ``avatar_start_interrupted`` (nothing billed) it asks again after the ``Retry-After`` given. Once ``stop`` is set, the call asks no more: it raises an ``AtmeeException`` with ``code="closed"`` instead of asking again. A request already sent is still awaited, since it may open a session that must then be ended. """ _check_wait_for(wait_for) _warn_avatar_version(avatar_version) payload: dict[str, Any] = {"livekitUrl": livekit_url, "livekitToken": livekit_token} if agent_identity: payload["agentIdentity"] = agent_identity if max_duration_seconds is not None: payload["maxDurationSeconds"] = int(max_duration_seconds) if metadata: payload["metadata"] = metadata params: dict[str, str] | None = ( {"waitFor": wait_for} if wait_for != "initializing" else None ) # Never retried on a timeout or a 5xx: the POST reserves and bills a # session as soon as the server accepts it, so asking again could open # a second render for the same room. Only the answers that say nothing # was reserved (`avatar_warming`, `avatar_start_interrupted`) are asked # again, within the start budget. loop = asyncio.get_running_loop() budget = await self._start_budget(avatar_id, wait_for) deadline = loop.time() + budget while True: if stop is not None and stop.is_set(): raise AtmeeException("the avatar session start was stopped", code="closed") try: data = await self._request( "POST", f"/v1/avatars/{avatar_id}/avatar_sessions", json=payload, params=params, retry=False, total_timeout=max(1.0, deadline - loop.time()), ) break except AtmeeAvatarWarmingError as e: pause = e.retry_after if e.retry_after is not None else _WARMING_RETRY_DEFAULT pause = min(max(pause, _WARMING_RETRY_BOUNDS[0]), _WARMING_RETRY_BOUNDS[1]) if loop.time() + pause >= deadline: raise logger.info( "atmee avatar render not started yet; asking again", extra={"code": e.code, "retry_in": pause}, ) # nothing is reserved while waiting: a stop ends the wait at once if stop is None: await asyncio.sleep(pause) else: try: await asyncio.wait_for(stop.wait(), timeout=pause) except asyncio.TimeoutError: pass session_id = data.get("sessionId") if not isinstance(session_id, str) or not session_id: raise AtmeeException( f"the Atmee API accepted the avatar session but returned no sessionId: {data!r}", code="invalid_response", ) return AvatarSessionInfo( session_id=session_id, status=str(data.get("status", "")), avatar_participant_identity=str(data.get("avatarParticipantIdentity", "")), agent_identity=str(data.get("agentIdentity", "")), room_name=str(data.get("roomName", "")), max_duration_seconds=int(data.get("maxDurationSeconds") or 0), raw=data, )Render
avatar_idinto the roomlivekit_tokengrants.livekit_tokenis a LiveKit access token you minted with YOUR project's secret for the avatar participant:kind=agent, aroomJoingrant for your room, and thelk.publish_on_behalfattribute naming your agent (:class:AvatarSessiondoes this for you). Returns as soon as the rendering worker acknowledged the start, or once the avatar joined withwait_for="avatar_joined".Any avatar renders as what it is, v1 or v2; the request names no generation. The call first reads the avatar's generation to know how long the start may take: a v1 start may take up to 1 minute (3 with
wait_for="avatar_joined"). The first v2 session after a quiet period can take minutes, so a v2 start may take up to 5 (7.5), and while the API answersavatar_warmingoravatar_start_interrupted(nothing billed) it asks again after theRetry-Aftergiven.Once
stopis set, the call asks no more: it raises anAtmeeExceptionwithcode="closed"instead of asking again. A request already sent is still awaited, since it may open a session that must then be ended. async def end_avatar_session(self, session_id: str) ‑> dict[str, typing.Any]-
Expand source code
async def end_avatar_session(self, session_id: str) -> dict[str, Any]: """Stop billing for a session now. Idempotent: ending an ended session answers normally with ``alreadyEnded: true``. Does not remove the avatar from your room — :meth:`AvatarSession.aclose` does that.""" return await self._request("POST", f"/v1/avatar_sessions/{session_id}/end")Stop billing for a session now. Idempotent: ending an ended session answers normally with
alreadyEnded: true. Does not remove the avatar from your room — :meth:AvatarSession.aclose()does that. async def get_avatar(self, avatar_id: str) ‑> livekit.plugins.atmee.api.AvatarInfo-
Expand source code
async def get_avatar(self, avatar_id: str) -> AvatarInfo: return await self._get_avatar(avatar_id) async def get_avatar_session(self, session_id: str) ‑> dict[str, typing.Any]-
Expand source code
async def get_avatar_session(self, session_id: str) -> dict[str, Any]: return await self._request("GET", f"/v1/avatar_sessions/{session_id}")
class AtmeeAvatarNotReadyError (message: str, *, status_code: int = 0, code: str | None = None)-
Expand source code
class AtmeeAvatarNotReadyError(AtmeeException): """The avatar cannot be rendered yet (HTTP 409 ``avatar_not_renderable``): it has no portrait, or its appearance is still being processed."""The avatar cannot be rendered yet (HTTP 409
avatar_not_renderable): it has no portrait, or its appearance is still being processed.Ancestors
- livekit.plugins.atmee.api.AtmeeException
- builtins.Exception
- builtins.BaseException
class AtmeeAvatarWarmingError (message: str, *, code: str, retry_after: float | None = None)-
Expand source code
class AtmeeAvatarWarmingError(AtmeeException): """The render could not start yet, and nothing was billed (HTTP 503): ``avatar_warming``, a v2 avatar is still getting ready to render, or ``avatar_start_interrupted``, the render ended before the avatar joined. :meth:`AtmeeAPI.create_avatar_session` asks again by itself within its start budget; this is raised only once that budget is spent. ``retry_after`` is the server's hint in seconds, when it sent one. """ def __init__(self, message: str, *, code: str, retry_after: float | None = None) -> None: super().__init__(message, status_code=503, code=code) self.retry_after = retry_afterThe render could not start yet, and nothing was billed (HTTP 503):
avatar_warming, a v2 avatar is still getting ready to render, oravatar_start_interrupted, the render ended before the avatar joined.:meth:
AtmeeAPI.create_avatar_session()asks again by itself within its start budget; this is raised only once that budget is spent.retry_afteris the server's hint in seconds, when it sent one.Ancestors
- livekit.plugins.atmee.api.AtmeeException
- builtins.Exception
- builtins.BaseException
class AtmeeException (message: str, *, status_code: int = 0, code: str | None = None)-
Expand source code
class AtmeeException(Exception): """An Atmee API call failed. ``status_code`` is the HTTP status (0 when the request never got an answer), ``code`` the machine-readable ``error`` field of the response body when the API sent one (``invalid_livekit_token``, ``avatar_not_renderable``, ``insufficient_credits``, ...), and ``message`` its human-readable text. """ def __init__(self, message: str, *, status_code: int = 0, code: str | None = None) -> None: super().__init__(message) self.status_code = status_code self.code = code self.message = message def __str__(self) -> str: parts = [self.message] if self.code: parts.append(f"[{self.code}]") if self.status_code: parts.append(f"(HTTP {self.status_code})") return " ".join(parts)An Atmee API call failed.
status_codeis the HTTP status (0 when the request never got an answer),codethe machine-readableerrorfield of the response body when the API sent one (invalid_livekit_token,avatar_not_renderable,insufficient_credits, …), andmessageits human-readable text.Ancestors
- builtins.Exception
- builtins.BaseException
Subclasses
- livekit.plugins.atmee.api.AtmeeAvatarNotReadyError
- livekit.plugins.atmee.api.AtmeeAvatarWarmingError
- livekit.plugins.atmee.api.AtmeeNoCapacityError
class AtmeeNoCapacityError (message: str, *, retry_after: float | None = None)-
Expand source code
class AtmeeNoCapacityError(AtmeeException): """Every rendering worker is busy (HTTP 503 ``no_capacity``). Never retried by the client: a render slot frees up on the order of seconds to minutes, not the milliseconds a transport retry covers. ``retry_after`` is the server's hint in seconds, when it sent one. """ def __init__(self, message: str, *, retry_after: float | None = None) -> None: super().__init__(message, status_code=503, code="no_capacity") self.retry_after = retry_afterEvery rendering worker is busy (HTTP 503
no_capacity).Never retried by the client: a render slot frees up on the order of seconds to minutes, not the milliseconds a transport retry covers.
retry_afteris the server's hint in seconds, when it sent one.Ancestors
- livekit.plugins.atmee.api.AtmeeException
- builtins.Exception
- builtins.BaseException
class AvatarInfo (avatar_id: str,
status: str,
kind: str | None = None,
name: str | None = None,
engine: str | None = None,
raw: dict[str, Any] = <factory>)-
Expand source code
@dataclass class AvatarInfo: """What ``GET /v1/avatars/{avatarId}`` reports about an avatar. ``kind`` (``conversational`` | ``render_only``) says whether the avatar has a voice and persona; ``engine`` is its generation as the API reports it (``"v1"``, ``"v2"`` or ``"v2hd"``; ``None`` when the API names none). """ avatar_id: str status: str kind: str | None = None name: str | None = None engine: str | None = None raw: dict[str, Any] = field(default_factory=dict, repr=False) @property def ready(self) -> bool: """Whether it can be rendered: built, and for a v2 avatar, serving a take.""" if self.status != "ready": return False return self.engine not in _V2_ENGINES or bool(self.raw.get("servingCandidateId"))What
GET /v1/avatars/{avatarId}reports about an avatar.kind(conversational|render_only) says whether the avatar has a voice and persona;engineis its generation as the API reports it ("v1","v2"or"v2hd";Nonewhen the API names none).Instance variables
var avatar_id : strvar engine : str | Nonevar kind : str | Nonevar name : str | Nonevar raw : dict[str, typing.Any]prop ready : bool-
Expand source code
@property def ready(self) -> bool: """Whether it can be rendered: built, and for a v2 avatar, serving a take.""" if self.status != "ready": return False return self.engine not in _V2_ENGINES or bool(self.raw.get("servingCandidateId"))Whether it can be rendered: built, and for a v2 avatar, serving a take.
var status : str
class AvatarSession (*,
avatar_id: str,
api_url: NotGivenOr[str] = NOT_GIVEN,
api_key: NotGivenOr[str] = NOT_GIVEN,
avatar_participant_identity: NotGivenOr[str] = NOT_GIVEN,
avatar_participant_name: NotGivenOr[str] = NOT_GIVEN,
max_duration_seconds: int = 3600,
metadata: dict[str, Any] | None = None,
wait_for: WaitFor = 'initializing',
conn_options: APIConnectOptions = APIConnectOptions(max_retry=3, retry_interval=2.0, timeout=10.0, liveness_timeout=6.0),
http_session: aiohttp.ClientSession | None = None,
avatar_version: NotGivenOr[str] = NOT_GIVEN)-
Expand source code
class AvatarSession(BaseAvatarSession[Literal["avatar_disconnected"]]): """An Atmee avatar session: render an Atmee avatar into your agent's room. Any Atmee avatar renders by its id: a **v1 avatar** is a talking head generated from a single portrait; a **v2 avatar** moves as a whole. Both are created in the Atmee studio or with the Atmee API. The first v2 session after a quiet period can take a few minutes to start, and :meth:`start` waits for it. The avatar joins the room as its own participant (``atmee-avatar-agent``) publishing on behalf of your agent; your agent's audio output is replaced by a data stream to it, and the avatar publishes synced video and audio. Billing runs while the avatar is in the room and stops when your agent leaves, the room closes, or :meth:`aclose` runs (which also removes the avatar participant); a ceiling of ``max_duration_seconds`` bounds it. Emits ``avatar_disconnected`` (the ``rtc.RemoteParticipant``) if the avatar participant leaves the room while the session is still running, for example because its rendering worker died. """ def __init__( self, *, avatar_id: str, api_url: NotGivenOr[str] = NOT_GIVEN, api_key: NotGivenOr[str] = NOT_GIVEN, avatar_participant_identity: NotGivenOr[str] = NOT_GIVEN, avatar_participant_name: NotGivenOr[str] = NOT_GIVEN, max_duration_seconds: int = DEFAULT_MAX_DURATION_SECONDS, metadata: dict[str, Any] | None = None, wait_for: WaitFor = "initializing", conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, http_session: aiohttp.ClientSession | None = None, avatar_version: NotGivenOr[str] = NOT_GIVEN, ) -> None: """ Args: avatar_id: the Atmee avatar to render, v1 or v2. api_key: your ``sk_atmee_...`` key; defaults to ``ATMEE_API_KEY``. api_url: API base; defaults to ``ATMEE_API_URL`` or the Atmee cloud. avatar_participant_identity: identity the avatar joins with (``atmee-avatar-agent``). Set it when one room hosts several avatars. max_duration_seconds: hard ceiling of the session (default 3600). metadata: free-form JSON attached to the session on the Atmee side. wait_for: ``"initializing"`` (default) returns from :meth:`start` once the worker acknowledged the start; ``"avatar_joined"`` blocks until the avatar is in the room. Anything else raises ``ValueError``. conn_options: connect timeout and retries of the API calls that are safe to repeat (reading the avatar before the start, ending the session); see :class:`AtmeeAPI`. http_session: an aiohttp session to reuse; the job's shared one by default, or a private one outside a job (released by :meth:`aclose`). avatar_version: deprecated and ignored; the avatar's own engine decides how it renders. """ _check_wait_for(wait_for) _warn_avatar_version(avatar_version) if not avatar_id: raise AtmeeException("avatar_id is required") super().__init__() self._avatar_id = avatar_id self._max_duration_seconds = int(max_duration_seconds) self._metadata = metadata self._wait_for: WaitFor = wait_for self._conn_options = conn_options # Resolved lazily by AtmeeAPI: the job's shared session inside a # worker, a private one (closed by aclose) outside. self._api = AtmeeAPI( api_key=api_key, api_url=api_url, conn_options=conn_options, session=http_session, ) self._avatar_participant_identity = avatar_participant_identity or _AVATAR_AGENT_IDENTITY self._avatar_participant_name = avatar_participant_name or _AVATAR_AGENT_NAME self.session_id: str | None = None """The Atmee avatar session id once :meth:`start` returned.""" self.session_info: AvatarSessionInfo | None = None """The full start response once :meth:`start` returned.""" self._end_task: asyncio.Task[None] | None = None self._end_lock = asyncio.Lock() self._close_lock = asyncio.Lock() self._ended = False self._started = False self._render_requested = False # aclose() was called: a start still waiting on its request then asks no # more (_stop_start) and ends what the request opens instead of wiring # it up. The request itself is never cancelled: the session it may open # could not be ended without the id its answer carries. self._closed = False self._stop_start = asyncio.Event() self._start_request: asyncio.Future[AvatarSessionInfo] | None = None self._settle_tasks: set[asyncio.Task[bool]] = set() self._room_for_events: rtc.Room | None = None self._agent_session_for_events: AgentSession[Any] | None = None self._aclose_task: asyncio.Task[None] | None = None @property def avatar_identity(self) -> str: return self._avatar_participant_identity @property def provider(self) -> str: return "atmee" async def start( self, agent_session: AgentSession[Any], room: rtc.Room, *, livekit_url: NotGivenOr[str] = NOT_GIVEN, livekit_api_key: NotGivenOr[str] = NOT_GIVEN, livekit_api_secret: NotGivenOr[str] = NOT_GIVEN, ) -> None: """Start rendering the avatar into ``room`` for ``agent_session``. Mints a LiveKit access token for the avatar participant with YOUR project's credentials (``LIVEKIT_URL`` / ``LIVEKIT_API_KEY`` / ``LIVEKIT_API_SECRET`` or the arguments), asks Atmee to render into the room, and routes the agent's audio to the avatar. Called before ``agent_session.start``, the avatar speaks from the agent's first word; called after it, the agent speaks with its own voice until the avatar takes over. A v1 avatar's start is acknowledged within seconds; the first v2 session after a quiet period can take a few minutes, and this waits for it. One-shot: each call creates a separately billed render, so a second ``start()`` on the same instance raises. After a failed start, create a new ``AvatarSession``. If :meth:`aclose` runs while the start is still waiting (the job shuts down, the agent session closes), the start asks no more, the render it opened is ended and this raises an ``AtmeeException`` with ``code="closed"``. If this call is cancelled, its request still finishes in the background and the render it opens is ended. """ if self._started: raise AtmeeException( "AvatarSession.start() was already called; create a new AvatarSession " "for another render" ) self._started = True await super().start(agent_session, room) try: await self._start_render( agent_session, room, livekit_url=livekit_url, livekit_api_key=livekit_api_key, livekit_api_secret=livekit_api_secret, ) except BaseException: # Undo what super().start() set up (listeners, join task). Once the # render was requested its outcome is uncertain, so close fully: # end a session that may exist and remove the avatar participant. # Before that, no avatar of ours can be in the room, and removing # the participant could disconnect another session's avatar that # uses the same identity. Either way the instance stays spent. try: if self._render_requested: await self.aclose() else: await self._release_without_removing_participant() except Exception: # never let a teardown error replace the real start failure logger.warning("cleanup after a failed atmee avatar start failed", exc_info=True) raise async def _release_without_removing_participant(self) -> None: async with self._close_lock: room = self._room if room is not None: room.off("connection_state_changed", self._on_connection_state_changed) # the base aclose() removes the avatar identity from a # connected room; with no room it only drops its listener # and cancels the join task self._room = None await super().aclose() await self._api.aclose() async def _start_render( self, agent_session: AgentSession[Any], room: rtc.Room, *, livekit_url: NotGivenOr[str], livekit_api_key: NotGivenOr[str], livekit_api_secret: NotGivenOr[str], ) -> None: livekit_url = livekit_url or (os.getenv("LIVEKIT_URL") or NOT_GIVEN) livekit_api_key = livekit_api_key or (os.getenv("LIVEKIT_API_KEY") or NOT_GIVEN) livekit_api_secret = livekit_api_secret or (os.getenv("LIVEKIT_API_SECRET") or NOT_GIVEN) if not livekit_url or not livekit_api_key or not livekit_api_secret: raise AtmeeException( "livekit_url, livekit_api_key, and livekit_api_secret must be set " "by arguments or environment variables" ) job_ctx = get_job_context(required=False) local_participant_identity = ( job_ctx.local_participant_identity if job_ctx is not None else room.local_participant.identity ) livekit_token = ( api.AccessToken(api_key=livekit_api_key, api_secret=livekit_api_secret) .with_kind("agent") .with_identity(self._avatar_participant_identity) .with_name(self._avatar_participant_name) .with_grants(api.VideoGrants(room_join=True, room=room.name)) # allow the avatar to publish audio and video on behalf of your local agent .with_attributes({ATTRIBUTE_PUBLISH_ON_BEHALF: local_participant_identity}) .with_ttl(timedelta(seconds=self._max_duration_seconds) + _TOKEN_TTL_SLACK) .to_jwt() ) logger.debug( "starting atmee avatar session", extra={"avatar_id": self._avatar_id, "lk.pii.room": room.name}, ) if self._closed: # closed before the render was requested: never open a billed render raise AtmeeException("the avatar session was closed before it started", code="closed") self._render_requested = True # The agent session can close without the job shutting down # (AgentSession.aclose(), an error), even while the start is still # waiting: end the render then too, or it would bill until # max_duration_seconds. self._agent_session_for_events = agent_session agent_session.on("close", self._on_agent_session_close) request = asyncio.ensure_future( self._api.create_avatar_session( self._avatar_id, livekit_url=livekit_url, livekit_token=livekit_token, agent_identity=local_participant_identity, max_duration_seconds=self._max_duration_seconds, metadata=self._metadata, wait_for=self._wait_for, stop=self._stop_start, ) ) self._start_request = request try: # waits without cancelling the request when start() itself is cancelled await asyncio.wait({request}) except asyncio.CancelledError: # The caller gave up on start() (the job shuts down, a timeout of # its own): let the request finish and end what it opens. self._closed = True self._stop_start.set() task = asyncio.ensure_future(self._settle_closed_start(request)) self._settle_tasks.add(task) task.add_done_callback(self._settle_tasks.discard) raise if self._closed: # aclose() ran while the request was waiting and left the teardown # to this start: end the session the API opened, if any, remove # the avatar and release the client, wiring nothing up. opened = await self._settle_closed_start(request) raise AtmeeException( "the avatar session was closed while it was starting" + ("; its render was ended" if opened else ""), code="closed", ) info = request.result() self.session_info = info self.session_id = info.session_id logger.info( "atmee avatar session started", extra={ "session_id": info.session_id, "status": info.status, "max_duration_seconds": info.max_duration_seconds, }, ) self._room_for_events = room room.on("participant_disconnected", self._on_participant_disconnected) agent_session.output.replace_audio_tail( _AvatarAudioOutput( room=room, destination_identity=self._avatar_participant_identity, sample_rate=SAMPLE_RATE, wait_remote_track=rtc.TrackKind.KIND_VIDEO, wait_playback_start=True, ), ) def _on_participant_disconnected(self, participant: rtc.RemoteParticipant) -> None: if participant.identity != self._avatar_participant_identity or self._ended: return # The avatar left while we are still running: its worker died or was # removed. The worker reports its own end to Atmee; ending here too # closes the gap for a worker that could not (idempotent). logger.warning( "atmee avatar participant left the room", extra={"session_id": self.session_id, "lk.pii.identity": participant.identity}, ) self.emit("avatar_disconnected", participant) self._schedule_end() def _on_agent_session_close(self, _: Any) -> None: if self._aclose_task is None: self._aclose_task = asyncio.create_task(self.aclose()) def _schedule_end(self) -> None: if self._end_task is None or self._end_task.done(): self._end_task = asyncio.create_task(self._end_session()) async def _settle_closed_start(self, request: asyncio.Future[AvatarSessionInfo]) -> bool: """Close everything once a start closed mid-request has its answer: end the session it opened, if any. Returns whether one was opened.""" await asyncio.wait({request}) info = None if not request.cancelled() and request.exception() is None: info = request.result() # failed or stopped requests opened nothing if info is not None: self.session_id = info.session_id async with self._close_lock: await self._aclose_locked() return info is not None async def _end_session(self) -> None: """Best-effort ``POST /v1/avatar_sessions/{id}/end``. Serialized, and marked done only once the API confirmed the end (the call is idempotent), so a failed or cancelled attempt can be retried by a later :meth:`aclose`. """ async with self._end_lock: if self._ended or not self.session_id: return try: await self._api.end_avatar_session(self.session_id) except AtmeeException as e: if 400 <= e.status_code < 500 and e.status_code not in (408, 429): # The session is unknown or already settled on the Atmee # side: nothing left to end, retrying would not help. self._ended = True logger.warning( "failed to end atmee avatar session; the worker's own report or the " "session ceiling will settle it", extra={"session_id": self.session_id, "error": str(e)}, ) return except Exception as e: # never let a billing hint break teardown logger.warning( "failed to end atmee avatar session; the worker's own report or the " "session ceiling will settle it", extra={"session_id": self.session_id, "error": str(e)}, ) return self._ended = True logger.debug("atmee avatar session ended", extra={"session_id": self.session_id}) async def _finish_end(self) -> None: # an end already under way (the avatar left on its own) finishes first, # so none is still running once the HTTP client is released if self._end_task is not None and not self._end_task.done(): try: await self._end_task except Exception: pass await self._end_session() async def aclose(self) -> None: """End the Atmee session and remove the avatar from the room. Registered as a job shutdown callback by :meth:`start`; call it yourself when running outside a job. Safe to call more than once and concurrently (the agent session's ``close`` event also triggers it): calls are serialized, so the HTTP session is never closed while another call is still ending the render. """ self._closed = True self._stop_start.set() async with self._close_lock: await self._aclose_locked() async def _aclose_locked(self) -> None: if self._room_for_events is not None: self._room_for_events.off("participant_disconnected", self._on_participant_disconnected) self._room_for_events = None if self._agent_session_for_events is not None: self._agent_session_for_events.off("close", self._on_agent_session_close) self._agent_session_for_events = None if self._start_request is not None and not self._start_request.done(): # A start is still waiting on its request, with no session to end # yet: once the request returns, the session it opened is ended and # everything here is closed (see _settle_closed_start). return # Remove the avatar participant and end the render side by side, so the # avatar leaves at once even when the end call is slow: at job shutdown # the process has only a few seconds. results = await asyncio.gather(super().aclose(), self._finish_end(), return_exceptions=True) for result in results: if isinstance(result, Exception): logger.warning("atmee avatar session cleanup step failed", exc_info=result) await self._api.aclose()An Atmee avatar session: render an Atmee avatar into your agent's room.
Any Atmee avatar renders by its id: a v1 avatar is a talking head generated from a single portrait; a v2 avatar moves as a whole. Both are created in the Atmee studio or with the Atmee API. The first v2 session after a quiet period can take a few minutes to start, and :meth:
startwaits for it.The avatar joins the room as its own participant (
atmee-avatar-agent) publishing on behalf of your agent; your agent's audio output is replaced by a data stream to it, and the avatar publishes synced video and audio. Billing runs while the avatar is in the room and stops when your agent leaves, the room closes, or :meth:acloseruns (which also removes the avatar participant); a ceiling ofmax_duration_secondsbounds it.Emits
avatar_disconnected(thertc.RemoteParticipant) if the avatar participant leaves the room while the session is still running, for example because its rendering worker died.Args
avatar_id- the Atmee avatar to render, v1 or v2.
api_key- your
sk_atmee_…key; defaults toATMEE_API_KEY. api_url- API base; defaults to
ATMEE_API_URLor the Atmee cloud. avatar_participant_identity- identity the avatar joins with
(
atmee-avatar-agent). Set it when one room hosts several avatars. max_duration_seconds- hard ceiling of the session (default 3600).
metadata- free-form JSON attached to the session on the Atmee side.
wait_for"initializing"(default) returns from :meth:startonce the worker acknowledged the start;"avatar_joined"blocks until the avatar is in the room. Anything else raisesValueError.conn_options- connect timeout and retries of the API calls that
are safe to repeat (reading the avatar before the start, ending
the session); see :class:
AtmeeAPI. http_session- an aiohttp session to reuse; the job's shared one by
default, or a private one outside a job (released by
:meth:
aclose). avatar_version- deprecated and ignored; the avatar's own engine decides how it renders.
Ancestors
- livekit.agents.voice.avatar._types.AvatarSession
- abc.ABC
- EventEmitter
- typing.Generic
Instance variables
prop avatar_identity : str-
Expand source code
@property def avatar_identity(self) -> str: return self._avatar_participant_identityThe participant identifier of the avatar
prop provider : str-
Expand source code
@property def provider(self) -> str: return "atmee"The provider of the avatar
var session_id-
The Atmee avatar session id once :meth:
startreturned. var session_info-
The full start response once :meth:
startreturned.
Methods
async def aclose(self) ‑> None-
Expand source code
async def aclose(self) -> None: """End the Atmee session and remove the avatar from the room. Registered as a job shutdown callback by :meth:`start`; call it yourself when running outside a job. Safe to call more than once and concurrently (the agent session's ``close`` event also triggers it): calls are serialized, so the HTTP session is never closed while another call is still ending the render. """ self._closed = True self._stop_start.set() async with self._close_lock: await self._aclose_locked()End the Atmee session and remove the avatar from the room.
Registered as a job shutdown callback by :meth:
start; call it yourself when running outside a job. Safe to call more than once and concurrently (the agent session'scloseevent also triggers it): calls are serialized, so the HTTP session is never closed while another call is still ending the render. async def start(self,
agent_session: AgentSession[Any],
room: rtc.Room,
*,
livekit_url: NotGivenOr[str] = NOT_GIVEN,
livekit_api_key: NotGivenOr[str] = NOT_GIVEN,
livekit_api_secret: NotGivenOr[str] = NOT_GIVEN) ‑> None-
Expand source code
async def start( self, agent_session: AgentSession[Any], room: rtc.Room, *, livekit_url: NotGivenOr[str] = NOT_GIVEN, livekit_api_key: NotGivenOr[str] = NOT_GIVEN, livekit_api_secret: NotGivenOr[str] = NOT_GIVEN, ) -> None: """Start rendering the avatar into ``room`` for ``agent_session``. Mints a LiveKit access token for the avatar participant with YOUR project's credentials (``LIVEKIT_URL`` / ``LIVEKIT_API_KEY`` / ``LIVEKIT_API_SECRET`` or the arguments), asks Atmee to render into the room, and routes the agent's audio to the avatar. Called before ``agent_session.start``, the avatar speaks from the agent's first word; called after it, the agent speaks with its own voice until the avatar takes over. A v1 avatar's start is acknowledged within seconds; the first v2 session after a quiet period can take a few minutes, and this waits for it. One-shot: each call creates a separately billed render, so a second ``start()`` on the same instance raises. After a failed start, create a new ``AvatarSession``. If :meth:`aclose` runs while the start is still waiting (the job shuts down, the agent session closes), the start asks no more, the render it opened is ended and this raises an ``AtmeeException`` with ``code="closed"``. If this call is cancelled, its request still finishes in the background and the render it opens is ended. """ if self._started: raise AtmeeException( "AvatarSession.start() was already called; create a new AvatarSession " "for another render" ) self._started = True await super().start(agent_session, room) try: await self._start_render( agent_session, room, livekit_url=livekit_url, livekit_api_key=livekit_api_key, livekit_api_secret=livekit_api_secret, ) except BaseException: # Undo what super().start() set up (listeners, join task). Once the # render was requested its outcome is uncertain, so close fully: # end a session that may exist and remove the avatar participant. # Before that, no avatar of ours can be in the room, and removing # the participant could disconnect another session's avatar that # uses the same identity. Either way the instance stays spent. try: if self._render_requested: await self.aclose() else: await self._release_without_removing_participant() except Exception: # never let a teardown error replace the real start failure logger.warning("cleanup after a failed atmee avatar start failed", exc_info=True) raiseStart rendering the avatar into
roomforagent_session.Mints a LiveKit access token for the avatar participant with YOUR project's credentials (
LIVEKIT_URL/LIVEKIT_API_KEY/LIVEKIT_API_SECRETor the arguments), asks Atmee to render into the room, and routes the agent's audio to the avatar. Called beforeagent_session.start, the avatar speaks from the agent's first word; called after it, the agent speaks with its own voice until the avatar takes over. A v1 avatar's start is acknowledged within seconds; the first v2 session after a quiet period can take a few minutes, and this waits for it.One-shot: each call creates a separately billed render, so a second
start()on the same instance raises. After a failed start, create a newAvatarSession. If :meth:acloseruns while the start is still waiting (the job shuts down, the agent session closes), the start asks no more, the render it opened is ended and this raises anAtmeeExceptionwithcode="closed". If this call is cancelled, its request still finishes in the background and the render it opens is ended.
Inherited members
class AvatarSessionInfo (session_id: str,
status: str,
avatar_participant_identity: str,
agent_identity: str,
room_name: str,
max_duration_seconds: int,
raw: dict[str, Any] = <factory>)-
Expand source code
@dataclass class AvatarSessionInfo: """The ``202`` body of ``POST /v1/avatars/{avatarId}/avatar_sessions``.""" session_id: str status: str avatar_participant_identity: str agent_identity: str room_name: str max_duration_seconds: int raw: dict[str, Any] = field(default_factory=dict, repr=False)The
202body ofPOST /v1/avatars/{avatarId}/avatar_sessions.Instance variables
var agent_identity : strvar avatar_participant_identity : strvar max_duration_seconds : intvar raw : dict[str, typing.Any]var room_name : strvar session_id : strvar status : str