LiveKit docs › Integrations › LiveKit Portal › Concepts --- # LiveKit Portal concepts > How LiveKit Portal roles, observations, and control handoff work. ## Overview LiveKit Portal is a toolkit for building robotics apps that run in LiveKit rooms. It provides classes and APIs for building robots and operators, along with tools for managing rooms and the data exchanged between them. It is built around four core concepts: 1. Robots and operators have distinct roles. 2. Robots publish data while operators subscribe to it. 3. Camera frames and robot state are synchronized into observations. 4. One operator controls the robot at a time. ## Roles LiveKit Portal defines two roles, selected by the class you construct. | Class | Publishes | Subscribes to | | `Robot` | Video frames, state | Actions | | `Operator` | Actions | Video frames and state, synchronized into observations | Each session contains one robot and any number of operators. A human teleoperator, an inference policy, a recorder, and a supervisory controller can all participate as operators in the same LiveKit room. Both roles declare the same observation and action schema using `add_video`, `add_state_typed`, and `add_action_typed`. Camera names, field names, field order, and data types must match. If the schemas differ, Portal rejects the data, making schema mismatches the first thing to check when observations or actions don't arrive. A participant's role is fixed when it is constructed. The `Robot` and `Operator` facades expose only their role's methods. The `send_action` method doesn't exist on a `Robot`, so calling it raises an `AttributeError`: ```python robot = Robot(cfg) robot.send_action({"j1": 1.0}) # AttributeError: 'Robot' object has no attribute 'send_action' ``` The unified `Portal`, on the other hand, exposes every method regardless of role. If you invoke the `send_action()` method on a `Portal` constructed with `Role.ROBOT`, it raises `PortalError.WrongRole` at runtime: ```python portal = Portal(PortalConfig("session-1", Role.ROBOT)) portal.send_action({"j1": 1.0}) # PortalError.WrongRole: operation not available for role Robot ``` ## Schema and data validation LiveKit Portal validates data against the declared schema and handles mismatches predictably: - **Schema mismatches are dropped, not raised.** Each packet carries a fingerprint of the ordered field names and data types. A peer whose schema does not match has its packets dropped, with a single warning per mismatched fingerprint. The other side continues running. - **Unknown fields are ignored.** Values sent for fields that are not in the declared schema are dropped, with a single warning per field name. - **Out-of-range values are limited to the type's range.** A value outside the range of its data type is replaced with the nearest value the type can represent, with a single warning per field. For example, `9999` sent into an `I8` field becomes `127`. A `NaN` value becomes `0` in an integer field and `false` in a `BOOL` field. - **Type mismatches raise immediately.** Sending a value whose Python type does not match its declared data type raises `PortalError.DtypeMismatch` before any packet is sent. An `int` is accepted for a float field, but a `bool` is rejected everywhere except a `BOOL` field. Because most of these behaviors are silent, check the schema and field names first when observations or actions don't arrive. ## The observation model Most robotics apps process a single observation for each control step: camera frames and robot state captured at the same point in time. Video and state are transported independently and typically arrive with different latencies. Video passes through encoding, transport, and decoding, while state travels over a data channel. As a result, the two streams arrive out of sync even when they were captured simultaneously. LiveKit Portal synchronizes these streams using timestamps attached by the sender. On the operator, frames and state with matching timestamps are combined into a single observation: ```python Observation( frames={"cam1": VideoFrameData, "wrist": VideoFrameData}, state={"j1": 0.1, "j2": -0.3, "gripper": True}, timestamp_us=1717171717000000, ) ``` An observation is delivered only after every declared camera has produced a matching frame. State that cannot be matched is reported through `on_drop`. The `obs.state` dictionary contains native Python values based on the declared schema. For example, `BOOL` fields become `bool`, and integer fields become `int`. The `obs.raw_state` dictionary exposes the same values as floating-point numbers for apps that write directly into NumPy arrays. Each `VideoFrameData` has four attributes: `data`, the packed RGB24 image bytes; `width` and `height` in pixels; and `timestamp_us`, the sender's capture timestamp. Convert the frame with the provided helper: ```python from livekit.portal import frame_bytes_to_numpy_rgb rgb = frame_bytes_to_numpy_rgb(bytes(frame.data), frame.width, frame.height) # rgb is uint8, shape (H, W, 3), RGB order. ``` > ℹ️ **Copy before mutating frame bytes** > > The helper returns a zero-copy view over the frame bytes. Call `.copy()` before modifying the array. ### How matching works For a state with timestamp `S`, a frame with timestamp `F` is considered a candidate when `|S - F|` falls within the synchronization window. Portal selects the closest matching frame from each camera, then takes one of three actions: - **Match**: Every camera has a matching frame, so an observation is delivered. - **Wait**: One or more cameras don't yet have a matching frame, but future frames could still satisfy the window. It continues buffering the state. - **Drop**: At least one camera has advanced beyond the synchronization window, making a match impossible. It discards the state and invokes `on_drop`. The synchronization window is determined by the `fps` and `tolerance` settings and defaults to 50 ms. See [Tuning](https://github.com/livekit/portal/blob/main/docs/04-tuning.md) for configuration guidance and [Synchronization](https://github.com/livekit/portal/blob/main/docs/reference/synchronization.md) for the complete matching algorithm. ### Handling drops The `on_drop` callback receives a list of state dictionaries. Each dictionary has the same structure as `obs.state` but does not include a timestamp. ```python def on_drop(dropped): # dropped is List[Dict[str, bool | int | float]] print(f"lost {len(dropped)} states") op.on_drop(on_drop) ``` A small number of dropped states during startup is expected while video streams begin producing frames. If drops continue during normal operation, increase the synchronization window or investigate delayed camera streams. ## Set the active operator The robot accepts actions from only one operator at a time. The **active operator** is a piece of robot state that identifies which operator can control the robot. The following diagram shows how actions are delivered based on the active operator. In this example, `policy-v1` is the active operator. ```mermaid flowchart TD P["operator
policy-v1"] H["operator
human-id"]G{"gate:
sender active?"}P -- actions --> G H -- actions --> GG -- "yes" --> OK["on_action fires"] G -- "no" --> DROP["dropped silently"]OK --> M["motors"]subgraph R["robot   active_operator = policy-v1"] G OK DROP M endstyle DROP stroke-dasharray: 4 4 ``` LiveKit Portal stores the active operator as an attribute on the robot's participant, and LiveKit mirrors it to every participant in the room. Any participant can read or change this value, but the robot's copy is the source of truth: it is what the control gate uses to decide whose actions to accept. Transferring control requires a single API call: ```python # A human preempts the policy. await human.set_active_operator(human.local_identity()) # ... teleoperate for a while ... # Hand control back. await human.set_active_operator("policy-v1") ``` Changing the active operator does not interrupt the action stream. The robot continues receiving actions without reconnecting or renegotiating the session. ### Behavior and state Keep the following behaviors in mind: - **The active operator is initially unset.** A newly connected robot drops all incoming actions until an operator claims control with `set_active_operator()`. - **Inactive operators receive no indication that their actions were ignored.** A call to `send_action()` succeeds even when the robot drops the action because another operator has control. Use `active_operator()` to determine whether your actions are currently accepted. - **The active operator persists across disconnects.** If the active operator disconnects, the robot continues to reference that identity. Reconnecting with the same identity resumes control. To transfer control elsewhere, another participant must call `set_active_operator()`. For the complete control-plane API and callback reference, see the [LiveKit Portal API reference](https://docs.livekit.io/reference/robotics/portal-api.md#active-operator). ## Multi-operator patterns Because operators are ordinary room participants, several patterns are available without additional APIs: | Pattern | Participants | Mechanism | | **Single operator** | Robot and one operator | The operator claims control at startup. | | **Human in the loop** | Robot, policy, and human | Either operator calls `set_active_operator()`. Executed actions remain continuous across the handoff. | | **Data recording** | Robot, policy, human, and recorder | The recorder enables action subscription and logs every executed action, labeled by `action.sender`. | | **Shadow evaluation** | Robot, active policy, and candidate policy | The candidate streams actions that the gate drops. Both streams are recorded for offline comparison. | | **Supervisor** | Robot, multiple operators, and a supervisor | The supervisor never claims control. It calls `set_active_operator()` only to route control between operators. | Recorders, shadow policies, and monitoring interfaces need to observe the actions the robot actually executed. By default, an operator only sends actions and does not receive them. Enable action subscription to receive the executed actions: ```python cfg = OperatorConfig("session-1") cfg.add_action_typed([("joint1", DType.F32)]) # required to deserialize actions cfg.set_action_subscription(True) op = Operator(cfg) op.on_action(lambda action: log.append(action)) ``` With action subscription enabled, the operator applies the same control gate as the robot: `on_action` and `on_action_chunk` fire only for the active operator's actions. An operator also receives its own actions, because LiveKit does not deliver a publisher's own data packets back to it and Portal delivers them locally instead. Label recorded actions with `action.sender`, which is set when the action passes the gate, rather than with `active_operator()`, which can change between sending an action and receiving it. ## Putting it together ```mermaid flowchart LR subgraph Robot["Robot host"] H[Hardware
cameras + motors] RP[Robot
publish frames/state
subscribe actions] H --> RP endsubgraph Cloud["LiveKit room"] V[(Video tracks)] S[(State stream)] A[(Action stream)] endsubgraph Operator["Operator host"] OP[Operator
subscribe + match
publish actions] M[Policy /
teleop / recorder] OP --> M M --> OP endRP -- stamped frames --> V RP -- stamped state --> S A --> RPV --> OP S --> OP OP -- actions --> A ``` The following sequence shows one control cycle from sensor capture through action execution: ```mermaid sequenceDiagram participant R as Robot participant L as LiveKit room participant O as Operator participant M as Policyloop every tick R->>R: read hardware R->>L: send_video_frame(cam1, frame) ts=T R->>L: send_state(joints) ts=T endL-->>O: video frames (variable latency) L-->>O: state packetNote over O: match frames to state
within the search windowO-->>M: on_observation({frames, state, ts}) M-->>O: action O->>L: send_action(action) L-->>R: on_action(action) if sender is active R->>R: drive the motors ``` ## Video frame format The `send_video_frame()` method accepts packed RGB24 image data. Pixels are stored in `R`, `G`, `B` byte order with no alpha channel. The image is tightly packed in row-major order, so `stride = width * 3` and the buffer size is exactly `width * height * 3` bytes. This corresponds to a NumPy `uint8` array with shape `(H, W, 3)` in RGB order, which is the output of `PIL.Image.convert("RGB")` and OpenCV's `cvtColor(frame, COLOR_BGR2RGB)`. > 🔥 **Frame dimensions must be even** > > Frame width and height must both be even. I420 chroma subsampling requires even dimensions, and odd values raise `PortalError.InvalidFrameDimensions`. On the default WebRTC transport, LiveKit Portal converts RGB frames to I420 using libyuv before passing them to WebRTC. Typical conversion costs are: | Resolution | Per frame | At 30 fps | | 640x480 | 0.3 to 0.9 ms | 1 to 3% of a core | | 1280x720 | 1 to 3 ms | 3 to 10% | | 1920x1080 | 2 to 6 ms | 6 to 20% | If your camera already produces I420 or NV12 frames, this conversion is unnecessary. For RGB and BGR sources, which include most cameras and Python image pipelines, the built-in conversion is typically the most efficient option. Applications that require lossless image data should use a byte-stream codec such as MJPEG, PNG, or RAW instead of the default WebRTC video path. See [LiveKit Portal API reference](https://docs.livekit.io/reference/robotics/portal-api.md#codecs). ### Frames must carry a timestamp Every frame processed by LiveKit Portal must include `user_timestamp` in its LiveKit packet trailer metadata. It sets this automatically for the tracks it publishes. Portal cannot synchronize frames from publishers that don't provide timestamps. Republish those streams through it or configure the upstream publisher to include user timestamp trailers. For more information, see [Timestamps and frame metadata](https://docs.livekit.io/robotics/media/video/metadata.md) and the [wire protocol](https://github.com/livekit/portal/blob/main/docs/reference/wire-protocol.md). ## Callbacks and threading Callbacks registered with `on_observation()`, `on_action()`, and related APIs execute on the asyncio event loop that was active when they were registered. They don't execute on LiveKit Portal's internal Tokio worker threads. Long-running callbacks block your app's event loop and can cause frame drops if processing falls behind. Keep callbacks lightweight and move computationally intensive work to separate tasks or threads. If a callback raises an exception, Portal logs the traceback and continues running. Exceptions don't terminate the session, so monitor app logs to detect callback failures. ## Next steps The following resources build on these concepts: - **[LiveKit Portal API reference](https://docs.livekit.io/reference/robotics/portal-api.md)**: Configuration, classes, callbacks, and control-plane APIs. - **[Examples](https://docs.livekit.io/robotics/integrations/portal/examples.md)**: Runnable examples using synthetic and physical robots. - **[Full LiveKit Portal documentation](https://github.com/livekit/portal/tree/main/docs)**: Tuning, metrics, troubleshooting, and the wire protocol. --- This document was rendered at 2026-10-07T16:53:14.813Z. For the latest version of this document, see [https://docs.livekit.io/robotics/integrations/portal/concepts.md](https://docs.livekit.io/robotics/integrations/portal/concepts.md). To explore all LiveKit documentation, see [llms.txt](https://docs.livekit.io/llms.txt).