Concepts

Overview

LiveKit Portal is a toolkit for building robotics apps that run in LiveKit rooms. It provides classes and APIs for building robots and operators, along with tools for managing rooms and the data exchanged between them. It is built around four core concepts:

1. Robots and operators have distinct roles.
2. Robots publish data while operators subscribe to it.
3. Camera frames and robot state are synchronized into observations.
4. One operator controls the robot at a time.

Roles

LiveKit Portal defines two roles, selected by the class you construct.

Each session contains one robot and any number of operators. A human teleoperator, an inference policy, a recorder, and a supervisory controller can all participate as operators in the same LiveKit room.

Both roles declare the same observation and action schema using add_video, add_state_typed, and add_action_typed. Camera names, field names, field order, and data types must match. If the schemas differ, Portal rejects the data, making schema mismatches the first thing to check when observations or actions don't arrive.

A participant's role is fixed when it is constructed. The Robot and Operator facades expose only their role's methods. The send_action method doesn't exist on a Robot, so calling it raises an AttributeError:

robot = Robot(cfg)
robot.send_action({"j1": 1.0})
# AttributeError: 'Robot' object has no attribute 'send_action'

The unified Portal, on the other hand, exposes every method regardless of role. If you invoke the send_action() method on a Portal constructed with Role.ROBOT, it raises PortalError.WrongRole at runtime:

portal = Portal(PortalConfig("session-1", Role.ROBOT))
portal.send_action({"j1": 1.0})
# PortalError.WrongRole: operation not available for role Robot

Schema and data validation

LiveKit Portal validates data against the declared schema and handles mismatches predictably:

• Schema mismatches are dropped, not raised. Each packet carries a fingerprint of the ordered field names and data types. A peer whose schema does not match has its packets dropped, with a single warning per mismatched fingerprint. The other side continues running.
• Unknown fields are ignored. Values sent for fields that are not in the declared schema are dropped, with a single warning per field name.
• Out-of-range values are limited to the type's range. A value outside the range of its data type is replaced with the nearest value the type can represent, with a single warning per field. For example, 9999 sent into an I8 field becomes 127. A NaN value becomes 0 in an integer field and false in a BOOL field.
• Type mismatches raise immediately. Sending a value whose Python type does not match its declared data type raises PortalError.DtypeMismatch before any packet is sent. An int is accepted for a float field, but a bool is rejected everywhere except a BOOL field.

Because most of these behaviors are silent, check the schema and field names first when observations or actions don't arrive.

The observation model

Most robotics apps process a single observation for each control step: camera frames and robot state captured at the same point in time.

Video and state are transported independently and typically arrive with different latencies. Video passes through encoding, transport, and decoding, while state travels over a data channel. As a result, the two streams arrive out of sync even when they were captured simultaneously.

LiveKit Portal synchronizes these streams using timestamps attached by the sender. On the operator, frames and state with matching timestamps are combined into a single observation:

Observation(
    frames={"cam1": VideoFrameData, "wrist": VideoFrameData},
    state={"j1": 0.1, "j2": -0.3, "gripper": True},
    timestamp_us=1717171717000000,
)

An observation is delivered only after every declared camera has produced a matching frame. State that cannot be matched is reported through on_drop.

The obs.state dictionary contains native Python values based on the declared schema. For example, BOOL fields become bool, and integer fields become int. The obs.raw_state dictionary exposes the same values as floating-point numbers for apps that write directly into NumPy arrays.

Each VideoFrameData has four attributes: data, the packed RGB24 image bytes; width and height in pixels; and timestamp_us, the sender's capture timestamp. Convert the frame with the provided helper:

from livekit.portal import frame_bytes_to_numpy_rgb

rgb = frame_bytes_to_numpy_rgb(bytes(frame.data), frame.width, frame.height)
# rgb is uint8, shape (H, W, 3), RGB order.

How matching works

For a state with timestamp S, a frame with timestamp F is considered a candidate when |S - F| falls within the synchronization window. Portal selects the closest matching frame from each camera, then takes one of three actions:

• Match: Every camera has a matching frame, so an observation is delivered.
• Wait: One or more cameras don't yet have a matching frame, but future frames could still satisfy the window. It continues buffering the state.
• Drop: At least one camera has advanced beyond the synchronization window, making a match impossible. It discards the state and invokes on_drop.

The synchronization window is determined by the fps and tolerance settings and defaults to 50 ms. See Tuning  for configuration guidance and Synchronization  for the complete matching algorithm.

Handling drops

The on_drop callback receives a list of state dictionaries. Each dictionary has the same structure as obs.state but does not include a timestamp.

def on_drop(dropped):
    # dropped is List[Dict[str, bool | int | float]]
    print(f"lost {len(dropped)} states")

op.on_drop(on_drop)

A small number of dropped states during startup is expected while video streams begin producing frames. If drops continue during normal operation, increase the synchronization window or investigate delayed camera streams.

Set the active operator

The robot accepts actions from only one operator at a time. The active operator is a piece of robot state that identifies which operator can control the robot.

The following diagram shows how actions are delivered based on the active operator. In this example, policy-v1 is the active operator.

LiveKit Portal stores the active operator as an attribute on the robot's participant, and LiveKit mirrors it to every participant in the room. Any participant can read or change this value, but the robot's copy is the source of truth: it is what the control gate uses to decide whose actions to accept. Transferring control requires a single API call:

# A human preempts the policy.
await human.set_active_operator(human.local_identity())

# ... teleoperate for a while ...

# Hand control back.
await human.set_active_operator("policy-v1")

Changing the active operator does not interrupt the action stream. The robot continues receiving actions without reconnecting or renegotiating the session.

Behavior and state

Keep the following behaviors in mind:

• The active operator is initially unset. A newly connected robot drops all incoming actions until an operator claims control with set_active_operator().
• Inactive operators receive no indication that their actions were ignored. A call to send_action() succeeds even when the robot drops the action because another operator has control. Use active_operator() to determine whether your actions are currently accepted.
• The active operator persists across disconnects. If the active operator disconnects, the robot continues to reference that identity. Reconnecting with the same identity resumes control. To transfer control elsewhere, another participant must call set_active_operator().

For the complete control-plane API and callback reference, see the LiveKit Portal API reference.

Multi-operator patterns

Because operators are ordinary room participants, several patterns are available without additional APIs:

Recorders, shadow policies, and monitoring interfaces need to observe the actions the robot actually executed. By default, an operator only sends actions and does not receive them. Enable action subscription to receive the executed actions:

cfg = OperatorConfig("session-1")
cfg.add_action_typed([("joint1", DType.F32)])   # required to deserialize actions
cfg.set_action_subscription(True)

op = Operator(cfg)
op.on_action(lambda action: log.append(action))

With action subscription enabled, the operator applies the same control gate as the robot: on_action and on_action_chunk fire only for the active operator's actions. An operator also receives its own actions, because LiveKit does not deliver a publisher's own data packets back to it and Portal delivers them locally instead. Label recorded actions with action.sender, which is set when the action passes the gate, rather than with active_operator(), which can change between sending an action and receiving it.

Putting it together

The following sequence shows one control cycle from sensor capture through action execution:

Video frame format

The send_video_frame() method accepts packed RGB24 image data. Pixels are stored in R, G, B byte order with no alpha channel. The image is tightly packed in row-major order, so stride = width * 3 and the buffer size is exactly width * height * 3 bytes.

This corresponds to a NumPy uint8 array with shape (H, W, 3) in RGB order, which is the output of PIL.Image.convert("RGB") and OpenCV's cvtColor(frame, COLOR_BGR2RGB).

On the default WebRTC transport, LiveKit Portal converts RGB frames to I420 using libyuv before passing them to WebRTC. Typical conversion costs are:

If your camera already produces I420 or NV12 frames, this conversion is unnecessary. For RGB and BGR sources, which include most cameras and Python image pipelines, the built-in conversion is typically the most efficient option.

Applications that require lossless image data should use a byte-stream codec such as MJPEG, PNG, or RAW instead of the default WebRTC video path. See LiveKit Portal API reference.

Frames must carry a timestamp

Every frame processed by LiveKit Portal must include user_timestamp in its LiveKit packet trailer metadata. It sets this automatically for the tracks it publishes.

Portal cannot synchronize frames from publishers that don't provide timestamps. Republish those streams through it or configure the upstream publisher to include user timestamp trailers. For more information, see Timestamps and frame metadata and the wire protocol .

Callbacks and threading

Callbacks registered with on_observation(), on_action(), and related APIs execute on the asyncio event loop that was active when they were registered. They don't execute on LiveKit Portal's internal Tokio worker threads.

Long-running callbacks block your app's event loop and can cause frame drops if processing falls behind. Keep callbacks lightweight and move computationally intensive work to separate tasks or threads.

If a callback raises an exception, Portal logs the traceback and continues running. Exceptions don't terminate the session, so monitor app logs to detect callback failures.

Next steps

The following resources build on these concepts: