Operator app

Overview

This guide walks you through building an operator app for teleoperation. It is scoped to simple human-interface apps built using JavaScript  or Flutter, but teleoperation is not limited to these languages.

Building a teleoperation app typically involves the following steps:

1. Define the control schema
2. Render the robot's video tracks
3. Capture operator input
4. Publish control commands

Define the control schema

A control schema defines the structure and meaning of the control commands the operator sends to the robot. It's the data contract for the control channel.

Define a small control schema before implementing either participant. The operator serializes commands to this schema before publishing them on the control track. The robot deserializes each command against the same schema before applying it to the local control system.

A useful control frame typically contains the following data:

• Sequence number: Detects gaps or replayed frames.
• Timestamp or expiration: Prevents stale input from moving the robot.
• Control values: Carries normalized axes, velocities, or other app-specific commands.
• Control lease ID: Identifies the active control lease held by the operator.

The following example control schema follows this guidance:

{
  "sequence_number": 1,
  "timestamp": 1717000000000,
  "control_values": {
    "steering": 0.5,
    "throttle": 0.5
  },
  "control_lease_id": "lease_7f3a"
}

Authentication and control leasing are out of scope for this guide. Use RPC for discrete actions that need a response, such as acquiring, renewing, or releasing exclusive control, changing operating mode, or starting calibration. Include the resulting control lease ID in each command frame when the robot enforces one active operator.

Render the robot's video tracks

Use a named video track for each camera view the robot publishes. The operator subscribes to the views it needs and renders the frames in the control interface.

For example, a mobile robot might publish front_camera and rear_camera tracks.

With automatic subscription enabled, handle the track subscription event and attach each video track to a platform renderer:

import { RemoteVideoTrack, RoomEvent } from 'livekit-client';

const videoContainer = document.getElementById('robot-video');

room.on(RoomEvent.TrackSubscribed, (track, publication) => {
  if (!(track instanceof RemoteVideoTrack) || !videoContainer) {
    return;
  }

  const videoElement = track.attach();
  videoElement.dataset.trackName = publication.trackName;
  videoContainer.appendChild(videoElement);
});

room.on(RoomEvent.TrackUnsubscribed, (track) => {
  track.detach().forEach((element) => element.remove());
});

import 'package:flutter/widgets.dart';
import 'package:livekit_client/livekit_client.dart';

class RobotVideoView extends StatefulWidget {
  const RobotVideoView({required this.robot, super.key});

  final Participant robot;

  @override
  State<RobotVideoView> createState() => _RobotVideoViewState();
}

class _RobotVideoViewState extends State<RobotVideoView> {
  TrackPublication? videoPublication;

  @override
  void initState() {
    super.initState();
    widget.robot.addListener(_onParticipantChanged);
    _onParticipantChanged();
  }

  @override
  void dispose() {
    widget.robot.removeListener(_onParticipantChanged);
    super.dispose();
  }

  void _onParticipantChanged() {
    final subscribedVideos = widget.robot.videoTrackPublications.where((publication) {
      return publication.kind == TrackType.VIDEO &&
          !publication.isScreenShare &&
          publication.subscribed;
    });

    setState(() {
      videoPublication = subscribedVideos.isEmpty || subscribedVideos.first.muted
          ? null
          : subscribedVideos.first;
    });
  }

  @override
  Widget build(BuildContext context) {
    final track = videoPublication?.track;

    return track is VideoTrack
        ? VideoTrackRenderer(track)
        : const SizedBox.shrink();
  }
}

Create a renderer for each subscribed track when displaying multiple camera views at the same time. The examples follow these official SDK implementations:

• JavaScript: Client SDK demo .
• Flutter: Video rendering example .

Capture operator input

Capture operator input from a gamepad, keyboard, control interface, or other input device. The input mapping is app-specific, but it should produce a consistent control state that can be published at a regular interval.

Common control surfaces include:

• Web: Arrow or WASD keys, the browser Gamepad API, or on-screen controls.
• Mobile: Touch joysticks, buttons, sliders, device motion, or a Bluetooth controller.
• VR and XR: Headset controllers, hand tracking, or spatial input through the Unity SDK.
• Desktop and industrial: USB gamepads, joysticks, or purpose-built control panels.

Normalize input into a consistent control state and keep input capture independent from the publishing loop. Reset the state when the control surface disconnects or loses focus.

Publish control commands

Publish the latest control state to the robot at a regular interval. Use a named data track when the client SDK supports it. In Flutter, use lossy data packets with a topic such as robot.control. Both approaches fit continuous, latency-sensitive commands such as steering, throttle, joint velocity, or pan and tilt.

The following examples encode the control frame as UTF-8 JSON and use lossy delivery so an old command isn't retransmitted ahead of newer input. This behavior fits continuously updated input, where the latest state matters more than receiving every intermediate value.

const encoder = new TextEncoder();
const controlTrack = await room.localParticipant.publishDataTrack({
  name: 'robot.control',
});

let sequenceNumber = 0;

function publishControlFrame(controlValues, controlLeaseId) {
  const frame = {
    sequence_number: ++sequenceNumber,
    timestamp: Date.now(),
    control_values: controlValues,
    control_lease_id: controlLeaseId,
  };

  controlTrack.tryPush({
    payload: encoder.encode(JSON.stringify(frame)),
  });
}

publishControlFrame(
  { steering: 0.5, throttle: 0.5 },
  'lease_7f3a',
);

import 'dart:convert';

import 'package:livekit_client/livekit_client.dart';

Future<void> publishControlFrame({
  required Room room,
  required int sequenceNumber,
  required Map<String, num> controlValues,
  required String controlLeaseId,
  required String robotIdentity,
}) async {
  final frame = {
    'sequence_number': sequenceNumber,
    'timestamp': DateTime.now().millisecondsSinceEpoch,
    'control_values': controlValues,
    'control_lease_id': controlLeaseId,
  };

  await room.localParticipant.publishData(
    utf8.encode(jsonEncode(frame)),
    reliable: false,
    destinationIdentities: [robotIdentity],
    topic: 'robot.control',
  );
}

await publishControlFrame(
  room: room,
  sequenceNumber: 1,
  controlValues: {'steering': 0.5, 'throttle': 0.5},
  controlLeaseId: 'lease_7f3a',
  robotIdentity: 'robot',
);

Update the sequence number, timestamp, and control values before each publish. If a lossy frame is dropped, continue with the next frame instead of retrying stale input. On the robot, validate each command and implement safe control behavior before applying it to an actuator.

Additional resources

Use these implementations as references for operator teleoperation patterns: