Video tracks | LiveKit Documentation

Overview

A video track is a stream of video that one participant publishes and others subscribe to. Your app captures raw frames from a camera or another local source and pushes them into a video source. LiveKit encodes each frame and sends it to every subscriber.

SDK comparison

LiveKit SDKs publish cameras through a high-level or a low-level API.

API level	SDKs	Usage
High-level	Browser JavaScript, Swift, Android, Flutter, React Native	One call opens the camera, requests device permissions, and publishes the track. To learn more, see Camera & microphone.
Low-level	Rust, Python, C++, Node.js, Unity	Create a video source, publish a track that wraps it, then push each frame to the source.

Some SDKs provide both. Swift, for example, also accepts app-produced frames through BufferCapturer.

This topic covers the low-level API.

How publishing works

A camera publication uses three objects:

Video source: Accepts raw frames from your app.
Local video track: Wraps the video source so LiveKit can publish it.
Publication: The result of the publish call.

Create the video source and publish the track before your capture loop. Then push frames to the video source for as long as the camera runs. Every subscribed participant receives the encoded track.

Publish a video track

The following examples publish a front_camera video track, then push frames into the video source. Your app supplies the pixel data for each frame.

use livekit::options::TrackPublishOptions;
use livekit::prelude::*;
use livekit::webrtc::video_frame::{
    I420Buffer, VideoFrame, VideoRotation,
};
use livekit::webrtc::video_source::native::NativeVideoSource;
use livekit::webrtc::video_source::{RtcVideoSource, VideoResolution};

const WIDTH: u32 = 1280;
const HEIGHT: u32 = 720;

let source = NativeVideoSource::new(
    VideoResolution { width: WIDTH, height: HEIGHT },
    false, // Camera content, not a screen share
);
let track = LocalVideoTrack::create_video_track(
    "front_camera",
    RtcVideoSource::Native(source.clone()),
);

let options = TrackPublishOptions {
    source: TrackSource::Camera,
    ..Default::default()
};

room.local_participant()
    .publish_track(LocalTrack::Video(track), options)
    .await?;

// Push one frame for every frame your camera produces.
loop {
    let mut frame = VideoFrame {
        rotation: VideoRotation::VideoRotation0,
        timestamp_us: 0, // Zero lets LiveKit set the timestamp
        frame_metadata: None,
        buffer: I420Buffer::new(WIDTH, HEIGHT),
    };

    let (stride_y, stride_u, stride_v) = frame.buffer.strides();
    let (data_y, data_u, data_v) = frame.buffer.data_mut();

    // Copy the Y, U, and V planes of the current camera frame
    // into data_y, data_u, and data_v. Each row is stride_y,
    // stride_u, or stride_v bytes.

    source.capture_frame(&frame);
}

Rust accepts the YUV buffer types only, which is why this example fills an I420Buffer. If your camera produces RGB, convert it first using the SDK's built-in yuv_helper conversions, such as abgr_to_i420.

from livekit import rtc

WIDTH = 1280
HEIGHT = 720

source = rtc.VideoSource(WIDTH, HEIGHT)
track = rtc.LocalVideoTrack.create_video_track("front_camera", source)
options = rtc.TrackPublishOptions(
    source=rtc.TrackSource.SOURCE_CAMERA,
)

publication = await room.local_participant.publish_track(
    track, options
)

# Push one frame for every frame your camera produces.
while True:
    # WIDTH * HEIGHT * 4 bytes of RGBA pixels from the camera.
    buffer = ...

    frame = rtc.VideoFrame(
        WIDTH, HEIGHT, rtc.VideoBufferType.RGBA, buffer
    )
    source.capture_frame(frame)

Python accepts both the YUV and RGB buffer types. The SDK converts an RGB buffer to I420 during capture_frame.

#include "livekit/local_video_track.h"
#include "livekit/video_frame.h"
#include "livekit/video_source.h"

constexpr int kWidth = 1280;
constexpr int kHeight = 720;

auto source = std::make_shared<livekit::VideoSource>(kWidth, kHeight);
auto track = livekit::LocalVideoTrack::createLocalVideoTrack(
    "front_camera", source);

livekit::TrackPublishOptions options;
options.source = livekit::TrackSource::SOURCE_CAMERA;

if (auto local_participant = room->localParticipant().lock()) {
  local_participant->publishTrack(track, options);
}

// Push one frame for every frame your camera produces.
while (true) {
  auto frame = livekit::VideoFrame::create(
      kWidth, kHeight, livekit::VideoBufferType::RGBA);

  // Copy kWidth * kHeight * 4 bytes of RGBA pixels from the
  // current camera frame into frame.data().

  source->captureFrame(frame);
}

C++ accepts both the YUV and RGB buffer types. The SDK converts an RGB buffer to I420 on capture.

import {
  LocalVideoTrack,
  TrackPublishOptions,
  TrackSource,
  VideoBufferType,
  VideoFrame,
  VideoSource,
} from '@livekit/rtc-node';

const WIDTH = 1280;
const HEIGHT = 720;

const source = new VideoSource(WIDTH, HEIGHT);
const track = LocalVideoTrack.createVideoTrack(
  'front_camera',
  source,
);

const options = new TrackPublishOptions();
options.source = TrackSource.SOURCE_CAMERA;

await room.localParticipant.publishTrack(track, options);

// Push one frame for every frame your camera produces.
while (true) {
  // Fill data with WIDTH * HEIGHT * 4 bytes of RGBA pixels.
  const data = new Uint8Array(WIDTH * HEIGHT * 4);

  const frame = new VideoFrame(
    data,
    WIDTH,
    HEIGHT,
    VideoBufferType.RGBA,
  );
  source.captureFrame(frame);
}

Node.js accepts both the YUV and RGB buffer types. The SDK converts an RGB buffer to I420 on capture.

Unity publishes from a Texture. Write your camera frames into the texture, and TextureVideoSource reads it back each Unity update:

IEnumerator PublishCamera(Room room)
{
    const int width = 1280;
    const int height = 720;

    var texture = new Texture2D(
        width, height, TextureFormat.RGBA32, false);
    var source = new TextureVideoSource(texture);
    var track = LocalVideoTrack.CreateVideoTrack(
        "front_camera", source, room);

    var options = new TrackPublishOptions
    {
        Source = TrackSource.SourceCamera
    };

    var publish = room.LocalParticipant.PublishTrack(track, options);
    yield return publish;
    if (publish.IsError) yield break;

    // Fill texture with width * height * 4 bytes of RGBA pixels, then
    // call texture.Apply(). The source reads it on each update.
    source.Start();
    StartCoroutine(source.Update());
}

Unity accepts the RGB buffer types RGBA, ARGB, BGRA, and RGB24. It doesn't accept the YUV buffer types.

LiveKit converts any YUV buffer type it can't encode directly. Capture I420 where your camera can produce it, to avoid a conversion on every frame.

Other participants receive the track through a room event, then read decoded frames from a video stream.

use futures_util::StreamExt;
use livekit::prelude::*;
use livekit::webrtc::video_stream::native::NativeVideoStream;

while let Some(event) = room_events.recv().await {
    let RoomEvent::TrackSubscribed {
        track: RemoteTrack::Video(track),
        ..
    } = event
    else {
        continue;
    };
    if track.name() != "front_camera" {
        continue;
    }
    tokio::spawn(async move {
        let rtc_track = track.rtc_track();
        let mut stream = NativeVideoStream::new(rtc_track);
        while let Some(frame) = stream.next().await {
            // Render the frame, run inference, or save it.
            // The pixels are in frame.buffer.
            let buffer = &frame.buffer;
            println!("{}x{}", buffer.width(), buffer.height());
        }
    });
}

Frames arrive in the buffer type the decoder produced, with no conversion. That includes Native on platforms with hardware decoding. Call to_i420() on frame.buffer when your code needs a known type.

import asyncio

from livekit import rtc


@room.on("track_subscribed")
def on_track_subscribed(
    track: rtc.Track,
    publication: rtc.RemoteTrackPublication,
    participant: rtc.RemoteParticipant,
):
    if track.kind != rtc.TrackKind.KIND_VIDEO:
        return
    if track.name != "front_camera":
        return
    asyncio.create_task(handle_video_track(track))


async def handle_video_track(track: rtc.Track):
    stream = rtc.VideoStream(track)
    async for event in stream:
        frame = event.frame
        # Render the frame, run inference, or save it.
        # The pixels are in frame.data.
        print(f"{frame.width}x{frame.height}")
    await stream.aclose()

Frames arrive in the buffer type the decoder produced. Pass format to VideoStream to receive a specific type instead.

Use setOnVideoFrameCallback, which handles the subscription and callback threading for you. Pass the publisher identity and track name:

room->setOnVideoFrameCallback(
    "robot", "front_camera",
    [](const livekit::VideoFrame& frame, std::int64_t timestamp_us) {
      // Render the frame, run inference, or save it.
      // The pixels are in frame.data().
      std::cout << frame.width() << "x" << frame.height() << "\n";
    });

// Later, when you no longer want frames from this track:
room->clearOnVideoFrameCallback("robot", "front_camera");

Frames arrive as RGBA, because VideoStream::Options defaults its format field to that type. Pass a different format to receive another type.

import { RoomEvent, TrackKind, VideoStream } from '@livekit/rtc-node';

room.on(RoomEvent.TrackSubscribed, async (track) => {
  if (track.kind !== TrackKind.KIND_VIDEO) {
    return;
  }
  if (track.name !== 'front_camera') {
    return;
  }

  const stream = new VideoStream(track);
  for await (const event of stream) {
    // Render the frame, run inference, or save it.
    // The pixels are in event.frame.data.
    const frame = event.frame;
    console.log(`${frame.width}x${frame.height}`);
  }
});

Frames arrive in the buffer type the decoder produced. Call frame.convert() to get a specific type.

VideoStream decodes frames into a RenderTexture, which you can assign to a material or a RawImage:

room.TrackSubscribed += (track, publication, participant) =>
{
    if (track is not RemoteVideoTrack video) return;
    if (video.Name != "front_camera") return;

    var stream = new VideoStream(video);
    stream.TextureReceived += texture =>
    {
        // Render the frame, run inference, or save it.
        Debug.Log($"{texture.width}x{texture.height}");
    };

    stream.Start();
    StartCoroutine(stream.Update());
};

VideoStream requests I420, then converts each frame into a RenderTexture.

Subscribe only while the video is in use, such as when an operator has the view on screen. Each subscription consumes bandwidth and decoding time. To control what each participant receives, see Subscribing to tracks.

Frame requirements

Your capture loop controls the rate, buffer type, and resolution of the frames you push. Each one affects how LiveKit encodes and delivers those frames.

Frame rate and buffering

The video source doesn't buffer frames. Each call to capture_frame submits one frame for encoding. Your capture loop sets the publish rate.

Even when the image doesn't change, push frames continuously. A participant who joins after the last frame has nothing to render until the next frame arrives.

Before the first capture_frame call, the video source sends a black frame 10 times per second. This behavior stops after your first captured frame.

Color format

Video codecs work in YUV, also known as YCbCr. A YUV frame keeps brightness in the Y plane and color in the U and V planes. Each plane is a separate region of memory with its own stride. NV12 is the exception, with one Y plane and a second plane that interleaves U and V. RGB keeps all channels interleaved in a single buffer.

Each buffer type belongs to one of the two color models:

Color model	Buffer types
YUV	`I420`, `I420A`, `I422`, `I444`, `I010`, `NV12`
RGB	`RGBA`, `ABGR`, `ARGB`, `BGRA`, `RGB24`

Each SDK differs in the buffer types it accepts and the ones it delivers. See the note under each example in Publish a video track and Subscribe to a video track.

Resolution

Set the video source resolution to the resolution your camera produces. LiveKit reads this value at publish time and uses it for:

The track width and height it reports to the server.
The simulcast layers it computes and advertises.

Pushing a frame of a different size doesn't raise an error, so a mismatch is easy to miss. LiveKit advertises simulcast layers for a size you never send and rescales every frame to the declared resolution, adding per-frame processing.

Next steps

Hardware encoder support

Use hardware video encoders to minimize encoding latency on supported platforms.

Timestamps and frame metadata

Attach timestamps and metadata to published frames to correlate video with sensor data.

Low latency

Tune the pipeline for the network conditions and jitter tolerance of your app.

Local video example

Rust app that captures from a local camera and publishes it as a video track.