Skip to main content
IntroductionBuild AgentsAgent FrontendsTelephonyWebRTC TransportManage & DeployReferences
XYouTubeSlack

Frame metadata

Attach and read per-frame metadata on video tracks.

On this page

OverviewPublishing frame metadataReading frame metadataConstraints

Overview

Frame metadata is a set of optional fields associated with individual video frames. Subscribers receive the metadata together with the frame it describes. You can use it for frame-accurate timing, latency measurement, and per-frame application data.

Each field is negotiated independently when the track is published, so a track includes only the fields you enable:

  • User timestamp: Wall-clock capture time in microseconds. Use it to measure end-to-end latency or align frames with an external clock.
  • Frame ID: A user-defined numeric identifier associated with a frame. For example, use a monotonically increasing value to detect dropped frames or correlate a frame with state computed elsewhere.
  • User data: Bytes defined by your application. Use this field to attach small, per-frame binary payloads that are relevant to each video frame. For example, you might send a list of numbers representing a robot arm's joint positions.
Video frames only

Frame metadata applies to video frames only. Audio frames don't carry frame metadata.

SDK support

The user timestamp and frame ID fields are available in the Rust, C++, Python, JavaScript, and Unity SDKs. The user data field is currently supported by the Rust SDK (publish and read) and the browser JavaScript SDK (read only).

Publishing frame metadata

Publishing frame metadata is a two-step process: enable the features you want when publishing the track so they're negotiated with the server, then provide the matching fields for each frame. A field is only serialized onto the wire if its feature was enabled at publish time.

How the fields are provided depends on the SDK. SDKs that publish from a backend raw video source (Rust, C++, Python, and Unity) can attach metadata to each captured frame. The browser JavaScript SDK captures frames itself, so it derives the timestamp and frame ID automatically once the feature is enabled.

Enable the features you need on TrackPublishOptions when publishing the track:

use livekit::options::{FrameMetadataFeatures, TrackPublishOptions};
use livekit::prelude::*;


let mut features = FrameMetadataFeatures::default();
features.user_timestamp = true;
features.frame_id = true;
features.user_data = true;


let options = TrackPublishOptions {
    source: TrackSource::Camera,
    frame_metadata_features: features,
    ..Default::default()
};


room.local_participant()
    .publish_track(LocalTrack::Video(track), options)
    .await?;

Then attach metadata to each frame before capturing it on the video source:

use livekit::webrtc::video_frame::{FrameMetadata, VideoFrame, VideoRotation};


let mut frame = VideoFrame {
    rotation: VideoRotation::VideoRotation0,
    timestamp_us: 0,
    frame_metadata: None,
    buffer, // your filled I420 (or other) buffer
};


// Every field is optional; set `None` to omit it for this frame.
frame.frame_metadata = Some(FrameMetadata {
    user_timestamp: Some(capture_time_us),
    frame_id: Some(frame_id),
    user_data: Some(payload_bytes),
});


source.capture_frame(&frame);

Enable the features you need on TrackPublishOptions, then publish the track:

#include "livekit/local_video_track.h"
#include "livekit/video_source.h"


auto source = std::make_shared<livekit::VideoSource>(width, height);
auto track = livekit::LocalVideoTrack::createLocalVideoTrack("camera", source);


livekit::TrackPublishOptions options;
options.source = livekit::TrackSource::SOURCE_CAMERA;
options.frame_metadata_features.emplace();
options.frame_metadata_features->user_timestamp = true;
options.frame_metadata_features->frame_id = true;


room->localParticipant().lock()->publishTrack(track, options);

Then attach metadata to each frame through VideoCaptureOptions when you capture it:

livekit::VideoCaptureOptions capture_options;
capture_options.timestamp_us = capture_time_us;
capture_options.metadata = livekit::VideoFrameMetadata{};
capture_options.metadata->user_timestamp_us = capture_time_us;
capture_options.metadata->frame_id = frame_id;


source->captureFrame(frame, capture_options);

Enable the features you need on TrackPublishOptions, then publish the track:

from livekit import rtc


source = rtc.VideoSource(width, height)
track = rtc.LocalVideoTrack.create_video_track("camera", source)


options = rtc.TrackPublishOptions(
    source=rtc.TrackSource.SOURCE_CAMERA,
    frame_metadata_features=[
        rtc.FrameMetadataFeature.FMF_USER_TIMESTAMP,
        rtc.FrameMetadataFeature.FMF_FRAME_ID,
    ],
)


publication = await room.local_participant.publish_track(track, options)

Then build a FrameMetadata for each frame and pass it to capture_frame:

metadata = rtc.FrameMetadata()
metadata.user_timestamp = capture_time_us  # wall-clock microseconds
metadata.frame_id = frame_id               # numeric identifier


source.capture_frame(frame, timestamp_us=timestamp_us, metadata=metadata)

Enable the features with WithPacketTrailerFeatures on TrackPublishOptions, and set a MetadataProvider on the source to supply the fields for each outgoing frame:

using LiveKit;
using LiveKit.Proto;


// Negotiate the per-frame fields you want with the server.
var options = new TrackPublishOptions
{
    Source = TrackSource.SourceCamera,
}.WithPacketTrailerFeatures(
    PacketTrailerFeature.PtfUserTimestamp,
    PacketTrailerFeature.PtfFrameId);


// Invoked once per outgoing frame. Return null to send no metadata.
source.MetadataProvider = () => new FrameMetadata
{
    UserTimestamp = captureTimeUs,  // wall-clock microseconds
    FrameId = frameId,              // numeric identifier
};


var track = LocalVideoTrack.CreateVideoTrack("camera", source, room);
yield return room.LocalParticipant.PublishTrack(track, options);

Reading frame metadata

On the subscribing side, metadata arrives attached to each received frame. Fields are present only when the publisher enabled the corresponding feature and set a value for that frame, so check each field before using it. Some SDKs also expose the negotiated feature set on the track publication.

The browser JavaScript SDK delivers frames through the browser video pipeline rather than a frame iterator. Instead of reading metadata off a frame object, applications look it up by RTP timestamp as frames are rendered. See the JavaScript tab for details.

use futures::StreamExt;
use livekit::webrtc::video_stream::native::NativeVideoStream;


// Inspect the features negotiated for this publication.
let features = publication.frame_metadata_features();


let mut stream = NativeVideoStream::new(video_track.rtc_track());
while let Some(frame) = stream.next().await {
    if let Some(metadata) = &frame.frame_metadata {
        if let Some(user_timestamp) = metadata.user_timestamp {
            // Compare against the local clock to measure latency.
        }
        if let Some(frame_id) = metadata.frame_id {
            // Detect gaps in the frame sequence.
        }
        if let Some(user_data) = metadata.user_data.as_deref() {
            // Decode your per-frame payload.
        }
    }
}

Read frames from a VideoStream. Each VideoFrameEvent carries an optional VideoFrameMetadata:

#include "livekit/video_stream.h"


auto stream = livekit::VideoStream::fromTrack(video_track, {});


livekit::VideoFrameEvent event;
while (stream->read(event)) {
    if (!event.metadata) {
        continue;
    }
    if (event.metadata->user_timestamp_us) {
        // Compare against the local clock to measure latency.
    }
    if (event.metadata->frame_id) {
        // Detect gaps in the frame sequence.
    }
}

Iterate a VideoStream. Each VideoFrameEvent exposes an optional metadata field:

from livekit import rtc


# Inspect the features negotiated for this publication.
features = publication.frame_metadata_features


stream = rtc.VideoStream.from_track(video_track)
async for event in stream:
    metadata = event.metadata
    if metadata is None:
        continue
    if metadata.HasField("user_timestamp"):
        # Compare against the local clock to measure latency.
        ...
    if metadata.HasField("frame_id"):
        # Detect gaps in the frame sequence.
        ...

Subscribe to a VideoStream and read the optional Metadata off each received frame:

using LiveKit;


var stream = new VideoStream(remoteVideoTrack);
stream.FrameReceived += frame =>
{
    var metadata = frame.Metadata;
    if (metadata == null) return;
    if (metadata.HasUserTimestamp)
    {
        // Compare against the local clock to measure latency.
    }
    if (metadata.HasFrameId)
    {
        // Detect gaps in the frame sequence.
    }
};
stream.Start();

Constraints

Keep the following limits in mind when designing around frame metadata:

  • Video only. Frame metadata is carried alongside video frames. Audio frames don't support it.
  • Size limit. All enabled metadata fields share a budget of approximately 232 bytes per frame. The fixed fields (user_timestamp and frame_id) consume part of this budget, reducing the space available for user_data when they're enabled.
  • Oversize metadata is dropped, not truncated. If user_data causes the frame metadata to exceed the size limit, the entire metadata payload is dropped before transmission rather than being truncated. Keep payloads small and fixed-size where possible.