Timestamps and frame metadata

Overview

Frame metadata provides optional per-frame fields for a video track, including a capture timestamp from your own clock, a frame ID, and a small binary payload. WebRTC's RTP timestamp is used for playback synchronization; it doesn't record when your sensor captured a frame or correspond to a ground-truth clock such as GPS or PTP. Frame metadata lets you correlate video frames with other data in your robotics system.

For how it works, the available fields, SDK support, and publish and subscribe examples in every SDK, see Frame metadata.

Use cases

Frame metadata supports robotics workflows that depend on per-frame correlation:

• Sensor alignment: Stamp each frame with a ground-truth timestamp so video can be aligned with LiDAR, IMU, and other sensor streams on a shared timeline.
• Glass-to-glass latency: Stamp each frame with its exposure time to measure latency across capture, encoding, network transmission, decoding, and rendering.
• Frame-accurate correlation: Include a frame ID so consumers can match a frame to inference results produced elsewhere without maintaining a separate synchronization channel.
• Device state: Attach per-frame state, such as the joint positions of the arm holding the camera, so it stays associated with the frame instead of requiring correlation later.

Additional resources