ESP32 microcontrollers

Overview

The ESP32 SDK  makes it possible to connect low-cost microcontrollers to LiveKit servers, so you can stream video, audio and data, and connect to voice and video AI agents.

This SDK supports audio streaming with the Opus codec and video streaming with the H.264 codec, as well as RPCs for interacting with hardware peripherals via AI agents.

Note that the ESP32 SDK has several important limitations. It supports only ESP32-S3 and ESP32-P4 chips. Media publishing and subscription support is limited as follows:

• Publishing: Supports up to 1 video track and 1 audio track.
• Subscribing: Supports 1 audio track. Video track subscriptions are not currently supported.

Use cases

The ESP32 SDK supports a variety of use cases for connected devices:

• Voice AI enabled devices
  • Children's interactive toys
  • Smart assistant devices
• AI enabled security cameras
• Video doorbell systems
• Agent-enabled kiosks
• Teleoperation of robots or drones

Desktop comparison

The ESP32 SDK is designed for resource-constrained embedded devices and differs from desktop SDKs in several important ways to optimize for memory and processing constraints.

Low-level capture and render

To allow the SDK to support any ESP32-based board, it delegates the responsibility of configuring the capture and render pipelines to your application.

Room state management

On other platforms, LiveKit retains room and participant state internally. In contrast, the ESP32 SDK maintains minimal state internally to reduce allocations your application might not need. Instead, mid-level events are dispatched to your application when changes occur, most notably, on_participant_info and on_room_info. When an update is received, your application can copy any fields it needs to retain; the event payload is deallocated after the handler returns.

Architecture

The ESP32 LiveKit SDK is built on top of core components from the Espressif ecosystem and is designed to be agnostic to the A/V peripherals on the board.

Underlying components

The ESP32 SDK is built on top of these core capture and render components:

• esp_capture : Handles capture from A/V input peripherals: camera and microphone (via audio codec)
• av_render : Handles rendering to A/V output peripherals: display and speaker (via audio codec)

Together, these components enable capture/render from a wide range of hardware and interfaces (for example, I2S, SPI, MIPI, USB).

SDK design

The LiveKit SDK itself is agnostic to the A/V peripherals on the board. When initializing a room, you provide pre-configured capture and render handles for the hardware available on your board using esp_capture_handle_t and av_render_handle_t, respectively.

Accessing A/V peripherals

To configure a capture and render handle, you'll need to obtain handles for the A/V peripherals available through your board. There are two options for this:

• Use a BSP (board support package): A board support package handles initialization of and provides access to onboard peripherals. Pre-made BSPs for popular Espressif and M5Stack development boards live in esp-bsp . Waveshare also maintains BSPs for their board in this repo . For custom boards, you can fork one of the existing BSPs and modify as necessary to support custom peripherals and pin assignments. Recommended for production.

• Use the Espressif codec_board component: The codec_board component provides out-of-the-box configuration for many popular development boards. Available at codec_board . Useful for rapid development and easily running the same application on multiple boards.

Development boards

To get started with the ESP32 SDK, you'll need a suitable development board equipped with the media peripherals required for your application.

Supported development boards: