Build custom audio visualizers | LiveKit Documentation

This demo uses your device's microphone.

Overview

Build your own shader-based audio visualizers that react to voice and agent state in realtime. Start with an existing visualizer like aura, customize the WebGL shader code, and wire it up with smooth animations. This guide shows you how to create visual effects that bring your voice agent to life.

Architecture

Audio visualizers use a three-layer architecture that separates concerns and keeps your code clean:

Component layer — React component with standard props like state, audioTrack, size, and color.
Hook layer — Custom hook that handles animation values and responds to agent state changes.
Shader layer — GLSL fragment shader that renders the visual effect with WebGL.

Start from an existing visualizer

The aura visualizer is a good starting point for custom shaders. Copy the component and hook to your project:

pnpm dlx shadcn@latest add @agents-ui/agent-audio-visualizer-aura

This installs three files:

components/agents-ui/agent-audio-visualizer-aura.tsx - Main component and shader
hooks/agents-ui/use-agent-audio-visualizer-aura.ts - Animation hook
components/agents-ui/react-shader-toy.tsx - WebGL shader renderer

Customize the shader code

Shader structure

Shaders are written in GLSL and embedded as template strings. The code follows ShaderToy conventions with a mainImage function that runs for every pixel:

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
  // Transform fragment coordinates to UV space (0-1)
  vec2 uv = fragCoord / iResolution.xy;
  
  // Center the coordinates (-0.5 to 0.5)
  vec2 pos = uv - 0.5;
  
  // Your shader code here
  
  // Output final color with alpha
  fragColor = vec4(color, alpha);
}

Built-in uniforms

The shader renderer gives you ShaderToy-compatible values you can use right away:

iTime - Elapsed time in seconds (float).
iResolution - Canvas resolution in pixels (vec2).
iMouse - Mouse position and click state (vec4).
iFrame - Frame count (int).
iDate - Current date and time (vec4).

Modify the visual effect

Edit the shader code to create your own look. Here's how to turn the circular aura into a pulsing grid:

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
  vec2 uv = fragCoord / iResolution.xy;
  float aspect = iResolution.x / iResolution.y;

  vec2 pos = uv - 0.5;
  pos.x *= aspect;

  float radius = 0.15 + 0.25 * uIntensity;

  float particleRadius = 2.0 / iResolution.y;
  float blur = 1.0 / iResolution.y;

  float minDist = 1e6;
  for (int i = 0; i < NUM_PARTICLES; i++) {
    float fi = float(i);
    float speedVar = 0.7 + 0.6 * hash(fi * 2.3) * (0.5 + uComplexity);
    float wobble = (0.04 + 0.12 * uComplexity) * sin(iTime * 1.5 + fi * 4.1);
    float angle = hash(fi * 1.1) * TAU - iTime * uSpeed * 0.25 * speedVar + wobble;
    float rBob = 1.0 + (0.02 + 0.06 * uComplexity) * sin(iTime * 2.2 + fi * 3.7);
    vec2 pPos = radius * rBob * vec2(cos(angle), sin(angle));
    float d = length(pos - pPos);
    minDist = min(minDist, d);
  }

  float particle = 1.0 - smoothstep(particleRadius - blur, particleRadius + blur, minDist);

  vec3 color = uColor * particle * uIntensity;
  fragColor = vec4(color, particle * uIntensity);
}

Helper functions

Use constants and helper functions to help clarify and document your code:

const float TAU = 6.28318;
const int NUM_PARTICLES = 100;

float hash(float n) {
  return fract(sin(n) * 43758.5453123);
}

Add custom uniforms

Pass values from React to your shader using uniforms. Control speed, scale, color, and any other parameter you want to animate.

Define uniform types

Add your uniforms to the ReactShaderToy component's uniforms prop:

<ReactShaderToy
  fs={shaderSource}
  uniforms={{
    uColor: { type: '3fv', value: hexToRgb(color) },
    uSpeed: { type: '1f', value: speed },
    uIntensity: { type: '1f', value: intensity },
    uComplexity: { type: '1f', value: complexity },
  }}
/>

Uniform type reference

Map JavaScript types to GLSL types:

Type	GLSL	JavaScript
`'1f'`	`float`	`number`
`'2f'`	`vec2`	`[number, number]`
`'3f'` / `'3fv'`	`vec3`	`[number, number, number]`
`'4f'` / `'4fv'`	`vec4`	`[number, number, number, number]`
`'1i'`	`int`	`number`
`'Matrix2fv'`	`mat2`	`number[]` (length 4)
`'Matrix3fv'`	`mat3`	`number[]` (length 9)
`'Matrix4fv'`	`mat4`	`number[]` (length 16)

Important

ReactShaderToy declares your custom uniforms for you, so don't add them in your shader code. Declaring them in both places will cause your shader to fail to compile.

Add component props

Make your visualizer configurable by exposing uniforms as React props:

interface CustomShaderProps {
  color: string;
  speed: number;
  intensity: number;
  complexity: number;
}

function CustomShader({
  color,
  speed = 5.0,
  intensity = 1.0,
  complexity = 0.5,
  ref,
  className,
  ...props
}: CustomShaderProps & ComponentProps<'div'>) {
  return (
    <div ref={ref} className={className} {...props}>
      <ReactShaderToy
        fs={shaderSource}
        uniforms={{
          uColor: { type: '3fv', value: hexToRgb(color) },
          uSpeed: { type: '1f', value: speed },
          uIntensity: { type: '1f', value: intensity },
          uComplexity: { type: '1f', value: complexity },
        }}
        onError={(error) => {
          console.error('Shader error:', error);
        }}
        onWarning={(warning) => {
          console.warn('Shader warning:', warning);
        }}
        style={{ width: '100%', height: '100%' }}
      />
    </div>
  );
}

Animate with Motion values

Use Framer Motion to create smooth transitions between agent states and respond to audio in realtime. The visualizer pulses when the agent is thinking, reacts to volume when speaking, and settles when idle.

Create the animation hook

Build a custom hook to manage motion values and react to agent state changes:

import { useCallback, useEffect, useRef, useState } from 'react';
import { animate, useMotionValue, useMotionValueEvent } from 'motion/react';
import type { AgentState } from '@livekit/components-react';

function useAnimatedValue<T>(initialValue: T) {
  const [value, setValue] = useState(initialValue);
  const motionValue = useMotionValue(initialValue);
  const controlsRef = useRef<AnimationPlaybackControlsWithThen | null>(null);
  
  useMotionValueEvent(motionValue, 'change', (value) => setValue(value));

  const animateFn = useCallback(
    (targetValue: T | T[], transition: ValueAnimationTransition) => {
      controlsRef.current = animate(motionValue, targetValue, transition);
    },
    [motionValue],
  );

  return { value, motionValue, controls: controlsRef, animate: animateFn };
}

export function useCustomVisualizer(
  state: AgentState,
  audioTrack?: LocalAudioTrack | RemoteAudioTrack
) {
  const { value: intensity, animate: animateIntensity } = useAnimatedValue(1.0);
  const { value: speed, animate: animateSpeed } = useAnimatedValue(5.0);
  
  const volume = useTrackVolume(audioTrack, {
    fftSize: 512,
    smoothingTimeConstant: 0.55,
  });

  return { intensity, speed, volume };
}

Respond to agent state

Map each agent state to specific animation values:

useEffect(() => {
  switch (state) {
    case 'idle':
    case 'disconnected':
      animateIntensity(0.3, transition);
      animateSpeed(1, { duration: 0 });
      return;

    case 'listening':
      // Gentle pulsing
      animateIntensity([0.5, 0.8], pulseTransition);
      animateSpeed(2.5, { duration: 0 });
      return;

    case 'thinking':
    case 'connecting':
      // Rapid pulsing
      animateIntensity([0.25, 0.5], pulseTransition);
      animateSpeed(4.0, { duration: 0 });
      return;

    case 'speaking':
      // Fast animation
      animateSpeed(2.5, { duration: 0 });
      return;
  }
}, [state, animateIntensity, animateSpeed]);

Respond to audio volume

Update values instantly when audio volume changes:

useEffect(() => {
  if (state === 'speaking' && volume > 0) {
    // Use duration: 0 for instant updates
    animateIntensity(0.3 + 0.7 * volume, { duration: 0 });
  }
}, [state, volume, animateIntensity]);

Connect hook to component

Wire up the hook in your component and pass animated values to the shader:

export function CustomAudioVisualizer({
  size = 'lg',
  state = 'connecting',
  color = '#000000',
  complexity = 0.5,
  audioTrack,
  className,
  ref,
  ...props
}: CustomVisualizerProps & ComponentProps<'div'>) {
  const { intensity, speed } = useCustomVisualizer(
    state,
    audioTrack as LocalAudioTrack | RemoteAudioTrack | undefined,
  );

  return (
    <CustomShader
      ref={ref}
      color={color}
      speed={speed}
      intensity={intensity}
      complexity={complexity}
      className={className}
      {...props}
    />
  );
}

Standard props

Keep your visualizer consistent with the rest of Agents UI. These props let developers swap visualizers without changing code:

Required props

export interface CustomVisualizerProps {
  /**
   * Current agent state.
   *
   * @defaultValue 'connecting'
   */
  state?: AgentState;
  
  /**
   * Audio track to visualize. Can be a local/remote audio track or a track reference.
   */
  audioTrack?: LocalAudioTrack | RemoteAudioTrack | TrackReferenceOrPlaceholder;
  
  /**
   * Size of the visualizer.
   *
   * @defaultValue 'lg'
   */
  size?: 'icon' | 'sm' | 'md' | 'lg' | 'xl';
  
  /**
   * Primary color in hexadecimal format.
   *
   * @defaultValue '#000000'
   */
  color?: `#${string}`;

  /** 
   * The complexity of the visualizer.
   *
   * @defaultValue 0.5
   */
  complexity?: number;
}

Size variants

Use cva from class-variance-authority to define size variants:

import { cva } from 'class-variance-authority';

export const CustomVisualizerVariants = cva(['aspect-square'], {
  variants: {
    size: {
      icon: 'h-[24px]',
      sm: 'h-[56px]',
      md: 'h-[112px]',
      lg: 'h-[224px]',
      xl: 'h-[448px]',
    },
  },
  defaultVariants: {
    size: 'lg',
  },
});

Apply variants to your component:

export function CustomAudioVisualizer({
  size = 'lg',
  state = 'connecting',
  color = '#000000',
  complexity = 0.5,
  audioTrack,
  className,
  ref,
  ...props
}: CustomVisualizerProps & ComponentProps<'div'>) {
  const { intensity, speed } = useCustomVisualizer(
    state,
    audioTrack as LocalAudioTrack | RemoteAudioTrack | undefined,
  );

  return (
    <CustomShader
      ref={ref}
      color={color}
      speed={speed}
      intensity={intensity}
      complexity={complexity}
      className={cn(CustomVisualizerVariants({ size }), className)}
      {...props}
    />
  );

Color conversion

Convert hex colors to RGB arrays for shader uniforms:

function hexToRgb(hexColor: string): [number, number, number] {
  const rgbColor = hexColor.match(/^#([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})$/);
  
  if (rgbColor) {
    const [, r, g, b] = rgbColor;
    return [r, g, b].map(c => parseInt(c, 16) / 255) as [number, number, number];
  }
  
  return [0, 0, 0]; // Default black
}

Usage example

Put it all together:

'use client';

import { useAgent } from '@livekit/components-react';
import { CustomAudioVisualizer } from '@/components/agents-ui/custom-audio-visualizer';

export function VoiceAgentInterface() {
  const { audioTrack, state } = useAgent();

  return (
    <CustomAudioVisualizer
      size="lg"
      state={state}
      complexity={0.75}
      audioTrack={audioTrack}
    />
  );
}

Performance tips

Target 60fps for smooth animations. A few optimizations make a big difference:

Cache texture lookups when you sample the same texture multiple times.
Use math tricks like mix(), step(), and smoothstep() instead of branching with if statements.
Limit expensive ops like sin(), cos(), and sqrt() inside loops.
Set precision to mediump on mobile to balance quality and performance.
Test on real devices to see how your shader performs where it matters.

Audio visualizer overview

Overview of all visualizer variants and usage examples.

AgentAudioVisualizerAura

Reference implementation of a shader-based visualizer.

Agent state

Learn about agent state transitions and lifecycle.