LiveKit docs › Test & Evaluate › Observability › Build your own › OpenTelemetry traces

---

# OpenTelemetry traces

> Export agent traces to any OpenTelemetry-compatible backend.

## Overview

Available in:
- [x] Node.js
- [x] Python

LiveKit Agents instruments each session with OpenTelemetry traces: the same spans used by [Agent insights](https://docs.livekit.io/testing/observability/insights.md). Set a tracer provider to export these spans to any OpenTelemetry-compatible backend.

The following example sends spans to [Langfuse](https://langfuse.com/), an open-source LLM observability platform. The same approach works for any backend that accepts traces over the OpenTelemetry Protocol (OTLP). To learn more, see [Other backends](#other-backends).

> 🔀 **Behavior change: Span changes in Agents 1.7.0 and 1.8.0**
> 
> Agents 1.7.0 adds an `lk.pii.` prefix to content attributes, and Agents 1.8.0 changes spans to follow the OpenTelemetry GenAI semantic conventions. After upgrading, dashboards and queries in your own backend that use the old span format no longer match and don't raise an error. See [Content attributes](#content-attributes) and [GenAI semantic conventions](#genai-conventions).

## Set environment variables

Create an API key pair in your Langfuse project settings, then add the following to your agent's `.env.local` file:

- `LANGFUSE_PUBLIC_KEY`: The public key for your Langfuse project.
- `LANGFUSE_SECRET_KEY`: The secret key for your Langfuse project.
- `LANGFUSE_BASE_URL`: The URL for your Langfuse instance, such as `https://cloud.langfuse.com` (EU) or `https://us.cloud.langfuse.com` (United States).

The example script reads these variables to build the OTLP endpoint and authentication headers that the exporter sends to Langfuse. To export to a different backend, set those values directly instead. See [Other backends](#other-backends).

## Trace a complete agent

Both examples send the `x-langfuse-ingestion-version` header to opt into Langfuse's realtime ingestion. Without it, spans can take up to 10 minutes to appear.

**Python**:

Call `setup_langfuse` before the session starts so the agent's spans route to Langfuse. Pass `metadata` to set attributes on every span. For example, set `langfuse.session.id` to the room name to group all spans from a session together in Langfuse:

```python
import base64
import os

from dotenv import load_dotenv
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.util.types import AttributeValue

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    cli,
    inference,
)
from livekit.agents.telemetry import set_tracer_provider

load_dotenv(".env.local")


def setup_langfuse(metadata: dict[str, AttributeValue] | None = None) -> TracerProvider:
    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk.trace.export import BatchSpanProcessor

    public_key = os.environ.get("LANGFUSE_PUBLIC_KEY")
    secret_key = os.environ.get("LANGFUSE_SECRET_KEY")
    base_url = os.environ.get("LANGFUSE_BASE_URL")

    if not public_key or not secret_key or not base_url:
        raise ValueError("LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL must be set")

    langfuse_auth = base64.b64encode(f"{public_key}:{secret_key}".encode()).decode()
    os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = f"{base_url.rstrip('/')}/api/public/otel"
    os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = (
        f"Authorization=Basic {langfuse_auth},x-langfuse-ingestion-version=4"
    )

    trace_provider = TracerProvider()
    trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    set_tracer_provider(trace_provider, metadata=metadata)
    return trace_provider


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice AI assistant.",
            llm=inference.LLM(model="openai/gpt-5.2-chat-latest"),
        )


server = AgentServer()


@server.rtc_session(agent_name="my-agent")
async def entrypoint(ctx: JobContext):
    # Route spans to Langfuse before the session starts.
    trace_provider = setup_langfuse(metadata={"langfuse.session.id": ctx.room.name})

    # Flush any remaining spans before the process exits.
    async def flush_trace():
        trace_provider.force_flush()

    ctx.add_shutdown_callback(flush_trace)

    session = AgentSession(
        stt=inference.STT(model="assemblyai/universal-3-6-pro", language="en"),

        tts=inference.TTS(model="fishaudio/s2.1-pro", voice="fa4c9eb3dccc4806b382b40d61c6b10a"),

        preemptive_generation=True,
    )

    await session.start(agent=Assistant(), room=ctx.room)
    await ctx.connect()


if __name__ == "__main__":
    cli.run_app(server)

```

For a larger example with fallback models and metrics logging, see the [OpenTelemetry trace example on GitHub](https://github.com/livekit/agents/blob/main/examples/voice_agents/otel_trace.py).

---

**Node.js**:

Install the OpenTelemetry SDK and an OTLP trace exporter alongside `@livekit/agents`:

```bash
pnpm add @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/exporter-trace-otlp-http

```

Call `setupLangfuse` before the session starts so the agent's spans route to Langfuse. Pass `metadata` to set attributes on every span. For example, set `langfuse.session.id` to the room name to group all spans from a session together in Langfuse:

```typescript
import {
  type JobContext,
  ServerOptions,
  cli,
  defineAgent,
  inference,
  telemetry,
  voice,
} from '@livekit/agents';
import { type Attributes } from '@opentelemetry/api';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { BatchSpanProcessor, NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import dotenv from 'dotenv';
import { fileURLToPath } from 'node:url';

dotenv.config({ path: '.env.local' });

function setupLangfuse(metadata?: Attributes): NodeTracerProvider {
  const publicKey = process.env.LANGFUSE_PUBLIC_KEY;
  const secretKey = process.env.LANGFUSE_SECRET_KEY;
  const baseUrl = process.env.LANGFUSE_BASE_URL;

  if (!publicKey || !secretKey || !baseUrl) {
    throw new Error('LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL must be set');
  }

  const auth = Buffer.from(`${publicKey}:${secretKey}`).toString('base64');
  const traceExporter = new OTLPTraceExporter({
    url: `${baseUrl.replace(/\/$/, '')}/api/public/otel/v1/traces`,
    headers: { Authorization: `Basic ${auth}`, 'x-langfuse-ingestion-version': '4' },
  });

  // A provider takes its span processors at construction. Include a FanoutSpanProcessor and
  // hand its `add` method to setTracerProvider so the framework can attach the processor that
  // applies `metadata` to every span.
  const fanout = new telemetry.FanoutSpanProcessor();
  const traceProvider = new NodeTracerProvider({
    spanProcessors: [new BatchSpanProcessor(traceExporter), fanout],
  });

  traceProvider.register();
  telemetry.setTracerProvider(traceProvider, {
    metadata,
    registerSpanProcessor: (processor) => fanout.add(processor),
  });
  return traceProvider;
}

export default defineAgent({
  entry: async (ctx: JobContext) => {
    // Route spans to Langfuse before the session starts.
    const traceProvider = setupLangfuse({ 'langfuse.session.id': ctx.room.name });

    // Flush any remaining spans before the process exits.
    ctx.addShutdownCallback(async () => {
      await traceProvider.shutdown();
    });

    const session = new voice.AgentSession({
      stt: new inference.STT({ model: 'assemblyai/universal-3-6-pro', language: 'en' }),
      llm: new inference.LLM({ model: 'openai/gpt-5.2-chat-latest' }),
      tts: new inference.TTS({
        model: 'fishaudio/s2.1-pro',
        voice: 'fa4c9eb3dccc4806b382b40d61c6b10a',
      }),
    });

    await session.start({
      agent: voice.Agent.create({ instructions: 'You are a helpful voice AI assistant.' }),
      room: ctx.room,
    });

    await ctx.connect();
  },
});

cli.runApp(new ServerOptions({ agent: fileURLToPath(import.meta.url), agentName: 'my-agent' }));

```

`registerSpanProcessor` also keeps [Agent insights](https://docs.livekit.io/testing/observability/insights.md) working: with LiveKit Cloud tracing enabled, the framework registers its own exporter on your provider, so spans reach both Langfuse and LiveKit Cloud. Without it, the framework turns off Cloud tracing and logs a warning.

For a larger example with fallback models and metrics logging, see the [OpenTelemetry trace example on GitHub](https://github.com/livekit/agents-js/blob/main/examples/src/otel_trace.ts).

## Span attributes

Agent spans carry attributes from two namespaces. The `lk.*` namespace holds LiveKit-specific data such as speech IDs, turn timings, and serialized metrics. The `gen_ai.*` namespace follows the OpenTelemetry GenAI semantic conventions, which most observability backends use for model usage and cost reporting.

### Content attributes

Attributes that carry conversation content, tool payloads, or participant data are named with an `lk.pii.` prefix. The prefix lets LiveKit Cloud strip them when [PII redaction](https://docs.livekit.io/testing/observability/pii-redaction.md) is turned on, and it's applied in the SDK regardless of whether you use LiveKit Cloud. To keep these attributes out of your own backend, see [Strip PII from exported traces](#strip-pii).

Agents 1.7.0 added the prefix to all of these attributes. Python and Node.js use the same 12 names:

| Previous attribute | Current attribute | Contents |
| `lk.participant_identity` | `lk.pii.participant_identity` | Participant identity |
| `lk.room_name` | `lk.pii.room_name` | Room name |
| `lk.user_input` | `lk.pii.user_input` | User input for the turn |
| `lk.instructions` | `lk.pii.instructions` | Agent instructions |
| `lk.chat_ctx` | `lk.pii.chat_ctx` | Serialized chat context |
| `lk.response.text` | `lk.pii.response.text` | LLM response text |
| `lk.response.function_calls` | `lk.pii.response.function_calls` | Function calls in the LLM response |
| `lk.function_tool.arguments` | `lk.pii.function_tool.arguments` | Tool call arguments |
| `lk.function_tool.output` | `lk.pii.function_tool.output` | Tool call return value |
| `lk.input_text` | `lk.pii.input_text` | Text sent to the TTS |
| `lk.user_transcript` | `lk.pii.user_transcript` | Final user transcript |
| `lk.amd.transcript` | `lk.pii.amd.transcript` | Transcript captured by answering machine detection |

The rename is silent for downstream consumers. A Langfuse view, Datadog monitor, or custom query that uses an old name returns no matches after you upgrade rather than raising an error. Update these queries as part of the upgrade. Attributes that don't contain content, such as `lk.speech_id`, `lk.job_id`, and the `gen_ai.usage.*` token counts, keep their existing names.

Agents 1.7.0 also moves roughly 50 structured log fields to the same naming convention. Most are plugin-level fields that contain transcripts, tool payloads, or raw provider messages. The affected fields differ by SDK:

- **Python:** keys in the `extra` dict on a log record, such as `chat_ctx` → `lk.pii.chat_ctx`, `arguments` → `lk.pii.arguments`, and `transcript` → `lk.pii.transcript`.
- **Node.js:** Pino child-logger fields, including `roomName` → `lk.pii.room_name` and `participant` or `participantIdentity` → `lk.pii.participant_identity`. These two renames are specific to Node.js.

If you consume agent logs through a [log drain](https://docs.livekit.io/deploy/agents/log-drains.md), update any filters or dashboards that use the old field names.

To tag your own attributes and log fields for redaction by LiveKit Cloud, see [Tag your own attributes](https://docs.livekit.io/testing/observability/pii-redaction.md#tag-your-own-attributes).

### GenAI semantic conventions

Agents 1.8.0 ([Python](https://github.com/livekit/agents/releases/tag/livekit-agents%401.8.0), [Node.js](https://github.com/livekit/agents-js/releases/tag/%40livekit%2Fagents%401.8.0)) changes spans to follow the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Update dashboards, alerts, and queries that use the previous span format:

| Previous | Current | SDK |
| Realtime `gen_ai.*` attributes on the `agent_turn` span | Realtime `gen_ai.*` attributes on a new `realtime_inference` child span | Python, Node.js |
| `gen_ai.provider.name` = `api.openai.com` | `gen_ai.provider.name` = `openai` | Python, Node.js |
| `gen_ai.provider.name` = `AWS Bedrock` | `gen_ai.provider.name` = `aws.bedrock` | Python, Node.js |
| `gen_ai.provider.name` = `google` | `gen_ai.provider.name` = `gcp.gen_ai` | Python |
| `gen_ai.system.message` event | `gen_ai.system_instructions` attribute | Python |
| `gen_ai.user.message`, `gen_ai.assistant.message`, and `gen_ai.tool.message` events | `gen_ai.input.messages` attribute | Python |
| `gen_ai.choice` event | `gen_ai.output.messages` and `gen_ai.response.finish_reasons` attributes | Python |
| Realtime `gen_ai.operation.name` = `chat` | Realtime `gen_ai.operation.name` = `generate_content` | Python |

In Python, reasoning token counts appear in both `gen_ai.usage.reasoning.output_tokens` and `gen_ai.usage.reasoning_tokens`. Use one of them. Adding the two double-counts.

### Cached input tokens

Available in:
- [ ] Node.js
- [x] Python

LLM request spans report token usage with the `gen_ai.usage.*` attributes from the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/). The `gen_ai.usage.cache_read.input_tokens` attribute reports the input tokens served from the prompt cache.

> ℹ️ **Cache reads are included in input token counts**
> 
> Per the OpenTelemetry GenAI semantic conventions, `gen_ai.usage.input_tokens` includes tokens reported separately as `gen_ai.usage.cache_read.input_tokens`. Adding the two values double-counts cached tokens. To get the number of input tokens not served from cache, subtract `gen_ai.usage.cache_read.input_tokens` from `gen_ai.usage.input_tokens`.

## Strip PII from exported traces

By default, exporters receive the full span, including conversation content, tool payloads, and participant data. To withhold that content from your own backend, pass `allow_pii=False` (Python) or `allowPii: false` (Node.js) when you set the tracer provider. A GenAI-aware backend can't display the conversation after you do this, because it never receives it.

**Python**:

```python
set_tracer_provider(trace_provider, metadata=metadata, allow_pii=False)

```

---

**Node.js**:

```typescript
telemetry.setTracerProvider(traceProvider, {
  metadata,
  registerSpanProcessor: (processor) => fanout.add(processor),
  allowPii: false,
});

```

The SDK installs the stripping through `registerSpanProcessor`. If you omit it, the SDK logs a warning and your exporters receive the full span.

If you don't call `set_tracer_provider` and the SDK adopts the global OpenTelemetry provider instead, set the `LIVEKIT_TELEMETRY_ALLOW_PII` environment variable to `0`. An explicit argument takes precedence over the environment variable.

This option doesn't change what LiveKit Cloud receives. The SDK exporter for [Agent insights](https://docs.livekit.io/testing/observability/insights.md) still gets the full span, and your project's [PII redaction](https://docs.livekit.io/testing/observability/pii-redaction.md) setting governs what LiveKit Cloud stores.

### What the SDK removes

With `allow_pii=False`, the SDK strips the following from each span before any exporter on the provider sees it:

- Attributes with a `pii` segment in the key, including all [content attributes](#content-attributes) and any [attributes you tag yourself](https://docs.livekit.io/testing/observability/pii-redaction.md#tag-your-own-attributes).
- GenAI content attributes: `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`, `gen_ai.tool.description`, `gen_ai.tool.definitions`, and `gen_ai.prompt.variable.*`.
- GenAI message events, such as `gen_ai.choice` and `gen_ai.user.message`.
- Exception details. `exception.message` and the error status of the span become `exception details redacted`, and `exception.stacktrace` is dropped.

Attributes that don't carry content remain, including `gen_ai.usage.*` token counts, model names, provider names, durations, and `lk.speech_id`. The SDK removes whole attributes. It doesn't scan or mask values.

> ℹ️ **Spans only**
> 
> `allow_pii` applies to spans. Log attributes are filtered only when [PII redaction](https://docs.livekit.io/testing/observability/pii-redaction.md) is turned on.

> ℹ️ **Project redaction applies to every exporter**
> 
> When your project or session has [PII redaction](https://docs.livekit.io/testing/observability/pii-redaction.md) turned on, the SDK removes the same fields from every exporter, including the exporter for LiveKit Cloud. Passing `allow_pii=True` doesn't override this.

### Turn off content capture

To remove `gen_ai.*` content attributes from all spans, including those sent to LiveKit Cloud, set the standard OpenTelemetry environment variable `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `false`. You can also call `telemetry.gen_ai.set_capture_content(False)` (Python) or `telemetry.genAI.setCaptureContent(false)` (Node.js) before the session starts. This affects only the `gen_ai.*` content attributes. The `lk.pii.*` attributes are still recorded, so use `allow_pii` to strip those.

## Other backends

The preceding pattern works for any backend that accepts OpenTelemetry traces over OTLP. To export elsewhere, point the exporter at the OTLP endpoint for that backend and set the authentication it requires:

- `OTEL_EXPORTER_OTLP_ENDPOINT`: The OTLP HTTP endpoint for the backend.
- `OTEL_EXPORTER_OTLP_HEADERS`: Any authentication headers the backend requires, such as an API key.

The rest of the agent stays the same: build a tracer provider, add a batch span processor with an OTLP exporter, and pass the provider to `set_tracer_provider` (Python) or `telemetry.setTracerProvider` (Node.js) before the session starts.

In Node.js you can also pass `url` and `headers` to the exporter instead of setting environment variables, as the preceding example does. The `url` option is used as-is, while `OTEL_EXPORTER_OTLP_ENDPOINT` has `/v1/traces` appended to it.

## Additional resources

- **[Build your own observability](https://docs.livekit.io/testing/observability/build.md)**: Collect session data with the Agents SDK or LiveKit Egress and send it to systems you run.

- **[Agent insights](https://docs.livekit.io/testing/observability/insights.md)**: View transcripts, traces, logs, and audio recordings in LiveKit Cloud.

- **[Data hooks](https://docs.livekit.io/testing/observability/data.md)**: Collect metrics, session reports, and recordings within the Agents SDK.

---

This document was rendered at 2026-10-31T04:02:50.888Z.
For the latest version of this document, see [https://docs.livekit.io/testing/observability/tracing.md](https://docs.livekit.io/testing/observability/tracing.md).

To explore all LiveKit documentation, see [llms.txt](https://docs.livekit.io/llms.txt).