AI, LLM, and Agent Observability

Agents, LLMs, vector stores, custom logic: visibility can’t stop at the model call.

Trace every agent run end to end and get the context you need to debug failures, optimize performance, and keep AI features reliable.

Try Sentry For Free Get a Demo

Tolerated by 4 million developers

Full LLM Observability

See everything on one pane of glass

Track all agent runs, error rates, LLM calls, tokens used, and tool executions. Monitor traffic patterns and duration metrics across your AI-powered features.

See Docs

Track Model Costs & Tokens

Monitor spending across models.

Compare costs across different models. See token usage breakdown by model, track input vs output tokens, and identify expensive operations.

See Docs

Agent tracing, end to end

Trace every agent run, end to end.

AI agents fail across a chain of steps, not a single line of code. Agent tracing stitches every LLM call, tool execution, and handoff into one connected trace, with full prompt and response context, token counts, cost, and timing on every span. Follow the trace to the exact tool or step that went wrong, and apply agent tracing best practices to keep runs reliable as they scale.

Agent tracing best practices

Getting started with Sentry is simple

We support every technology (except the ones we don't).
Get started with just a few lines of code.

Install sentry-sdk from PyPI:

Bash

pip install "sentry-sdk"

Add OpenAIAgentsIntegration() to your integrations list:

Python

import sentry_sdk
from sentry_sdk.integrations.openai_agents import OpenAIAgentsIntegration

sentry_sdk.init(
    # Configure your DSN
    dsn="https://examplePublicKey@o0.ingest.sentry.io/0",
    # Add data like inputs and responses to/from LLMs and tools;
    # see https://docs.sentry.io/platforms/python/data-management/data-collected/ for more info
    send_default_pii=True,
    integrations=[
        OpenAIAgentsIntegration(),
    ],
)

The vercelAIIntegration adds instrumentation for the ai SDK by Vercel to capture spans using the AI SDK's built-in Telemetry. Get started with the following snippet:

JavaScript

Sentry.init({
  // Configure your DSN
  dsn: 'https://<key>@sentry.io/<project>',
  tracesSampleRate: 1.0,
  integrations: [
    Sentry.vercelAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

To correctly capture spans, pass the experimental_telemetry object with isEnabled: true to every generateText, generateObject, and streamText function call.

JavaScript

const result = await generateText({
  model: openai("gpt-4o"),
  experimental_telemetry: {
    isEnabled: true,
  },
});

"Sentry played a significant role in helping us develop [Claude] Sonnet"

Since adopting Sentry, Anthropic has seen:

10-15%

increase in developer productivity

600+

engineers rely on Sentry to ship code

20-30%

faster incident resolution

What is AI observability?

AI observability is the practice of understanding what your AI features are actually doing in production, across LLM calls, agent runs, tool executions, and the application code around them. Instead of treating the model as a black box, it connects prompts, responses, token usage, latency, errors, and cost to the rest of your application's traces and logs, so you can debug failures down to the specific call that caused them.

What is LLM observability, and how is it different from AI observability?

LLM observability focuses specifically on the behavior of large language model calls: prompts, completions, token counts, model latency, and cost. AI observability is broader: it covers LLM calls plus the agents, tools, retrieval steps, and custom logic that wrap them. In practice you want both, which is why Sentry tracks individual LLM calls and stitches them into full agent traces in one place.

What's the difference between AI monitoring and AI observability?

What is agent tracing?

Agent tracing follows a single AI agent run from start to finish as a connected trace, capturing every LLM call, tool execution, and handoff as a span with its inputs, outputs, token counts, cost, and timing. Because AI agents make decisions across many steps, a flat log isn't enough; agent tracing shows you the full execution path so you can see exactly which step failed or stalled. See how agent tracing works in Sentry.

How do I get observability for AI agents?

Can I monitor and trace MCP servers?

Is AI observability included in Sentry's free plan?

Fix what's broken with LLM Observability

Get started with the only LLM observability platform that gives developers tools to fix application problems without compromising on velocity.

Try Sentry For Free Get a Demo

Solutions

Products