← Back to Cookbook

Use production AI agent conversations in your evals

Your users are already generating the best eval dataset you'll ever have. Export their real conversations from Sentry and score them in the eval tool of your choice.

Jump to recipe
Features
AI Observability ,  Tracing ,  CLI ,  Sentry MCP
SDKs
Next.js ,  Node.js
Category Workflow
Time
30-45 minutes
Difficulty
Advanced
Steps
8 steps

Before you start

SDKs & packages
Accounts & access
  • Sentry account with tracing enabled on your project
  • An eval platform account — this recipe uses Braintrust, and step 8 covers open-source and local alternatives
Knowledge
  • Basic familiarity with how evals work (dataset → task → scorer)
  • Comfort running TypeScript scripts (tsx or similar)

1
Instrument your AI agent with Sentry

If your agent isn't reporting to Sentry yet, add the SDK and enable AI agent monitoring. With the Vercel AI SDK, add vercelAIIntegration to your server config and make sure tracesSampleRate is above zero. Set recordInputs and recordOutputs to true — the whole point of this recipe is capturing what users actually said and what the model answered, so opt out only on routes that handle sensitive data.

Sentry also has dedicated integrations for OpenAI, Anthropic, LangChain, and other libraries if you're not on the Vercel AI SDK.

AI agent monitoring for Next.js
TypeScript
import * as Sentry from '@sentry/nextjs';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  tracesSampleRate: 1.0,
  integrations: [
    Sentry.vercelAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

2
Tag every conversation with a conversation ID

A chat conversation spans many requests, and each request produces its own trace. Calling Sentry.setConversationId() in your chat route stamps every AI span from that request with gen_ai.conversation.id, so Sentry can group multi-turn conversations together — and so you can export them as complete conversations instead of disconnected fragments.

Use the same ID your frontend already threads through the chat (most chat SDKs, including the Vercel AI SDK's useChat, send one).

AI Conversations documentation
TypeScript
import * as Sentry from '@sentry/nextjs';
import { streamText } from 'ai';

export async function POST(req: Request) {
  const { messages, id: conversationId } = await req.json();

  // Groups every AI span from this request under one conversation
  Sentry.setConversationId(conversationId);

  const result = streamText({
    model: yourModel,
    messages,
    experimental_telemetry: { isEnabled: true },
  });

  return result.toUIMessageStreamResponse();
}

3
Explore your conversations in Sentry

Once traffic flows, open Explore → Conversations in Sentry. Each conversation shows the user's inputs, the model's outputs, which model was used, cost, token counts, and every tool call — and because it's all trace-connected, you can zoom out to the full waterfall from page request to API route to database queries.

This view is your dataset browser: skim it to understand what users actually ask, which conversations thrash, and which models they were routed to. That's the data your evals should run on.

AI Conversations documentation
Sentry AI Conversations view showing a table of production conversations with user, first input, LLM call count, tool calls, errors, cost per conversation, and last message time

4
Export untruncated spans with the Sentry CLI

The Sentry Events API exposes every gen_ai span with its full payloads — including complete gen_ai.request.messages and gen_ai.response.text. The sentry api command handles auth for you and works against any /api/0/ path.

First list conversation IDs (sorted by token volume — the heaviest conversations are the most interesting eval cases), then fetch each conversation's spans. Pages cap at 100 rows, so paginate with cursor=0::0 — long conversations that exceed one page would otherwise silently lose their tool results and final answer.

The Sentry MCP server is great for exploring this data interactively from your agent, but its search results are truncated for context efficiency — use the Events API for the full payloads your dataset needs.

Sentry CLI api command reference
Bash
# List the most token-heavy conversations from the last 30 days
sentry api "/api/0/organizations/YOUR_ORG/events/?dataset=spans&project=YOUR_PROJECT_ID&statsPeriod=30d&query=has:gen_ai.conversation.id&field=gen_ai.conversation.id&field=sum(gen_ai.usage.total_tokens)&sort=-sum(gen_ai.usage.total_tokens)&per_page=50"

# Fetch all spans for one conversation (paginate past 100 with cursor=0:100:0)
sentry api "/api/0/organizations/YOUR_ORG/events/?dataset=spans&project=YOUR_PROJECT_ID&statsPeriod=30d&query=gen_ai.conversation.id:conv_abc123&field=span.op&field=span.status&field=span.duration&field=timestamp&field=gen_ai.request.model&field=gen_ai.request.messages&field=gen_ai.response.text&field=gen_ai.usage.total_tokens&field=gen_ai.cost.total_tokens&field=gen_ai.tool.name&sort=timestamp&per_page=100"

5
Assemble spans into eval-ready records

Stitch each conversation's spans into one record: input is the first user message (parsed from gen_ai.request.messages), output is the final assistant response, and everything else becomes metadata your scorers can use — model, tokens, cost, tool calls, tool errors, and a link back to the Sentry conversation.

One subtlety that matters: sum tokens and cost from gen_ai.chat and gen_ai.generate_content spans only. gen_ai.invoke_agent spans aggregate their children, so including them doubles every figure. Also filter out your own noise — seeded demo data and smoke tests — by conversation-ID convention before they poison the dataset.

AI agents span conventions
TypeScript
// Actual model-call spans. invoke_agent spans aggregate their
// children — counting them double-counts every token and dollar.
const CLIENT_OPS = new Set(['gen_ai.chat', 'gen_ai.generate_content']);

function assemble(conversationId: string, spans: SentrySpan[]) {
  const clientCalls = spans.filter((s) => CLIENT_OPS.has(s['span.op']));
  const tools = spans.filter((s) => s['span.op'] === 'gen_ai.execute_tool');
  const lastChat = clientCalls
    .filter((s) => s['gen_ai.response.text'])
    .sort((a, b) => Date.parse(a.timestamp) - Date.parse(b.timestamp))
    .at(-1);

  return {
    id: conversationId,
    input: firstUserMessage(spans), // parse gen_ai.request.messages
    output: responseText(lastChat), // parse gen_ai.response.text
    metadata: {
      model: lastChat?.['gen_ai.request.model'],
      total_tokens: sum(clientCalls, 'gen_ai.usage.total_tokens'),
      cost_usd: sum(clientCalls, 'gen_ai.cost.total_tokens'),
      tool_calls: tools.length,
      tool_errors: tools.filter(
        (s) => s['span.status'] && s['span.status'] !== 'ok',
      ).length,
      sentry_url: `https://YOUR_ORG.sentry.io/explore/conversations/${conversationId}/`,
    },
  };
}

6
Load the conversations into Braintrust

Push each assembled record into a Braintrust dataset. Setting id to the conversation ID makes inserts upserts — re-running the export syncs new conversations instead of duplicating old ones. Keep the production answer in metadata: your first experiment should score what the agent *actually said* in production, not a re-generation.

You can also log each conversation as a trace in Braintrust's logs so its clustering and topic features work on your production traffic.

Braintrust Dataset SDK reference
TypeScript
import { flush, initDataset } from 'braintrust';

const dataset = initDataset('my-project', {
  dataset: 'prod-conversations',
});

for (const conv of conversations) {
  dataset.insert({
    id: conv.id, // upsert key — re-runs overwrite, never duplicate
    input: conv.input,
    metadata: {
      ...conv.metadata,
      production_output: conv.output,
    },
  });
}

await flush();

7
Score real answers with scorers and a judge

Now run an experiment. The task is a passthrough that returns the production output, so you're scoring reality. Write deterministic scorers for anything you can verify with code — if your agent cites entities from your database, groundedness is a lookup, not a judgment call. Telemetry metadata gives you free scorers too: token efficiency and tool success rate come straight from the export.

Save the LLM-as-judge for the one dimension code can't reach — was the answer actually helpful? — and calibrate it periodically against human labels. When you're ready to compare models or prompts, swap the passthrough task for a live call to your agent and run the same dataset through it.

Braintrust TypeScript SDK reference
TypeScript
import { Eval, initDataset } from 'braintrust';

Eval('my-project', {
  data: initDataset('my-project', { dataset: 'prod-conversations' }),
  // Passthrough: score what the agent actually said in production
  task: async (input, hooks) =>
    String(hooks.metadata?.production_output ?? ''),
  scores: [
    // Deterministic: every entity the answer cites must exist in your DB
    async ({ output, metadata }) => ({
      name: 'grounded',
      score: await citationsExistInDb(output, metadata),
    }),
    // Free from telemetry: tight answers score 1, token thrash decays to 0
    ({ metadata }) => ({
      name: 'efficiency',
      score: Math.max(
        0,
        Math.min(1, (120_000 - Number(metadata?.total_tokens ?? 0)) / 105_000),
      ),
    }),
  ],
});

8
Swap in the eval stack that fits you

Nothing above is Braintrust-specific except step 6. The pipeline is three separable stages — extract (Sentry Events API), assemble (plain TypeScript), load (your tool's SDK) — so switching platforms means rewriting one function. Solid options:

The fastest way to adapt it: hand the spec to your coding agent. This prompt captures everything this recipe covered, including the traps.

  • Braintrust (cloud) — logs, datasets, and experiments with upserts via dataset.insert({ id }); what this recipe uses.
  • Langfuse (open source, self-hostable) — langfuse.dataset.createItem() per record, then dataset.runExperiment().
  • promptfoo (local-first CLI) — write records to a JSON tests file and run npx promptfoo eval; scorers are inline JavaScript assertions.
  • Arize Phoenix (open source, runs locally) — upload a DataFrame with client.datasets.create_dataset() and run experiments against it.
Markdown
Export my production AI conversations from Sentry into <EVAL TOOL> as an
eval dataset.

Extract: use the Sentry Events API through `sentry api` with dataset=spans.
List conversations with query "has:gen_ai.conversation.id", then fetch each
conversation's spans with query "gen_ai.conversation.id:<id>", requesting
span.op, span.status, span.duration, timestamp, gen_ai.request.model,
gen_ai.request.messages, gen_ai.response.text, gen_ai.usage.*,
gen_ai.cost.total_tokens, and gen_ai.tool.name. Paginate with
cursor=0:<offset>:0 — pages cap at 100 rows.

Assemble one record per conversation: input = first user message, output =
final assistant response, metadata = model, total tokens, cost, tool calls,
tool errors, and a link back to the Sentry conversation. Sum tokens and cost
from gen_ai.chat and gen_ai.generate_content spans only — invoke_agent spans
aggregate their children and double-count. Filter out test and seeded
conversations by ID convention.

Load: upsert into <EVAL TOOL> keyed on the conversation id so re-runs sync
instead of duplicating. Then write scorers: deterministic checks for anything
verifiable against my data (grounding is a lookup), and an LLM judge only
for genuinely subjective quality.

That's it.

Your evals run on real users now.

Every model comparison, prompt tweak, and cost decision is measured against what your users actually asked — not what an LLM guessed they might ask.

  • Instrumented an AI agent so every conversation lands in Sentry with model, cost, tokens, and tool calls
  • Exported untruncated gen_ai spans through the Sentry CLI and Events API
  • Assembled raw spans into eval-ready records with input, output, and telemetry metadata
  • Loaded production conversations into Braintrust as an upsert-safe dataset
  • Scored real answers with deterministic scorers plus an LLM judge for subjective quality
  • Adapted the same pipeline to Langfuse, promptfoo, or Arize Phoenix

Pro tips

  • 💡 Sort the export by total tokens descending — the heaviest conversations are where agents thrash, and they make the most revealing eval cases.
  • 💡 Tag records from telemetry alone (thrash, tool-error, repeated-tool-call) so you can slice experiments by failure mode without hand-labeling anything.
  • 💡 Promote a production answer to the dataset's expected field only after it passes your deterministic checks — verified-clean conversations become golden rows for regression testing.
  • 💡 Scorers are code, and code goes blind silently. Keep a self-test that plants known hallucinations and asserts they get caught, so a scorer regression fails loudly instead of inflating your scores.
  • 💡 Real and synthetic data aren't either/or: production conversations cover what users actually do, synthetic cases cover what they haven't done yet. Use both.

Common pitfalls

  • ⚠️ Summing tokens or cost across all gen_ai spans — gen_ai.invoke_agent spans aggregate their children, so including them doubles every figure. Verify your totals against the Sentry Conversations UI.
  • ⚠️ Ignoring pagination on the Events API — pages cap at 100 spans, and a long conversation's tail holds its tool results and real final answer.
  • ⚠️ Exporting your own noise — seeded demo data and smoke-test conversations will poison the dataset unless you filter them out by ID convention.
  • ⚠️ Using an LLM judge for things code can check — asking a judge "is this grounded?" invites a hallucinated verdict when the answer is a database lookup away.
  • ⚠️ Leaving recordInputs/recordOutputs on for routes that handle sensitive data — you're exporting these payloads to a third-party eval platform, so scrub PII before it leaves Sentry.

Frequently asked questions

No. Only the load stage is Braintrust-specific — the extract and assemble stages work with any target. Langfuse and Arize Phoenix are open source and self-hostable, and promptfoo runs entirely locally from a JSON file. Step 8 has the mapping for each.

Both work — they hit the same Sentry APIs. The MCP server is ideal for interactive exploration from a coding agent, but its responses are truncated for context efficiency. For bulk export with full message payloads, call the Events API through sentry api (or plain HTTP with an auth token).

Prompts and outputs are only captured when recordInputs/recordOutputs are enabled, and Sentry supports server-side data scrubbing before storage. Remember the export sends this data to your eval platform too — apply the same scrubbing standards there, or self-host the eval tool.

No — it complements them. Synthetic datasets are guesses about user behavior; production conversations are the ground truth of it. Real traffic tells you what to fix today, synthetic cases protect flows real users haven't exercised yet.

Fewer than you'd think. A few dozen real conversations sorted by token volume will surface your agent's actual failure modes — thrashing, tool errors, hallucinated citations — faster than hundreds of synthetic cases.

Yes, two ways. If your app already routes users to different models, the exported metadata includes the model per conversation — group your experiment by it. To test a model your users haven't touched, swap the passthrough task for a live call to your agent with the candidate model and run the same dataset.

What's next?

AI Solution

Monitor and trace AI agents, LLM calls, and tool executions in production. Catch errors, latency, token costs, and budget overruns with agent tracing and full trace context
Learn about AI Solution →

Monitor AI agent costs in Next.js

Set up the instrumentation side of this recipe in depth — token usage, per-model costs, and agent traces in Sentry.
Read the recipe →

Set up alerts on agent regressions

Alert on token spikes, tool error rates, or cost thresholds so you catch degradations between eval runs.
Go to Alerts →

Fix it, don't observe it.

Get started with the only application monitoring platform that empowers developers to fix application problems without compromising on velocity.

Try Sentry for Free Explore our Sandbox