# Build AI agent evals from production conversations

> Export real AI agent conversations from Sentry with the Sentry CLI and turn them into eval datasets in Braintrust, Langfuse, promptfoo, or Phoenix

**URL:** https://sentry.io/cookbook/agent-conversations-to-evals/

---

Export your users' real AI agent conversations from Sentry and turn them into eval datasets you score in Braintrust, Langfuse, promptfoo, or Phoenix. Real traffic becomes your ground-truth test set.

**Time:** 20-30 minutes | **Difficulty:** Intermediate

## What You'll Learn

- Instrumented an AI agent so every conversation lands in Sentry with model, cost, tokens, and tool calls

- Exported complete production conversations with one Sentry CLI command

- Mapped each conversation to an eval record with input, output, and telemetry metadata

- Loaded production conversations into Braintrust as an upsert-safe dataset

- Scored real answers with deterministic scorers plus an LLM judge for subjective quality

- Adapted the same pipeline to Langfuse, promptfoo, or Arize Phoenix

**Topics:** AI Observability, Tracing, CLI, Agent Tracing

**SDKs:** Next.js, Node.js

## Before you start

**SDKs & packages**

- An app with an AI agent or chat feature (this recipe uses a [Next.js app built with the Vercel AI SDK](https://github.com/getsentry/nextjs-conf-scheduler))

- Sentry JavaScript SDK 11 or later

- Vercel AI SDK 7 and Node.js 22+ (required by AI SDK 7)

- [Sentry CLI](https://cli.sentry.dev/) 0.44 or later, installed and authenticated (`sentry auth login`)

**Accounts & access**

- [Sentry account](https://sentry.io/signup/) with tracing enabled on your project

- An eval platform account. This recipe uses [Braintrust](https://www.braintrust.dev/), and step 8 covers open-source and local alternatives

**Knowledge**

- Basic familiarity with how evals work (dataset → task → scorer)

- Comfort running TypeScript scripts (`tsx` or similar)

## Steps

### 1. Instrument your AI agent with Sentry

If your agent isn't reporting to Sentry yet, add the SDK and enable tracing. In Sentry JavaScript SDK v11, `vercelAIIntegration` is enabled by default once `tracesSampleRate` is above zero, and it records inputs and outputs by default. That's what this recipe needs, because the whole point is capturing what users actually said and what the model answered. Opt out only on routes that handle sensitive data, with `telemetry: { recordInputs: false, recordOutputs: false }` on those AI SDK calls.

Sentry also has dedicated integrations for OpenAI, Anthropic, LangChain, and other libraries if you're not on the Vercel AI SDK.

```typescript
import * as Sentry from '@sentry/nextjs';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // Tracing must be on. vercelAIIntegration is enabled by default
  // and records inputs and outputs unless you opt out.
  tracesSampleRate: 1.0,
});
```

[AI agent monitoring for Next.js](https://docs.sentry.io/platforms/javascript/guides/nextjs/agent-tracing/)

### 2. Tag every conversation with a conversation ID

A chat conversation spans many requests, and each request produces its own trace. Calling `Sentry.setConversationId()` in your chat route stamps every AI span from that request with `gen_ai.conversation.id`, so Sentry can group multi-turn conversations together. It also lets you export them as complete conversations instead of disconnected fragments.

Use the same ID your frontend already threads through the chat (most chat SDKs, including the Vercel AI SDK's `useChat`, send one).

```typescript
import * as Sentry from '@sentry/nextjs';
import { streamText } from 'ai';

export async function POST(req: Request) {
  const { messages, id: conversationId } = await req.json();

  // Groups every AI span from this request under one conversation
  Sentry.setConversationId(conversationId);

  const result = streamText({
    model: yourModel,
    messages,
    telemetry: { functionId: 'chat-agent' }, // names the agent in Sentry
  });

  return result.toUIMessageStreamResponse();
}
```

[AI Conversations documentation](https://docs.sentry.io/product/agents/conversations/)

### 3. Explore your conversations in Sentry

Once traffic flows, open [Explore → Agents](https://sentry.io/orgredirect/organizations/:orgslug/explore/agents/) in Sentry and select the **Conversations** tab. Each conversation shows the user, the tools used, cost, and message count. Open one to read the full transcript of inputs, outputs, and tool calls. Because it's all trace-connected, you can follow the trace link out to the full waterfall from page request to API route to database queries.

This view is your dataset browser. Skim it to understand what users actually ask, which conversations thrash, and which models they were routed to. That's the data your evals should run on.

[AI Conversations documentation](https://docs.sentry.io/product/agents/conversations/)

### 4. Export conversations with the Sentry CLI

The `sentry agent-conversation list` command returns one row per conversation. Sentry computes the totals for you: LLM calls, tool calls, tool errors, tokens, and cost. With `--json`, each row also includes `firstInput` (the first user message) and `lastOutput` (the final assistant response) in full. Those are the input and output of an eval record, so you don't need to fetch raw spans or stitch them together.

By default, the command returns 25 conversations from the last 7 days. Set `--period` for a longer window and `--limit` for up to 1,000 conversations. Use `--fields` to keep only the fields you need. To read one conversation turn by turn, with the model and tool calls of each turn, use `sentry agent-conversation view`.

```bash
# Export up to 1,000 conversations from the last 30 days
sentry agent-conversation list YOUR_ORG \
  --period 30d \
  --limit 1000 \
  --json \
  --fields conversationId,title,firstInput,lastOutput,flow,llmCalls,toolCalls,toolErrors,toolNames,totalTokens,totalCost,webUrl \
  > conversations.json

# Read one conversation turn by turn: model, tokens, and tool calls per turn
sentry agent-conversation view YOUR_ORG/CONVERSATION_ID --json
```

[Sentry CLI agent-conversation command reference](https://cli.sentry.dev/commands/agent-conversation/)

### 5. Map conversations to eval records

Map each row to one record. `input` is `firstInput`, `output` is `lastOutput`, and the other fields become metadata your scorers can use: agent, tokens, cost, tool calls, tool errors, and a link back to the conversation in Sentry.

Drop rows where `firstInput` or `lastOutput` is `null`. Those conversations come from routes where input or output recording is off. Also drop your own noise, like seeded demo data and smoke tests, before it gets into the dataset. Then sort by `totalTokens` so the heaviest conversations come first. That is where agents thrash, and those conversations make the most useful eval cases.

```typescript
import { readFileSync } from 'node:fs';

type Conversation = {
  conversationId: string;
  title: string | null;
  firstInput: string | null;
  lastOutput: string | null;
  flow: string[]; // agent names
  llmCalls: number;
  toolCalls: number;
  toolErrors: number;
  toolNames: string[];
  totalTokens: number;
  totalCost: number;
  webUrl: string;
};

const { data } = JSON.parse(readFileSync('conversations.json', 'utf8')) as {
  data: Conversation[];
};

// Replace with your own convention for test and seeded conversations
const isTestTraffic = (id: string) => id.startsWith('test-');

const records = data
  .filter((c) => c.firstInput && c.lastOutput && !isTestTraffic(c.conversationId))
  .sort((a, b) => b.totalTokens - a.totalTokens)
  .map((c) => ({
    id: c.conversationId,
    input: c.firstInput,
    output: c.lastOutput,
    metadata: {
      title: c.title,
      agents: c.flow,
      llm_calls: c.llmCalls,
      tool_calls: c.toolCalls,
      tool_errors: c.toolErrors,
      tools: c.toolNames,
      total_tokens: c.totalTokens,
      cost_usd: c.totalCost,
      sentry_url: c.webUrl,
    },
  }));
```

[Conversation JSON fields](https://cli.sentry.dev/commands/agent-conversation/)

### 6. Load the conversations into Braintrust

Push each record into a Braintrust dataset. Setting `id` to the conversation ID makes inserts upserts, so re-running the export syncs new conversations instead of duplicating old ones. Keep the production answer in metadata. Your first experiment should score what the agent *actually said* in production, not a re-generation.

You can also log each conversation as a trace in Braintrust's logs so its clustering and topic features work on your production traffic.

```typescript
import { initDataset } from 'braintrust';

const dataset = initDataset({
  project: 'my-project',
  dataset: 'prod-conversations',
});

for (const record of records) {
  dataset.insert({
    id: record.id, // upsert key: re-runs overwrite, never duplicate
    input: record.input,
    metadata: { ...record.metadata, production_output: record.output },
  });
}

await dataset.flush();
```

[Braintrust TypeScript SDK reference](https://www.braintrust.dev/docs/sdks/typescript/api-reference)

### 7. Score real answers with scorers and a judge

Now run an experiment. The task is a passthrough that returns the production output, so you're scoring reality. Write deterministic scorers for anything you can verify with code. If your agent cites entities from your database, groundedness is a lookup, not a judgment call. Telemetry metadata gives you free scorers too: token efficiency and tool success rate come straight from the export.

Save the LLM-as-judge for the one dimension code can't reach (was the answer actually helpful?), and calibrate it periodically against human labels. When you're ready to compare models or prompts, swap the passthrough task for a live call to your agent and run the same dataset through it.

```typescript
import { Eval, initDataset } from 'braintrust';

// Tune to the token count of a normal conversation for your agent
const TOKEN_BUDGET = 15_000;

// Fields written by the export script
type ConversationMetadata = {
  production_output: string;
  total_tokens: number;
  tool_calls: number;
  tool_errors: number;
};

Eval<string, string, void, ConversationMetadata>('my-project', {
  data: initDataset({ project: 'my-project', dataset: 'prod-conversations' }),
  // Passthrough: score what the agent actually said in production
  task: async (input, hooks) => hooks.metadata.production_output,
  scores: [
    // Deterministic: every entity the answer cites must exist in your DB
    async ({ output, metadata }) => ({
      name: 'grounded',
      score: await citationsExistInDb(output, metadata),
    }),
    // From telemetry: 1 within budget, then decays as tokens grow
    ({ metadata }) => ({
      name: 'efficiency',
      score: Math.min(1, TOKEN_BUDGET / Math.max(1, metadata.total_tokens)),
    }),
    // From telemetry: share of tool calls that succeeded
    ({ metadata }) => ({
      name: 'tool_success',
      score: metadata.tool_calls ? 1 - metadata.tool_errors / metadata.tool_calls : 1,
    }),
  ],
});
```

[Braintrust TypeScript SDK quickstart](https://www.braintrust.dev/docs/sdks/typescript/quickstart)

### 8. Swap in the eval stack that fits you

Nothing above is Braintrust-specific except step 6. The pipeline has three separate stages: extract (`sentry agent-conversation list`), map (plain TypeScript), and load (your tool's SDK). To change platforms, you rewrite one function. Good options:

The fastest way to adapt it: give the spec to your coding agent. This prompt has everything this recipe covered, including the traps.

- **[Braintrust](https://www.braintrust.dev/)** (cloud): logs, datasets, and experiments with upserts via `dataset.insert({ id })`. This is what this recipe uses.

- **[Langfuse](https://langfuse.com/docs/evaluation/experiments/datasets)** (open source, self-hostable): `langfuse.dataset.createItem()` per record, then `dataset.runExperiment()`.

- **[promptfoo](https://www.promptfoo.dev/docs/configuration/test-cases/)** (local-first CLI): write records to a JSON tests file and run `npx promptfoo eval`. Scorers are inline JavaScript assertions.

- **[Arize Phoenix](https://arize.com/docs/phoenix/get-started/get-started-datasets-and-experiments)** (open source, runs locally): upload a DataFrame with `client.datasets.create_dataset()` and run experiments against it.

```markdown
Export my production AI conversations from Sentry into <EVAL TOOL> as an
eval dataset.

Extract: run `sentry agent-conversation list <org> --period 30d --limit 1000
--json` (Sentry CLI 0.44 or later). Each item in `data` is one conversation
with conversationId, title, firstInput, lastOutput, flow, llmCalls,
toolCalls, toolErrors, toolNames, totalTokens, totalCost, and webUrl.
Sentry computes the totals per conversation. Do not fetch or sum raw spans.

Map one record per conversation: id = conversationId, input = firstInput,
output = lastOutput, metadata = the other fields. Skip rows where firstInput
or lastOutput is null (recording was off). Filter out test and seeded
conversations by ID convention. Sort by totalTokens, descending. For
per-turn detail or the model of each turn, run
`sentry agent-conversation view <org>/<id> --json`.

Load: upsert into <EVAL TOOL> keyed on the conversation id so re-runs sync
instead of duplicating. Then write scorers: deterministic checks for anything
verifiable against my data (grounding is a lookup), and an LLM judge only
for genuinely subjective quality.
```

## Summary

**Your evals run on real users now.**

Every model comparison, prompt tweak, and cost decision is measured against what your users actually asked, not what an LLM guessed they might ask.

## Pro Tips

- Sort the export by total tokens descending. The heaviest conversations are where agents thrash, and they make the most revealing eval cases.

- Tag records from telemetry alone (`thrash`, `tool-error`, `repeated-tool-call`) so you can slice experiments by failure mode without hand-labeling anything.

- Promote a production answer to the dataset's `expected` field only after it passes your deterministic checks. Verified-clean conversations become golden rows for regression testing.

- Scorers are code, and code goes blind silently. Keep a self-test that plants known hallucinations and asserts they get caught, so a scorer regression fails loudly instead of inflating your scores.

- Real and synthetic data aren't either/or: production conversations cover what users actually do, synthetic cases cover what they haven't done yet. Use both.

## Common Pitfalls

- Summing tokens or cost from raw spans yourself. `gen_ai.invoke_agent` spans aggregate their children, so a naive sum doubles every figure. The CLI totals are already correct per conversation, so use them as they are.

- Exporting with the defaults. `sentry agent-conversation list` returns 25 conversations from the last 7 days unless you set `--limit` and `--period`.

- Treating `lastOutput` as the final answer in every case. It is the last assistant message Sentry recorded, so a long or interrupted run can end on a step in the middle. Use `sentry agent-conversation view` to check the turns of a conversation before you score it.

- Running an older Sentry CLI. The `agent-conversation` command needs version 0.44 or later.

- Exporting your own noise. Seeded demo data and smoke-test conversations will poison the dataset unless you filter them out by ID convention.

- Using an LLM judge for things code can check. Asking a judge "is this grounded?" invites a hallucinated verdict when the answer is a database lookup away.

- Recording inputs and outputs on routes that handle sensitive data. They're on by default, and you're exporting these payloads to a third-party eval platform, so opt out on those routes or scrub PII before it leaves Sentry.

## FAQ

**Do I have to use Braintrust?**

No. Only the load stage is Braintrust-specific. The extract and map stages work with any target. Langfuse and Arize Phoenix are open source and self-hostable, and promptfoo runs entirely locally from a JSON file. Step 8 has the mapping for each.

**Should I use the MCP server or the CLI?**

Both work, and they read the same Sentry data. The MCP server is good for interactive exploration from a coding agent, but it truncates responses to save context. For bulk export with full inputs and outputs, use `sentry agent-conversation list --json`.

**What about user privacy and PII?**

In Sentry JavaScript SDK v11, prompts and outputs are recorded by default. Turn them off globally with `dataCollection: { genAI: { inputs: false, outputs: false } }` in `Sentry.init()`, or per integration with `recordInputs: false` and `recordOutputs: false`. Sentry also supports server-side data scrubbing before storage. Remember the export sends this data to your eval platform too, so apply the same scrubbing standards there, or self-host the eval tool.

**Does this replace synthetic eval datasets?**

No, it complements them. Synthetic datasets are guesses about user behavior; production conversations are the ground truth of it. Real traffic tells you what to fix today, synthetic cases protect flows real users haven't exercised yet.

**How many conversations do I need before this is useful?**

Fewer than you'd think. A few dozen real conversations sorted by token volume will surface your agent's actual failure modes (thrashing, tool errors, hallucinated citations) faster than hundreds of synthetic cases.

**Can I compare different models with this?**

Yes, two ways. If your app already routes users to different models, get the model of each turn with `sentry agent-conversation view  --json`, add it to the record metadata, and group your experiment by it. To test a model your users haven't touched, swap the passthrough task for a live call to your agent with the candidate model and run the same dataset.

## Next Steps

- [solution/ai-observability](https://sentry.io/solutions/ai-observability/)

- [Monitor AI agent costs in Next.js](/cookbook/monitor-ai-agent-costs-nextjs/): Set up the instrumentation side of this recipe in depth: token usage, per-model costs, and agent traces in Sentry.

- [Set up alerts on agent regressions](https://sentry.io/orgredirect/organizations/:orgslug/monitors/alerts/): Alert on token spikes, tool error rates, or cost thresholds so you catch degradations between eval runs.

---

*Source: [sentry.io/cookbook/agent-conversations-to-evals/](https://sentry.io/cookbook/agent-conversations-to-evals/)*