← Back to Cookbook

Monitor AI agent spend with dashboards and alerts

Turn standard GenAI spans into a shared spend dashboard, then detect fixed-budget breaches and unusual cost spikes.

Watch the video Jump to recipe
Features
AI Observability ,  Agent Tracing ,  Dashboards ,  Alerts ,  CLI
Category Monitoring
Time
20 minutes
Complexity
Intermediate
Steps
5 steps

Before you start

Telemetry
  • A Sentry project receiving AI agent spans with standard gen_ai.* attributes
  • At least a few model calls with token data
Accounts & access
  • Permission to create dashboards and detectors in your Sentry organization
  • A Sentry auth token for the CLI
Tools
  • The Sentry CLI installed
  • Bash, curl, and Python 3

1
Check the fields that drive the dashboard

Open Trace Explorer and inspect a model-call span. Confirm that it contains gen_ai.operation.type:ai_client, gen_ai.usage.total_tokens, and gen_ai.response.model. Add user.id and user.username in your instrumentation if you want per-user widgets, and set gen_ai.conversation.id if you want the most expensive conversations table. The dashboard filters to ai_client spans because one turn can carry cost on both the model call and its parent agent span. Summing both would count the same cost twice.

Sentry AI agent monitoring

2
Install and authenticate the Sentry CLI

Install the current Sentry CLI, then authenticate it. sentry login stores the token in the CLI configuration so the dashboard script can call your organization APIs.

Sentry CLI authentication
Bash
curl -sL https://sentry.io/get-cli/ | sh
sentry login

3
Create the dashboard and detectors

Download the reusable script from Sentry's agent tracing examples, make it executable, and pass your organization and project slugs. The script creates one dashboard and two detectors. It stops before making changes when items with the same names already exist, and it removes anything from the current run if a later API call fails.

Dashboard script source
Bash
curl -O https://raw.githubusercontent.com/getsentry/sentry-agent-tracing-examples/main/dashboards/llm-spend-per-user.sh
chmod +x llm-spend-per-user.sh
./llm-spend-per-user.sh <org-slug> <project-slug>

4
Review the spend dashboard

Open the URL that the script prints, or select LLM Spend per User in Dashboards. If the user or conversation widgets are empty, add user.id, user.username, or gen_ai.conversation.id to your model-call spans.

Sentry Dashboards
The LLM Spend per User dashboard in Sentry with total spend, tokens, active users, spend charts, and top spenders

5
Tune the spend monitors

Open Monitors and adjust LLM spend rate high and LLM spend anomaly for your traffic and budget. To route notifications during creation, set WORKFLOW_ID and, optionally, OWNER=user: or OWNER=team: before you run the script. Watch the alert workflow in the demo video.

Sentry Monitors and Alerts
Sentry Monitors filtered to the LLM spend rate and anomaly Metric monitors

That's it.

Agent spend is visible before it becomes a surprise.

Your team can now see who and what drives AI cost, then respond when hourly spend crosses a limit or leaves its normal range.

  • Checked that your AI spans contain the fields used by the dashboard
  • Created eight spend and usage widgets with the Sentry CLI
  • Avoided double-counting cost from parent agent spans
  • Created static and anomaly spend detectors

Pro tips

  • 💡 Start with the included thresholds in a test project, then replace them with limits that match your model mix and traffic.
  • 💡 Set user.id to a stable internal identifier and user.username to a useful display value. Do not use email addresses unless your data policy allows them.
  • 💡 Use TITLE to give a second dashboard a distinct name when you need separate views for environments or teams.
  • 💡 Add an alert workflow so a detector routes to the team that can reduce spend or stop a faulty agent.

Common pitfalls

  • ⚠️ Removing the gen_ai.operation.type:ai_client filter from cost or token widgets. Parent agent spans can repeat the model-call values and double the totals.
  • ⚠️ Assuming Sentry receives a dollar value from the SDK. Sentry derives gen_ai.cost.total_tokens from token counts and its model price list.
  • ⚠️ Running the script again without deleting or renaming the existing dashboard and detectors. The duplicate-name guard stops the run on purpose.
  • ⚠️ Treating the starting thresholds as production defaults. A suitable hourly limit depends on your traffic, models, and budget.

Frequently asked questions

No. It works with any Sentry project that receives the standard GenAI span attributes used by the queries.

That is the current Sentry field name for derived model cost. Its value is in US dollars, not tokens.

Your model-call spans need user.id. The top-spenders and conversation tables also use user.username and gen_ai.conversation.id.

Yes. Connect an existing alert workflow with WORKFLOW_ID, and set OWNER to a Sentry user or team when you create the detectors.

What's next?

Instrument a Flue agent

Send Flue conversations, tools, logs, and terminal failures to Sentry.
Open the Flue recipe →

Instrument an Eve agent

Send Eve conversations and tool calls to Sentry over OTLP.
Open the Eve recipe →

Browse the agent examples

See complete Flue, Eve, and dashboard examples in one repository.
Open the examples on GitHub →

Fix it, don't observe it.

Get started with the only application monitoring platform that empowers developers to fix application problems without compromising on velocity.

Try Sentry for Free Explore our Sandbox