Automate daily AI agent error triage with Claude Routines and Sentry
A Claude Routine reads your agent's overnight conversations, flags tool errors and reasoning failures, and files tickets before you've had your coffee.
Before you start
Accounts & access
- Sentry account with a project running an AI agent
- Claude account with access to Claude Routines
- Linear account (or another project tracker) for filing tickets
SDKs & packages
- A Sentry SDK with Agent Tracing enabled in your AI agent's codebase
Knowledge
- Basic familiarity with AI agent architectures (tool calls, multi-turn conversations)
- Comfort writing prompts for LLMs
1 Enable Agent Tracing for your AI agent
Before you can triage your agent's behavior, Sentry needs to capture it. Set up Agent Tracing in your agent's codebase so every conversation, tool call, and model interaction is recorded as a trace.
Sentry has Agent Tracing guides for Python, Node.js, Next.js, React Native, and more. Pick the one that matches your stack and follow the setup instructions. Set tracesSampleRate high enough to capture the conversations you care about. For triage, you want comprehensive coverage. If you're working at higher volumes, Sentry's sampling strategies let you keep costs down while still capturing enough data to surface errors reliably.
2 Verify conversations appear in Sentry
Once your agent has handled a few conversations, open Explore > Agents in Sentry. You should see your agent's conversations listed with duration, message counts, errors, cost, and tool calls.
Each conversation shows user inputs, model outputs, tool calls, errors, latency, and cost. This is the data your Claude Routine will query.
3 Configure the Sentry MCP connector in Claude
Your Claude Routine needs to authenticate with Sentry to query your agent's traces. Set up the Sentry MCP connector in Claude so the routine can call Sentry APIs directly.
If you want the routine to file tickets automatically, enable the Linear MCP connector (or whichever project tracker you use). The routine will search for existing tickets before creating new ones to avoid duplicates.
Sentry MCP server documentation
4 Create the Claude Routine with a triage prompt
Create a new Claude Routine and set it to run daily (mornings work well since the routine analyzes overnight activity). The prompt should tell Claude to use the Sentry MCP to pull aggregate stats, sample conversations with and without errors, and file tickets for new findings.
Keep the prompt intentionally plain. The Sentry MCP handles the data retrieval. Let the LLM handle the judgment: deciding whether a pattern is new, whether an error is self-correcting, and whether a conversation's conclusion actually follows from its reasoning.
Use the Sentry MCP to look at the last 24hr of <product>
conversations and analyze the results. In particular look for errors
that might be happening during the <product_stage>, say if our tools
are broken. But also sample some success conversations as well to see
if they look correct with secondary inspection.
If you come across new findings/errors, record them in a Linear ticket
for the <linear_project> project. 5 Review and tune the routine's output
After the first few runs, review what the routine flags. A good triage routine catches the following categories of problems.
That last category, reasoning failures, is why sampling successful conversations matters. A conversation can complete without a single tool error and still reach the wrong conclusion. That only shows up when something checks the reasoning against the verdict.
- Tool call failures: tools returning errors, especially at higher-than-normal rates
- Hallucinated inputs: the agent guessing at parameters (like repo names or project slugs) instead of looking them up
- Latency anomalies: conversations with unusually long hangs before an error, which may signal backend issues distinct from the agent's own mistakes
- Agent loops: conversations with far more tool calls than typical, suggesting the agent is thrashing
- Cost anomalies: runs that are significantly more expensive than the baseline
- Reasoning failures: conversations that completed without tool errors but reached the wrong conclusion
Analyzed the last 24h of Seer conversations (~445-551 total,
~11k tool calls). No verdict-quality problems found. Spot-checked
several conversations and the agent's reasoning was well-grounded,
correctly downgrading when it lacked repo access rather than
fabricating.
Did find an efficiency issue worth tracking: ~21% of conversations
(83/400 sampled) hit at least one tool error, mostly self-corrected
retries. Root causes:
- Search Code sometimes gets fed malformed repo_name values
(e.g. appending a monorepo subpath onto the real slug, like
"foo/bar/applications/integrations" instead of "foo/bar"),
causing "repository not found" until it retries.
- <Tool> sometimes gets a guessed-wrong project_slug (e.g.
"mobile") before self-correcting on retry with the right
slug + time window.
- Two conversations showed 30-56s hangs before an opaque
"internal error", a real backend latency signal, distinct
from the guessing pattern above, worth an engineering look.
Filed Linear ticket AIML-123 with full details, example
conversation links, and error-rate tables.That's it.
Your agent's overnight errors are already triaged.
By the time you sit down with coffee, a Claude Routine has read your agent's conversations, flagged the broken ones, and filed tickets for anything new.
- Set up Agent Tracing so every AI agent conversation lands in Sentry with tool calls, errors, and latency data
- Connected a Claude Routine to Sentry via the MCP connector for automated querying
- Built a daily triage routine that pulls aggregate stats, samples conversations, and files tickets for new findings
- Tuned the routine to catch tool failures, hallucinated outputs, agent loops, and cost anomalies
Pro tips
- 💡 Keep the routine's prompt short and focused on judgment. The Sentry MCP handles data retrieval, so the prompt only needs to describe what to look for and where to file findings.
- 💡 Have the routine search your project tracker for existing tickets before filing new ones. This prevents duplicates for known issues.
- 💡 Sample both errored and successful conversations. Reasoning failures only surface when you check whether the agent's conclusion actually follows from its work.
- 💡 Sort sampled conversations by tool call count or token usage. The heaviest runs are where agents thrash and reveal the most about failure modes.
Common pitfalls
- ⚠️ Only looking at conversations with tool errors. A conversation can complete cleanly and still reach the wrong conclusion.
- ⚠️ Hardcoding org-specific details in the routine prompt instead of using variables. This makes the routine harder to share or adapt.
- ⚠️ Writing overly detailed prompts that try to specify exact MCP queries. Let the LLM figure out how to query Sentry. Focus the prompt on what to look for, not how to look for it.
- ⚠️ Not searching for existing tickets before filing. The routine will create duplicates for issues that are already tracked.
Frequently asked questions
What's next?
Fix it, don't observe it.
Get started with the only application monitoring platform that empowers developers to fix application problems without compromising on velocity.