Reducing Time-to-Resolution from Days to Hours with Custom Error Context

Testmo is a unified software testing solution that lets engineering teams ship better software by managing test cases, exploratory testing sessions and test automation integration from a single platform. Building a tool designed for software testers carries high user expectations, and without error context, the team faced challenges monitoring and improving the quality of their applications.

Read on to learn how enriching errors with context and improving their workflow with out-of-the-box and custom instrumentation they:

• Reduced time to resolution from days to hours
• Significantly reduced user-facing errors
• Improved developer workflow and triaging
• Improved their CRM by proactively monitoring and resolving issues

We’ve significantly reduced the time between discovery and resolution of new issues. In many cases we’ve reduced time to resolution from days to hours. Dennis Gurock, CTO & Co-Founder, Testmo.

Errors without context are just… logs

Testmo offers a low-friction, low-touch solution for engineering teams of all sizes with a high bar for UX, application availability, performance and stability. In many cases, users who encounter issues when trying out new software don’t report them, they either find a workaround – not ideal – or, worst case scenario, simply give up.

The team originally relied on extensive logging tools and processes to review and report issues, but found that most of these solutions and application logs lacked the context needed to enrich errors. Compounding that, they couldn’t easily replicate errors locally and found it difficult to work their way back from an error notification to see the steps that led up to the issue for quick resolution.

They also lacked the visibility needed to proactively monitor and improve the quality of their applications and risked missing issues customers might run into.

A lightweight solution to identify and fix issues in production

Straight off the bat Testmo knew they needed a SaaS solution with SDKs supporting Laravel, PHP and JavaScript, and instrumentation to monitor deployments to their Docker-based environments. Beyond that, whatever they chose had to integrate well with their business-critical systems, marketing site, lead/trial registration process, internal portal and billing services.

We wanted to cover everything that our users and team members interact with to ensure an efficient and smooth user experience without surprises.

Dennis’ team built their platform with an extensive testing strategy, including backend, end-to-end and unit testing. Ultimately though, experience taught them that testing alone couldn’t account for the subtle but impactful differences between testing, staging and production environments, and various user configurations.

They needed a lightweight solution with automatic and custom instrumentation that could support a small but mighty team with the tools to proactively monitor and resolve issues from the front-to-the backend. Finally, and simply put, Testmo wanted a vendor with a “great reputation among developers” and open-source code to support them as they grew.

We researched various error monitoring solutions and Sentry provided the best experience out-of-the-box, offered great integrations with our existing stack and was easy to customize with important context.

Monitoring across testing, staging and production environments with rich context

Sentry’s out-of-the-box integrations meant that Testmo could deploy and start monitoring quickly. On the frontend they implemented Sentry’s JavaScript SDK alongside their custom frameworks and, using tags, added detailed context to errors. On the backend and at the API layer, PHP and Laravel support let them enrich every event with the stack trace and original component attached to an issue.

We didn’t have to make many changes in order to integrate Sentry into our codebase. Language and framework support is comprehensive and well-documented and we were able to quickly start monitoring our testing, staging and production environments.

Alerts help them stay on top of the volume and frequency of new errors and with Sentry’s Slack integration the relevant channels are notified of new issues in a platform purpose-built for collaboration. Once an issue is assigned, breadcrumbs let them replicate errors in their staging and development environments, working their way back from the point that an error occurred.

For more context, Testmo integrates sourcemaps from their CI pipeline to view source code context obtained from stack traces; and use custom tags to configure the level of details included in breadcrumbs and error logs depending on the log level of the application, customer tenant and environment.

This tailored workflow gives the team:

• Visibility at the code level with detailed stack traces, even when using transpiled frontend code
• A clear timeline of the actions leading up to an error with breadcrumbs and,
• Rich context to quickly figure out where in their environment the bug was introduced

Customizing how they use Sentry while letting out-of-the-box solutions take care of the grunt work has helped Dennis’ team take on difficult to diagnose issues and today they’re able to find, replicate and resolve errors that none of their other reporting and logging tools have been able to detect. They’ve even noticed a shift in sales and CRM, since they’re proactively monitoring and resolving issues before a user reports them.

By using Sentry we have been able to significantly reduce the time between discovery and resolution of new problems. In many cases, we are now able to resolve issues within hours, not days. Sentry’s full stack traces, breadcrumbs and enriching errors with tags have proven invaluable.

Putting context to work

Testmo relies heavily on breadcrumbs, and the context they’ve been able to attach to errors has already paid off. The following event trail shows how a user opened the Delete dialog to remove an entry, before confirming the dialog by clicking the submit button. The frontend code correctly sent a request to delete the entry, refreshed the view’s fragment and updated the browser’s location:

Next, though, the code sent another delete request for the same entry, confirmed by their API with a 404 error. Reviewing the stack trace they figured out that when hiding the delete dialog after the successful action, there was a brief period when the user could – and did – click the button again during the dialog’s hide animation.

A streamlined workflow and ability to enrich errors with custom tags set the groundwork for them to quickly filter out the log level, customer tenant and environment in which this particular issue occurred.

Testmo’s true north is to provide a robust, stable and fast application. Sentry has helped them tailor a workflow that makes sense for their team, with fit for purpose instrumentation and the tools to enrich errors with context.

In addition to our extensive testing strategy, Sentry is a key component for us to monitor and improve the quality of our applications and forms an integral part of our software quality strategy to ensure that users have a great experience using our product.

Key results:

• Reduced time to resolution from days to hours
• Significantly reduced user-facing errors
• Improved CRM by proactively monitoring and resolving issues
• Improved developer workflow and triaging process

More about Testmo

You’ve seen how Dennis’ team uses Sentry for code-level observability and monitoring the health of their applications. Now check out how Testmo helps engineering teams ship better software with a unified solution for managing test cases, sessions and automation.