Saving millions with real-time code observability

About Power Home Remodeling Group

While Power Home Remodeling sells windows, roofing, siding, doors, and solar solutions to homeowners, walking into their offices, you may confuse them with being a technology company. In business since 1992, Power has grown to over $700 million in revenue annually.

In 2008, the technical team first launched Nitro, the application powering their entire business. Today, it’s a thirteen-year-old monolith Ruby-on-Rails application designed to function as a Component-Based Rails Application, allowing developers to separate code into different product areas (sales, human resources, contact center, and many more). Nitro now has well over one-hundred components.

Since implementing Sentry, Power has seen the following benefits:

• Saved millions of dollars in revenue by using Discover to learn from past issues to lessen the impact of future service outages.
• Reduced time to triage by 1 hour/week per principal developer with Code Owners.
• Increased issue ownership and proactivity across development teams.

Seeing and solving issues before they become downtime

During the most active periods of the day, Nitro downtime can cost Power up to $1M an hour in lost revenue. While the engineering team at Power can’t afford errors, slowdowns, or crashes, the reality is that bugs happen, so they need to be able to find and fix them quickly. However, Power’s previous error monitoring solution provided limited notifications and filtering, resulting in monitoring delays.

Our process before [Sentry] was very manual. End users would contact our customer service desk highlighting an error, and then there was a lot of back-and-forth to figure out how to reproduce issues…the monitoring service couldn’t keep up with us.

Wade Winningham, Principal Developer

Integrating Sentry across teams to build a comprehensive application monitoring system

As the engineering team continued to grow, they not only required a real-time view of what was happening in their application but also needed a product that was innovating and could scale with their needs.

They chose Sentry because of its dynamic alerting, detailed issue context, breadth of integrations, and responsive customer support.

Sentry has quickly become an integral part of the Power workflow. It gives the development team real-time visibility into what’s really going on in Nitro while listening to feedback and building new products and features to improve how we monitor and take action on issues. Now, rarely is there an issue we don’t already know about or context we don’t have to go fix it.

Wade Winningham, Principal Developer

Today 12 different teams across engineering, QA, UX, and product–all with unique workflows and goals–use Sentry.

The flexibility of Sentry’s platform allows us to build different connection points for each workflow. Together, this builds a cohesive, comprehensive application monitoring system within Power Home Remodeling that would not be possible without Sentry.

Wade Winningham, Principal Developer

However, with so many different teams shipping code, they needed a better issue ownership solution. Enter the Sentry Code Owners GitHub integration.

Wade’s team was already using GitHub to help link components in Nitro to specific teams or individuals for code reviews. Before this feature, a smaller group of engineering leads would review the Issues list in Sentry to see the issues that occurred within 24 hours for anything active or urgent and assign unassigned items to specific teams.

However, if team assignments changed or were not updated, errors risked going unnoticed by developers for weeks.

With the Code Owners integration, the engineering leads didn’t have to worry about updating and mapping team assignments correctly, saving them 15-20 minutes a few times a week on issue management and saving developers countless hours troubleshooting issues not owned by them.

Sentry’s integration with GitHub’s Code Owners has streamlined how we assign issues to our developers and promotes autonomy amongst our Scrum teams. Before, we had a buildup of unassigned issues taking our principal developers 15-20 minutes a few times a week to assign and manage. Now, with Sentry’s Code Owners, issues are assigned instantly, removing these tasks from our principal developer’s workload and allowing our teams to self-manage. The bottom line is that we are fixing issues faster and keeping our error rates low.

Jenny Gray, Vice President of Application Development and Infrastructure

Troubleshooting issues and identifying patterns faster with Stats and Discover

Sentry is also a key component of driving cross-team discussions. Every two weeks, everyone in Power Home Remodeling’s Business Technology Department (developers, QA staff, SREs, executives) meets to learn how the team is doing, share new things in the industry, and get the latest department updates.

Wade includes Sentry charts to report on the time between meetings.

One chart is the billing summary from Stats which shows the number of events they’ve received for the current billing period.

If the chart has a steep incline upwards, it alerts them to investigate what’s driving the increase. Tracking their usage trends saves the team money and headaches from sharp usage shifts.

The other chart is the Top 5 Events by Volume built in Discover. This chart shines a spotlight on the recent, worst offenders, raising awareness and urgency to the broader group. These issues get reported on a departmental basis, and teams are encouraged not to stay on this list. Reviewing this chart during the meeting helps incentivize teams to take ownership of issues and solve them quickly.

The ‘top events by volume’ chart has been critical in our bi-weekly departmental meetings to raise awareness and urgency to issues across the engineering organization. Some of these can highlight problems with our codebase or reflect service outages where we can learn how those services impacted our code and then analyze how to lessen the impact of those outages in the future, which has saved us potentially millions.

Wade Winningham, Principal Developer

What’s next: Expanding Discover and Releases usage

Even with Sentry integrated so tightly across teams, Power plans to expand and apply more Sentry features into its development workflow.

The technology group has used Discover to focus on a few problem areas, such as drilling in to view affected users by issue as well as tracking the top issues over time. Next, Wade and his team plan to use more custom queries to help prioritize issues.

Last, as more teams deploy, Power plans to start using Releases. Releases will help them quickly detect bad releases and make rollback decisions, prioritize the issues to solve across versions, and track when a release is ready to roll out to a wider audience.

Results

Sentry has helped Power Home Remodeling fix issues faster and keep error rates low. This streamlined issue management workflow has removed buildup of unassigned issues and saved principal developers 15-20 minutes of triaging multiple times a week.

Power has uncovered key insights into codebase issues and future service outages. While this saves millions in revenue from potential outages, it also increases ownership and proactivity to fix issues across development teams.

Sentry has even helped the QA team identify and assign user-flagged issues quickly to help increase their awareness of an issue’s severity and impact to prioritize issues and keep the application running.