Volley’s Non-Trivial Pursuit of a Smooth User Experience

Volley is the leading creator of voice-controlled entertainment for Amazon Alexa and Google Home. With games like Song Quiz, Yes Sire, and Are You Smarter Than a 5th Grader?, Volley entertains millions of families on smart speaker devices every month.

With all those voices blurting out answers at their voice-controlled devices, Volley engineers knew that they needed complete visibility into their code. The only way their customers could find out if they’re smarter than fifth graders was if the Volley team was smarter than their code’s errors.

The Challenge

Even though Song Quiz was already successful, Volley’s team of engineers wanted to rebuild Song Quiz from the ground up. They also wanted to implement a new voice engine that would make iteration easier and faster throughout their codebase. This meant corralling many different technologies being used in many different ways, including AWS, Phaser.js, and Node.js with Typescript.

But after pushing this new code, the Volley team were left scratching their heads as they noticed dipping Apdex scores, spiking memory, and increasing transaction times. That was when they turned to Performance, our client-side performance monitoring solution. After taking a look at their transaction summary, the team quickly realized that LaunchRequest (which initializes the voice skill) was one of their slowest interactions.

The Process

Using our Express SDK, the Volley team observed that LaunchRequest was waiting for an API call to update the user’s subscription status. It quickly became evident that the API call to check the user’s subscription status was responsible for much of this delay. Now that they could clearly see what was going on, it was clear what they needed to do next: move the subscription status check to those sections of the code which needed it.

After removing the subscription check from the initial launch code, the team revelled in seeing how many user interactions sped up — LaunchRequest in particular.

The Results Are In

Once Volley made this one simple change in production, they noticed:

A. Steady and high APDEX B. Faster user interactions C. Cutting time to launch in half

After seeing these results, the Volley team transitioned from using Performance as a reactive service that flags errors to proactive one which lets them get out in front of potential issues. They began defining a series of metric alerts so that they would be notified of any performance issues. They used custom messages to get a detailed look at how Song Quiz was performing on user devices so they could continue to deliver great user experiences — even during peak hours. And they used Discover to query longer-term trends in performance. For Volley, Sentry went from a break-glass-in-case-of-emergency device to a core tool of their daily operations.

Sentry has become part of my daily routine. Every morning, I check to see if there are any new issues, and look at yesterday’s performance to see if any of our latest changes affect it negatively or positively.

Devin Ozel, Sr. Software Engineer, Volley.

When it comes to solving a user’s experience, there’s no such thing as partial credit. It’s why Volley not only needed to solve the problems their code was throwing at them today, but also to identify possible performance opportunities in the future. And it’s why Sentry is the right answer to a question that’s always stumped developers: what is going on with my code?