How Optimizely uses Sentry
For the third edition of Customer Stories, we spoke to Travis Beck, a software engineer at Optimizely. Optimizely enables businesses to deliver continuous experimentation and personalization across websites, mobile apps, and connected devices. Marketers, developers, and product managers have delivered over 700 billion experiences tailored to the needs of their customers thanks to Optimizely.
(Editor’s note: Optimizely lets customers run experiments everywhere — across every device, every channel, every customer touchpoint. So long as an electronic device will work wherever you happen to be standing, you can run an experiment there. Which means not while you’re swimming, but absolutely if you’re on a water-tight submarine that has WiFi.)
(Creative Director’s note: You can run experiments while swimming if your device is in a plastic bag.)
We have a lot of goals at Optimizely, but making our customers happy is the number one thing. Maybe the biggest impact we can make is to have a stable, fast product that people can fundamentally rely on. I personally want people who are using Optimizely to feel like, “Whoa, this makes it so easy to get my work done.”
One way we accomplish this is through the speed of our product. I really see speed as a differentiating feature. And to ensure the overall snappiness of the application, we have to start with measurements. It’s pretty hard to improve the speed of anything without actually measuring that speed first. That obviously means building dashboards, looking at the performance of the stuff you care about, and then diving into places that seem like they’re problems. Then doing that over and over again. The sticky part is deciding where to focus your measurement and improvement.
At Optimizely, we focus very heavily on the 99th percentile of customers, those people who are the heaviest users of our application. We find we have a certain number of customers who are ridiculous power users, and these ridiculous power users are, like, fundamentally our most important and engaged customers. And since they’re pushing the limits of our product, ensuring things run smoothly for them is generally a good way to ensure things run smoothly for everyone.
With these customers, we’re looking at request latency for their API requests, render times for entire pages, speed of downloads; the issues for our heaviest users tend to be problems with scale. This person or company is running thousands more experiments than the average user. Based on their high-touch interaction with product, we sometimes have to fundamentally rethink the way that we’re exposing features to customers. Maybe it’s not okay to just give people back all their data. Maybe we have to page it or chunk it up in some way. How can we be smarter about how the product is organized?
We have a real mantra here: “low effort / high impact.” We want to provide the most value possible while doing the least amount of work. That doesn’t mean we’re not working hard — our biggest concern is that we’re building high-quality software — it just means we’re not trying to do every single thing ourselves. We want to invest our own time and energy in places that are gonna be the most impactful.
We value transparency as a cultural value, too. So it’s really important to us that everyone can see what everyone else is doing. We don’t want artificial walls. It’s very normal that other engineers and I build dashboards and then share them with the entire company.
There are a lot of technically-minded people across the organization (even in marketing and sales), and I feel like the more you spread information around, the more likely you are to get feedback and help. I like having help, you know? I’m reasonably smart, but it’s not like I’m a perfect person who has the right answer to everything.
This is also an area where monitoring and Sentry really help out. Monitoring is basically an insurance policy that alerts us if things aren’t working properly. Our goal is to be on top of things long before a customer has even started thinking about an issue like, “Oh, this is a weird thing. Maybe I should send a support ticket into Optimizely.” We want to be the type of team that is actively responsive and fixes problems before a customer even has the chance to encounter them.
We get a ton of value from Sentry because we’re notified (through Slack and elsewhere) every single time an exception gets thrown in our production application. That means if a user sees an error, we’re notified immediately. “Hey, a user just saw an error!” And we’re able to jump on the problem.
Personally, I feel like if you build a thing and you release it, part of your ownership of that thing is making sure it continues to work. That’s just the way it is. It’s part of the job.
As noted above, one of the biggest ways Optimizely uses Sentry is to tackle errors before the vast majority of their customers even have the chance to encounter them. And one way they do that is by funneling notifications about those errors to a channel in Slack, so they’re immediately made aware of the issue and can take action.
Sentry has dozens of integrations, several of which (like Slack and PagerDuty) are also used for notifications. The Slack integration is configured at the project level. Which means you can choose any project you’d like to connect with Slack and then visit its Project Settings. From there:
- Go to All Integrations
- Find Slack in the list and click Enable Plugin
- Add your Slack webhook URL. Slack can walk you through generating this webhook
- Enter the channel you’d like to use for error notifications
All set. Now you’ll be notified of every new issue through the Slack channel you’ve chosen.
Now, maybe you don’t want to be notified of every single new issue. Maybe you only want to be notified of issues that are of a certain level of severity. If so, you’ll be happy to hear you have full control over which sorts of errors ping you on Slack, thanks to our highly customizable alert rules. Read our short guide to creating your own alert rules.