Welcome back to another Sentry Snack of the Week. As promised from last week, this week we’re joining Adam as he describes a performance improvement that was made by changing the default behavior of Flagr.
How’s it going everyone. My name is Adam and I am one of the Engineering Managers here at Sentry. And today, I’m goin to tell you a quick story about caching, Python objects, and JSON blobs, oh my! Like most companies, we use feature flags to show which features to show to customers. Initially, we used a home grown solution to do this, but after a few years it started to show its age and we decided to switch to an open source solution called Flagr.
While implementing this, we started to notice a few different performance regressions. And since we build a performance monitoring tool, we decided to use it and really dig into what was going on.
Now, on any given page we may have hundreds or thousands of flags being checked, And what we do is we take all of these flags, pass them to Flagr, and send in some extra context around the user and the organization that they are a part of.
Flagr then sends us back a giant JSON blob with all of the details for each individual flag. Basically, whether or not the user can view that flag. The problem was that Flagr’s deserialization code was turning that entire JSON blob into a Python object. This deserialization is actually really expensive to do. So when we removed it and just access the JSON blob itself we noticed a reduction in time to parse all of these flags by about 500 milliseconds.
The next problem we saw was around caching. Because previously the deserialization was taking so much time, we didn’t actually know that this was a problem until we fixed that. Calling Flagr now is faster than checking and storing in the cache. And something that previously took 900 milliseconds was now only taking 200 milliseconds. Sometimes with performance monitoring, especially at scale, it’s less about finding that one specific performance issue, and more about peeling back the layers until you uncover a number of different things. Things that you wouldn’t expect to be slow, like caching, Can sometimes bite you in the butt.
And don’t forget to like, subscribe and follow us on YouTube. You don’t want to miss anymore of these Sentry Snacks of the Week.
Director of DevRel, Sentry
Here’s a quick look at how Sentry handles your personal information (PII).×
We collect PII about people browsing our website, users of the Sentry service, prospective customers, and people who otherwise interact with us.
What if my PII is included in data sent to Sentry by a Sentry customer (e.g., someone using Sentry to monitor their app)? In this case you have to contact the Sentry customer (e.g., the maker of the app). We do not control the data that is sent to us through the Sentry service for the purposes of application monitoring.Am I included?
We may disclose your PII to the following type of recipients:
You may have the following rights related to your PII: