Fetch Feature Flags Faster

Welcome back to another Sentry Snack of the Week. As promised from last week, this week we’re joining Adam as he describes a performance improvement that was made by changing the default behavior of Flagr.

How’s it going everyone. My name is Adam and I am one of the Engineering Managers here at Sentry. And today, I’m goin to tell you a quick story about caching, Python objects, and JSON blobs, oh my! Like most companies, we use feature flags to show which features to show to customers. Initially, we used a home grown solution to do this, but after a few years it started to show its age and we decided to switch to an open source solution called Flagr.

While implementing this, we started to notice a few different performance regressions. And since we build a performance monitoring tool, we decided to use it and really dig into what was going on.

Now, on any given page we may have hundreds or thousands of flags being checked, And what we do is we take all of these flags, pass them to Flagr, and send in some extra context around the user and the organization that they are a part of.

Flagr then sends us back a giant JSON blob with all of the details for each individual flag. Basically, whether or not the user can view that flag. The problem was that Flagr’s deserialization code was turning that entire JSON blob into a Python object. This deserialization is actually really expensive to do. So when we removed it and just access the JSON blob itself we noticed a reduction in time to parse all of these flags by about 500 milliseconds.

The next problem we saw was around caching. Because previously the deserialization was taking so much time, we didn’t actually know that this was a problem until we fixed that. Calling Flagr now is faster than checking and storing in the cache. And something that previously took 900 milliseconds was now only taking 200 milliseconds. Sometimes with performance monitoring, especially at scale, it’s less about finding that one specific performance issue, and more about peeling back the layers until you uncover a number of different things. Things that you wouldn’t expect to be slow, like caching, Can sometimes bite you in the butt.

And don’t forget to like, subscribe and follow us on YouTube. You don’t want to miss anymore of these Sentry Snacks of the Week.

< Watch the previous Snack

Watch the Next Snack >

Featuring

• 

  Adam McKerlie

• 

  Sarah Guthals

  Director of Developer RelationsSentry