How Panenco Improved API Performance and Reduced Response Time by up to 90%

Panenco is a full-service digital product studio with a team of more than 50 experts specializing in software development, product management, data science, design, and marketing. Founded in 2014 and headquartered in Leuven, Belgium, they’ve built up a Product as a Service portfolio spanning multiple industries, including healthcare, fintech, and education.

Customer success forms a critical function of Panenco’s daily operations and mitigating user friction is a key focus. But as the business – and number of users – grew, customer feedback became less detailed and technical . Resolving issues took longer and at times felt like “fishing expeditions,” with teams skimming through raw logs.

We historically handled customer feedback in a reactive manner, focusing on being easily reachable for all audiences and ensuring quick response times.

Sebastiaan Viaene, Senior Software Engineer, Panenco

Monitoring an increasingly complex environment

As a studio, the business operates in product teams, each with a designer, product manager, frontend, and backend engineers. Teams deal with multiple products, users, and related transactions on a daily basis, and Panenco hosts more than 80% of the products they build, compounding existing complexities.

So, the team set out to find an APM solution that would give them the ability to:

• Clearly define issue ownership, while maintaining autonomy
• Enrich errors with detailed context
• Proactively monitor, identify and fix issues in production
• Quickly see failed API calls
• Manage releases and,
• Monitor endpoint performance

We needed an efficient error and performance monitoring solution that we could implement across our entire portfolio.

Ruben Opdenakker, Product Manager, Panenco

Panenco usually hosts a React frontend supported by a Node backend both written in Typescript. They’re not afraid of using other frameworks and languages on the backend as well. The database of choice is PostgreSQL. Any solution they choose would need detailed SDK support with the flexibility to allow custom instrumentation.

Rather than spending more time on handling an increasing amount of incoming requests, we started looking for ways to prevent them altogether and spot broken code, crashes, and failed API calls before users are impacted, says Sebastiaan.

Impact, not just volume

Implementing Sentry took the team just 2 days, and they started building a workflow that best serves their operational model.

Custom tags let chapters configure the level of detail included in breadcrumbs and error logs depending on the log level of the application and environment. By integrating source maps from their CI pipeline they get additional source code context obtained from stack traces.

The team also configured alerts based on a set response time threshold for each transaction. Measuring the actual response time a user experienced as well as the number of times the threshold was exceeded, Sentry generates a “user misery” score that helps them prioritize issues based on their impact.

Configuring response time thresholds

Sentry’s Slack integration automatically routes errors to the relevant product channel which has helped teams further prioritize issues and assign ownership to the right people while reducing overall noise.

Panenco deploys new releases multiple times a week. Developers log release dates in Sentry, and map them to their code repository commits in Github and to team members who worked on a particular release. This lets them track whether errors were triggered before or after a specific release, making it easier to pinpoint the exact issue.

Finally, in combination with tools like k6 and Locust, and with the goal of becoming more proactive about resolving user-facing issues, teams regularly simulate extreme use cases to stress-test their products and share feedback through monthly retrospectives.

Sentry is great at ringfencing issues, which takes a lot of pain out of triaging. It’s become almost a cultural thing on the team, we don’t build any features without having it in place, says Ruben.

A roadmap for new projects

Panenco chose QIT online as the first product test case for both error and performance monitoring. With processes defined, teams had access to automatic error and exception reports, alerts on crucial performance issues impacting user experience, and could quickly investigate and resolve them before they caused any “misery.”

Over the past few months, we made error and performance monitoring integral to our operating model. We’ve seen improvements in user experience and boosted performance while reducing user friction, says Sebastiaan.

Taking performance monitoring a step further

Visibility into backend performance, and the ability to detect endpoints with the highest response times meant developers had already cut the time it took to find ‘the root cause of slow’ and improve application performance.

But, being the tinkerers they are, the team wanted to take it a step further and figured there’d be even more value in taking a closer look at response times. The idea? Break response times down to the main tasks that the backend had to execute for each endpoint.

We look at endpoints if there is a lot of user misery. Look at spans to see what takes long - look at the high volume stuff to see where we can make it better.

So, they integrated performance monitoring with their CICD setup and created a separate repository containing the stress-testing code written in Locust. A scheduled pipeline executes a stress testing session on a daily basis, targeting the staging backend server, and generates a report on completion.

Sentry captures a percentage of these endpoint calls, which developers then use to review their response times. The team also granted Sentry access to automatically create issues on GitLab for their backend and/or to link existing issues to Sentry reports, minimizing the manual overhead for the issue tracker.

Now our developers can easily identify and resolve performance bottlenecks (e.g. querying the database while iterating over a collection rather than prefetching), resulting in up to orders of magnitude fewer database queries, and shorter response times.

A proactive, new approach

As a business, Panenco brought together a talented group of problem-solvers with the goal of building high-quality digital products in partnership with companies and entrepreneurial individuals. As a team, they’ve taken that same dynamic and implemented Sentry with a clear focus on product development, performance and user experience. This line from their founding statement pretty much sums it up: “The company’s success will be the sum of the product successes.”

We use Sentry to monitor API and endpoint performance and have seen up to a 90% reduction in API request execution time for our highest-volume portfolio products. Customer success is a critical function of our business’ daily operations, and performance gives us a clear idea of how we’re tracking against our “frictionless user experience” goal, says Ruben.

Panenco partners with industry experts, research units, ambitious corporates, scale-ups and others alike. If you’re looking for a software partner that embodies product quality as part of their culture, go check them out!