Background
We have a mission-critical application used across multiple locations. It is a legacy .NET MVC application that leverages Redis for caching and SQL Server for data storage. The application is hosted on Azure App Service Premium V2, running on the highest P3 plan, which costs approximately $580 per instance. During peak hours, it operates on three instances, while during off-peak hours, it scales down to two instances.
The application was initially developed by experienced developers, ensuring a solid foundation. However, over time, its maintenance was handed over to junior L4 developers and the L3 support team. With each new enhancement, performance gradually deteriorated, leading to inefficiencies and skyrocketing resource consumption — ultimately transforming it into a resource-hungry monster. 🚀💰🔥
Over the past month, the application became increasingly unstable, suffering from severe performance issues and multiple high-priority incidents. While adding more instances could have provided a quick fix, it would have significantly increased costs — pushing us beyond budget, which wasn’t an option. Instead, we needed a smarter, cost-effective optimization strategy to restore stability without overspending.
My manager initiated a stabilization project to tackle these performance issues and appointed me as the Lead to drive the effort. It was my responsibility to identify inefficiencies, optimize resource usage, and bring the application back to peak performance — all while keeping costs under control.
Identifying inefficiencies
I began by identifying the root causes of the performance issues, dividing my investigation into three key areas: Redis, the database, and the App Service. After thoroughly analyzing Redis and the database, I found both to be clean and functioning optimally, ruling them out as potential culprits. With those eliminated, my focus shifted entirely to the App Service, where the real issue seemed to lie.
Azure had a built-in code profiler, but due to restrictions on enabling it in the Production environment, I had to find an alternative approach.
I had two profiling options: Glimpse and MiniProfiler. While Glimpse was the more powerful choice, it had a major limitation — it only worked with _layout.cshtml, whereas our application had multiple layout pages. Due to this constraint, I opted for MiniProfiler, which provided the flexibility needed to effectively investigate performance bottlenecks across different layouts.
I used the following MiniProfiler packages and set up the basic configuration required to get it up and running:
MiniProfiler.Mvc4
MiniProfiler.EF6
I added the necessary settings to enable profiling across the application. This allowed me to track slow-running API calls, and rendering times, helping to pinpoint the exact performance bottlenecks.
The findings were shocking! 🚨
N+1 Query Problem: Several parts of the application suffered from the classic N+1 query issue, causing excessive database calls instead of efficient joins.
Inefficient Filtering & Searching: A significant amount of filtering was happening in memory rather than on the database side. The app was unnecessarily loading massive datasets from SQL views and then filtering them in memory before returning the results.
Broken Pagination: Pagination logic was also being handled in memory, instead of being executed efficiently at the database level using OFFSET and FETCH.
Client-Side Query Execution: Many LINQ queries were using .NET functions that forced query evaluation on the client-side, leading to unnecessary data transfers and heavy processing on the app server rather than leveraging SQL Server’s optimized execution plans.
It became clear that fixing these inefficiencies was crucial to stabilizing performance and reducing resource consumption.
The Results
All findings were thoroughly documented and addressed by the team. Once the optimizations were implemented, we migrated the changes to PROD, and the impact was immediate and drastic — CPU usage dropped by 4x, and memory consumption plummeted by 10x! 🚀
We closely monitored the application’s health for a month and, seeing sustained improvements, made further optimizations. We reduced the instance count even during peak hours to just two and also downgraded the App Service plan, significantly cutting costs while maintaining stability.
Key Takeaway:
This entire experience underscores why code reviews matter — not just for functionality but also for writing optimized, efficient code that prevents performance bottlenecks.
If you’re interested in a deep dive into the exact MiniProfiler configurations, the issues we discovered, and how we fixed them, drop a comment, and I’ll write a detailed follow-up article! 🚀🔥
Top comments (0)