The Mistake We've Made with Our Open-Source Product: Premature Optimization

We started FeatBit (an open-source feature flag platform) almost 4 years ago, and it has been open-sourced for 2 and a half years. Premature optimization is one of the biggest mistakes we've made.

We don’t regret the optimizations we implemented, as some customers appreciate them. However, our growth path could have been easier if we had started simpler instead of focusing on premature optimization.

This is our original blog - Keep open-source product simple and scalable. To make it easier to read, I’ve copied my content here.

Introduction

We are FeatBit, an open-source feature flag platform. This article discusses the mistakes we made and the lessons we learned while building and maintaining an open-source product.

Premature optimization is the root of all evil

FeatBit was designed as a self-hosting-friendly service, and it has remained that way until today. On the other hand, FeatBit wanted to provide customers with a performance guarantee. So, we put in a lot of effort at the very beginning of our open-source journey:

1. We offer two versions to suit different needs:
   • Standard: Supports high concurrency and connections, but does not include experimentation analysis.
   • Professional: Based on Standard, with added capabilities to handle large data volumes for experimentation analysis.
2. Our platform uses WebSocket connections for real-time feature flag updates, and we’ve made efforts to ensure:
   • High performance.
   • Scalability for high concurrency.
   • High availability for continuous uptime.
3. We provide several ready-to-use open-source deployment solutions, such as:
   • Docker Compose.
   • Kubernetes.
   • Terraform.

These efforts seemed like the right things to do at the time. However, after receiving feedback from customers, we realized that we had made several mistakes:

• Maintenance Complexity.
• Premature Optimization.

Maintenance Complexity

To ensure high performance, scalability, and availability, we introduced Redis, MongoDB, Kafka, and ClickHouse as databases. However, the big challenges we faced are:

1. MongoDB is well-known, but it's not widely used by most of our target customers.
2. Many of our target customers lack experience with Redis.
3. Kafka and ClickHouse are even less relevant; many B2B customers don’t need them because they don’t deal with large volumes of data.

It doesn’t make sense to require customers to learn and manage technologies that FeatBit doesn’t directly provide, as this can be a dealbreaker when deciding whether to adopt our product. Especially when the maintainers are Ops teams, not the developers and product managers who will actually use FeatBit.

Premature Optimization

We might not have built such a complex system at the beginning of our product development.

1. REST APIs are sufficient for many use cases. We could have supported WebSocket after receiving enough feedback.
2. We didn't need to focus on high performance for individual calculation resources. Making it scalable would have been enough.
3. A multiple-region high availability solution wasn't necessary at the start. Edge agents might have been simple enough.
4. We didn't need to add too many features (such as A/B experimentation) initially. Focusing on feature flags might have been sufficient.
5. We didn't need to design a microservice architecture at the beginning. A monolithic architecture might have been enough.
6. And so on.

By considering these points, we could have started with a much simpler architecture, allowing us to focus more on marketing and enabling users to easily deploy and use FeatBit in their production environments.

Pivot to Simplicity

We recognized our mistakes, but time has passed. Fortunately, the complex system we built allowed us to gain highly valued paid customers. However, to acquire more customers, we need to simplify our product so it can be used by a broader audience.

Making decisions is always difficult, because supporting a simpler version means we won't have time to develop new features. So, we conducted further investigation before making a decision:

• We identified which databases are most commonly used by our target customers.
• We spoke with potential customers to understand their actual needs.

Based on this, we decided to:

1. Provide a standalone version that uses only PostgreSQL as the database, removing Redis, MongoDB, Kafka, and ClickHouse.
2. Support both MongoDB and PostgreSQL as primary databases, catering to different customer needs.
3. Improve the FeatBit Agent to simplify many scenarios:
   • Customers who don't want to self-host the entire FeatBit system can host an agent to keep data in their own environment.
   • Customers who don't want to self-host a multi-region FeatBit system can deploy agents in different regions to maintain high availability and scalability.

Conclusion

Premature optimization is detrimental to acquiring early customers. However, if you've already built a complex system and received enough customer feedback, it's never too late to simplify your solution.

In the future, we may reduce the number of microservices and simplify our system. This will not only improve deployment and maintenance but also make it more developer-friendly.