We've all seen some of our fellow colleagues entering euphoria and la-la-land as we mention technologies such as ...
- Kafka
- Pulsar
- ActiveMQ
- Solace
- Etc ...
However, the dirty secret of our industry is that 99.999999999999% of the times some developer implements a message broker, the result becomes 1,000 times more unnecessary complexity, implying your project becomes undeliverable, and your employer ends up wasting money for nothing.
Say after me; We don't need a message broker!
I have seen the same thing happen over and over again. Some developer with a hole in his CV is given the task of solving some problem. The problem could easily have been solved with a MySQL database and a couple of CRUD endpoints, but because the developer is trying to fill holes in his CV, he ends up implementing it using 37 micro services, with cross process communicating, through some "hot and cool message broker thing", orchestrating this architectural nightmare from an astronaut's perspective. Using a message broker when you don't need it, is like building a space shuttle to go shopping at SafeWay.
Kafka is MADNESS unless you need it, and you almost never need it!
Kafka, Solace, ActiveMQ, and all associated technologies results in added complexity, and unless you absolutely need it, this added complexity is literally like stealing from your employer. Your employer pays you for your time. You wasting time configuring Pulsar when it is not needed, ensures your employer is literally flushing his money down the toilet, while you mess around with 100+ YAML files, each 1,000 lines of code, forcing you to spend 3 weeks simply to read the documentation before you can apply things correctly. And yes, simply configuring Pulsar to run correctly easily takes a month, even for a senior developer.
For the record, I don't mean to pick on Pulsar specifically here, it's an amazing piece of tech, however just like all the other message broker implementations, is is almost never, ever, ever, ever, ever, ever, ever, ever needed!
The above implies that unless you really need a message broker, you have effectively stolen your salary from your employer, by wasting your time doing unnecessary things. Quite frankly, I find it fascinating that employers even pay people such as this ...
You should be ashamed of yourself, and somebody should fire you!
For the record, I will be sending this article to your manager ... ;)
Top comments (16)
Well, it really depends on what you want to achieve, and what your bottlenecks are. Message broker solutions are optimized for the exact purpose of managing/transforming/pushing messages efficiently, while writing your own service with a traditional database and a simple REST server won't be optimized at all for the same purpose.
But I agree with the fact that you leveraged in a previous article of yours : never over-engineer things ! Existing solutions have been developed with a specifc use case in mind (or even trying to solve too many use cases). So it rarely fits well your needs and you usually can create your own microservice for your own usage : it will be cheaper and faster to implement and maintain.
Thank you, and yes of course, that was my exact point. If you need a message broker, by all means use one - However, my experience is that you rarely need one, and another of my experiences is that way too many apps have one ... :/
I have no other explanation for the latter than the "coolness factor", which of course is a serious waste of resources, and quite frankly lack of respect for colleagues and employers, which will have to wait sometimes as much as 10x time before they have solved their business requirements ... :/
I'm not really against anything, I'm just advocating "the middle road", as in avoid making things more complicated than what's required ...
I completely agree with you. Many bad choices are made because something is the "new shit".
Anyway, I want to have your thoughts on something I analysed during the past year (I have not implemented it yet, for some reasons that are not really interested here). I worked on a headless CMS system where I wanted to have a kind of pub/sub mechanism. It is mostly a subscribe service where there is only one publisher: the PostgreSQL server that notifies listeners when something has changed on some tables (this is handled by procedures). I need this so the clients are notified in realtime to improve user experience. And, of course, this would be implemented with "circuit breaker" in mind: this service is just the icing on the cake and it is not needed for the system to work reliably.
Now. What are your thoughts about this ? I'm really interested on those.
YAGNI comes to mind, as in "You Ain't Gonna Need It". It seems you've already concluded down that path yourself as you're writing the same in your own comment ...?
It's always simpler to implement request/response type of services in a multi machine/process context. Adding out of process pub/sub to the mix, especially if it's just "the icing on the cake", sounds like something that only adds complexity ...
But, don't take my advice, you know this better than me, and it depends upon the problem at hand ...
I thought some more about this, and in a CMS, headless or not, it's rarely important to have very fresh data - Implying if the data is stale for 5 minutes or not, is almost irrelevant. Cache in these regards would probably solve a lot of your problem, since from the client's point of view, pulling (quite frequently) is a no brainer.
There's only one place in our own stuff that we've got pub/sub stuff, which is during creation of a cloudlet. See screenshot below. And even there it's more for "show", and/or to ensure the client gets feedback, since it's a process that might take a couple of minutes - However, even in these parts we're using "simple web sockets" and no fancy message broker ...
You can see it with your own eyes if you create a cloudlet here ...
I'm replying here to your both comments ;)
This doesn't add any complexity on the backend since the LISTEN/NOTIFY mechanism is baked by default with first service (which is a simple REST server). But yes, on the frontend it adds its layer of complexity.
Why we need LISTEN/NOTIFY ? Because it is needed to plug other services like a log service, a webhooks service, or anything else. All those services are actually needed in the system, but for simplicity it is better to separate them from the base service since they are not needed to run the first layer.
In relation to the pub/sub service, it is indeed icing on the cake and could be completely removed from the stack. At first, I wanted it to have consistency in cache data hosted by the clients (which come from the REST service). This is needed to support offline connectivity for the mobile application mostly. But, in reality, offline connectivity is really hard to build and, IMHO, offline-first is a chimera. The only way to do it well is by caching the data when it's requested by the user iteratively (and not all at once). Because of that, it is impossible to foresee what data the user will need when its device becomes offline and the only way to have access to the right data is by improving the UX of the application so the user is responsible of what he'll need and must retrieve the relevant data beforehand.
When I first analyse the pub/sub service, I wasn't aware of this. But now that I'm thinking this through it seems this service does not meet any real need. It is just cool to have data automatically updated on clients when someone modifies it. And developing something that is just cool to have but at the price of hundred hours of work, well... Why bother :D
Hahahahaha, I love the way your brain made its way to a conclusion here ... :D
Feels like we're building up to Microservice CV driven development (imo the big idea is bounded contexts anyway, an extension of cohesion).
That said there are instances where niche technologies do save companies money:
Ultimately many "niche" solutions have a similar fate to this:
Most tech have applications where it’s justified. Just make sure you ONLY use it when it’s justified is my punchline really 😊
Hi Thomas! I'm curious: are you writing this controversial article series to get all these things out of your chest, for personal branding or to give Aista some exposure? If it's the latter, is it working? (By "working" I mean driving traffic to Aista's website or getting clients for the product).
As an ex-digital marketer, I always wondered if a marketing action like this would work. Sure it's doing for your personal brand (if you are looking for to be the controversial and over-the-top language guy haha), as I saw you got 12.000 readers recently, congrats! But I wonder if that translates to business meaningful numbers (at least in your case).
Thanks a lot!
Thank you for a very smart question, and the answer is yes, as in all of the above. As to traffic exposure, I can tell you it's working much less than you'd think. We got an additional roughly 60 page views the day I had the most readers last week. However, there is another explanation too, which is that every time I show Magic to business owners their answer is; "That sounds interesting, let me go get my best dev guy and put into our meeting."
The "best dev guy" ALWAYS ends up attacking our stuff, ridiculing it because it's "too simple", etc. This has left me with a feeling of that it's "them against me". And as I just told my marketing manager, who loves this series for the record ...
FYI, I don't mind building up my online persona as "the dev guy who makes everything stupidly simple" ... ;)
Haha :D
I agree, and I probably should have created an article with the name of "How did we end up here?" - However, in the end it would be speculations, even though I personally can see it clearly as the sun. I've aligned my company's incentives towards the direct opposite of complexity though, so change is possible. Now all I've got to do is to profit from it, and others will follow ... ;)
Thx mate :)
Psst, agree, it's "Snake Oil" (complexity) basically ... :/
Do you think it is the developers? I rather see management selecting the most complex frameworks and tech, and developers struggle with it.
For me it's mostly been driven by developers, but I assume it can be different for others. However, the problem is "bad marketing", something we can see from the NoSQL movement, where some few companies and projects started marketing NoSQL, until it gained a life of its own, and you're now considered a "bad developer" unless you choose NoSQL instead of RDBMS.
It's a herd mentality thing, resulting in that everybody chooses the same as everybody else for fear of standing out in the crowd.