How Organizational Complexity Scales

This is a follow-up to a previous post: https://dev.to/nhumrich/microservices-the-worst-technical-decision-you-will-ever-make-jff

In the previous post, I mentioned how one con to monolithic architecture is that you have higher organizational cost. A buddy of mine asked me to explain "organizational cost" and give some examples. Well, I am finally writing this post over a year later, sorry Devin.

In order to talk about organizational complexity (and more specifically, how it changes with microservices on monolith approaches), I first want to bring up a story that maybe you are all too familiar with.
The other day, I was organizing my home office, and had a lot of extra cables that have piled up from various devices. There were several cables and related peripherals that I wanted to save. Naturally, I threw them all into a single bin. The very next day, I needed a cable while at my companies' office. The IT department had all the cables neatly sorted so that I could find the cable I needed fast.

It would take much longer to find a specific cable in my one bin full of cables. It would also, however, take longer to put them away. Especially now that I have them all in a single bin, it would take more effort to organize them than to leave them alone. This is, literally, organization complexity. You will also notice that both situations are probably correct for their use case. I don't need a fully organized set of shelves for my home office, because only I use them, and I grab cables rarely. The IT office at my work office has people asking for cables probably every day. If they had a single giant bin full of cables, it would take them all day to find all the cables they needed.

It is somewhat hard to define or quantify this organization complexity, but we can see that if we compare the two solutions, there is a cost associated with the task that in other words would not be there if we had "organized" things differently.

Time to garden

Now, let's imagine that we have a community garden, about 5 acres of land. Different people help each other to plant all varieties of plants and vegetables. At first, everyone helps share the load. Most of the land is unused, but the garden keeps growing. The garden is quite a success, and more and more people start planting in the garden. Weeds continue to naturally grow. As weeds affect all plants, most people help out and start to weed, but there are always a couple that don't quite feel worth getting rid of. As the garden keeps growing and growing, some weeds start to affect only certain spots of land. There are now so much weeds, and used land, that gardeners start to realize that their efforts weeding might be futile. There is no way that a single gardener can possibly weed everything themselves. They start to focus on only their plants. If the weeds are in their patch of land, they weed it, otherwise, it gets left untouched.

Something else funny starts to happen. The plants start to cross-pollinate, and there becomes weird hybrid plants that start to pop up. People are not sure who's plants they are, since they could belong to multiple people.
As people continue to plant more plants, it becomes harder to know who's planted what where. It becomes obvious that unless someone takes over and controls the rate at which weeds and hybrid plants grow, it will become a mess. The city comes in and starts prioritizing certain weeds and hybrid plants to get done. Since people are generally confused about whose plants are whose, and whose land is whose, it takes more and more city officials to keep weeds and hybrids under control. They spend a majority of their time telling gardeners which weeds to weed.

Eventually, the city officials realize that this is taking too much time, and they aren't getting any other initiatives done. They spray paint lines on the ground, and assign each gardener a plot of land. Each gardener is responsible for their own plot of land. The city no longer cares as much about individual weeds. Gardeners are free to let their own plot fill full of weeds as much as they want. This causes conflict because some gardeners do a superb job keeping their plot of land clean and free of weeds, but the weeds from the plot next to them keep coming over. City officials decide to ask another city how they deal with this problem. The other city has installed fences and road between each assigned plot of land. This prevents weeds from creeping over and it gives very clear ownership. There is a downside that it makes it much harder for gardeners to work together and share water, as they have to install weird passages between their fences.

But enough of the metaphors...

As you likely realized, this story is not really about gardens. It is a representation of how software teams function. When a growing dev team continues to work in a single code base, the lines between bugs and features get blurred. Due to multiple initiatives happening in the company, it's no longer feasible for devs to work in all areas. They start to specialize around certain products/features. Teams working on different features start to step on each-other's toes. Certain bugs and technical debt comes up, and it becomes less clear which teams should work on what. Leadership starts drawing lines of ownership. Ownership reduces communication overhead, because instead of a leader telling people what they should work on, it becomes clear who should work on something based on who owns a thing. However, these "lines" are just suggestions. It becomes far too easy to reach across the line, and grab something that you need. This overcomplicates code and continues to blur the lines.

Microservices are a very heavy-handed approach that essentially forces boundaries and prevents blurred ownership lines. Crossing boundaries is difficult, so people do it less often. More importantly, ownership can almost never be misunderstood. So the amount of time spent cross-communicating is far less.

Now some actual Examples

Here are some real examples I have seen and how they can play out in each scenario.

1. You need to upgrade your tool/language/framework/thing. Let's say for now, it's a version of node. In a monolith, if we want to update node, we just need to do it once! This sounds easy and simple, especially compared to the microservices approach where we would have to do it for each microservice, dramatically increasing the amount of work. However, the problem with the monolith approach is it touches everything. Upgrading is a very dangerous thing, as every feature that exists could potentially be effected. In order to be safe, it's probably better to make sure that every single team has a chance to test their things for breakages. There also isn't a way to test this type of change behind a feature toggle. This means that there are now two branches being kept up to date. As the rest of the engineering team keeps working on features, the person working on the upgrade has to keep merging things in over and over, potentially continuing to resolve merge conflicts. Another option is to "stop the world" and make the entire department work on the upgrade, so there are no conflicts. This is a hard problem to organize. Its organization complexity. In the microservices world, even though you have to "do more work", coordinating that work is very easy. Each team upgrades their own service when they get around to it. One team might do it this week, another team might do it 3 months from now after their big deadline. They don't have to coordinate or communicate with any other team, so the whole project becomes pretty easy. The communication overhead is near zero.
2. You have a bug in production that is affecting a lot of users. You don't know exactly what is causing the issue, you just know it's at a specific endpoint. You ask the team that owns that endpoint to look at it, but the bug is in a specific library they haven't touched. Before you know it, half of the department is in a war-room discussing who might know what changed, rather than actually fixing the bug. Finally, they find the root cause, the person who made the change fixes the issue. And in the meantime, you have paid half of the department to fix this issue. In a microservices world, you notify the owners of the endpoint. They realize there is a 500 coming from another service. That other services team has already been notified because of the 500. They already have a fix in place. More than half the department wasn't even aware anything happened.
3. You have a feature you want to build. The team you think would own it is working on other higher priority teams. You assign it to a different team because "it doesn't really matter too much" if a different team builds it. They spend a lot of time ramping up, and build it in a way the other team wouldn't approve of, which is a problem because long term they will be the owners. In a microservices world, the team you want to work on it is too busy, so it goes on the backlog, and doesn't get worked on immediately. Either that, or ramp up time and boundaries will be obvious, and you make the decision if it's worth having another team work on it, knowing there are barriers to entry, and that you need to keep that team involved.

Now, all of these are extreme examples. You certainly can organize teams to function better. At the end of the day, a perfectly organized monolith and perfectly designed microservices look exactly the same. The difference is that one is about building a culture of expectations, and the other is about enforcement. From a culture point of view, that makes microservices sound a bit heavy-handed. However, it's more of an acceptance of reality. Pressures in the business will force bad decisions. If you don't believe me, ask if you have ever had technical debt.
Microservices is an intentional approach to prevent the business from ever putting you in a spot where really for organization is considered. It forces certain boundaries, despite business pressure.
In all situations, those boundaries prevent you from taking on too much technical debt. The difference is, in some cases, you might need the debt, and in other cases, the debt wasn't truly needed.

“I have been asked what I mean by “word of honor.” I will tell you. Place me behind prison walls—walls of stone ever so high, ever so thick, reaching ever so far into the ground—there is a possibility that in some way or another I might be able to escape; but stand me on the floor and draw a chalk line around me and have me give my word of honor never to cross it. Can I get out of that circle? No, never! I’d die first.”
― Karl G. Maeser

What Karl had wrong, is that he didn't consider the fact that someone else might push him. "whoops, sorry mate", and now Karl left the circle. Intentions do not accurately represent future results.

At the end of the day, the only difference between microservice proponents and the rest is that microservice proponents dont trust anyone to stay inside the lines. This has nothing to do with trust in the individuals themselves, but rather, it's because they recognize mistakes happen. Business pressures will cause people to violate sound principles, even if subconcious. There are some mistakes so costly, you prevent humans from doing them, even if you trust the indivuduals completely. If you transfer $1 million to a bank, you hire armed gaurds. Why don't you just ask your most trusted friend? or do it yourself? Because the issue isn't your trust in the persons character, its the trust that nothing else outside their control will happen.

Microservice architectures reduce organizational complexity by enforcing boundaries with electric fences. If you feel your organization is disciplined enough to respect boundaries at all costs, without the need for microservices, than you should probably avoid microservices. But for the rest of us, microservices are the only invevitable architecture.

Do you have the discipline to have many cooks in a large kitchen? Or would you prefer to have many kitchens? As for me, I think there is a thing as too many cooks.