"Low coupling, high cohesion", "information hiding": Well known design principles. Nevertheless, they are not taken seriously in many software architectures, especially when it comes to the interaction between the frontend and the backend.
APIs are valuable to hide information
By providing an API systems decide which information they expose to the outside world and which information they keep secret (Information hiding).
Why is that important? Because the broader an API is the more expensive it is to maintain. Think about the extreme of exposing every implementation detail. Every change in the system might break the clients which use the code. That's the reason why we strive for having small APIs. I wrote a post about this topic in the context of Event Sourcing.
Regardless of its size an API prevents us from evolving our system in a free manner. We always have to document the API carefully and consider not to break our clients. Changes have to be introduced in a backward compatible way or we have to align the deployments of our systems which quickly ends up in a deployment monolith hell.
API as product
But why do systems provide so many APIs, if it's so hard to maintain them? One reason might be that the API is the system's Unique Selling Point. The API is the product und you earn money by providing this API. In this case it's worth the efforts to document it, to use techniques like hypermedia and care about backward compatibility. Your clients will appreciate it and your API product shines.
Separation of frontend and backend
I argue that those API products are rare. Way more often the existence of a broad API is a sign for badly chosen boundaries - a violation of the "Low coupling - High cohesion" principle.
While this investigation applies to backend-to-backend communication, too, I would like to focus on the frontend. I question that we need an API to connect the backend and the frontend.
It's natural that the backend and the frontend are highly cohesent, because whenever some new data needs to be displayed on the frontend the backend needs to be adjusted. So why do we try to decouple them by putting a wall (an API) in between them?
Most of the time such a design is driven by organizational or technical circumstances. There are dedicated backend and frontend teams, but I wonder if the overhead of maintaining an API is always taken into account when deciding to separate the teams.
The desire for using client-side rendering
Another driver is the desire to implement the frontend with a modern Javascript UI library like Angular, React or Vue.js. In contrast to server-side rendered (SSR) approaches these libraries render the view at client-side (CSR) in the browser and rely on services (REST, GraphQL,...) provided by the backend to retrieve data and perform actions.
Self contained and CSR are no contradiction
Although I think that SSR is highly underrated these days, I can understand that the frontend requirements might be addressed better by a modern Javascript library.
But... using such a library does not necessitate an API! Think about delivering the service as a self-contained system, including the backend AND the frontend. Sure, you have a technology and environment break as the backend code is executed server-side and the frontend code is executed in the user's browser. This break requires the usage of HTTP in order to synchronize both environments.
I call this a bridge rather than an API as you can evolve the backend services hand in hand with the frontend. There is no need to document an API carefully or to deal with breaking changes, because there is just one client out there. And this client is well-known, because it's located in the very same system. The consequence is that the backend and frontend are always deployed as a whole and those documentation and compatibility challenges disappear.
Conclusion
Technology should not be the driver for system design. Regarding the interaction between the backend and the frontend take the requirements into account and decide if you want to go for a monolithic frontend or self-contained systems. "Low coupling - high cohesion" is a powerful guideline!
If you decide for a self-contained system it's up to you to choose a server-side or client-side rendering approach. Although CSR and SPA (Single Page Application) often are lumped together you can use CSR while still having a per-module frontend.
References
I had the idea of writing this post while listening to the SoftwareArchitekTOUR Podcast - Episode 82 (German) with Stefan Tilkov and Eberhard Wolff. Thanks for the inspiration!
Good resources to get deeper into self-contained systems and micro frontends:
Self-Contained Systems
Micro Frontend
Top comments (6)
While I agree with the conclusion, I'm struggling to find a coherent idea that leads to it.
You need an API. An API, a programming interface (as you also point out) hides implementation details and allows decoupling on whatever level you're exposing it. Having an API is essential regardless on whether you create an old-school monolith, a loosely coupled but still self contained system or you're providing your API externally as REST.
Whether you do CSR or SSR, in a specialised world it won't be the same people doing your data interaction as well as the pretty UX. One way or another there will be an API in-between, whether it's between people in the same team or different teams or different departments or if someone else entirely works on the UI, whether it's CSR or SSR.
"Every change in the system might break the clients which use the code." - that's the only really wrong thing I can point out. Any API, whether exposed externally or not, must be versioned and breaking changes must be made in a new version unless you are in a position to deprecate on the fly.
I agree that it's unlikely that the UX expert is not the same person as the backend engineer, but I argue that they should work together closely.
There are techniques like pair/mob/ensemble programming to facilitate this idea.
I also I agree on your last chapter: In every API you have to deal with versioning and breaking changes. That's why I recommend to avoid an API in the first place. ;-)
But how can you avoid an API ? Unless you're going for writing SQL queries in your display logic, you will have an API somewhere. Even if it's a simple service-like class that returns a dataset then it's technically an API. And as long as it's used by a developer other than you, there should be a contract of compatibility that ideally should include versioning (for the long run).
The issue you are raising is not solved by not having an API. It's solved (ideally) by replacing code-enforced contracts by communication. However, that's not practical in the long run because a clean API is self-documenting whereas direct communication backed by spaghetti code isn't.
Technically you might call it an API, because you have a technology switch between the frontend and the backend.
Think about an internal class in your backend module. Do you always evolve it in a backward compatible way? Hopefully not as you can use the refactoring capabilities of your IDE or the compiler or your unit tests help you to keep the internals consistent.
It's all the same with the frontend and backend commination. It's an internal implementation detail which you don't expose to the rest of the world.
Exactly, so unless you want to mish-mash all the login together, there will be an API. That's not the question.
The question is how you manage it's evolution. That's really up to you and how you want to manage interactions with those who consume your API, whether it's your fellow Dev next door or someone in a different company.
There are dozens of approaches.
I like versioning. I version up when I no longer want to maintain compatibility and add deprecation messages to relevant methods and classes. It's the core method of any software package and translates well particularly to DDD.
What's the alternative? Once I need to provide something everyone has to drop what they're doing and sync with me (or the other way around). We can't work asynchronously anymore and thus lose a core way of attaining agility. Any cohesive team should have the level of trust that people will respect the contract outlined by those pieces of code they output and provide timely notice so that others can handle their domain.
Refactor by IDE? That ensures code th as the won't break, not logic. If I need to add extra data on something a method returns, the IDE can make sure the code won't break but it can't speak to the actual handling of that. My consumer expects X and nothing more but if I need to add something else in a way they should be aware of, they'll get a new version of my method, the old will get a deprecation notice and everything is clear. There's no overhead and people can get to planning changes at their own pace alongside other priorities.
I agree. You list benefits of having an API. These arguments are valid. I do not want to question that APIs are an important tool, but I argue that they are often used without taking the costs into account.