I get anxious when working on back-end code. Data loss, security breaches and scalability/performance issues seem to be lurking around every corner if I don't pay attention. Sure, I can come up with a solution to most problems thrown my way, but with every design choice there's always the thought that maybe I didn't think it through and I forgot about a critical part of the system and it will have serious ramifications in the future.
This feeling is not only at my day job where I work on a fairly complex Rails app, but also in my hobby of game design and development — sketching out ideas and hacking together a prototype is one thing, but building it so that the code is maintainable, scalable, secure and easily understood from the start feels like black magic.
Of course, architecture and design patterns are not only a back-end thing. I've learned to write more or less maintainable code over the years and always seek ways to make it more flexible and concise, but the end result is just cosmetics anyway and the real magic happens elsewhere, so it doesn't feel as important.
Aside from just reading and understanding the source code I'm working on and familiarizing myself with source code of other, open-source applications, what are other good ways to learn the "correct mindset" of building systems? I'm most interested in how to think about the system as a whole and how to divide it into parts that make the most sense, given the overall problem the system is trying to solve; How to think about data flow, single responsibility, patterns that can be applied etc. How and where should I even begin?
Top comments (20)
For more back-end stuff, here's a few links I enjoyed (and still studying):
The System Design Primer
Awesome Scalability
Backend Best Practices
youtube.com/watch?v=3dLrSA-T7OY
Awesome Scalability --> Top
Thanks Rodrigo!
I learn from examples, in the last few months I gathered an youtube playlist with real "big" examples architectures.
If you want to go deep and also train for an interview, the System design from the interviewBit is the best resource I could find (as of quality).
As for the books, if you are interested how the distributed systems work (databases, cloud services, back-back end stuff), this is the "bible"
Unfortunately most of the "architecture" books are old, built with the Java/C++ monolith set of mind (outdated), so there aren't so many good ones left out there, but a few were mentioned in the other comments. There are some micro-services dedicated ones, maybe a few for Lambda, but I haven't found one that presented all types of "current" types of architectures, with examples.
If you are into gaming systems, a new book was launched, I haven't read it but it seems very promising and has good reviews: Game Programming Patterns
Lots of good reading material here, but nobody has mentioned Mr Fowler yet, martinfowler.com/design.html :)
If you are thinking about security (and who isn't?) while feeling a bit like an imposter, then I would suggest a rummage round Pluralsight to find an overview and maybe some deep dives in areas of interest (such as web application security - at work we use Troy Hunt's course as formal accreditation for peer reviewers). I also heartily recommend talks by Kelly Shortridge such as this on resilience: youtube.com/watch?v=ux--pHFpeac she has an ability to keep things understandable while covering a subject very clearly.
Like Ben, I have been at this quite a while and completed many projects in my life. I can't really nail one single thing down that made me become confidant in my architectural abilities but I believe it just came down to the experience of doing it more than several times. I think it really comes down to confidence in your abilities from experience.
The few tips I can offer from experience are...
Get a whiteboard if you don't have one already. This is where I draw out my initial thoughts about a project and it's processes and architecture then later use it to visualize flows and relationships.
Read, Read, Read... Pick up some good books like Eloquent Ruby, Design Patterns: Elements of Reusable Object-Oriented Software, and other books that help you build your skills in a thoughtful way.
Don't get overly anxious about your code or architecture. As you complete more projects through the years your confidence will grow as will your knowledge. Go with what you know NOW.
Good luck to you!
I'm very slowly working on a roguelike game, with the intent of using the Entity-Component-System pattern myself instead of letting a game engine handle it for me. This has already taught me many useful lessons on algorithmic complexity with many moving parts, composition of smaller parts to create a larger whole, and separation of concerns and decoupling. Everything very useful on the front-end as well and as a bonus, I can work on something I really enjoy creating. Not as complex as writing a kernel from scratch, though :)
It doesn't need to be a kernel you write, I believe that if you write any sufficiently complex piece of software from scratch with the intent of making it scalable and extensible, you will come out on the other side with a much better idea of design patterns and clean code architecture. While I have not built a kernel, my baby is a static site generator, Orchid, and a complex game can also be a great example.
Projects with a clear focus that is comprised of many disparate pieces are the best, as you have to find ways to manage each system independently while also having them play nicely together as an entire system. A kernel runs processes, but that requires filesystems, networking, and memory management, which are all very different. Orchid builds large static websites, but requires knowledge of parsers, filesystem abstraction, and I even implemented a simple HTTP server for an admin panel. A game has a pretty clear goal of playing the game, but involves a rendering/graphics engine, a game logic engine, data storage and state management, maybe even AI.
To do this, you'll have to give yourself strict constraints and promise to stick by them, which is much easier said than done. You have to go slow and constantly refactor, and find that single code path which everything else is attached to. If you don't explicitly find this path, then you run the risk of doing too much with too little governance, which is where code rot starts to creep in. In strongly-typed languages like Java, you can have the compiler enforce this with package boundaries and multi-module Gradle projects. In other languages, you'll have to be the one enforcing these rules of which methods are allows to be called, which fields are allowed to be changed, etc.
In building Orchid, I actually kind-of "rediscovered" several GoF design patterns, just because I got backed into a corner and needed to really think about how to find my way out. And when you start working on a way to get out of the corner, actually take the time to go and apply that same solution to other areas in the app, and see if it stands the test of scaleability. If it doesn't, refactor and find a better way. If it does, you've now got a new tool in your toolbelt to help you solve even harder problems.
Most of the resources on here are links to half baked material.
But check this: educative.io/collection/5668639101...
It is not free, but will teach you a crazy amount by learning from the big players and their architectural patterns. There are a few free chaptere to give you an idea of the flow of the course. Hope that helps.
This is amazing - I read through the first few preview chapters and really liked how the information is presented. Thanks!
Check The Architecture of Open Source Applications out!
It's wonderfully well written, though each one is in a different style.
Yes, this is one of the things I'm currently reading. Good stuff!
I've read tons of books about the topic but if I have to pick one it would be Release it from Michael Nygard who has also tons of good material about the topic .
After a few years of working in software industry, I agree that software architecture or system design areextremely important in the beginning of a project. It definitely determines the success of a project.
Actually, there are many design pattern available for us to study. The below is my list of Design Pattern for study. If you have list, please tell me.
Developing a functional programming mindset will go a long way to making you a better engineer, since software architecture and engineering are extensions of programming. I'd say, study how functional languages (Clojure, Elixir, Haskell) solve the classic problems of statelessness, concurrency, and distributed architecture.
Here are a couple more resources:
pragprog.com/book/swdddf/domain-mo...
pragprog.com/book/mkdsa/design-it
Best of luck.