For the last 4 years my field of expertise has been in highly scalable web applications, focusing on clients built with web components communicated with restful web services written in Javascript that persist to lightning fast postgres instances.
More recently I have taken the path of Principle Engineer and am currently building out an enterprise service platform. This is leveraging good practice from the microservices community and aims to provide the platform on which the company can build out customer facing products taking full advantage of common agile methodologies. Its all about speed of change!
I see the role of Principle Engineer to be one of leadership. What do I mean by this? First that it affords me the time to fully investigate the best solutions for a given set of business requirements, within the constraints of the current Engineering proposition. Then it becomes a job of collaboration with the team to bring them on the journey you have made. If you cannot provide compelling arguments for the decisions you have made, and get the team on board, you haven't got the right answer! It is not the role of the Principle Engineer to dictate the solution but to light the way. The hardest skill to learn is when to disagree and commit (in the Amazon parlance). It is only possible if you do not take your position as one of manager. Only a teacher and collaborator will succeed.
Top comments (3)
What are some of the key trends you see as important to highly scalable web apps in the next few years?
Hi there Ben. Thanks for engaging with my #ama :)
First off, in an attempt to avoid any flame wars, nothing I say is definitive. I'll just give a few opinions and hopefully some are helpful.
Second, trends in tech are hard to spot. And can be quick to change. The industry "bible" on trending tech is the Technology Radar, put together by ThoughtWorks. These people know a thing or two about tech, so I will expand on some of the relevant "adopts" for this topic.
Lets pick a few technologies I think you should be using/trying/reading about if you are interested in building scalable web apps right now.
TL;DR
Looking at the strong adoption of cloud compute, microservices and serverless products in scalable web applications I think moving forward we will see more focus on the technologies that suit these patterns the most. A few to look at, in my opinion, are:
A deeper dive
Tools
Before we get started on web app specifics, let's think about some of the tools you should be investing in.
Typescript
It seems to me if your journey in to development was through web applications it is possible that your first experience of a programming language was JavaScript. If so, you will have been spared exposure to a system called strongly typed languages that facilitate a thing called Type Checking. This is done at compile time or, even better, by you code editor!
Others who made the journey from, for example, C# or Java into the world of web, will often be found lamenting the loss of the ability to statically declare the type of the data intended to be pointed to in a variable. Like the philosopher in the allegory of the cave, there is not going back.
And then we got Typescript. You should use it.
Terraform
Something that is clear now is that cloud compute is the solution for almost all infrastructure requirements in the modern era. There are use cases for owning and managing your own tin, but even then it will be running something like VSphere and internally consumed as cloud compute anyway.
As the microservice flavour of SOA shows more and more that it can be a great choice for scalable web applications, the adoption of Automated Infrastructure through Infrastructure as Code (IaC) propositions will be on your radar. Right now, Terraform is becoming a strong contender in this space, mostly due to its large and active community. At Kano we leverage a lot of Amazon services and in conversations with their engineers it seems that Terraform is even quicker to react to changes in the AWS service catalogue than the Cloud Formation team (anecdotal only, I offer no data on this point!).
So check out Terraform, it will be important as you build you web application infrastructure.
Micro Frontends
Anyone who has built (or more horrifyingly, taken over) a single page application (SPA) will be able to tell you tales of code splitting and routing delights. There are tools dedicated to helping with some of the optimisations that are essential for any SPA of more than a little complexity. I am interested in how to open up access to the wealth of web applications we now have to developing markets (see below). So we need to reduce the size of downloads, we need to think about data transfer and low powered devices running our apps.
Micro Frontends do two things to help with this. The separation of your client application code into services will allow targeted application loading. But it also will set up a mindset of building out components that communicate through published APIs, and message passing. This facilitates the leveraging of other emerging patterns, such as agent based UI designs that allow you to launch certain UI rendering tasks in web workers and get back the state of a UI component from the render process asynchronously. I think. Maybe?
Golang for serverless
I built a lot of middle tier i nodejs using a framework called express. SO much so I wrote an opinionated library to build them out in the pattern I was using.
However, as I started to use AWS' Serverless product I started to see that maybe node is not the best solutions for a lambda.
The popularity of node can be attributed to a range of things, but the value proposition is its non-blocking IO implementation. This allows for a single threaded application to accept and handle potentially thousands of connections (as long as you are using it for the right things). The design of lambda means that each invocation of a function is tied to one request. Once the request is completed it becomes available for the next request. Any additional requests that arrive to a serverless deployment are handled by a separate lambda in its own deployment (container). So the main advantage of node being build on the V8 engine is lost.
Now we want to keep the number of concurrent lamda invocations down, as they each cost money. SO we want the function running inside our lambda to be as fast as it can be, often from a cold start.
Compare Javascript and python to Golang in a head to head race on getting stuff done and you will see why I think golang is a good fit for my lambda code! So give it a go.
Plus, career wise, it is a good tech to ad to your polyglot CV.
Web Assembly
This one could be good for web apps. But as of yet I have not done a lot of work with it. But watch this space. Plus there is a tool that combines web assembly and typescript: AssemblyScript
What I would like to happen
I have always been interested in how we can take all the richness and complexity in web applications of today and make them available to the billions of people who have a digital device but do not have the latests versions nor the highest bandwidths. I was hopeful that Progressive Web Apps and We Components would be a path to achieve this. So please keep those in mind as you move forward with all this.
Oh, and Flutter of course.