When The Swarm Isn't Enough (An Incredible Journey)

Exit Docker Swarm

About 2.5 years ago I was tasked with an impossible mission, much like John Wick. My task was to take a monolithic system and attempt to break it into pieces and convert those pieces into micro-services. Well there were a couple problems with that.

1. There was so much cross pollination of the database you would of thought growing a super weed was inevitable. In other words, developers were going around the service api and DAO layers and interacting with the database direct.
2. Written using Java 1.2 Standards need I say more
3. Unit/Functional/Integration tests were shit
4. There is more but I don't want to list it out.

So we decided to rewrite the whole application from scratch, we used OpenApi Spec to define our API calls, we used Java Spring Architecture for each micro-service. We even created a few JS micro-services that use express and some other npm packages.

At the time were were told the micro-services needed to be deployed as VM's in openstack (I will admit it wasn't fun this approach). Then we got our first customer and we had 2 weeks to deploy our application into a customer VMware environment. This means no openstack, yeah oh shit! So what did we do?

Well I was experimenting with use Docker Compose at the time and we took the container nestle tea plunge and deployed our app using Docker Compose. This actually worked out pretty well minus the scalability being static.

We then took the natural next step and enable Docker Swarm and scaled up our services. We have manager nodes and worker nodes and the application is working quite alright. One major issue we have found with Docker Swarm is utilizing its ingress networking. We are finding stale DNS entries. In addition there isn't elastic horizontal scaling. So we started our search. Enter K8's........

Enter Kubernetes

We spoke with our group and everyone was on board with making the move to Kubernetes. So me and my architecture team got together and we enlisted the help of a consulting firm. We worked for 10 weeks with a brilliant group of 4 people (6 of us total) to learn how to do infrastructure as code in AWS(cloud formation, terraform), to setup vertical scaling and horizontal scaling. We even in a matter of 4 weeks out of the 10 were able to do a bare metal deployment using kubespray.

The rebuilding of the application, CI/CD, Container orchestration all happened over the past 2.5 years. I have learned so much about dev ops, application threads, and shared memory. There are so many 3rd party products we use it is incredible. It's a wonder I feel like my head is full everyday.

Thanks for taking the time to read my incredible journey.