I have written a new Elixir book:
Deploying Elixir: Advanced Topics
It is based on the feedback and suggestions I got from the readers of my previous book "Deploying Elixir".
They asked me to write more about clustering in Elixir and deploying to Kubernetes.
I'll take you from zero to deploying a Kubernetes cluster in the three big Cloud Providers.
In this book:
- I'll show you how to create an Elixir Release from your application and Dockerize it
- Learn what an Erlang cluster is and how to create one
- We explore the different strategies to create clusters using the
libclusterlibrary - I'll show you how to deploy it to a Kubernetes cluster using minikube
- When you feel comfortable scaling your app I'll show you how to deploy to Amazon Web Service, Microsoft Azure, and Google Cloud Platform
You can download a sample chapter here:
If you prefer a video version, I presented part of chapter 4 in the this Elixir meetup talk:
Get the full book here:
Top comments (4)
With a beginner beginner point of view, I understand that a BEAM cluster is P2P with TCP connection between running nodes. It seems against nature to need to wrap your Elixir code into containers and use k8. Besides the automatic scaling, is it because sizes of Erlang P2P clusters are limited that k8 is attractive? Also, the minimal running cost is way higher with k8. Are there other technical difficulties to run a BEAM cluster that pushes towards container orchestration?
The automatic scaling is in fact the point. k8s provisions the servers, orchestrates the loads and does a lot more things that only connecting nodes. BEAM can't do that, you have to physically provision a node, connect it, assign some storage, ensure that is in enough availability zones so that your app is resilient to failures. With just a Erlang cluster, if a node dies, it dies forever. You have to restart it with tools/scripts outside the beam. If you want to share storage between nodes, you can't do that with beam. What I want to say is, as impressive as the BEAM is, it is quite limited in what it can do to keep your cluster healthy and elastic.
Of course that's not a requirement for every application, but when you have that requirement, k8s solves the problem and BEAM doesn't.
Ok! I imagine you can form a cluster of containers (not sure how they communicate) but if I read you, it doesn't make any sense. Just have a dockerized app at let k8 do the piping and scaling. Now, if your application is modest, I believe you have a cheap and robust implementation using a small cluster, knowing that you have to keep an eye on you nodes. BEAM looks more like Docker Swarm, isn't it?
Yes, you're right.
For small clusters and if you don't really need the added complexity of k8s and are willing to do the monitoring/provisioning of nodes, a simple Erlang cluster of your Elixir application is a perfectly valid solution. I think for some internal application in a company that has on premises servers, that's is a solution that works very well.
k8s is not a solution for everyone, and certainly not for small apps, or apps with very few users or that don't need high availability.