Best DX for MongoDB Replica set

MongoDB at Woovi

We run a MongoDB replica set in production as this is the recommended setup for redundancy and data availability.
Our MongoDB node is a pod in our Kubernetes with persistent volumes, so the pod can restart without losing any data.

We also have 1 hidden member on our replica set that is used for backup and Metabase, so it does not affect production workflow. A hidden member does not accept any reads from replica set connection.

There are some features that are only available when running MongoDB in replica set mode like transactions and change streams. This happens because these features require to access the oplog only available in replica set config.

We are using change streams to make our architecture more event driven and to update our Elasticsearch index to power our Woovi search feature. We are going to write in the next blog post the deep dive of our search architecture at scale.

We focus a lot on DX (developer experience), so running a MongoDB replica set should be easy and fast for everyone in our team, let's see how we achieve that.

MongoDB Docker Compose Setup

Everybody at Woovi has a macbook m1 to make sure they are productive.
Running docker using Docker for Mac was heavy and slow, we switched to OrbStack and it feels that we are not even running docker anymore, it feels native.

To run a minimal MongoDB Replica set we decided to run 2 mongodb nodes, docker compose makes it very easy to setup

version: '3.8'
services:
  mongo1:
    container_name: mongo1
    image: mongo:latest
    command: --bind_ip_all --replSet rs0
    ports:
      - '28017:27017'
    volumes:
      - mongodb_data_container:/data/mongod1db
      - ./scripts/mongodb/rs-init.sh:/scripts/rs-init.sh
    networks:
      mongo-network:
        aliases:
          - mongo1.localhost
        ipv4_address: 172.20.0.2
    links:
      - mongo2
  mongo2:
    container_name: mongo2
    image: mongo:latest
    command: --bind_ip_all --replSet rs0
    ports:
      - '29017:27017'
    volumes:
      - mongodb_data_container:/data/mongod2db
    networks:
      mongo-network:
        aliases:
          - mongo2.localhost
        ipv4_address: 172.20.0.3

volumes:
  mongodb_data_container:
  cache:
    driver: local

networks:
  mongo-network:
    name: mongo-network
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/16
          gateway: 172.20.0.254

Let's deep dive on this docker compose file.

We declared 2 services: mongo1, and mongo2.
Each of them are mapped to different ports in localhost, 28017 and 29017.
Each of them have their own data in their own volumes
Each of them have a fixed IP 172.20.0.2 and 172.20.0.3, fixed IP is important to make mongodb drive to resolve mongo1 and mongo2 properly.

We also need a script to configure the replica set

rs-init.sh

#!/bin/bash

mongosh <<EOF
const config = {
    "_id": "rs0",
    "version": 1,
    "members": [
        {
            "_id": 1,
            "host": "mongo1:27017",
            "priority": 2
        },
        {
            "_id": 2,
            "host": "mongo2:27017",
            "priority": 1
        },
    ]
};
rs.initiate(config, { force: true });
rs.status();
EOF

This script will start mongosh (mongo cli shell), and configure the replica set members.
As they are in the same network inside docker, they can find each other using their names, mongo1 and mongo2.

Our final script to make this setup only need one command is mongodb-rs.sh

docker-compose up -d

sleep 5

docker exec mongo1 ./scripts/rs-init.sh

It will start the docker compose, it will wait for 5 seconds and it will configure the replica set.

The connect URI would be like this

mongodb://mongo1:27017,mongo2:27017/db?replicaSet=rs0

However your host machine can't resolve mongo1 or mongo2, so we need to "simulate" the DNS.

We have this command to add the fixed IP to our /etc/hosts, so our host machine can resolve mongo1 properly.

sudo -- sh -c "echo 172.20.0.2 mongo1 >> /etc/hosts"
sudo -- sh -c "echo 172.20.0.3 mongo2 >> /etc/hosts"

In Conclusion

This sounds like a lot of work, and it is.
It takes us some time to find the perfect configuration that works well for everybody, but it is worth it.
As we reduced the cost of the setup to everyone in the team avoiding wasting a lot of time.

At Woovi we invest a lot in building blocks that will reduce the cost of us shipping new code faster to production.
The 1 command to setup a MongoDB replica set environment is one of our example of this investment.

Scaling in not only adding more pods to your Kubernetes in production, it is all about reducing the cost of developing new software on top of the existing one.

---

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Comments

[Joe Mainwaring]

Excellent writeup on self-hosting a replica set.

And for those who may be reading this and thinking "Oh my god this is too complicated", MongoDB does offer a managed service option - with some of the best customer service I have encountered across my many vendor relationships. Worthwhile to consider if you don't have the resources to allocate towards self-hosted.

[Azriiii]

awesome article... Thank you so much .
Still have one question:
How did you setup the databases authentications ?
This setup is just intended to launch a mongodb cluster with two nodes ... without configuring the databases authentication.

[Sibelius Seraphini]

there is no authentication in this setup, as is this mostly for localhost or development

for authentication there are many approaches, check here mongodb.com/docs/manual/core/authe...

you could create a script to create a basic user to access the database mongodb.com/docs/manual/tutorial/c...