The Adventures of Blink S2e2: Database, Contained

Hey friends!

Last week we started Season 2 here on the Adventures of Blink, in which I intend to explore developer tools and devops practices by building an application from the ground up. But I didn't give any details!

So it's now time to spill the tea...

This season, we're going to build the game "Hangman", where there's a word or phrase and you have to figure out what it is by guessing one letter at a time. If you guess a letter that doesn't appear in the word or phrase, then you draw a part of the man... and if you draw the whole man before you guess the overall phrase, you lose.

So how does Hangman help us learn dev tools, you might ask? Let's talk about its construction, because this version of hangman will be a bit overpowered 😬. Here's what we're going to do:

• Have a Mongo database holding the various phrases we're going to use and potentially configuration data for the frontend as well.
• Put all the database logic in a REST api that runs separate from our front-end implementation.
• Implement our frontend as a python application.
• Construct unit tests for our api as well as for our frontend.
• Use GitHub actions to run unit tests when we push our code.
• Run all components of the application in Docker containers configured to work together.
• Build an in-app editor for people to add and remove phrases from the database.
• Experiment with Generative AI tools to create phrases for the game to keep it infinitely fresh.

TL/DR: Youtube

Don't feel like reading? Click here... and don't forget to like & subscribe!

Step 1: MongoDB

The first thing we're going to do is build our database. We're building this app from the bottom up... so data is our foundation. So let's spin up mongo in a container!

a. Create a project folder with a database folder inside it

We're going to start by creating a project folder. Then within our project folder, we'll make a subfolder for our database, like this:

b. Create an .env file in project root

In the project root directory, we're going to make a file called .env. This file will be used to hold the global environment variables needed to build our container structure. It doesn't live in the database section because ultimately we're going to have multiple containers in this project and this file will be available to all of them, to act as a central global configuration file.

The contents of .env that we'll start with will contain the database admin credentials.

It's vital that you don't commit this .env file to source control! Pushing your credentials into GitHub/GitLab/etc can be very dangerous. We'll talk about having a .gitignore later, when we're ready to push our new project. For now, just be aware that .env is going to be created by everyone who builds and runs the project from source - so you probably want to make notes about it in your README.md file!

Here are the contents for our .env file:

MONGO_INITDB_ROOT_USERNAME=your_username
MONGO_INITDB_ROOT_PASSWORD=your_password

Of course, you're changing your_username and your_password to be whatever values you want to use.

c. Create a Dockerfile for mongo

Now that we have created an .env to protect our credentials, we can start working on building our database container. To do this, we're going to create a Dockerfile. Dockerfiles are always named "Dockerfile"... with no extension. This file will be created inside the database folder... because any folder can only have a single Dockerfile, we have to keep the various container definitions separated in the code.

Here's what will be in our Dockerfile, along with comments explaining each part:

# We're building on top of the latest version of mongodb's official docker container
FROM mongo:latest

# Set environment variables for MongoDB credentials
# These will be sourced from the .env file at runtime
# There's another configuration that will link these up
# Right now it doesn't work, we'll fix it later!
ENV MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
ENV MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}

# Mongo's default port is 27017, so once it's running inside the container, we have to allow traffic to that port from outside of the container... or nothing will be able to access the database.
EXPOSE 27017

# Define a command to run MongoDB
CMD ["mongod"]

Um, Blink - you know that docker won't save any data in the container, right?

If you're new to containers, you might not realize this: while a container is running, it's read only. If you make changes, they'll disappear when the container stops. Not really useful for a database!

Docker provides a way to work around this - you can create Volumes. A volume is sort of like a "mount point" in a container. It points at a storage location outside the container, giving you a persistent storage point. We'll see how to set up volumes in the next section.

d. Docker-Compose

I've mentioned a couple of times along the way that we'll have more than one docker container in this project. But we can't depend on end users to know what order to start them in, or what configurations connect them to each other. Docker provides a means of orchestrating all the containers in your app called docker-compose. This is controlled by a yaml file in the project root called docker-compose.yml. Here is how we're going to build ours (again with comments to explain what it does):

version: '3.8'

# Each component of our app will be a 'service'
services:
  mongo:
    # The 'mongo' service is created by telling Docker to build
    # whatever it finds in the 'database' folder.
    build: ./database

    # It will construct a container named 'mongo_container' 
    # which will automatically try to restart if it's not 
    # running, unless we manually stop it.
    container_name: mongo_container
    restart: unless-stopped

    # It will ingest the .env file from here in our project root.
    env_file:
      - .env

    # It will use the location ./database/mongo_data as the folder
    # where mongo will store the persistent data files - this is 
    # where your database's contents will be so that the container
    # doesn't lose it when it stops.  The mongo_data folder will
    # be created if it doesn't exist, and then when you start
    # mongo for the first time, it will be filled with all the data
    # files that mongo requires.
    volumes:
      - ./database/mongo_data:/data/db

    # This allows the database port to be mapped to the outside
    # world.  Without this, the docker-compose environment would
    # be able to access the database but we wouldn't be able to 
    # reach it from the host file system.
    ports:
      - "27017:27017"

e. Try it out!

We've now reached a point where you have enough working to be able to run this and see things happen.

Open a terminal, go to the project root folder, and run this:

Windows/Linux:
docker-compose up --build

Mac:
docker compose up --build

You should see a bunch of startup logging as docker pulls the latest mongo, builds your app, and starts it up.

Once things stop scrolling, you're running! Because of the way we're testing this, that terminal window has to keep the app active - we'll learn later how to make it run in the background.

You can verify that your container is reachable from your host OS by opening a browser and going to http://localhost:27017. You should get a plain text message about trying to access Mongo... and that's victory for now!

After you've validated, return to your terminal and press Ctrl-C to stop your container.

For reference, here's what you should have in your project folder so far:

f. Initialize the database

Our database is still empty. We need to initialize it. If you're used to working with standard Relational Database Management Systems (RDBMS), you're probably used to thinking of "tables". In Mongo, these are called "collections".

We need to create a script that will be executed when the container starts up - it needs to verify whether the database has already been created, and if not, create it.

Below is our script, for starters - we still don't actually know the structure of our database yet, so this is sort of a skeleton example. As we define the structure of our database, we'll edit this script to make it complete. This is valuable because our database config will be stored as code, making it easier to support in the long term.

Store this in the database folder, alongside the Dockerfile. Call it init_mongo.sh:

#!/bin/bash

# Load environment variables from .env
source /env/.env

# Database and Collection names
DB_NAME="hangman"
COLLECTION_NAME="phrases"

# Command to check if the collection exists
COLLECTION_EXISTS=$(mongo --username $MONGO_INITDB_ROOT_USERNAME --password $MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase admin --quiet --eval "db.getSiblingDB('$DB_NAME').getCollectionNames().includes('$COLLECTION_NAME')")

# Check if the collection exists
if [ "$COLLECTION_EXISTS" = "false" ]; then
  echo "Collection $COLLECTION_NAME does not exist. Initializing..."
  # Command to create the collection
  mongo --username $MONGO_INITDB_ROOT_USERNAME --password $MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase admin --eval "db.getSiblingDB('$DB_NAME').createCollection('$COLLECTION_NAME')"
  echo "Collection $COLLECTION_NAME created."
else
  echo "Collection $COLLECTION_NAME already exists."
fi

This is the beginning of our database initialization script. The next thing we have to do is modify our Dockerfile to run the script when the container starts:

FROM mongo:latest

ENV MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
ENV MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}

# Copy the initialization script into the container
+ COPY init_mongo.sh /docker-entrypoint-initdb.d/init_mongo.sh

# Ensure the script is executable
+ RUN chmod +x /docker-entrypoint-initdb.d/init_mongo.sh

EXPOSE 27017

CMD ["mongod"]

Note: Above is a view that shows the change as a "diff" so you can see what's being changed easily - the '+' characters are simply denoting that the line is being added to the file. Having + at the beginning of these commands won't actually work in Docker!

g. What about the table structure???

If you've worked with Relational Database Management Systems (RDBMS) before, you're likely thinking that we need to start database design work next. Mapping the structure of the tables, what fields are present, what data types each of those fields holds... and if you're deep into the database world you might be thinking of foreign keys and normal forms and all that stuff.

But when you use Mongo... you don't. Mongo is a schema-less database - meaning that a collection is just a named object that can hold data: data of any sort. So even though we're planning on a data set that will (at least at first) contain 4 fields:

We don't have to do anything to define this until we start actually writing data into it. 🤯🤯🤯

Wrapping up

This season's off to a wild start, isn't it? So far, we've built a Docker container using a Dockerfile, we've used YAML to configure a docker-compose job, and we wrote a bash script that will be the foundation of our database initialization process. We're 3 languages and 2 major technologies deep already, and we haven't even begun actually writing our app! 😳

Friends, this is a typical day in DevOps - shifting between languages and technologies as fluidly as possible, all in single-minded pursuit of our goal.

Stay tuned as we continue to build out our project and learn about lots of other new things along the way. See you next time!