Using Docker Compose to Build Environments

In our previous posts, we explored how Docker containers and Dockerfiles can be used to package applications neatly. But what happens when your applications become more complex, with multiple components and intricate configurations? Imagine building an online store, where you have separate microservices for the frontend, backend, payment processing, order management and analytics. Each of these microservices might be developed using different programming languages and technologies, and they all need to be built, packaged, and configured correctly.

This is where Docker Compose comes to the rescue. It's a powerful tool specifically designed for managing applications that run in multiple Docker containers. You can define your entire application stack in a single YAML file, including each microservice, its configuration, and how they all interact with each other. With Docker Compose, you can spin up your entire complex application with just a single command, making it super convenient for development, testing, CI/CD pipelines, and even production environments.

The essential features of Docker Compose can be grouped into three categories:

• Isolation: Imagine running multiple instances of your application, each completely isolated from the others. Docker Compose makes this possible, allowing you to replicate your entire application stack on various environments like developer machines, CI servers, or shared hosts. This not only optimizes resource utilization but also simplifies management by reducing operational complexity.
• Stateful Data Management: Say your application needs to store data on disk, like a database. Docker Compose takes care of managing the data volumes associated with your containers, ensuring that your data persists across different runs. This makes it much easier to work with applications that rely on persistent storage.
• Iterative Design: Docker Compose works with a clear configuration that defines all the containers in your application. You can easily add new containers to this configuration without disrupting existing ones. For example, if you have two containers running and decide to add a third, Docker Compose won't touch the first two. It will simply create and connect the new container, making it a breeze to expand your application iteratively.

Docker Compose's ability to isolate multiple instances of your application, manage stateful data, and support iterative development makes it a must-have tool for handling complex applications with multiple containers. In this chapter, we'll take a closer look at how Docker Compose can help you manage the entire lifecycle of your application, from setup to deployment.

We'll start by diving deep into the Docker Compose command-line interface (CLI) and the structure of Compose files. Then, we'll explore different ways to configure your applications using Compose and how to define dependencies between different services within your application stack.

Since Docker Compose is a core tool in the Docker ecosystem, gaining both technical knowledge and practical experience with it will be a valuable addition to your skillset. By the end of this chapter, you'll be well-equipped to handle even the most intricate multi-container applications with ease.

Docker Compose CLI

Docker Compose works hand-in-hand with Docker Engine to orchestrate multi-container applications. It uses a command-line tool called docker-compose to communicate with the Engine. On Mac and Windows, this tool is conveniently bundled with Docker Desktop. However, if you're running a Linux system, you'll need to install docker-compose separately after installing Docker Engine. The good news is that it's a single executable, making the installation quite straightforward.

You can find instructions on installing the Compose Plugin here.

Docker Compose CLI Commands

The docker-compose command can handle all aspects of an application's life cycle that use multiple containers. It is possible to start, stop, and restart services with the subcommands. You can also see what's going on with the running stacks and get their logs. The following posts will give you hands-on examples with the most important commands. In the same way, the following command can be used to see a sample of all the features:

docker-compose --help

And you should see something like the following:

Usage:  docker compose [OPTIONS] COMMAND

Define and run multi-container applications with Docker.

Options:
      --ansi string                Control when to print ANSI control
                                   characters ("never"|"always"|"auto")
                                   (default "auto")
      --compatibility              Run compose in backward compatibility mode
      --dry-run                    Execute command in dry run mode
      --env-file stringArray       Specify an alternate environment file.
  -f, --file stringArray           Compose configuration files
      --parallel int               Control max parallelism, -1 for
                                   unlimited (default -1)
      --profile stringArray        Specify a profile to enable
      --progress string            Set type of progress output (auto,
                                   tty, plain, quiet) (default "auto")
      --project-directory string   Specify an alternate working directory
                                   (default: the path of the, first
                                   specified, Compose file)
  -p, --project-name string        Project name

Commands:
  build       Build or rebuild services
  config      Parse, resolve and render compose file in canonical format
  cp          Copy files/folders between a service container and the local filesystem
  create      Creates containers for a service.
  down        Stop and remove containers, networks
  events      Receive real time events from containers.
  exec        Execute a command in a running container.
  images      List images used by the created containers
  kill        Force stop service containers.
  logs        View output from containers
  ls          List running compose projects
  pause       Pause services
  port        Print the public port for a port binding.
  ps          List containers
  pull        Pull service images
  push        Push service images
  restart     Restart service containers
  rm          Removes stopped service containers
  run         Run a one-off command on a service.
  scale       Scale services
  start       Start services
  stop        Stop services
  top         Display the running processes
  unpause     Unpause services
  up          Create and start containers
  version     Show the Docker Compose version information
  wait        Block until the first service container stops
  watch       Watch build context for service and rebuild/refresh containers when files are updated

Run 'docker compose COMMAND --help' for more information on a command.

There are three key docker-compose commands crucial for managing the lifecycle of applications. Here's a breakdown of these commands and how they fit into the overall lifecycle:

• docker-compose up: The docker-compose up command initializes and starts the containers as defined in your configuration file. You can either build container images from scratch or use pre-built images available in a registry. For long-running services, like web servers, it’s often practical to run the containers in detached mode by using the -d or --detach flags. This allows the containers to run in the background, freeing up your terminal for other tasks. For a full list of options and flags available with this command, you can use docker-compose up --help.
• docker-compose ps: The docker-compose ps command provides a snapshot of the containers and their current status. This is particularly useful for diagnosing issues and performing health checks on your containers. For example, if you have a two-container setup with a backend and a frontend, you can use docker-compose ps to see the status of each container. It helps you identify whether any of your containers are down, not responding to health checks, or have failed to start due to misconfiguration.
• docker-compose down: The docker-compose down command stops and removes all the resources, including containers, networks, images and volumes.

Docker Compose File

The Docker Compose CLI manages and configures multi-container applications. Typically, these settings are saved in a file called docker-compose.yml. Docker Compose is a powerful tool, but its efficacy is dependent on the quality of the configuration. As a result, understanding how to create and fine-tune these docker-compose.yml files is critical, requiring careful attention to detail.

A docker-compose.yaml file consists of four main sections:

name: myapp
services:
    - ...
networks:
    - ...
volumes:
    - ...

Name

Docker Compose specifies that the top-level name property will be used as the project name if you don't set one yourself. You can change this name, and if the top-level name element is not set, Docker Compose will set a default project name that will be used. Also note that the project name is exposed for interpolation and we can use the COMPOSE_PROJECT_NAME to access this name.

name: myapp

services:
    some-service:
        image: busybox
        command: echo "I'm running ${COMPOSE_PROJECT_NAME}"

Services

A service is a general term for a computer resource in an app that can be changed or grown without affecting other parts of the app. A group of containers backs up the services. The platform runs the containers based on replication needs and placement restrictions. A Docker image and a set of runtime arguments describe a service because it is backed by a container. With these arguments, all containers in a service are made in the same way.

name: myapp

services:
    server:
        image: nginx:latest
        ports:
            - 8080:80

This section of the docker-compose.yaml file defines a service named server, which is set up to run a Nginx web server. The image: nginx:latest directive instructs Docker to utilize the latest version of the Nginx image from Docker Hub. The ports section maps port 8080 on the host system to port 80 within the container. This configuration allows you to reach the Nginx server via http://localhost:8080 on your host machine.

Networks

Networks let services communicate with each other. By default Δοψκερ Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by the service's name. The top-level networks element lets you configure named networks that can be reused across multiple services.

services:
  frontend:
    image: example/webapp
networks:
    - front-tier
    - back-tier

This section of the docker-compose.yaml file creates a service named frontend and uses the Docker image example/webapp. This image is anticipated to include the application code and environment required to operate the frontend component of a web application. The networks section states that this service is connected to two networks: front-tier and back-tier. By connecting the frontend service to these networks, it can communicate with other services on the same networks, allowing for easier interaction across different components of a multi-container application.

Volumes

Volumes in Docker are used to handle persistent data that must survive the lifecycle of a container. Volumes in Docker Compose provide a common way for services to mount and use persistent storage areas. You can construct named volumes that are reusable across several services in your application by specifying them at the top level in a docker-compose.yaml file. This configuration not only helps to guarantee data consistency and integrity, but it also simplifies storage resource management by allowing volumes to be configured and allocated centrally.

services:
  backend:
    image: example/database
    volumes:
      - db-data:/etc/data

  backup:
    image: backup-service
    volumes:
      - db-data:/var/lib/backup/data

volumes:
  db-data:

In this docker-compose.yaml configuration, the volumes section defines a named volume called db-data. This volume is then used by two different services: backend and backup.

The backend service uses the example/database image and mounts the db-data volume to the /etc/data directory inside the container. This setup allows the backend service to store and persist data in the db-data volume.

The backup service, which uses the backup-service image, also mounts the same db-data volume but maps it to a different directory inside its container, /var/lib/backup/data. This configuration enables the backup service to access and potentially back up the data stored by the backend service.

Creating a Web Server with Docker Compose

Web servers running within containers frequently require various initial operations before they can begin serving content. These tasks could include setting up configurations, downloading files, or installing dependencies. Docker Compose makes this process easier by allowing you to specify all of these operations as part of a multi-container application configuration. In this exercise, you will build a preparation container specifically for creating static files such as index.html. Following that, a server container will be configured to serve the static files. With the proper network configuration, this server will be accessible from your host system. You'll also manage the full application lifecycle with various Docker Compose commands, which will make it easier to set up and run your containers.

Create a folder named compose-server and navigate to it:

mkdir compose-server
cd compose-server

---

Create a folder named init and navigate to it:

mkdir init
cd init

---

Create a bash script to generate a simple HTML page

#!/usr/bin/env sh

# Check if a directory named 'data' exists
if [ -d "data" ]; then
    # If it does, delete it and all its contents
    echo "Removing existing 'data' directory..."
    rm -rf data
fi

# Create a new directory named 'data'
echo "Creating a new 'data' directory..."
mkdir data

# Create a new file named 'index.html' inside the 'data' directory
echo "Creating a new file 'index.html' inside the 'data' directory..."
touch data/index.html

# Append HTML content to the 'index.html' file
echo "Appending HTML content to the 'index.html' file..."
echo "<h1>Welcome to Docker Compose! </h1>" >> data/index.html
echo "<img src='https://www.docker.com/wp-content/uploads/2021/10/Moby-logo-sm.png' />" >> data/index.html

---

Create a Dockerfile with the name Dockerfile and the following content:

# Use the busybox base image
FROM busybox

# Copy the prepare.sh script to /usr/bin/prepare.sh in the image
ADD prepare.sh /usr/bin/prepare.sh

# Make the prepare.sh script executable
RUN chmod +x /usr/bin/prepare.sh

# Set the entry point for the container to be the prepare.sh script
# This means that when the container starts, it will run the prepare.sh script
ENTRYPOINT ["sh", "/usr/bin/prepare.sh"]

This Dockerfile starts by using the busybox base image, which is a minimal and lightweight Linux distribution commonly used for small utilities and embedded systems. It then adds a script named prepare.sh from the build context to the /usr/bin/prepare.sh path within the image. To ensure that this script can be executed, the Dockerfile uses the RUN command to modify its permissions, making it executable with chmod +x. Finally, the ENTRYPOINT instruction is specified to define the default command that will be run when the container starts. In this case, it sets the entry point to execute the prepare.sh script using the sh shell. This setup means that every time the container is launched, it will automatically run prepare.sh, allowing for pre-defined setup or configuration tasks to be carried out as the container starts.

---

Change the directory to the parent folder with the cd .. command and create a docker-compose.yaml file with the following content

name: "compose-server"

services:
  init:
    build:
      context: ./init
    volumes:
      - static:/data

  server:
    image: nginx
    volumes:
      - static:/usr/share/nginx/html
    ports:
      - "8080:80"

volumes:
  static:

This Docker Compose file sets up a multi-container application with two services: init and server.

1. Service Definitions:
   • init: This service builds an image from a Dockerfile located in the ./init directory. The Dockerfile in this context likely contains instructions to prepare or generate static files. The volumes section maps a named volume, static, to the /data directory inside the container. This allows the init service to write data to this volume, which can then be accessed by other services.
   • server: This service uses the official nginx image to run an NGINX web server. It also mounts the same named volume, static, but this time to the /usr/share/nginx/html directory inside the container. This is where NGINX expects to find static files to serve. The ports section maps port 80 of the container to port 8080 on the host machine, making the web server accessible via http://localhost:8080 on the host.
2. Volumes:
   • The volumes section at the bottom defines a named volume called static. Named volumes are managed by Docker and provide persistent storage that is shared among containers. In this case, the static volume is used to transfer static files generated by the init service to the server service.

---

Start the application with the following command:

docker-compose up --detach

And you should see an output similar to the following:

[+] Running 8/8
 ✔ server 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                               41.0s
   ✔ efc2b5ad9eec Pull complete                                                                                   32.9s
   ✔ 8fe9a55eb80f Pull complete                                                                                   29.3s
   ✔ 045037a63be8 Pull complete                                                                                    0.9s
   ✔ 7111b42b4bfa Pull complete                                                                                    1.6s
   ✔ 3dfc528a4df9 Pull complete                                                                                    3.0s
   ✔ 9e891cdb453b Pull complete                                                                                    5.3s
   ✔ 0f11e17345c5 Pull complete                                                                                    6.2s
[+] Building 5.3s (9/9) FINISHED                                                                         docker:default
 => [init internal] load build definition from Dockerfile                                                          0.1s
 => => transferring dockerfile: 457B                                                                               0.0s
 => [init internal] load .dockerignore                                                                             0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [init internal] load metadata for docker.io/library/busybox:latest                                             4.5s
 => [init auth] library/busybox:pull token for registry-1.docker.io                                                0.0s
 => [init internal] load build context                                                                             0.0s
 => => transferring context: 771B                                                                                  0.0s
 => CACHED [init 1/3] FROM docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85  0.0s
 => => resolve docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7   0.0s
 => [init 2/3] ADD prepare.sh /usr/bin/prepare.sh                                                                  0.0s
 => [init 3/3] RUN chmod +x /usr/bin/prepare.sh                                                                    0.5s
 => [init] exporting to image                                                                                      0.1s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:8d2660840cab8d1e01e7cb9ad823fb2a2e3141b73c6cc0cce53c91355a6d52a6                       0.0s
 => => naming to docker.io/library/compose-server-init                                                             0.0s
[+] Running 4/4
 ✔ Network compose-server_default     Created                                                                      0.3s
 ✔ Volume "compose-server_static"     Created                                                                      0.0s
 ✔ Container compose-server-init-1    Started                                                                      0.1s
 ✔ Container compose-server-server-1  Started                                                                      0.1s

This command creates and starts the containers in detached mode. It starts by creating the compose-server_default network and the compose-server_static volume. It then builds the init container using the Dockerfile from the previous step, downloads nginx, and starts the containers.

---

Check the status of the application with the docker-compose ps command:

docker-compose ps

and you should see the following:

NAME                      IMAGE     COMMAND                                          SERVICE   CREATED         STATUS         PORTS
compose-server-server-1   nginx     "/docker-entrypoint.sh nginx -g 'daemon off;'"   server    3 minutes ago   Up 3 minutes   0.0.0.0:8080->80/tcp

---

Open the http://localhost:8080/ on your browser. If everything went as planned you should see the following:

---

Stop and remove all the resources with the following command if you do not need the application up and running:

docker-compose down

And you should see the following output:

[+] Running 3/3
 ✔ Container compose-server-server-1  Removed                                                                      0.6s
 ✔ Container compose-server-init-1    Removed                                                                      0.0s
 ✔ Network compose-server_default     Removed                                                                      0.3s

In this exercise, we successfully created and configured a multi-container application using Docker Compose. The docker-compose.yaml file was employed to define both networking and volume options, which are crucial for inter-container communication and data persistence. We demonstrated how to use Docker Compose CLI commands to build and manage the application. These commands included starting and stopping containers, checking their status, and removing them when no longer needed.

Summary

This blog post highlighted Docker Compose's ability to manage and orchestrate multi-container applications effectively. Building on our earlier explorations of Docker containers and Dockerfiles, we looked at how Docker Compose simplifies the setup and administration of complicated applications with various components. Docker Compose, which defines your whole application stack in a single YAML file, enables you to spin up complex environments with a single command, making it ideal for development, testing, and production.

We've highlighted Docker Compose's major capabilities, such as its ability to separate container instances, handle stateful data using named volumes, and facilitate iterative development. The practical examples offered demonstrated how to effectively develop, manage, and troubleshoot applications using Docker Compose CLI commands. We also went over the key components of a docker-compose.yaml file, like service definitions, network setups, and volume management.

Looking ahead, our next post will go over the configuration choices accessible in the Docker Compose environment. We'll go over advanced settings and approaches for further optimizing and tailoring your application installations, ensuring that you can fully utilize Docker Compose's capabilities to meet your individual requirements.

See you next Monday!

Comments

[Charles Brown]

This was a really informative post! The way you broke down the Docker Compose file was very helpful for understanding its components. Could you maybe provide more examples of using Docker Compose in a CI/CD pipeline in a future post? I think that would be a great follow-up!

[Kristijan Kanalaš]

A very good post! I would just like to add that docker-compose is deprecated for the newer version docker compose. See the following for more info: docs.docker.com/compose/migrate/