In cloud-native applications, we typically use environment variables to manage configuration settings. These variables offer a flexible way to adjust settings without modifying the source code. They are stored on Linux-based systems and can be accessed by applications to control their behavior.
For example, suppose your application relies on a LOG_LEVEL environment variable to determine the level of detail in its logs. If you change this variable from INFO to DEBUG and restart the application, you'll start seeing more detailed logs, which can help you diagnose issues more effectively.
This approach is also handy for deploying the same application across different environments, such as staging, testing, and production. You can simply adjust the environment variables for each environment to meet its specific needs.
Similarly, when using Docker Compose to manage containers, you configure services by setting environment variables for each container. This keeps your configurations consistent and easy to manage across different environments.
There are three methods of defining environment variables in Docker Compose arranged from the most stable, with the least frequent changes, to the most dynamic, with the most frequent changes:
- Using the Compose file
- Using shell environment variables
- Using the environment file
If the environment variables you need for your containers don’t change often, it's a good idea to keep them in your docker-compose.yaml file. For sensitive information like passwords, it’s safer to set these variables directly in your shell environment before running the docker-compose command. However, if you have a lot of variables or if they differ between your testing, staging, and production environments, it's more manageable to put them in .env files and then reference those in your docker-compose.yaml.
In the services section of your docker-compose.yaml file, you can specify environment variables for each service. For instance, you can set environment variables like LOG_LEVEL and METRICS_PORT for the server service like this:
services:
server:
image: nginx-custom
environment:
- LOG_LEVEL=debug
- METRICS_PORT=8445
If the environment variables are not defined directly in the docker-compose.yaml file, Docker Compose can still retrieve their values from the shell environment. For example, if you want the HOSTNAME environment variable for the server service to be set from the shell, you can ensure it's available by exporting it in your shell before running Docker Compose. Here's how it might look:
services:
server:
image: nginx-custom
environment:
- HOSTNAME
And exporting the HOSTNAME variable to the shell:
export HOSTNAME=myhost
docker-compose up
If the shell running the docker-compose command doesn't have a value set for the HOSTNAME environment variable, the container will start with HOSTNAME as an empty environment variable.
You can also store environment variables in .env files and configure them in your docker-compose.yaml files. For example, you might have a database.env file that looks like this:
DB_HOST=localhost
DB_PORT=5432
DB_USER=myuser
DB_PASSWORD=mypassword
In the docker-compose.yaml file, you can specify the .env file for each service using the env_file field. Here's how you would configure it for a service:
services:
my-service:
image: my-image
env_file:
- ./database.env
When Docker Compose sets up the server service, it will apply all the environment variables specified in the database.env file to the container. In the following example, you'll get hands-on experience configuring an application using all three methods for setting environment variables in Docker Compose.
Configuring Services with Docker Compose
In this example, you'll set up a Docker Compose application using various methods for configuring environment variables. You’ll start by defining two environment variables in a file named print.env. Next, you'll add one environment variable directly in the docker-compose.yaml file and set another one from the Terminal on the fly. By the end, you’ll see how these four environment variables from different sources are combined and used in your container.
Create a folder named configuration-server and navigate into it using the cd command
mkdir configuration-server
cd configuration-server
Create an .env file with the name display.env and the following content, using your favorite code editor. I am going to use VSCode:
VAR_FROM_ENV_FILE_1=HELLO
VAR_FROM_ENV_FILE_2=WORLD
Create a docker-compose.yaml file similar to the following:
services:
display:
image: busybox
command: sh -c 'sleep 5 && env'
env_file:
- display.env
environment:
- ENV_FROM_COMPOSE_FILE=HELLO
- ENV_FROM_SHELL
The docker-compose.yaml file defines a print service using the busybox image. It runs a command that sleeps for 5 seconds and then prints the environment variables. Environment variables are configured in three ways: from the print.env file, which contains key-value pairs; directly in the Compose file with ENV_FROM_COMPOSE_FILE=HELLO; and via shell environment variables with ENV_FROM_SHELL, which will be set at runtime if available.
Export the ENV_FROM_SHELL variable to the shell with the following command:
export ENV_FROM_SHELL=WORLD
Use the docker-compose up command to start the app. The output should be similar to the following:
docker-compose up
error during connect: this error may indicate that the docker daemon is not running: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dconfiguration-server%22%3Atrue%7D%7D": open //./pipe/docker_engine: The system cannot find the file specified.
After this moment of embarrassment, you will remember to start the actual Docker daemon, and the output should be the following:
display Pulling
ec562eabd705 Already exists
display Pulled
Network configuration-server_default Creating
Network configuration-server_default Created
Container configuration-server-display-1 Creating
Container configuration-server-display-1 Created
Attaching to configuration-server-display-1
configuration-server-display-1 | HOSTNAME=c791b58e8a00
configuration-server-display-1 | SHLVL=1
configuration-server-display-1 | HOME=/root
configuration-server-display-1 | VAR_FROM_ENV_FILE_1=HELLO
configuration-server-display-1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
configuration-server-display-1 | VAR_FROM_ENV_FILE_2=WORLD
configuration-server-display-1 | ENV_FROM_COMPOSE_FILE=HELLO
configuration-server-display-1 | ENV_FROM_SHELL=WORLD
configuration-server-display-1 | PWD=/
configuration-server-display-1 exited with code 0
The output comes from the display container defined in the docker-compose.yaml file, which runs the env command to display its environment variables. The results show two variables from the display.env file: VAR_FROM_ENV_FILE_1 with the value HELLO, and VAR_FROM_ENV_FILE_2 with the value WORLD. Additionally, the ENV_FROM_COMPOSE_FILE variable defined directly in the docker-compose.yaml file has the value HELLO. Finally, the ENV_FROM_SHELL variable, set in the shell before running Docker Compose, appears with the value WORLD.
In this example, you set up a Docker Compose application using various methods for configuration, such as Docker Compose files, environment definition files, and shell-exported values. This flexibility allows you to deploy the same application across different platforms.
Since Docker Compose handles multi-container setups, it's essential to define how these containers depend on each other. The next section will cover how to manage these interdependencies within Docker Compose applications.
Service Dependency
Docker Compose is used to manage multi-container applications, which are defined in docker-compose.yaml files. Even though the containers operate as separate microservices, it's common to have them depend on each other.
For example, in a two-tier application with a PostgreSQL database and a Java backend, the backend needs the PostgreSQL service to be running before it can connect and function correctly. Docker Compose allows you to control the startup and shutdown order of these services, ensuring that dependencies are handled properly.
services:
init:
image: busybox
pre:
image: busybox
depends_on:
- init
main:
image: busybox
depends_on:
- pre
In the above example, the main container depends on the pre container, and the pre container depends on the init container. Docker Compose will start the containers in the order of init, pre and main. Also, the containers will be stopped in reverse order: main, pre and then init.
In the following example, the order of containers will be used to fill a file and serve it with a web server.
Service Dependency with Docker Compose
In this exercise, you'll configure a Docker Compose application with four containers. The first three containers will run in sequence to generate a static file, which will then be served by the fourth container. This setup demonstrates how to manage and coordinate dependencies between services in Docker Compose.
Create a folder named service-dependency-example and navigate into it using the cd command
mkdir service-dependency-example
cd service-dependency-example
Create your docker-compose.yaml similar to the following
# This is the main configuration file for Docker Compose
# It defines the services that are to be run, and how they should be configured.
# Each service is defined as a key-value pair, where the key is the name of the service,
# and the value is a dictionary of configuration options.
#
# The services section is where you define the services that you want to run.
# Each service is defined as a key-value pair, where the key is the name of the service,
# and the value is a dictionary of configuration options.
#
# The configuration options for a service can include things like the image to use,
# the command to run, the ports to expose, and the volumes to mount.
#
# In this file, we have defined four services:
# - clean: This service simply cleans out the static volume.
# - init: This service initializes the static volume with a simple HTML file.
# - pre: This service simply adds data to the HTML file
# - server: This service runs an Nginx server, which serves the content from the static volume.
#
# The volumes section is where you define the volumes that are to be used by the services.
# In this case, we have defined a single volume called "static", which is mounted by all three services.
services:
clean:
image: busybox
command: rm -rf /data/*
volumes:
- static:/data
init:
image: busybox
command: "sh -c 'echo Hello from init >> /data/hello.html'"
volumes:
- static:/data
depends_on:
- clean
pre:
image: busybox
command: "sh -c 'echo Hello from pre >> /data/hello.html'"
volumes:
- static:/data
depends_on:
- init
server:
image: nginx
volumes:
- static:/usr/share/nginx/html
ports:
- "8080:80"
depends_on:
- pre
volumes:
static:
This Docker Compose file defines four services and a single volume named static. The volume is shared among all services. The clean service starts by removing the index.html file from the volume. The init service then creates and populates index.html. Afterward, the pre service appends an additional line to index.html. Finally, the server service serves the content from the static volume.
Start the application with the docker-compose up command. The output should look like the following:
Container service-dependency-clean-1 Created
Container service-dependency-init-1 Recreate
Container service-dependency-init-1 Recreated
Container service-dependency-pre-1 Recreate
Container service-dependency-pre-1 Recreated
Container service-dependency-server-1 Recreate
Container service-dependency-server-1 Recreated
Attaching to service-dependency-clean-1, service-dependency-init-1, service-dependency-pre-1, service-dependency-server-1
service-dependency-clean-1 exited with code 0
service-dependency-init-1 exited with code 0
service-dependency-pre-1 exited with code 0
service-dependency-server-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
service-dependency-server-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
service-dependency-server-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
service-dependency-server-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
service-dependency-server-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
service-dependency-server-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
service-dependency-server-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
service-dependency-server-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
service-dependency-server-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: using the "epoll" event method
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: nginx/1.27.0
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: OS: Linux 5.10.102.1-microsoft-standard-WSL2
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker processes
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 29
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 30
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 31
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 32
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 33
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 34
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 35
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 36
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 37
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 38
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 39
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 40
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 41
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 42
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 43
service-dependency-server-1 | 2024/08/13 06:53:43 [notice] 1#1: start worker process 44
service-dependency-server-1 | 172.21.0.1 - - [13/Aug/2024:06:53:48 +0000] "GET / HTTP/1.1" 200 31 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" "-"
The output shows that Docker Compose creates the containers in the order of clean, init, pre and server.
Open http://localhost:8080 in the browser:
The output shows that the clean, init and pre containers work in the expected order.
Return to your terminal and hit Ctrl + C to close the application gracefully. You will notice that in the logs, that the containers stop in the reverse order.
Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
Container service-dependency-server-1 Stopping
Container service-dependency-server-1 Stopped
Container service-dependency-pre-1 Stopping
Container service-dependency-pre-1 Stopped
Container service-dependency-init-1 Stopping
Container service-dependency-init-1 Stopped
Container service-dependency-clean-1 Stopping
Container service-dependency-clean-1 Stopped
In this example, we set up a Docker Compose application with interdependent services to demonstrate how Docker Compose manages the startup and operation of containers in a specific order. This feature is crucial for building and orchestrating complex multi-container applications.
Summary
In this post, we explored the use of environment variables to manage configurations in cloud-native applications, specifically within Docker Compose. We demonstrated how to configure applications using environment variables defined in Docker Compose files, shell environments, and .env files, allowing for flexible and adaptable deployments across different environments like staging, testing, and production.
We also delved into the management of service dependencies in Docker Compose, showing how to control the order in which containers start and stop. We created a Docker Compose application with four interdependent services that worked together to generate and serve a static file, highlighting the importance of managing dependencies in complex multi-container applications.
And with that we covered how to compose environments using Docker Compose. In the next few posts we are going to delve into Docker Networking.
See you next Monday!
Top comments (2)
This was a really good 15 part article on docker, This is just too good.
There are more parts coming my friend!