How to copy a file from a Docker container to the host 📋

Recently, we needed to download a file from a Docker container. Since the file was not inside a bind mount, we couldn't directly access the remote server to get the file on the file system.

So we quickly found the docker cp utility to copy the file on our host and voila, we had the file on our computer. ☺️

docker cp CONTAINER_NAME:/opt/service/FILENAME_TO_OBTAIN .

A simple example of the docker cp command, where the name of the container is known.

Unfortunately, we didn't know the exact name of the container, as containers created by docker compose are, by default, suffixed with a randomly generated hexadecimal string, such as .eq6l28g5mwsenh0gvag7zcg0j. 😶‍🌫️

And because we deployed our stack in swarm mode, we couldn't use the container_name option as it is ignored in this mode...

So, to obtain the full name of the container, we used the docker ps command, which lists the containers, along with its --format option to get only its full name.

docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX"

An example of the docker ps command, where we know the prefix of the container name.

If we combine these commands, we can have only one command to run to download our file.

docker cp "$(docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX")":"/opt/service/FILENAME_TO_OBTAIN" .

The resulting command, where we can copy a file from a container to our host.

It worked fine on our local computer, but the real file we were interested in was on a remote server... And this is where it all gets complicated for us. 🤮

Constraints

• We wanted an automated solution where we could just provide some information and we would get the file. 🤖
• The container was hosted on a random server from a pool of Docker machines. 🌈
• We had to be privileged users to run certain Docker commands. 👑

After a lot of guesswork and mistakes, we finally managed to get a working solution under these constraints, but it took longer than expected. Let's dive into some of the issues we encountered. 😌

Issues

How to run sudo commands on a remote server

In my opinion, this step is the most complicated part of our final script. Please be patient with me as I explain some of the things we have done below. 🤢

# Define and export some variables that will be used in this part
export SERVER=SERVER_HOSTNAME
export FILENAME=FILENAME_TO_OBTAIN
export CONTAINER_PREFIX=ranb2002_system
export PASSWORD=PASSWORD_TO_BE_ROOT

# Create a template where the environment variables will be replaced
TEMPLATE=$(
    envsubst <<'EOF'
    # Authenticate as sudo and pass the password without indentation
    sudo -S whoami
$PASSWORD

    # Delete the previous file if it exists on the server
    rm -f "${FILENAME}"

    # List all Docker containers on this server
    CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')

    # Check if we have a container for our application on this server
    CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${CONTAINER_PREFIX}")

    # Otherwise, continue to the next server
    if [-z "${CONTAINER_NAME[0]}" ]; then
      echo "The container with name ${CONTAINER_PREFIX} was not found on this server ($(hostname))... :("
      exit
    fi

    echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"

    # Copy the file from the container to the current host
    sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .

EOF
  )

# Connect to the server and run all commands from the template
ssh -T "${USER}@${SERVER}" <<< "$TEMPLATE"

The script runs a large number of commands on a remote server, where some of them require root privileges.

• To run multiple commands via SSH, we use a here document structure that lists all the commands we want to run on the remote host.
• Since the variables were not replaced inside this structure, we encompassed it with the envsubst command.
• To make the shell variables accessible to envsubst, we had to export them.
• Since we wanted to capture the container name in a variable, we ended up splitting the commands because our one-line command didn't work well with sudo.
• Since my user was not able to run Docker commands on this server, we had to gain root privileges. We have tried many (many) attempts and we managed to get a working solution by outputting our password to the first sudo -S command. The following commands don't need to have the password again. 🥺
• For some reason still unknown to me, the value of the CONTAINER_NAME variable is an array... So, to access it, we had no choice but to use CONTAINER_NAME[0]... 🤷

As you may have noticed, we didn't need to pass our password when running the ssh command, because we are using SSH public keys. If you want to install SSH keys on your servers, check out my article on how to automate ssh-copy-id! 😉

How to transfer the file from the remote server to our host

This part was easy to find, using sshpass and scp commands.

# Copy the file from the remote server to our host
sshpass -p "${PASSWORD}" scp "${USER}@${SERVER}":"/home/${USER}/${FILENAME}" .

Using sshpass to suppress the password prompt when connecting to the remote server.

Since we had previously installed an SSH public key on the server and we are using the same username on our host and the server, we can rewrite the previous command as follows.

scp "${SERVER}":"~/${FILENAME}" .

A small command when you can make some assumptions.

Simpler, right? 😉

How to find the server where the application container runs

To browse over a cluster of servers, we have provided the list as an array, along with a simple switch case. It's always useful when we need it to have a switch case example in Bash , so I'm putting it here for future reference. 😋

case "${ENVIRONMENT}" in
  DEV)
    SERVERS=(
      "DEV_SERVER_1_HOSTNAME"
      "DEV_SERVER_2_HOSTNAME"
    )
  ;;

  TEST)
    SERVERS=(
      "TEST_SERVER_1_HOSTNAME"
      "TEST_SERVER_2_HOSTNAME"
    )
  ;;

  *)
    SERVERS=(
      "PRODUCTION_SERVER_1_HOSTNAME"
      "PRODUCTION_SERVER_2_HOSTNAME"
    )
  ;;
esac

for SERVER in "${SERVERS[@]}"; do
  # All previous commands
done

A switch case that lists the servers we needed with a for-loop.

Unfortunately, since we are not able to know on which server we need to transfer the file from, we have to run all the previous commands on each server... 😵‍💫

So, we get a lot of scp: /home/ranb2002/[...]: No such file or directory error because only one server has the wanted file... 😒

How to ensure the arguments are provided

Since our script could fail if some arguments were not supplied or were incorrect, we added some conditions to verify these cases.

Note : if the password is incorrect or if the user is not a privileged user, the script will still fail because we haven't taken some time to figure out how to prevent this.

#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""

# Make sure we have your password
if [-z "${1}"]; then
  echo "You must provide your password!"
  exit
fi

# Export the password to an environment variable
export PASSWORD="${1}"

# Make sure we have the environment
if [-z "${2}"]; then
  echo "You must provide the environment!"
  exit
fi

export ENVIRONMENT="${2}"

# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
  #
  echo "The environment provided is incorrect!"
  exit
fi

Some checks we made at the start of our script.

Bonus : if for some reason you need to transform an uppercase value to its lowercase equivalent, here is the Bash command you can use: LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}'). 😜

Final solution

If we put together all the pieces together, our final script looks like below. 🧩

#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""

# Make sure we have your password
if [-z "${1}"]; then
  echo "You must provide your password!"
  exit
fi

# Export the password to an environment variable
export PASSWORD="${1}"

# Make sure we have the environment
if [-z "${2}"]; then
  echo "You must provide the environment!"
  exit
fi

export ENVIRONMENT="${2}"

# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
  #
  echo "The environment provided is incorrect!"
  exit
fi

LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}')
export DOCKER_STACK="stack-${LOWER_ENVIRONMENT}_service"

export FILENAME="FILENAME_TO_DOWNLOAD"

case "${ENVIRONMENT}" in
  DEV)
    SERVERS=(
      "DEV_SERVER_1_HOSTNAME"
      "DEV_SERVER_2_HOSTNAME"
    )
  ;;

  TEST)
    SERVERS=(
      "TEST_SERVER_1_HOSTNAME"
      "TEST_SERVER_2_HOSTNAME"
    )
  ;;

  *)
    SERVERS=(
      "PRODUCTION_SERVER_1_HOSTNAME"
      "PRODUCTION_SERVER_2_HOSTNAME"
    )
  ;;
esac

for SERVER in "${SERVERS[@]}"; do
  TEMPLATE=$(
    envsubst <<'EOF'
    # Authenticate as sudo and pass the password with no indentation
    sudo -S whoami
$PASSWORD

    # Delete the previous file if it exists
    rm -f "${FILENAME}"

    # List all Docker containers on this server
    CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')

    # Check if we have a container for our application on this server
    CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${DOCKER_STACK}")

    # Otherwise, continue to the next server
    if [-z "${CONTAINER_NAME[0]}" ]
    then
      echo "The container with name ${DOCKER_STACK} was not found on this server ($(hostname))... :("
      exit
    fi

    echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"

    # Copy the file from the container to the current host
    sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .
EOF
  )

  # Connect to the server and run all commands from the template
  ssh -T "${SERVER}" <<< "$TEMPLATE"

  # Copy the file from the server to our computer
  # shellcheck disable=SC2140
  scp "${SERVER}":"~/${FILENAME}" .
done

A complicated script just to download a file from a Docker container, right?

Hope this post is useful to you! Stay safe! 🍻