How to close attack vectors for exposed secrets in Docker

Cybernews recently reported 5,500 out of 10,000 public docker images contained 48,000+ sensitive secrets - a combination of harmless and potentially vulnerable API keys. This report illustrates why it's imperative that security and platform teams know the most common attack vectors for their Docker containers and understand how to close them.

This post will provide a brief checklist of the various attack vectors into your Docker containers specifically originating from exposed secrets.

Docker and exposed secrets

Let’s quickly examine the relationship between container runtime and registry. When we spin-up a container, an image is pulled from the registry via APIs and is deployed. This is visualized below:

Image Source

The high number of secrets from the Cybernews report is attributed to developers re-using packages from a registry containing sensitive secrets. Secrets are commonly found in the container image metadata - the environment variables and filesystem. Also, source code leakage could allow attackers to generate newer valid tokens that could provide unauthorized system access.

Attack surface:

An attack surface is a collection of all vulnerable points an attacker can use to enter the target system. Attackers skillfully exploit these vulnerable points in technology and human behavior to access sensitive assets.

We need to understand two Docker concepts as we continue this discussion:

Filesystem: In Docker, each layer can contain directory changes. The most commonly used filesystem, OverlayFS, enables Docker to overlay these layers to create a unified filesystem for a container.

Layers: Docker images are created in layers - i.e., each command on the DockerFile corresponds to a layer.

With that context, let’s understand and analyze how exposed secrets can affect these Docker image attack vectors.

Docker image layers

Secrets explicitly declared in the Dockerfile or build arguments can easily be accessed via the Docker image history command.

#terminal

docker image history <image>

This represents one of the simplest methods for an attacker to capitalize on a secret.

Filesystem

This Dockerfile demonstrates a scenario where sensitive data like SSH private key and secrets.txt are added to the container's filesystem and later removed.

#Dockerfile
FROM nginx:latest

# Copy in SSH private key, then delete it; this is INSECURE,
# the secret will still be in the image.
COPY id_rsa .
RUN rm -r id_rsa

ARG DB_USERNAME
ENV DB_USERNAME =$DB_USERNAME
ARG DB_PASSWORD
ENV DB_PASSWORD =$DB_PASSWORD
ARG API_KEY
ENV API_KEY =$API_KEY

# Expose secrets via a publicly accessible endpoint (insecure practice)
RUN echo "DB_USERNAME=$DB_USERNAME" > /usr/share/nginx/html/secrets.txt
RUN echo "DB_PASSWORD=$DB_PASSWORD" >> /usr/share/nginx/html/secrets.txt
RUN echo "API_KEY=$API_KEY" >> /usr/share/nginx/html/secrets.txt
RUN rm /usr/share/nginx/html/secrets.txt

CMD ["nginx", "-g", "daemon off;"]

Docker uses layer caching - hence, the secret is still available in one of the layers. An internal attacker can also extract individual layers of a Docker image, stored as tar files in registries, which enables them to uncover hidden secrets.

After creating a Dockerfile, developers mistakenly use build arguments to create an image. For the above dockerfile, the secrets are input as arguments.

terminal

docker build \
--build-arg DB_USERNAME=root \
--build-arg DB_PASSWORD=Xnkfnbgf \
--build-arg  API_KEY=PvL4FjrrSXyT7qr \
-t myapp:1.0 .

While convenient, it is not secure since the arguments also get embedded in the image. A simple docker history --no-trunc can expose the secret values. Developers should either use multi-stage builds or secret managers.

Environment variables

Apart from Docker image access, unauthorized access to the source code of the docker image can provide additional attack vectors. The .env files are primarily used to store secrets such as API tokens, database credentials, and other forms of secrets that an application needs. When attackers have access to secrets in the .env, they can make unauthorized accesses that the secret allows.

Dockerfile

DockerFile is a standard file that contains execution instructions to build an image while spinning up containers. Hard-coding secrets into DockerFile creates a significant attack surface. When attackers access the DockerFile, they can see hard-coded secrets, the base image, the list of dependencies, and critical file locations. Developers need to use appropriate secret managers to reference variables.

Docker-compose.yml 

Docker-compose defines networks, services, and storage volumes. When an attacker views the file, they can understand the application architecture and exploit or disrupt its operation.

#docker-compose.yml
services:
  web:
    image: my-web-app:latest
    ports:
      - "80:80"
    networks:
      - app-network
  db:
    image: postgres:latest
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: example_db_password
networks:
  app-network:
volumes:
  db-data:

In the above docker-compose.yml, the postgres database password is hardcoded. The password can easily be accessed with the docker exec command as shown below:

#terminal
docker exec -it <container id or name> /bin/bash
env

Apart from secrets, an attacker can also analyze the volume mappings and identify potential points of weakness. If they discover that the database volume (db-data) is also mounted to the host filesystem, they could exploit and perform a container breakout attack, gaining access to the underlying host system.

CI/CD config files

CI/CD configuration files such as .gitlab-ci.yml, Azure-pipelines.yml , Jenkinsfile,  etc., contain instructions for building, testing, and deploying applications. The logs generated in CI/CD pipeline can contain debugging and logging information. If a developer includes a logging statement that inadvertently prints a sensitive secret, it can lead to unauthorized exposure and compromise. Such secret leaks need to be detected so that developers can fix their source code.

Developers also tend to leave the registry login credentials in the CI CD configuration files. Consider the following gitlab-ci.yml.

# .gitlab-ci.yml

variables:
  DOCKER_IMAGE_TAG: latest
  DOCKER_REGISTRY_URL: registry.example.com
  DOCKER_IMAGE_NAME: my-docker-image
  DOCKER_REGISTRY_USER: adminuser <-- should use $CI_REGISTRY_USER  
  DOCKER_REGISTRY_PASSWORD: secretpassword <-- should use $CI_REGISTRY_PASSWORD 

# Jobs
build:
  stage: build
  image: image_name:stable
  script:
    - docker build -t $DOCKER_REGISTRY_URL/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG .
    - docker login -u $DOCKER_REGISTRY_USER -p $DOCKER_REGISTRY_PASSWORD $DOCKER_REGISTRY_URL
    - docker push $DOCKER_REGISTRY_URL/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG

In the configuration above, the developer makes a docker registry login using a hardcoded username and password, leading to unwarranted secret exposure. A good development practice is to integrate CI/CD environments with secret managers like Hashicorp Vault.

Detecting secrets

Older or unused Docker images can contain unpatched vulnerabilities or outdated dependencies, posing security risks to the enterprise. Regularly scanning and removing unused images helps mitigate these risks by reducing the attack surface and ensuring that only secure images are deployed.

Enterprises also need to be actively using secret scanners to detect secrets in docker images, whether they’re stored in Docker Hub, JFrog Artifactory, AWS ECR, or any other repository. There are numerous secret scanning tools and infrastructure as code scanning tools as well as secrets management platforms that provide a more high-scale, holistic answer to these secrets detection, management, encryption, and storage challenges.

I recommend that enterprise users buy a mature platform with plenty of successful case studies, rather than embark on an expensive, high-skill DIY project to detect and manage secrets.