What a mysterious bug taught us about how Docker stores registry credentials

We recently ran into a mysterious bug that required hours of digging into the
arcane details of Docker's registry credentials store to figure out. Although
in the end the fix turned out to be easy, we learned a thing or two along the
way about the design of the credentials store and how, if you're not careful,
it can be configured insecurely.

Blimp, sometimes needs to pull private images from a
Docker registry in order to boot those images in the cloud. This typically
works fine, but unfortunately, when some users started
Blimp, they were getting the following error message:

Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials

At first, we were completely baffled by this cryptic message and had no clue it
was related to our handling of credentials. To understand how we figured it
out, first you need to know a little about how modern Docker credentials are
handled.

Docker's External Credentials Store

The recommended way to store your Docker credentials is in an external
credentials store. In your Docker config file, which is usually located at
~/.docker/config.json, there are two fields you can use to configure how
Docker gets and stores credentials: credsStore and credHelpers.

credsStore tells Docker which helper program to use to interact with the
credentials store. All helper programs have names that begin with
docker-credential- -- the value of credsStore is the suffix of the helper
program.

For example, if you work on a Mac laptop, you might decide to use the Mac OS
keychain. The name of the helper program to use the keychain is
docker-credential-osxkeychain. So your config.json would include the
following:

{
  "credsStore": "osxkeychain"
}

If you want to see what credentials Docker currently has for you, you can use list. For example:

docker-credential-osxkeychain list

The result is a list of pairs of servers and usernames. For example:

{
  "http://quay.io":"kklin",
  "https://index.docker.io":"kevinklin"
}

You may also notice credHelpers in your config.json. These helpers are similar
to credsStore, but are used to generate short lived credentials. For example,
if you use gcr, gcloud installs a credHelper that uses
your Google login to get tokens. This way, Docker never has your Google
credentials directly -- the docker-credential-gcloud acts as a middleman
between Docker and your Google credentials.

Once again, here's the error message our users were getting:

Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials

We were able to run the docker-credential-osxkeychain list and get
commands to see the credentials for 1234.dkr.ecr.us-east-1.amazonaws.com, so
why were we getting an error that there weren't any credentials??

In the Beginning: Docker Stores Your Registry Password In Your Config File

It turns out that external credentials stores weren't
added to Docker until version 1.11,
in 2016. Before 1.11, Docker stored credentials via a config field called
auths. This field is stored in the same file as the credStore: ~/.docker/config.json.

Whenever you logged into a registry, Docker would set the value of auths to
your password. For
example,
your config file might contain the following:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}

What we learned the hard way is that there's a quirk with Docker's login
command. When you log in using docker login, Docker adds an entry via the
credsStore and in auths, using slightly different server names. Your
credentials are properly stored in the credentials store, but the entry in
auths doesn't contain the username or password. The result looks something
like this:

{
"auths": {
  "https://index.docker.io/v1/": {}
}

The problem is that Blimp grabs credentials from both auths and
credsStore. So it was passing two copies of the credentials to the Docker
image puller -- one with the correct username and password, and one without the
password at all.

Unfortunately, Docker preferred the https:// version of the credential, and
attempt to pull the image with the empty credential. Thus, the no basic auth
credentials error.

Once we figured out that the problem was that an empty duplicate entry was
getting added to the insecure store, it was easy to fix the
problem. All we
needed to do was add an if statement to skip empty credentials:

    addCredentials := func(authConfigs map[string]clitypes.AuthConfig) {
        for host, cred := range authConfigs {
            // Don't add empty config sections.
            if cred.Username != "" ||
                cred.Password != "" ||
                cred.Auth != "" ||
                cred.Email != "" ||
                cred.IdentityToken != "" ||
                cred.RegistryToken != "" {
                creds[host] = types.AuthConfig{
                    Username:      cred.Username,
                    Password:      cred.Password,
                    Auth:          cred.Auth,
                    Email:         cred.Email,
                    ServerAddress: cred.ServerAddress,
                    IdentityToken: cred.IdentityToken,
                    RegistryToken: cred.RegistryToken,
                }
            }
        }
    }

A Potential Docker Credentials Security Risk

In the process of uncovering this bug, we noticed a potential security risk
that you may not be aware of. As we learned, it's best practice to use an
external store to store your external registry credentials. However, depending
on how and when you installed Docker it's possible you could still be using the
legacy auths method. If you are, your ~/.docker/config.json might look
something like this:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}

This may look reasonable secure, the passwords appear to be a garbled bunch of
gibberish. Surely those passwords are encrypted, right?

Guess again. All Docker did was encode the passwords using base64. And as David
Rieger pointed out on Hacker
Noon,
base64

may look like encryption on first glance, but it's not. Base64 is a scheme
for encoding, not encryption. You can simply copy the base64 string and
convert it to ASCII in a matter of seconds.

That seemingly secure password of aGVzdHVzZXI6dGVzdHBhc3N3b3Jk? All you need to do to read the password is base64 decode it:

$ echo aGVzdHVzZXI6dGVzdHBhc3N3b3Jk| base64 -D
hestuser:testpassword

The Moral of Our Story: Double Check Your Docker Credentials' Security

So that's the bad news: if Docker config file isn't properly set up, Docker is
storing your credentials password in plain text.

The good news is that it's easy to fix the problem.

All you and your team members need to do is take a quick look at
~/.docker/config.json. If it contains an auths password, get rid of it and
switch over to using a credentials store. To do so, just download the
appropriate docker-credential- helper for your system, and update the
credsHelper field in ~/.docker/config.json.

Hope that helps!

Read the Top 5 common mistakes when writing Docker Compose